Forums

Parallel Cobra test suite

General discussion about Cobra. Releases and general news will also be posted here.
Feel free to ask questions or just say "Hello".

Parallel Cobra test suite

Postby Charles » Sat Sep 26, 2009 5:11 am

For those of you hacking on the Cobra compiler, you are no doubt familiar with running "testify" to see how your changes have affected the regression test suite. I have just added a -testify-threads:N option. Here are the times I get for running the entire suite on my new quad core Nehalem machine under Mac OS X Snow Leopard and Mono 2.4:

1 thread, 6:07
2 threads, 3:16
3 threads, 2:37
4 threads, 2:18
5 threads, 2:13

I haven't tested on Windows yet. If you have a Core 2 Duo, Quad Core or other such machine, please give this a whirl. I'm interested in hearing results including what op sys and VM you are running.

It used to take 25:00+ on my 2006 MacBook Pro. So ~2:20 is a nice speed up. :-)

The improvement is due to CobraSharp.dll, -testify-threads:N and new hardware.

How To Invoke

Posix:
Code: Select all
cd workspace/Source
testify -testify-threads:2

Windows:
Code: Select all
cd workspace\Source
testify -testify-threads:2

How It Works

When used, Cobra will build a thread-safe queue of the directories to be run (100-basics, 110-basics-two, 120-classes, etc.), then spawn N worker threads. Each thread will repeatedly grab a directory name from the queue and spawn another cobra.exe process to run -testify on just that directory. After the process is finished, the thread will look for more work to do from the queue, or gracefully end if there is nothing left.

After all threads have joined, the individual results files are concatenated into one and the individual files deleted.

Misc

There is probably room for improvement. For example, the number of tests and failures is not tallied. Also, the code is fresh, so there could be bugs. Patches are welcome.

There is a new -testify-results:filename option if you don't like the "r-testify" default.
Charles
 
Posts: 2515
Location: Los Angeles, CA

Re: Parallel Cobra test suite

Postby hopscc » Sun Sep 27, 2009 1:54 am

Pentium 4 Cpu 3.00 GHz, 1 GB Ram Win Xp Pro SP3

Std (no -testify-threads) 849 tests, 00:12.26.2
-testify-threads:2, 00:10:04.4531250
-testify-threads:4, 00:10:24.4375000


Can we lose the extra cruft re the thread implementation (threadpool, thread starting/tid/thread run time) in the output which makes it harder to see the pass/fail status for each test as
( or shortly after) its running...
hopscc
 
Posts: 632
Location: New Plymouth, Taranaki, New Zealand

Re: Parallel Cobra test suite

Postby todd.a » Sun Sep 27, 2009 5:28 am

ArchLinux distro

Xen VM: 2.6.24-24-xen
x86_64 Dual-Core AMD Opteron

-testify-threads:4
real 8m59.711s
user 9m9.750s
sys 1m6.652s

-testify-threads:2
real 6m22.894s
user 9m13.619s
sys 1m3.668s

Not as good as Chuck's quad core but still a significant improvement.
todd.a
Site Admin
 
Posts: 81
Location: Chicago, IL

Re: Parallel Cobra test suite

Postby hopscc » Mon Sep 28, 2009 5:07 am

Heres a patch for the multithreaded testify runner to
a) remove the extraneous cruft unless running at verbose level 2 or more
b) trap some exceptions that were ocasionally ocurring on my system when run MT
c) accumulate and emit the total number of tests and total number of failures and suppress the sub process thread timeits
Attachments
mt-testify-cleanup.patch
(3.8 KiB) Downloaded 346 times
hopscc
 
Posts: 632
Location: New Plymouth, Taranaki, New Zealand

Re: Parallel Cobra test suite

Postby Charles » Tue Sep 29, 2009 12:11 am

Patch gives 6 of 6 hunks failed. This probably has to do with the fact that somehow TestifyRunner.cobra ended up with non-standard line endings. For example on Moc:
Code: Select all
~/Projects/Cobra/workspace-b/Source $  le -r *.cobra
Line Endings

Action: Report

found text file (unix)   : Attributes.cobra
...
found text file (unix)   : SyntaxHighlighter.cobra
found text file (dos)    : TestifyRunner.cobra
found text file (unix)   : Tokenizer.cobra
...


I won't have time tonight to make the changes and my week is extra busy, but I'll take a look later.

-Chuck
Charles
 
Posts: 2515
Location: Los Angeles, CA

Re: Parallel Cobra test suite

Postby hopscc » Tue Sep 29, 2009 12:18 am

Oh Pox - My fault - I probably didnt run it through my edit-sanitise cycle
I'll repost a cleaned version
hopscc
 
Posts: 632
Location: New Plymouth, Taranaki, New Zealand

Re: Parallel Cobra test suite

Postby hopscc » Tue Sep 29, 2009 12:30 am

Here ya go - this one definitely passes the patchTst chk
Attachments
mt-testify-cleanup.patch
(3.8 KiB) Downloaded 347 times
hopscc
 
Posts: 632
Location: New Plymouth, Taranaki, New Zealand

Re: Parallel Cobra test suite

Postby eric.sellon » Thu Oct 01, 2009 8:18 pm

I finally tried on my laptop this against revision 2193

Core 2 Duo T6400
3 GB RAM
Ubuntu 9.04
Mono 2.0.1

1 thread
real 6m34.972s
user 5m46.946s
sys 0m26.698s

2 threads
real 3m43.291s
user 6m28.000s
sys 0m24.906s

3 threads
real 3m57.373s
user 6m26.824s
sys 0m24.914s

4 threads
real 3m58.527s
user 6m29.332s
sys 0m25.282s
eric.sellon
 
Posts: 24

Re: Parallel Cobra test suite

Postby Charles » Fri Oct 02, 2009 7:39 pm

webnov8 wrote:ArchLinux distro

Xen VM: 2.6.24-24-xen
x86_64 Dual-Core AMD Opteron

-testify-threads:4
real 8m59.711s
user 9m9.750s
sys 1m6.652s

-testify-threads:2
real 6m22.894s
user 9m13.619s
sys 1m3.668s

Not as good as Chuck's quad core but still a significant improvement.

Actually I don't see what your time is without the -testify-threads option... so I don't know what improvement you got.
Charles
 
Posts: 2515
Location: Los Angeles, CA

Re: Parallel Cobra test suite

Postby Charles » Fri Oct 02, 2009 7:40 pm

eric.sellon wrote:I finally tried on my laptop this against revision 2193
...

Sweet. Thanks for the numbers.
Charles
 
Posts: 2515
Location: Los Angeles, CA

Next

Return to Discussion

Who is online

Users browsing this forum: No registered users and 74 guests

cron