Parallel Cobra test suite
Posted: Sat Sep 26, 2009 5:11 amFor those of you hacking on the Cobra compiler, you are no doubt familiar with running "testify" to see how your changes have affected the regression test suite. I have just added a -testify-threads:N option. Here are the times I get for running the entire suite on my new quad core Nehalem machine under Mac OS X Snow Leopard and Mono 2.4:
1 thread, 6:07
2 threads, 3:16
3 threads, 2:37
4 threads, 2:18
5 threads, 2:13
I haven't tested on Windows yet. If you have a Core 2 Duo, Quad Core or other such machine, please give this a whirl. I'm interested in hearing results including what op sys and VM you are running.
It used to take 25:00+ on my 2006 MacBook Pro. So ~2:20 is a nice speed up.
The improvement is due to CobraSharp.dll, -testify-threads:N and new hardware.
How To Invoke
Posix:
Windows:
How It Works
When used, Cobra will build a thread-safe queue of the directories to be run (100-basics, 110-basics-two, 120-classes, etc.), then spawn N worker threads. Each thread will repeatedly grab a directory name from the queue and spawn another cobra.exe process to run -testify on just that directory. After the process is finished, the thread will look for more work to do from the queue, or gracefully end if there is nothing left.
After all threads have joined, the individual results files are concatenated into one and the individual files deleted.
Misc
There is probably room for improvement. For example, the number of tests and failures is not tallied. Also, the code is fresh, so there could be bugs. Patches are welcome.
There is a new -testify-results:filename option if you don't like the "r-testify" default.
1 thread, 6:07
2 threads, 3:16
3 threads, 2:37
4 threads, 2:18
5 threads, 2:13
I haven't tested on Windows yet. If you have a Core 2 Duo, Quad Core or other such machine, please give this a whirl. I'm interested in hearing results including what op sys and VM you are running.
It used to take 25:00+ on my 2006 MacBook Pro. So ~2:20 is a nice speed up.
The improvement is due to CobraSharp.dll, -testify-threads:N and new hardware.
How To Invoke
Posix:
- Code: Select all
cd workspace/Source
testify -testify-threads:2
Windows:
- Code: Select all
cd workspace\Source
testify -testify-threads:2
How It Works
When used, Cobra will build a thread-safe queue of the directories to be run (100-basics, 110-basics-two, 120-classes, etc.), then spawn N worker threads. Each thread will repeatedly grab a directory name from the queue and spawn another cobra.exe process to run -testify on just that directory. After the process is finished, the thread will look for more work to do from the queue, or gracefully end if there is nothing left.
After all threads have joined, the individual results files are concatenated into one and the individual files deleted.
Misc
There is probably room for improvement. For example, the number of tests and failures is not tallied. Also, the code is fresh, so there could be bugs. Patches are welcome.
There is a new -testify-results:filename option if you don't like the "r-testify" default.