Alter Unit Test Assert behavior to continue until exit

[torial]

Currently when the application is run, all unit tests will run until an assert fails. This (on Windows) gets funneled up and Windows Error Reporting gives an option to report to MS or to debug the application or cancel.

Suggested changes:
* For tests only, make it so asserts don't throw a terminating exception, but do trigger the application to exit.

* Allow all tests to be run before triggering the application to exit. This doesn't include other asserts in the same test, rather any other "test" blocks for other functions or classes.

Possible implementation ideas:

Auto wrap the test case in a try/catch block, if the assert exception occurs, set an app exit flag, and log the assert info for the report. This should make it possible to run all tests. Then check the flag to see if the application should terminate.

[Chuck]

Thanks for the ticket.

What about an interface? When the program is invoked, how does one specify what the behavior should be? In other words, how should be do test flags or test control?

[torial]

Currently the application will continue if it passes all its tests, or throw an untrapped exception if it fails an assert. So the primary change I'm suggesting is that the application exits cleanly w/o throwing an exception, and reports to the Err stream/console that tests have failed.

The secondary change would be to make it run all the tests and indicate which ones failed if it exits the app.

In terms of flags, I'm not sure I understand. I think at some level I think it would be cool to have a tool that parses the cobra code, finds which classes and methods (and other stuff?) has test code, and provides a simple list of them (e.g. class WordMatcher? has a test, so it is simply called WordMatcher?, and method addTargetWords of the class has a test, so it is called WordMatcher?.addTargetWords), show the status (e.g. green for success, red for fail), and give any test that is associated with it. I guess I don't see that as the scope of this ticket :-)

[torial]

err.. "any test that is associated" should read "any test result that is associated"

[torial]

Chuck, I'm assigning this to you so you'll see it. If that is improper protocol, let me know.

I'm not sure this is the right location, I'm getting the original code from CobraCore.cobra (and it could be old old..). I'm also not brave enough to try to run the build, as I'm afraid of mucking up what I have working :-) Maybe in the future, but not yet!

            def runAllTests
                """
                Run all Cobra `test` sections in all assemblies using reflection to locate them.
                """
                # start = DateTime.now
                _runAllTests(Assembly.getEntryAssembly, Dictionary<of String, String>())
                    Application.Exit
                # duration = DateTime.now.subtract(start)
                # trace duration

            def _runAllTests(ass as Assembly, found as Dictionary<of String, String>)  # CC: found should be Set<of String>
                name = ass.getName.toString
                if found.containsKey(name)
                    return
                found.add(name, name)
                if _skipAssemblyNameForTesting(name)  # saves some time
                    return

                # print 'Testing assembly:', ass
                for type in ass.getExportedTypes
                    members = type.getMember('RunTestsIfNeeded')
                    if members and members.length and members[0] inherits MethodInfo
                        method = members[0] to MethodInfo
                        # trace type, method
                        if type.containsGenericParameters
                            # TODO: this is quite awful. tests in generic types are being skipped!
                            # potential solution 1: move type test out of the type into a "sister" type: private __FooTest
                            # potential solution 2: require a constructed type in Cobra: `test Foo<of int> ...` and then put that in a method attribute in the gen C# and use that type
                            continue
                        method.invoke(type, nil)

                # traverse further assemblies
                for assName in ass.getReferencedAssemblies
                    subAss = Assembly.load(assName)
                    _runAllTests(subAss to !, found)}}}

What if it were changed to something like:

            def runAllTests
                """
                Run all Cobra `test` sections in all assemblies using reflection to locate them.
                """
                # start = DateTime.now
                if !_runAllTests(Assembly.getEntryAssembly, Dictionary<of String, String>())
                    Application.exit
                # duration = DateTime.now.subtract(start)
                # trace duration

            def _runAllTests(ass as Assembly, found as Dictionary<of String, String>) as bool  # CC: found should be Set<of String>
                name = ass.getName.toString
                if found.containsKey(name)
                    return true
                found.add(name, name)
                if _skipAssemblyNameForTesting(name)  # saves some time
                    return true

                result = true

                # print 'Testing assembly:', ass
                for type in ass.getExportedTypes
                    members = type.getMember('RunTestsIfNeeded')
                    if members and members.length and members[0] inherits MethodInfo
                        method = members[0] to MethodInfo
                        # trace type, method
                        if type.containsGenericParameters
                            # TODO: this is quite awful. tests in generic types are being skipped!
                            # potential solution 1: move type test out of the type into a "sister" type: private __FooTest
                            # potential solution 2: require a constructed type in Cobra: `test Foo<of int> ...` and then put that in a method attribute in the gen C# and use that type
                            continue
                        try
                            method.invoke(type, nil)
                        catch exc as Exception
                            result = false

                # traverse further assemblies
                for assName in ass.getReferencedAssemblies
                    subAss = Assembly.load(assName)
                    result = _runAllTests(subAss to !, found) and result
                return result

[hopscc]

Heres a patch for Torials idea corrected for syntax.

[hopscc]

new patch file fixed line endings

[Chuck]

I'll take a look.

[Chuck]

What I meant by interface and flags, is how do we let the developer pass options to the test runner?

For example, how does the developer specify whether tests should be run in the current way (which is useful when using a debugger) or the proposed way (which is useful to get the whole picture)?

Other interesting parameters to a test runner include tests to skip, or alternatively, a complete list of tests to run.

I wonder if what we really need is some kind of basic TestRunner? class which the developer can subclass and then inform Cobra to use. That still leaves the question of how to specify the TestRunner? class, but it at least gets it down to one point.

[Chuck]

I have a partially complete TestRunner? class constellation that will fix this ticket as well as make the test running customizable by developers.

[Chuck]

Done. This is now the default behavior for unit tests... the exception is printed and execution continues with the next unit test.

One or more failures causes all output to be displayed.

I did not use the patch only because we were overdue to break out running tests to a TestRunner? class family.