Regression test problems re tests with CultureInfo assumptions

[Chuck]

Reported by jonathandavid:

About the failing tests:

So far I've been able to fix the following ones:

(29) 062-string-substitution.cobra
(85) 120-primitive-type-members.cobra
(101) 310-primitive-members.cobra
(133) 510-number-parse.cobra
(267) 100-number-float32.cobra
(269) 100-number-float64.cobra
(273) 106-number-decimal.cobra

I've fixed all of them by adding CultureInfo??.invariantCulture when calling toString or parse. Otherwise, my locale turned "3.2" into "3,2", which wasn't expected by the tests.

I encountered my first problem with test "(130) 504-sharp-error.cobra". This test assumes that the C# will produce error messages in English, which is not the case in my installation.

A similar thing happens with "(445) 500-cannot-infer.cobra", and with "520-debug.cobra". I think we can relax these tests a little bit so that they don't look for specific messages, but rather for things that won't change on a non-English locale. For example, instead of looking for a warning that contains "related to", we can look for a warning that contains simply the line that causes the warning "(3)". If you want I can perform these modifications, and once all tests pass in my installation, create a patch and submit it here.

A different kind of problems occurs with "(639) 410-use-lowernames-dll.cobra". This took me a while to figure out. Apparently, the test is referencing a lowernames.dll file that should be in the current path. Instead, there is a lowernames.cs file. I got the test to pass by manually compiling this .cs file, which generates the necessary dll, but this is not done automatically when I install from source and run -testify. I don't know how to solve this problem, can you help me?

Also, there seems to be a minor issue with the testify script itself. When a test fails, the test count is incremented twice, and the test name is written twice. For example:

...

(taken from ticket:146)

[Chuck]

• Can you submit a patch for the test cases you already fixed so I can apply it now? It doesn't sound like they depend on other upcoming fixes.
• Regarding the tests with C# english messages, the real fix is to have Cobra catch those errors instead of letting them fall through. Well with the exception of 504-sharp-error.
• I'm not sure what the problem is with 410-use-lowernames-dll.cobra. Did you say what your platform is? Op sys, .NET version?
• I'm not too worried about the extra increment in testify, but if you submit a patch to fix it, I'll certainly accept it. The running twice is intentional, as you probably know. It runs verbosely to help with diagnosis.

[Chuck]

Yes, you did say your platform in a prior thread. XP and .NET 2.0.

I'd like a patch for the fixes you have so far since they don't seem affected by the outstanding problems.

[jonathandavid]

Fixes culture-related issues on several regression tests

[Chuck]

Applied patch in changeset:1953

[Chuck]

Regarding "A similar thing happens with "(445) 500-cannot-infer.cobra", and with "520-debug.cobra". ..." if these messages are localized then they are coming from the C# compiler. The fix I would prefer at this point is to generate the error or warning from Cobra. We'll need to eventually anyway. (And even when Cobra gains localized error messages, we could have testify run under English for its verification purposes.)

Regarding "A different kind of problems occurs with "(639) 410-use-lowernames-dll.cobra". ..." I notice you say "when I install from source and run -testify"! I don't think anyone has ever done that before. I run directly from a workspace, not from the installed version of Cobra. But then I don't see Tests in my C:\Cobra\Cobra-<version> so I'm not sure I follow you here.

Regarding "Also, there seems to be a minor issue with the testify script itself. ..." I think we can leave that as is. The second run is intentional to provide more debug info. The extra increment is no big deal.

[Charles]

@jonathandavid, do you still experience the remaining problems?