small Tokenizer cleanup patch

[hopscc]

Heres a patch to Tokenizer.cobra to add some commented timing instrumentation to nextToken to evaluate various changes
plus some code cleanup/refactoring of nextToken itself.
No additional tests, no relnotes comments

[Chuck]

Overall, looks nice. I'd like a new patch, though:

• Use doc strings instead of top comments
  
• Use more descriptive var names like avgTicks instead of avt. I know it seems unimportant at the time, but it really helps others read your code.
  
• I don't think you need (tokens.count to int) instead of tokens.count unless there is a bug to report
  
• I find it easier to disable code with if false than commenting all the lines out, but it's not a requirement for the new patch. With if false you still get syntax highlighting and it's very easy to re-enable the code (even easier than removing /# and #/ from two different locations).
  

[Chuck]

Never mind a new patch as I'm taking care of this myself as part of applying the "no warn" patch.

[Chuck]

Done. The directive is:

    # .no-warnings.

The space between # and . is not relevant. Case-sensitive.

Also, I set up a reg-ex for this format. If we have other future directives, it will be easy to recognize them.

Maybe "%% foo" should be converted over?

[Chuck]

Woops that's the comment for ticket:37 which also included these code cleanup improvements.