Request For Help: cobra2html via Pygments

[Charles]

Now that Cobra is going open source, I'll be putting out some "Request For Help" notices. I'm still on the hook for deeply rooted bugs and features in the compiler because I know the code base best. Those are going to keep me pretty busy. And while you're welcome to work on those too, I imagine some people might enjoy easier, auxiliary tasks that are also very important.

Here's one:

In the new release (0.7.4) under the Supplements directory, you will find a cobra2html.py file which sort of works. Except it's just using the Python syntax highlighter from Pygments which is a nice project that supports around 25 different languages. The goal is to provide a Cobra highlighter by writing one from scratch or copying the Python one and modifying it. Pygments is open source so we can do so.

Your contribution would be distributed with Cobra under the MIT open source license, or if that conflicts with Pygments for some reason, we could mark that file under their choice of the BSD.

Btw this would hopefully become the means by which we'd get highlighting on the discussion board, upcoming wiki and the online code samples (How To and Samples).

And yes we could investigate doing a Cobra implementation of Pygments if someone gets the itch. It might make a nice comparison to see the contracts, builtin unit tests, extra speed and so forth. And also to see what things about Python are still more convenient than Cobra.

But leveraging Pygments is a great place to start.

[khjklujn]

I took a brief look at pygments, but regexes are definitely not one of my strong points. However, it did occur to me that cobra already provides me with a lexer, and while this class isn't particularly elegant, it will produce syntax highlighted htmlized cobra code.

[Charles]

I'll check it out. It's still desirable to have Pygments support as it drives other canned applications like Trac.

For Cobra, I want the nodes to actually participate in syntax highlighting at some point. Words like "is" have more than one use (for "is": declarations and binary operator). Keywords can be used as method names in which case you would not want them highlighted as keywords. etc.

But using the lexer is a good start.

I still welcome a Pygments plugin for anyone who is still interested.

[themaniac]

With a huge health warning, given that it's my first time writing a lexer and I'm still learning cobra, please try

hg clone http://scm.themaniac.org/hg/pygments/

as a first attempt at writing a highlighter.

It worked with my little test file, but what would really help me is if you could put together allsyntax.cobra, a file containing all cobra's syntax in as condensed form as possible. Would obviously help others trying to learn the language too...

I fully acknowledge it's not finished (in particular I probably need to break down what subtype all the keywords are), but I hope it's better than nothing. I will continue to work on it as time allows.

Of course the next project going begging is writing a port of epydoc for cobra (unless you've already got ideas of how you're going to generate documentation?)

[Charles]

I'll try to take a look tonight. Ultimately, Cobra itself should be able to do both highlighting and doc gen, but even if highlighting were there, having Pygments is still needed because it drives other applications like Trac.

I don't have an allsyntax.cobra, but if you check out a workspace from http://cobra-language.com/source/ you will find hundreds of programs under Tests/. The larger source files like Source/CobraParser.cobra and Source/Expr.cobra make awesome test cases. See also EditorSupport if you haven't already.

-Chuck

[themaniac]

Fair cop - shouldn't have been so lazy. In fact, have been grabbing pieces of syntax from your howto section to try to create my own mini allsyntax.cobra with as many pathological cases as possible. The implementation is now looking pretty reasonable (my first attempt was indeed hideously broken), but I'll keep looking for bugs / missing language. In any case, it is now genuinely at a stage where it is better than nothing

Known bug:

Code: Select all

var _possessions = List<of Object>()
   """
      These are my possessions
   """


doesn't correctly register that there's a docstring rather than a triple-quoted string there, but given that there could be almost anything after the =, I can't currently think how to store enough state to parse the rest of the line correctly.

[Charles]

For "var _foo = <expr>" there is not going to be anything else on that line after the expression except a comment. Then it's a question of whether the next line is indented or not.

Python has this too, but does not require (or even accept) the indentation. The following is valid Python code.

Code: Select all

class A:
   
   b = 5
   """
   Hi there
   """


print A
"""
what?
"""
print A.b

Cobra does not support doc strings or standalone multi-line strings after statements like Python does.

HTH

[themaniac]

I think it's going to be a bit of a mission to upgrade the pygments lexer so that it is able to understand context to that degree.

I realise that once you've got

Code: Select all

var foo as String =


you know that the next line could have a doc string on it, but you need to parse the rest of the line before returning to the doc string. Think I'll have to spend more time with the source...

Otherwise, I still need to do more with the 'has' keyword as support is currently pretty basic, but I don't really understand what it means, so I'll have to look through your examples to see how it gets used.

[Charles]

"has" is used to add .NET attributes to a declaration. This is documented in the 0.7 release notes.

[Charles]

Okay, I tried it out.

For the time-challenged, here are some steps (sorry for the unix bias if you're on Windows):

1. Install Mercurial if you don't already have it ("hg" at the command line).
2. hg clone http://scm.themaniac.org/hg/pygments/
3. cd pygments
4. cp /path/to/Cobra/Source/Enums.cobra .
5. ./pygmentize -f html -O full,style=colorful -o Enums.cobra.html Enums.cobra

To crank out a separate stylesheet with a "cobra" prefix:

pygmentize -f html -S colorful -a .cobra > cobra-styles.css

I'm not sure how to get the ".cobra" context into the HTML generation. (I'm time-challenged too.)

Also, "colorful" is just one style. "emacs" is another. I'm not sure how to get the list of all available styles.

The docs for the Pygments command line are at http://pygments.org/docs/cmdline/

Regarding the Cobra highlighter, it's off to great start! The few issues I see from looking at the highlighted Enum.cobra are:

• The highlighter is confused by generics.
• Doesn't recognize the nilable suffix ("?" in "int?")
• Doesn't recognize the "!" in "foo to !" which is a typecast to non-nil.
• Declared method names get special highlighting, but not property names.