Request For Help: cobra2html via Pygments

[Charles]

The parsing problem with the ">=" you described above is now fixed.

[Charles]

First, a note for Subversion users. There is an "hg update" command, but what you really want is "hg pull". The fact that "hg update" exists might mislead you into thinking you're up-to-date when you're not.

I tried the latest on CobraParser.cobra. Overall, looking very nice! Here are some code bits it had trouble with:

# vari is a keyword which can precede an argument type
# if it's easier to just assume it can precede any type, that's fine
# when running Pygments, we can assume the code is valid. we don't need error checking
# so if looser definitions help, by all means take the easy route
    def optional(whatTypes as vari String) as IToken?

# a question mark can follow a generic type
    var _leftStack as Stack<of Expr>?

# generic types can be nested.
# also true in C# and VB if you need to consult what they did.
# VB also uses "Of" but with parens instead of angle brackets
    var _opStackStack as Stack<of Stack<of String>>?

# a typecast to non-nilable can use ! instead of restating the underlying type
    return _globalNS to !

# types can be qualified. that's true in C# and VB too so you might see what they do
    def makeList(t as System.Collections.IList, u as System.Collections.IList) as System.Collections.IList

# don't know why this errored. The arg list ends on a generic type, but that's not so strange.
    def nameForDeclGenericParams(token as IToken, paramList as List<of GenericParam>) as String

# this line had problems at the comma
    def commaSepExprs(terminators as IList<of String>, isSpaceAgnostic as bool) as List<of Expr>

Maybe any kind of symbol after a generic type is problematic. Possibilities include at least ? , and )

I started a Related Projects wiki page with this highlighter as the first entry.

Note that the wiki front page reads "This wiki is read only for now while we figure how to integrate user management between the forums and Trac (or decide to have two sets of user accounts)." If you want something posted there let me know here or via email (see "Contact" on the main nav of the Cobra site).

[Charles]

One more thing. The method names get wrapped with the .nf style. Can we wrap the property names too? Currently they are just "py" which isn't even in the stylesheet.

[themaniac]

That's weird - I see none of your problems on my version - are you sure you've updated? (iirc you have to do a pull followed by an update to get the latest code with mercurial - I don't have much experience of it - I only use it because it's what pygments use; istr pull pulls the stuff to your local repository and then update merges it with what you're currently looking at).

I'm not quite sure I understand your property point, but I've added a colour to the colorful stylesheet so now properties get some styling - hope that's what you were looking for

[Charles]

Before I started testing I did this:

Code: Select all

$  hg pull
pulling from http://scm.themaniac.org/hg/pygments/
searching for changes
adding changesets
adding manifests
adding file changes
added 8 changesets with 11 changes to 2 files
(run 'hg update' to get a working copy)
$

And apparently ignored that last statement that said I should run 'hg update'. So I tried that now:

Code: Select all

$  hg update
2 files updated, 0 files merged, 0 files removed, 0 files unresolved

And now all the explicit errors are gone. Yay!

Check out the styles for a declared method name vs. a declared property name (I put -----> at the relevant locations):

Code: Select all

   <span class="k">def</span> -----> <span class="nf">init</span><span class="p">(</span><span class="nv">fileName</span> <span class="k">as</span> <span class="n">String</span><span class="p">,</span> <span class="nv">msg</span> <span class="k">as</span> <span class="n">String</span><span class="p">)</span>
      <span class="k">base</span><span class="n">.init</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
      <span class="n">_fileName</span> <span class="o">=</span> <span class="n">fileName</span>
      <span class="n">_lineNum</span> <span class="o">=</span> <span class="mi">1</span>

   <span class="k">get</span> -----> <span class="py">hasSourceSite</span> <span class="k">as</span> <span class="kt">bool</span> <span class="kd">is</span> <span class="kd">override</span>
      <span class="k">return</span> <span class="k">true</span>


The "init" gets '<span class="nf">' but the property gets '<span class="py">'. Where is the "py" coming from? There is no stylesheet definition for it and of course it stands for "Python" but this should all be Cobra-oriented by now.

Also "py" gets used for 4 of the first 5 declared variable names in the Parser class. The incorrectly marked variable names are for any line of the form "var _foo = expr" (as opposed to "var _foo as Type").

I'm using CobraParser.cobra like so:

Code: Select all

./pygmentize -f html -O full,style=colorful -o CobraParser.cobra.html CobraParser.cobra

[themaniac]

py stands for property, not python, and I've now added it to the colorful stylesheet by editting styles/colorful.py

The correspondence is shown in tokens.py http://scm.themaniac.org/hg/pygments/?f=c5455256c8a8;file=pygments/token.py;style=gitweb

If vars aren't consistently set to the same thing I'll try to fix it.

[themaniac]

all vars are now shown as Variable - was just a silly typo / c+p-o

[Charles]

I gave it another test drive this morning. Looking really good; I saw no errors. Feedback:

-- If "nf" is for "named function" then "py" should probably be called "np" for "named property". In my opinion, it should also default to the same color as named function.

-- The boolean literals "true" and "false" should be broken out so they can be colored different than keywords like "while". For example, you might color literals such as ints, strings and true/false the same color (but a different color than keywords).

[Charles]

And I guess the same thing for "nil" if it's not already.

[themaniac]

'py' for property isn't my choice. Assuming the endpoint is to submit this back into pygments, I reckon it'd be pretty bad manners to change it to np, though I agree it would be more logical for Name.Property (they don't even have the excuse that np is already taken).

I reckon true/false/nil are better described as Keywords than anything else (Name/Literal/String/Number/Operator etc) so I've added a Keyword.Boolean and Keyword.Null (I reckon null is more common than nil, but I may be in a minority - doesn't really matter). (I'd be surprised if those additions didn't get rejected by the core pygments team though / rejected unless I change *all* the other languages to reflect them). The colours are chosen in styles/colorful.py so you can make them whatever you want (I reckon that colorful should probably make everything different; the other stylesheets can be more sane).

btw, I like the way you can use _ as a separator in numbers (e.g. 1_000). However, I think it'd be very bad style to write e.g.

Code: Select all

thousand = 1_0__0___0


I know it's just sugar, but perhaps we could warn if the _s don't separate powers of a thousand?