Need planning for Cobra (my opinion)

[eric.sellon]

I apologize for the length of this post. Further I apologize if I step on any toes or cover material already trodden (I've only been aware of Cobra for three weeks now so I don't know everything that's been done/previously discussed). I just wanted to get discussion and planning started.

As Cobra approaches a version 1 release, I think there are some things that need to be worked out:
1) Front End - Back End architecture
2) Portability of Cobra programs (Cobra libraries)

Topic 1: Front End - Back End architecture
The whole idea of adding back ends to Cobra is really cool. If the language takes off, I think you can expect back ends to developed as projects independent of the compiler (really this means independent of the front end as you can't have a compiler without a front and a back end). This is totally natural. Just look at Python. The reference implementation is CPython but the ports to .NET (IronPython) and the JVM (Jython) is sort of equivalent to a new back end for Cobra. There's even CLPython which compiles Python programs into executables via Common Lisp (I haven't tried it yet, but I love the idea).
In order to allow easy integration and creation of new back ends, there needs to be a document that specifies the language (versioned and tied to the front end) and a document that describes how to implement a back end (especially in terms of dealing with the front end and what it is required to be there outside the code generation as well as how to bootstrap a new back end). A versioned front end allows the back end to say what version(s) of the front end it will work with (and that essentially means what version(s) of the language it supports). I envision that the front end will have basically three phases: tokenizing, parsing, and validation (like an XML validator, it would make sure the program was well-formed....note: this does not mean correct). The back end would do the rest. The main reason that the front end wouldn't do more is because it wouldn't be aware of the specific platform's APIs (and it shouldn't be).
The question is how would the front end and back ends be compiled? Would the front end be one executable (and which back end would you build it with...probably multiple ones) and the back end another? Or would you build one executable for each platform (in that case the back end could basically be a library...or the front end could be....but probably not since you'll want basically the same switches available no matter the platform)?
Also from a command-line perspective, what about building libraries and linking statically or dynamically (passed on to the back end)? Include/source directories?

Topic 2: Portability of Cobra Programs
Obviously it is very important to allow access to platform-specific libraries. However, writing portable programs is important too. This portablilty is obviously achieved with having standard parts to the language (especially the standard library). Picking the contents of the standard library greatly impacts what Cobra third party libraries get written. I would suggest that part of the standard library is written in Cobra and comes with the front end and is pulled from a common directory by the back end. Other parts will be implemented in a platform-specific manner (possibly with the use of third-party libraries like CPython using tkinter) and will be provided by the back end. I'm thinking Cobra.Core for pure Cobra stuff, Cobra.Standard that will mix Cobra with native platform implementation and Cobra.StandardExtended that is most likely pure native platform implementation. Also a back end would have to provide all of Cobra.Standard to be considered a basic Cobra platform. Implementing Cobra.StandardExtended would allow it to be considered a full-featured Cobra platform. For example, sorting algorithms might be in Core or Standard while Web and GUI namespaces might be in StandardExtended. The GUI one makes a good example: it could be implemented with WinForms in .NET/Mono, with Swing on the JVM, with Cocoa for Objective-C, and with a third-party library (such as Qt or something built on top of SDL) for C++.

Conclusion
Basically planning on these issues needs to be done. This planning should be captured in some sort of documentation. Obviously it's too early to have to make everything a change proposal (or PEP like Python), but certainly planning and design documentation would be good things.

[Charles]

-- Actually Python is a really poor example of multiple back-ends. To my knowledge, the implementations of CPython, IronPython and Jython share nothing (other than some pure Python modules for the std lib). The obvious problem is duplication of effort. The next problem is that they fall out of date with each other. Sometimes all three can be on different versions. Also, you have dialects like Pyrex for doing things like adding static types, but then your Pyrex code can't run under CPython, Jython, etc. because it's not the same language--it's a dialect.

Contrast that with Cobra where I'm addressing these problems by having a self-hosted compiler and multiple back-ends. Once the JVM back-end is mature, I'll be adding this to the "Comparison to Python" page.

-- There are some architecture notes in <workspace>\Developer\ImplementationNotes.text although it is somewhat out of date.

-- A language spec would be nice, but between the test suite and the Cobra implementation, we can get by. I certainly don't want to see people writing a lexer, parser, symbol binder, etc. from scratch instead of reusing the ones we have.

-- I agree the architecture needs more documentation including a "How To Add A Back-End", but note that multiple back-ends are new to Cobra and undergoing changes as I work on the JVM back-end. I'm not ready to document that.

-- I do heartily agree though that we should empower people to work on back-ends independently. I hope someone takes up ObjC and JavaScript, for example.

-- Regarding phases, check out <workspace>\Source\Phases. Also, see <workspace>\Source\Compiler.cobra for where the phases are specified and created. Note that the back-end object is given the opportunity to add more phases.

-- Your idea of versioning the front-end is intriguing. Will have to think more about that.

-- The back-end is baked into the Cobra command line executable. I certainly want to see the ability for it to be a library (.dll, .class, etc.) in the future.

-- In terms of bootstrapping a new back-end, you would typically do cross compilation where you run Cobra under a mature platform (currently .NET), but specify -back-end: to point it to a different one (currently -back-end:jvm). If you can get all test cases to pass then in theory, you should be able to compile the compiler and voila! Cobra is now fully hosted on the new platform. Of course, I'm sure there will be snags, especially for the first back-end. The ones after that should get progressively easier.

-- How should additional back-end-specific arguments be passed? For example, let's say I have "cobra -back-end:javascript foo.cobra" but I also want to pass some specific options to the javascript back-end?

-- I refuse to comment on "Topic 2" because it should be a separate forum topic.
I'll copy+paste it and reply there.

[Charles]

Btw, no problem on lengthy posts. Your interest in Cobra is great. I just ask that you do a separate post for each topic so that they can be treated separately.

[eric.sellon]

Your point about Python implementations falling out of date with each other was part of my reason for suggesting versioning the front end.

Actually I did read your text file on the compiler's implementation before I posted (not that I mentioned that). That was one of my reasons for posting as I was somewhat concerned by what I read. I didn't come out and say it in the first post, but what I read made me concerned that each back end would require front end changes. It sounded like the Tokenizer was relying on a .NET type. Also the bind uses phase seemed to be in the front end. That made me wonder how using a platform's API would work. One other thing that worried me was that a change was checked in for the JVM back end to use Java's println function for print. I didn't look closely to which file was affected but I thought it might be a front end file.

I don't mean for this to be a "sky is falling" type of post. My thought was that the earlier we looked at these issues (if they are really concerns and not products of an active imagination) the better.

I agree with much of what you wrote. I was simply throwing out possible solutions to possible problems.

One other thought I'd like to leave you with is perhaps we skip doing a JavaScript back end. Instead I suggest doing a haXe back end. The reason is that they have multiple back ends and, being lazy, I think it would be more efficient to leverage what they've already done. They currently have back ends for JavaScript, PHP, Flash, and NekoVM (their own virtual machine). They are working on a C++ back end. I don't really know anything about the language; I've only done the "Hello World" demo in Flash.

[Charles]

There are still dependencies in the compiler on .NET types, but that doesn't prevent the implementation of cross compilation which is the next step. Things actually used to be much worse--everything you see in BackEndClr\*.cobra used to be mixed in with the main compiler!

I'm not familiar with println being used at this point. I can't even grep it in the code so if you recall where you saw it, let me know.

Re: haXe, if it can handle all of Cobra's features either directly, or with some extra code gen, then it certainly looks like a nice option. Note that the compiler is GPL not LGPL. I think this means we are forever banned from linking the haXe compiler and Cobra compiler into one binary. See http://haxe.org/doc/license

Even if we use haXe, I want Cobra to maintain a multi-back-end approach to keep our options open.

[eric.sellon]

I glossed over the commit comments real fast. I think I saw the one about a change for newLine made for portability .... so I was just misremembering what I saw.

Even if we use haXe, I want Cobra to maintain a multi-back-end approach to keep our options open.

I wouldn't expect that to change. Right now their back-ends don't overlap Cobra's C# one (nor the in-progress JVM or the planned Obj-C). Further I don't see it as a big deal if an overlap occurs eventually.
As for the license, I would think it would be ok to translate into haXe and then call their compiler (like we do for C#). Of course I'm not a lawyer. We don't need to go down the haXe path, I was just throwing out an idea.

It sounds like you're aware of the dependencies on .NET in the compiler. Are you planning (once you can cross compile the JVM back end) to compile the whole compiler with the JVM back end? That was one of my questions (not quite worded that way....it was a more general question then) in the first post.

Thank you for your previous responses. As I said before, I'm just trying to open some discussion. I'm not trying to be critical. I think you realize that, but it doesn't hurt to reiterate the point.

[Charles]

I glossed over the commit comments real fast. I think I saw the one about a change for newLine made for portability .... so I was just misremembering what I saw.

Even if we use haXe, I want Cobra to maintain a multi-back-end approach to keep our options open.

I wouldn't expect that to change. Right now their back-ends don't overlap Cobra's C# one (nor the in-progress JVM or the planned Obj-C). Further I don't see it as a big deal if an overlap occurs eventually.
As for the license, I would think it would be ok to translate into haXe and then call their compiler (like we do for C#). Of course I'm not a lawyer. We don't need to go down the haXe path, I was just throwing out an idea.

Sure. If someone wants to pursue it, that will be interesting.

It sounds like you're aware of the dependencies on .NET in the compiler. Are you planning (once you can cross compile the JVM back end) to compile the whole compiler with the JVM back end? That was one of my questions (not quite worded that way....it was a more general question then) in the first post.

Yes, the idea is that the compiler would self compile on any platform. Just not there yet.

Thank you for your previous responses. As I said before, I'm just trying to open some discussion. I'm not trying to be critical. I think you realize that, but it doesn't hurt to reiterate the point.

And likewise for my posts.