Java Back End

[Charles]

What are you thoughts about generating Java source vs. using the ASM library?

[hopscc]

Java is easier ( ) cos

• I know java, I dont know JVM (ASM) bytecode
• I can look at the generated source without needing disassembly and see where it's wrong and 'fix' it if needed
• I can tweak the gen source, compile and run it to explore what needs to be generated
• I can copy the C# code generation
• Its easier/faster to turn cobra src -> java src than cobra src to bytecodes - closer level of abstraction
• I can get something useful working faster with gen java->compile source ->run
• I can get better code off a java compiler than I can from my own ASM generated (handrolled) jvm bytecodes

The downside is

• Its likely that direct conversion to bytecode and execution is a faster process than conversion to java src + compile + run (once correct).
• You will need a java compiler

So for the moment I'm sticking with java.
Theres nothing precluding evolving into ASM use once the java gen+compile+run version is working and in fact it might be useful/necessary for such things as Properties...

[Charles]

Understandable.

Btw there is some notion of "properties" in the Java community. See http://download.oracle.com/javase/tutorial/javabeans/properties/.

I haven't studied them in depth to know how well they map back and forth with Cobra/.NET properties.

Also, under the hood, the .NET VM has no real notion of properties. Everything is a method such as get_Foo and set_Foo with an attribute attached to indicate that the method is special.

[hopscc]

OK - That took a little longer than expected....

Theres a mongo patch ( in 2 parts) on ticket:275 that provides an initial stab at a java back end for cobra. Its working for relatively simple cobra programs but theres still a whole lot missing wrt both code generation and features.

The patch took longer than I anticipated cos there were a whole lot of changes pushing backend dependencies/differences into a separate code block that can be swapped rather than wired directly through the codebase, New Types to abstract but allow handling of the back end native Types and some type name mappings (compiler to native). I probably still havent got them all but its better than it was.

More than that though since its a cross compiler at this point, was getting the java class info/descriptions from a java package or jarfile nto the cobra compiler running under .Net for use when inferring types and verifying call signatures.
What I chose to do here was provide a (java) tool (PkgSig) to inspect a package or jarfile and generate a text signature file for its contents ( a pkgsig file ).
The java backend looks for these .sig files (on startup for rt.jar cf mscorlib.dll or if referenced directly) and parses them into a set of Objects describing the class/methods/fields/properties etc. These objects are used to populate the namespace records when referenced just the same as the .Net backend.

Theres the start of a java runtime library ( CobraLang.dll equiv as CobraLang.jar) currently all done as java code but its very minimal at the moment. Assertions are currently done using the java assert handling but this is temporary...

It seems to compile and run the first 15 or so files ( up to 032-*) of the Test/100-* source files - at least thats the ones I've got to so far...
Theres still large chunks of java code gen not implemented - that'll probably be the next push but it should fall out of getting more tests working now that the java class symbol lookup/matching seems to be going.

Property support so far seems to be trivial ( I've only done input so far mapping getters and setters from classes jarfiles) - codegen will be just remapping back to java getter and setters from cobra code,

Generics dont seem to be causing any problems so far which probably means I'm missing something.

Other areas that are deficient (in no particular order)
- cobra to java source file naming - probably move to generating a separate file per class as javac likes that
- currently just gens java class files - should extend to make a complete app jarfile (at least for multi file programs)
- Cleanup the debug/info/status output
- handling of javac output needs to be better along with mapping of cobra line numbers to java code lines
- Testing (inline code) and Testify runner
- attribute/annotation handling
- Assertions and Exceptions

Possibly problematic issues are
Mapping entrypoints throwing (java) Exceptions to throwing RuntimeExceptions instead somehow
Implementing Delegates
Events vs Observers
Partial classes should be relatively easy ( additional accumulate phase) as should Extension class support
(extra lookup if not already doing so)

The backends (.Net, java and ObjC stub) are currently done as separate partial classes which is fine if a little bloating, I anticipate it should be relatively simple to eventually convert these to demand loadable/referenced Objects.


A Question for you Chuck.
Whats the criteria in scanClrType.cobra in _fixNilableMemberSigs for determining if a Class member needs 'fixing' ?
i.e whether it should be referenced in that method.
is this fixing them to be nilable or not nilable ?

[Charles]

First, thanks for making headway on this very non-trivial task.

Regarding the Java library scanning, what do you think about using IKVM to read it more directly? I think you could use the Java reflection classes to do so, which is what the compiler will ultimately end up doing once it is running on the JVM.

Regarding the partial classes, they were easy to implement, but definitely increased the bloat. They probably should have been done as visitors instead. This would also allow an abstract base class where some code could be shared, and then subclasses could implement abstract methods concretely. This would eliminate the "branch" statement in _callNativeScanMethod, for example.

.NET (as well as JVM) has no notion of nilable vs. non-nilable for reference types. So Cobra is forced to treat all reference types read from a DLL (at least a non-Cobra DLL) as nilable. This means that the .replace method for String would be:

class String

    def replace(find as String?, replacement as String?) as String?

In practice, the truth is that .replace always returns a String and never returns null/nil. The .fixNilableMemberSigs makes the adjustment, always in the direction of making things non-nil.

The classes and members you see there were hand-picked by me based on experience. A more systematic approach would be preferable. For example, the guys who work on contracts at Microsoft created such annotations maybe as far back as 3 years ago. I learned about this when I saw their presentation on Spec# at a Lang.NET conference.

Also, it would be preferable if the types and members were listed externally to the compiler, perhaps in a way that relates to the name of the library. But so far that hasn't been much demand in this area.

[hopscc]

I had a look at IKVM at one point but erred on the side of not adding an additional library dependency to the compiler initially.
I was also a little worried about how long such a package/jarfile load/scan takes - sensibly as it turns out.
The class load/parse is pretty localised so I figured I could revisit at a later stage if desired
- perhaps preparatory to a native Java implementation
- The intentionis that the functionality of PkgSig.java will eventually migrate to as close to the compiler as it can get.
One advantage of having a .sig file is that it acts as a pregenerated cache ( jython for example does this from an intial scan of the referenced class files - an initial run is s-l-o-w). You can get an idea of the overhead by running pkgsig on rt.jar (admittedly the most pathological case)
There are of course some disadvantages...

( besides I needed a refresher in writing java code - since using cobra I have gained a simplifying tendency to elide ';' and '{', '}' )
Now days java code just looks crufty...)

The longterm idea is to generate a separate package/Objects as a loadable module that handles the details of the backend implementations.
Visitor makes sense here but I wanted to get the backend dependencies identified and something up and working relatively quickly (ha!)
so I just beat on the existing architecture with a changed and added set of methods indirected through a backend 'switcher' and linked by a similar naming form.


yeees - well spotted re _callNativeMethodScan.
The use of _callNativeMethodScan was a quick hack to save me using the backEnd indirection and having to create 3 versions of every method call (in 3 places ) - I was getting a taste of success when I came to this and just wanted to get something going.
To be consistent with the rest it should be (will be) rolled into the BackEnd(s)

THx for folding the patch into the based source tree so quickly that'll make the continuing/follow on work much easier.
Subsequent patches will be bits of consolidation/cleanup and supporting more new stuff.


re fixNilable:
OK so basically the approach here would be to take a stab at which system lib class methods should be forced nonnilable and adjust on demand.
I saw the comment re pushing this out externally and agree but couldn't get a handle on what the exact initial criteria of choice was ....

Onward....

[hopscc]

Another question for ya Chuck.
I've probably asked this before but I seem to have a (blindness) issue with this everytime I come across it and I cant remember or find the answer

Symbols/Classes from used/referenced Dlls (Assemblies) arent bound ( bindImp, bindInt ) via the BindXXXPhases calling down through the Modules contents
- Modules.cobra comment in AssemblyModule bindInt/bindImp says "It's too expensive to scan all types in a DLL. Do them as needed."

Where is the "binding as needed" done? - what/where triggers these bind phases on (accessed) types when they are referenced ?

Is it Boxes.cobra Box._prepLibraryBox ??

I'm banging my head against a Exception in the compiler running with java backend where it throws a requireException in
AbstractMethod.ConstructedFor due .didBindInt being false..
( on AbstractMethod for 'entrySet')

Code: Select all


Unhandled Exception: Cobra.Lang_ert_fa10e98832857485b6fe8402ece2c12e.RequireException:
sourceSite = C:\home\hops\src\cobra\wkspace\Source\Members.cobra:329 in AbstractMethod.constructedFor for object Method-sh(52663, name=entrySet, didBindInh=false, didStartBindInt=true, didBindInt=false, didBindImp=false, token=Token.empty, name=entrySet, isNames=['public', 'abstract'], parentBox=Interface-mi(3101, name=Map<of,>, didStartBindInh=true, didBindInh=true, didStartBindInt=true, didBindInt=true, didBindImp=false, token=Token.empty, didBindInh=true, isGeneric=true, isGenericDef=true, needsConstruction=false, 3101), returnTypeNode=NilableTypeProxy-mi(52662, didBindInh=false, didBindInt=false, didBindImp=false, innerTypeProxy=JvmTypeProxy-sh(52661, didBindInh=false, didBindInt=false, didBindImp=false, jvmType=JavaInterface java.util.Set<java.util.Map$Entry<K, V>> is abstract public inherits java.util.Set`1 , 52661), 52662), returnType=nil, implementsType=nil, 52663)
info       = nil
this       = Method-sh(52663, name=entrySet, didBindInh=false, didStartBindInt=true, didBindInt=false, didBindImp=false, token=Token.empty, name=entrySet, isNames=['public', 'abstract'], parentBox=Interface-mi(3101, name=Map<of,>, didStartBindInh=true, didBindInh=true, didStartBindInt=true, didBindInt=true, didBindImp=false, token=Token.empty, didBindInh=true, isGeneric=true, isGenericDef=true, needsConstruction=false, 3101), returnTypeNode=NilableTypeProxy-mi(52662, didBindInh=false, didBindInt=false, didBindImp=false, innerTypeProxy=JvmTypeProxy-sh(52661, didBindInh=false, didBindInt=false, didBindImp=false, jvmType=JavaInterface java.util.Set<java.util.Map$Entry<K, V>> is abstract public inherits java.util.Set`1 , 52661), 52662), returnType=nil, implementsType=nil, 52663)
    .didBindInt = false

Its showing this (methods not bound) for methods/members of Maps, Collections and Lists all of which are differently done in java being both Interfaces and Generic...
Other Classes seem to be bound OK - its just them...
I cant find where/why they arent being bound correctly till I work out where the binding is supposed to happen in the first place....

[Charles]

Originally, the compiler just did .bindInt on all types everywhere, but there are hundreds of types in the .NET std lib (and potentially in 3rd party libraries), so I had to switch to the "on-demand approach" which is what _prepLibraryBox is.

_prepLibraryBox checks if the box in question is a native type that has not done, for example, .bindInt. It is called from within Box.cobra at key points--anywhere that a method such as .constructedTypeFor could not do its job properly if the interface were not bound (implementation is irrelevant for library types).

You can try throwing an extra call to _prepLibraryBox somewhere in Box, such as .memberForName or wherever you're having trouble. Just be sure the testify results don't change--I found this code to be very sensitive.

This is one of the more complex aspects of the compiler because of the relationships of the binding, phases, library vs. non, generics, etc. Every time I worked on it, I had to diagram out the relevant parts of the object/node graph and think hard about what was going on. Lots of trace statements too.

Btw it's neat seing "java.util.Map" in a Cobra stacktrace!

Now I have a question. When I compile this program:

# j.cobra
class X
    def main
        java = 'foo'
        print java
        @help java

I get this error:
j.cobra(4): error: Incompatible types. Cannot assign value of type String on the right to Type on the left.

I put @help on java, and the compiler thinks its type is System.Type and definition is "passthrough". Do you know why?

[hopscc]

re bindiing - joy. Thx for the info tho'

Re your question
Whoops leakthrough

I did some creative mods to the compiler source to get qualified java namespace refs compiling prior to having any sort of java package/namespace lookup/handling .
Looks like I missed removing this in cleanup when that capability came online.

In Expr.cobra round line 1158.

Code: Select all

Index: Expr.cobra
===================================================================
--- Expr.cobra  (revision 2554)
+++ Expr.cobra  (working copy)
@@ -1155,10 +1155,10 @@
                                                        .throwError('You must refer to non-underscored members with a leading dot (.). Member name is "[_name]".')
                        # hops
                        #print _name, canBeUndottedMember, _definition
-                       if _definition is nil and _name == 'java'  # TODO hops: handle thru .findDefinition
-                               #_definition = NameSpace(.token.copy('ID', 'java'), 'java')
-                               _definition = .compiler.passThroughType
-                               #print _name
                        if _definition is nil and (not _superNode inherits BinaryOpExpr or .binarySuperNode.op<>'ASSIGN')
                                .throwUnknownIdError
                                throw FallThroughException()


I'll include a testcase for trapping this to the source next jvm-backend patch.

Whats with the second warning ( j.cobra) ( since it pops up the help page anyway)

Code: Select all

 1093 xx:...cobra/Tst/100-basics> cobc1 j.cobra
j.cobra(6): warning: @help at "Help_for_ja_a.html".
j.cobra(6): warning: @help expression ignored due to not being a viable statement.
Compilation succeeded - 2 warnings
foo


[Charles]

I created @help as a pass through directive that leaves the expression in place. In other words, if you write:

x = @help y

You get the help file and you still get "x = y". If you write "@help y" then you would get the help file and an error that "y" cannot be a statement. I tweaked the compiler to switch that to a warning. Another approach could be to give no message at all since the developer is probably not expecting anything out of that line.