IDE binding internals

[thriwkin]

terminology...'symbol table of the IDE'...

"symbol definition" examples:
- "class B inherits A is partial"
- "def m(a as int) as bool is private, nonvirtual"

"symbol reference" examples:
- "b = B()"
- ".m(1)"

"Symbol table" is the common term used in the theory of compiler construction.
If all symbol defs are stored in a table (commonly a tree-like structure)
then the symbol refs can be found in this table. This is called "resolving" or "name binding".

"Symbol table of the IDE":
- in SD4: all classes derived from ICompliationUnit, IEntity.
- in SD5/MD/XS, one is this: all classes derived from IUnresolvedFile, IUnresolvedEntity.
The C# LB stores the symbol defs into this, as the next posting will show.

"Symbol table of the Cobra compiler":
- all nodes of the AST that are used to store symbol defs.

(I must stop now, no time any more, can be continued, may be in some days.
But I would like to post a code excerpt, it took me hours, and because of this: no time anymore.)

[thriwkin]

CSharpBinding MD/XS: parsing wherto?

So, that's a lot of stuff I just typed but it doesn't really explain how project references work in other bindings without requiring an actual assembly which is what your original question was about. The answer, [...]

The answer is in the source code of your "Reference implementation".
It parses like so:

Code: Select all

symbol definitions --(own parser)--> NR AST --(conversion visitor)--> symbol table of the IDE


The following excerpts are from the latest version I could download, from XS,
the dots (...) mean: "omitted", because distracting for the question: "parsing whereto?"


// -----------------------------------------------------------------
// project: "CSharpBinding.csproj"
// folder: "monodevelop-4.2.2\main\src\addins\CSharpBinding"
// assembly: "MonoDevelop.CSharpBinding.dll"
// -----------------------------------------------------------------

// file: "CSharpBinding.addin.xml"

Code: Select all

...
<Extension path = "/MonoDevelop/TypeSystem/Parser">
    <Parser class="MonoDevelop.CSharp.Parser.TypeSystemParser" mimeType = "text/x-csharp" />
</Extension>
...


// file: "TypeSystemProvider.cs"

Code: Select all

namespace MonoDevelop.CSharp.Parser {
    public class TypeSystemParser : MonoDevelop.Ide.TypeSystem.TypeSystemParser {
        public override ParsedDocument Parse (
            bool storeAst,
            string fileName,
            System.IO.TextReader content,
            MonoDevelop.Projects.Project project = null)
        {
            var parser = new ICSharpCode.NRefactory.CSharp.CSharpParser (...);
            ...
            var result = new ParsedDocumentDecorator ();
            ...
            var unit = parser.Parse (content, fileName);    // PARSE :  C# text -> NR.Ast
            ...
            var pf = unit.ToTypeSystem ();    // CONVERT :  NR.Ast -> IUnresolvedFile
            ...
            result.ParsedFile = pf;
            ...
            return result;
        }
    ...
}


// -----------------------------------------------------------------
// project: "ICSharpCode.NRefactory.CSharp"
// folder: "monodevelop-4.2.2\main\external\nrefactory\"
// assembly: "ICSharpCode.NRefactory.CSharp.dll"
// -----------------------------------------------------------------

// file: "SyntaxTree.cs"

Code: Select all

namespace ICSharpCode.NRefactory.CSharp    {
    public class SyntaxTree : AstNode {
        ...
        public CSharpUnresolvedFile ToTypeSystem () {
            ...
            var v = new TypeSystemConvertVisitor (this.FileName);
            v.VisitSyntaxTree (this);    // NR.Ast -> IUnresolvedFile
            return v.UnresolvedFile;
        }
        ...
    }
}


// file: "TypeSystemConvertVisitor.cs"

Code: Select all

namespace ICSharpCode.NRefactory.CSharp.TypeSystem {
    public class TypeSystemConvertVisitor : DepthFirstAstVisitor<IUnresolvedEntity> {
        ...
       
    // ---- Compilation Unit ----
        public override IUnresolvedEntity VisitSyntaxTree (SyntaxTree unit) {
            unresolvedFile.Errors = unit.Errors;
            return base.VisitSyntaxTree (unit);
        }
       
    // ---- Type definition ----
        public override IUnresolvedEntity
        VisitTypeDeclaration(TypeDeclaration typeDeclaration)
        {
            var td = currentTypeDefinition = CreateTypeDefinition(typeDeclaration.Name); 
                // td: CSharpUnresolvedTypeDefinition
            td.Region = ... ;
            td.BodyRegion = ... ;
            ...
            td.Modifiers = ...;
            td.Kind = ...;
            ...
            td.Attributes = ...;
            td.TypeParameters = ...;
            foreach (AstType baseType in typeDeclaration.BaseTypes) {
                td.BaseTypes.Add(...);
            }
            foreach (EntityDeclaration member in typeDeclaration.Members) {
                // foreach member def of this type def
                member.AcceptVisitor(this);
            }
            currentTypeDefinition =
              (CSharpUnresolvedTypeDefinition)currentTypeDefinition.DeclaringTypeDefinition;
            ...
            return td;
        }
       
    // ---- Field definition ---  simplified ---
        public override IUnresolvedEntity
        VisitFieldDeclaration(FieldDeclaration fieldDeclaration)
        {
            ...
            DefaultUnresolvedField field = null;    // IUnresolvedField
            f = new DefaultUnresolvedField(currentTypeDefinition, vi.Name);
            f.Region = ...;
            f.BodyRegion = ...;
            f.Attributes = ...;
            f.Modifiers = ...;
            f.ReturnType = ...;  // ITypeReference
            currentTypeDefinition.Members.Add(f);
            ...
            return f;
        }
        ...
    }
}


nerdzero,
this is an example of "essentially the same":
all LBs I had inspected map the symbol defs to the symbol table of the IDE
-- except your LB.
And you did peruse this?

[nerdzero]

The answer is in the source code of your "Reference implementation".

Correct. I had originally posted links to the various files containing these code snippets but my post was already super long so I thought we could discuss it later after you had taken a look; but it seems you already knew the answer. Cool.

this is an example of "essentially the same":
all LBs I had inspected map the symbol defs to the symbol table of the IDE
-- except your LB.

Okay, you mean essentially the same in the sense that they all use NRefactory for resolution. I don't think the D add-in does though. There's some code to convert the AST but it's commented out:
https://github.com/aBothe/Mono-D/blob/m ... er.cs#L153
https://github.com/aBothe/Mono-D/blob/m ... er.cs#L242

And you did peruse this?

Yes, definitely. Some of that code is burned into my retinas.

(I must stop now, no time any more, can be continued, may be in some days.
But I would like to post a code excerpt, it took me hours, and because of this: no time anymore.)

No need to spend more time on defining a symbol table. Thanks for taking the time to help me understand what you meant. I am familiar with the concept but wanted to know more about what you were calling the symbol table of the IDE.

I think one important distinction between a traditional symbol table used by a compiler and a "symbol table" used by a language binding is that the compiler only cares about resolving identifiers (symbols) whereas an IDE language add-in must also be able to resolve text locations to nodes and types for the purposes of creating tooltips and generating completion options within a certain scope: https://github.com/icsharpcode/NRefacto ... ile.cs#L61

[thriwkin]

I wrote
"...all LBs I had inspected..."
feed the symbol table (of the IDE).

Before I inspect the source code of an LB, I test if it implements the features I am interested in.
This was the case for these LBs:
- SD4: C#, Boo, IronPython
- SD5: C#

Mono-D
Good example.
First I would test the features, and if ok, then I would inspect the source code.
In this case, because you pointed to it, reverse: first the source code.

I skimmed over the text:
https://github.com/aBothe/Mono-D/blob/master/MonoDevelop.DBinding/Parser/DParserWrapper.cs#L153

- does not feed the symbol table (of the IDE)
Hence: will not have a class browser -- not interesting.

- the out-commented code feeds some old symbol table
("cu = new CompilationUnit(...) ... ConvertDParserToDomNode(...)"...)
Hence: an LB that is incompletely ported -- not interesting.

Presumably: if this author has figured out how to do this with the new symbol table, then he will do it.

[nerdzero]

- does not feed the symbol table (of the IDE)
Hence: will not have a class browser -- not interesting.

Well, it doesn't use NRefactory to do it but it still has a class browser: https://github.com/aBothe/Mono-D/blob/m ... ion.cs#L27 See attached screenshot as well.

You stated earlier you had stopped working on the SD4 binding for Cobra. Want some help updating it for SD5? MD and SD are using a common library so we could probably share a common core between the two add-ins.

[thriwkin]

Mono-D

... but it still has a class browser ...

This is not the "Class browser", it is the "Document Outline"!

I chosed the name Class browser because this is the common name for it,
and because the commands are different:
- in SD4: "View > Classes"
- in XS: "View > Pads > Classes"
- in XS: "View > Pads > Document Outline"

In the prior message I "predicted" it,
now I made an empircal test:
- On Windows, XS 4.2.2(2), with the "Add-in Manager", I installed the "D Language Binding version 1.3.4",
- and made this example you show.
- "View > Pads > Classes" displays that what he gets from this LB: nothing.

[nerdzero]

I see. I misunderstood. You are right then; it doesn't have a class browser.

[thriwkin]

A class browser -- for free.

Gradually I remember why I think
1. the class browser is very usefull, and
2. parsing into the 'symbol table of the IDE' is necessary.

1. Some years ago (it feels like 300 years ago) I wrote a parser for the Cobra language,
and I had to explore a hierarchy of 185 types (all derived from INode) -- with an editor (SciTE).
Later I found that SD had a class browser.
Like that one of XS, it can
- show all types in a tree view.
But additionally it can
- search for a given type and
- show the "Base types" and "Derived types".

2. Later I started to write a Cobra-SD binding.
I coded:
- create/open a project (MSBuild Task, .targets file, ...)
- smart indenting
- syntax highlighting
==> still no class browser
- parsing: map the symbol definitions in the source text to the 'symbol table of the IDE'
==> class browser

I never had to do anything for this class browser!
Just parsing like I had seen it in these SD-LBs: C#, Boo, IronPython.
Or like the parser of the C# binding for MD/XS.

* * *

Now exploring the types of the Cobra compiler is much easier as with the editor.
An example:
- "Where does the parser of the Cobra compiler store a `class Foo...` definition? "
To figure this out, I can
- open a .sln file in SD4 with the Cobra compiler source,
- open "View > Classes",
- search for "INode",
- click on "Derived types", ...,
- until the type node "Class" is expanded.

Now I realize -- that is a bad example, so many "class" words:
'Class' is the name of a `class` type, displayed by the class browser in the "View > Classes" pad.
But nevertheless:

ClassBrowser-(SD4-Cobra)-INode.jpg (71.17 KiB) Viewed 23040 times

[nerdzero]

Well, maybe it is worth suspending development on my parser and just write some classes that inherit from the various classes here instead: https://github.com/icsharpcode/NRefacto ... ementation and override as needed. Then write a visitor that converts the resolved declarations in the CobraModules to these classes. Then we'd get the class browser and some other features for free like you said; just pay for shipping and handling. I could probably just modify the existing visitor in the add-in instead of making a new one too. I'll give this a shot.

This wouldn't fix the issue with project references though as I'd still lack a resolver. I'm currently relying on Cobra.Compiler.dll to do the type binding and I can't seed it with existing parsed modules. I have to run the compiler phase that reads referenced assemblies. I don't think I could have it parse the files from multiple projects in one go either because they might have different compiler options, like different '-namespace' or '-number' values. Also, performance would get even worse than it is now.

[Charles]

Just a side note that if your editor/IDE has project-wide find, then you can find declarations pretty easily by searching for "class Foo" or "def bar".

Cobra declarations for types, enums, methods, etc. always follow the form "<keyword> <id>...". This is different than C-based languages which are "<type> <id>..." for methods which prevents this kind of easy search.

This doesn't get you other benefits like "all derived types", but it lets you find declarations pretty quickly. See also: http://cobra-language.com/trac/cobra/wi ... erantCtags

Regarding the Cobra compiler, I have an enhancement on the way that caches assemblies and therefore will make the IDE more responsive. It's almost done except for a couple of test cases that broke.