MultiList

[Charles]

jaegs, your observations and comments on MultiList are interesting.

Re: .getHashCode, I was being lazy. I thought I could get away with it for awhile because it seems rare to use the hash code of something mutable like MultiList.

Re: the Mac, there is at least TextMate and gedit. Lately, I've actually been doing gedit on Ubuntu on Parallels on Mac... love that Cobalt theme.

Also, feel free to whip up an editor or IDE using Cobra+MonoMac.

I think Hops and I covered your questions, but if I missed something, let me know.

[jaegs]

re: getHashCode
Upon further investigation, I see that supporting a hash function for a mostly mutable class would be incorrect and the MultiList exception should remain.
objA.equals(objB) => hash(objA) == hash(objB) also implies that hash(objA) @time1 == hash(objA) @time2 which is impossible if all the class's data fields are mutable.

A question I forgot to ask - is there existing syntactic support in Cobra for multidimensional slicing such that

ml[a:b][c:d][e:f]
# and / or
ml[a:b, c:d, e:f]
# would call a MultiList function like
ml.slice( ranges as vari Pair<or int>)

I know that lists have slicing but I'm not sure where to look in the source. Thanks.

[Charles]

Correct on the hashing. Plus I think it would be rare to use multi-lists as keys in dictionaries or members of sets. As a workaround, one could always cook up an immutable class that can be initialized from a multi-list.

Regarding multi-dim slicing, we don't have that right now.

If you want to dig into the slicing code, look for SliceExpr in the Cobra compiler. The parser creates instances of it. The main definition for it is in Expr.cobra and the C# code gen is in CobraWorkspace/Source/BackEndClr/SharpGenerator.cobra. There you will see that it calls CobraLangInternal.CobraImp.GetSlice which is implemented in CobraWorkspace/Cobra.Lang/Native.cs.

There are overloads of GetSlice on various types like IList<of T>, string, etc. and also Object.

You could introduce a new ISliceable interface and add an overload for that plus a check for it in the overload on Object (which already checks for list, string, etc.). This would give you a hook for allowing classes to implement their own slicing.

To support multiple dims, either the parser would have to detect that, or SliceExpr could detect it in _bindImp. Then the back-end would have to be enhanced to support it. We'd have to decide if a different interface would be used for that, or what.

It's definitely doable.

Do you know what Python does for these cases?

ml[a:b][c:d][e:f]
ml[a:b, c:d, e:f]

[jaegs]

Just to show some progress, since I've been pretty busy lately, I have a -- don't hate -- Python prototype of how slicing, permuting, and retrieving elements should be. Since it mostly works, I'm going to start on the Cobra code now. It was unfortunately too much to figure out MultiLists and a new programming language simultaneously.

[Charles]

No hating here. Incremental steps are fine. Thanks for working on this.

[jaegs]

BTW, gedit runs on Mac and cobalt looks pretty good. Here's the link.
To add the Cobra highlighting, navigate to /Applications right click on gedit and select "Show package contents" and go down a few directories.

[jaegs]

---deleted---

[jaegs]

I would like to show what I have so far for MultiList. Most everything seems to work but I could probably use a few more tests. There are tons of more methods that could be implemented, it's just a matter of determining which are the most important. Let me know if you have any questions, suggestions, or comments.

All methods with side effects, like "permute" return "this" which lets you chain methods together. For example

ml = MultiList<of int>(1,2,3).fill(...).permute(...).toString

A slice that is one dimension of data does not automatically turn into a list -- I think that would require dynamic typing.

I'll probably make some more changes in the next few days.

[Charles]

Looks like some great progress. I only had time to peek at it and not even run it, but here is some minor feedback and questions:

result_.append((for i in numBrackets get c']' ).toArray) _
# I think that could be:
result_.append(String(c']', numBrackets)) _

# In this method decl:
        def _generateIndices( shape_ as IList<of int> ) as IList<of int>*
# Why does the shape_ parameter have a trailing underscore?

# Instead of "top comments":

        # Permute comes from MATLAB
        # <!-- m --><a class="postlink" href="http://www.mathworks.com/help/techdoc/ref/permute.html">http://www.mathworks.com/help/techdoc/ref/permute.html</a><!-- m -->
        # Numpy has swapaxes.
        def permute( order as IList<of int> ) as MultiList<of T>

# We use doc strings:

        def permute( order as IList<of int> ) as MultiList<of T>
            """
            Permute comes from MATLAB
            <!-- m --><a class="postlink" href="http://www.mathworks.com/help/techdoc/ref/permute.html">http://www.mathworks.com/help/techdoc/ref/permute.html</a><!-- m -->
            Numpy has swapaxes.
            """

# These will show up in -doc output.

# This also maintains the convention that Cobra declarations
# start with a keyword (class, def, get, set, struct, etc.)
# followed by a name.

# Re:
        cue enumerate as T*
            for index in _generateIndices(_inversePermuteList(_shape) )
                yield this[(index to List<of int>).toArray]
# Why can't it be the simpler?
        cue enumerate as T*
            for item in _data, yield item
# Is it because of strides or something?

# For public methods:
        def fill( start as int, data as T[]) as MultiList<of T>
            require 
                not _isReadonly 
                not _isPermuted
# use public members in the contracts:
        def fill( start as int, data as T[]) as MultiList<of T>
            require 
                not .isReadonly 
                not .isPermuted

-- In class TestMultiList, feel free to have multiple "test" sections. They will all be run. Sometimes a really long test section can get confusing regarding the use of vars or what's expected prior to each subsection.

-- I suppose we can move .zip to CobraCore so that code that uses it will be portable when it is implemented for JVM as well. Or would an operator make sense? "stream1 * stream2"

-- Maybe I'm overthinking performance, but I noticed that _owner is non-nilable:

# ...
        var _owner as MultiList<of T>
# which means it has to be initialized with this:
                _owner = this
# which creates an immediate pointer/reference cycle and
# therefore more work for any garbage collector
# I would normally do this:
        var _owner as MultiList<of T>?

[jaegs]

I've incorporated your suggestions. Is there a place in the repo I can put the file until it's complete? I still need to work on tests.

# Re:
        cue enumerate as T*
            for index in _generateIndices(_inversePermuteList(_shape) )
                yield this[(index to List<of int>).toArray]
# Why can't it be the simpler?
        cue enumerate as T*
            for item in _data, yield item

# Is it because of strides or something?

Very important question. If the MultiList is the result of a slice (AKA a view), then it does not own a copy of the _data field. It shares _data with the original ML (AKA the owner). Not all the elements of the owner are present in the view, however.

Additionally, changing the order of the dimensions (permuting) changes the order of the elements. Permuting does not modifying _data.

To determine which elements actually are in the view and their order, I chose to generate all possible indices in order (see _generateIndices which is a recursive generator). This procedure is theoretically correct. I don't know if it's the most efficient. I didn't bother to read any existing libraries to see how they do it. Boost is a messy template library and Numpy has to bridge the gap between C and Python. Cobra, as a language, definitely makes implementing a multidim array library easier. Hopefully it can best Numpy in terms of speed too -- I'll have to test that.

I could put a clause such that if the ML is an umpermuted owner, than it could return an enumerator directly over _data.

I suppose we can move .zip to CobraCore so that code that uses it will be portable when it is implemented for JVM as well. Or would an operator make sense? "stream1 * stream2"

The stream operator makes sense. However, unless the language supports triples, stream1 * stream2 * stream3 would probably have to return a Pair<of T1, Pair<of T2, T3>>*

LINQ actually has a Zip function (link). It takes two IEnumerables and a selector function. Can LINQ be used in Cobra? It would combine nicely with a clean lambda syntax.

As I side note, I am not a fan of LINQ's query syntax. There are some operations that can only be performed with the fluent syntax forcing one to combine syntaxes. For example

Code: Select all

 (from x in lst where x == 5 select lst).FirstOrDefault();

For comparison, Python's Zip function takes a list of iterables. It does not have a selector function argument. (link)

Back to ML's: I would like to switch many of the methods in the MultiList class to take arrays as argument instead of lists. However, I feel that without array comprehensions the code would get more complicated.

Do you know what Python does for these cases?

Code: Select all

ml[a:b][c:d][e:f]
ml[a:b, c:d, e:f]

Python treats both cases the same. It also supports combining slices and indices:

Code: Select all

ml[a:b,c]


It's probably just important that

Code: Select all

ml_ = ml[:][:][:]
#doesn't get compiled into
ml2 = ml[:]
ml3 = ml2[:]
ml_ = ml3[:]
#but rather ends up being exactly the same, execution-wise as
ml_ = ml[:,:,:]

When it comes to operator overriding, I wonder if the best solution is to introduce general operator overriding support throughout the language. MultiList could benefit from having other operators. Ex. multilist1 + multilist2 would return a new multilist where each element is the sum of the corresponding elements in the left and right MLs.

Operators are kind of like cues in that they have special abilities built into the language. However, there are too many operators to reuse the cue keyword. Instead, the keyword "oper" could be used.

As a side note, do you think "toString" should be a cue? It's kind of built into the language. Ex. "print" calls the toString method.

Here's an example of how the + operator could be overrided

class X<of T>
  oper +( other X<of T>) as X<of T>
    newData = []
    for i in _data.count
      newData.add( _data[i] + other._data[i])
   return X<of T>( newData)

Since operator would now be methods, the following should be possible

(5).+(7).*(8).-(3)./(4)

Which is cool because this escapes the default order of operation


Classes should then be able to override and overload square brackets

#indexing
oper [](indices as vari int)  as T
  pass

#slicing
oper [](ranges as vari Pair<of int>) as X<T>
  pass