Ticket #288: MultiListUpdate1.patch
| File MultiListUpdate1.patch, 20.1 KB (added by jaegs, 12 years ago) |
|---|
-
Source/Cobra.Core/MultiList.cobra
1 namespace Cobra.Core X1 namespace Cobra.Core 2 2 3 class MultiList<of T>3 class MultiList<of A> 4 4 """ 5 This class provides an n-dimensional multi-list container. 5 This class provides a generic n-dimensional multi-list containing any other type. 6 The shape of a MultiList is a list of ints specifying the length of each dimension. 7 Dimensions are called axes. 8 The count is the total number of elements in the ML. 6 9 7 All methods that return MultiList<of T> are in place except for slice 8 which returns a readonly view and clone which returns a shallow copy. 10 For efficiency reasons, the MultiList class avoids copying its underlying 11 data as much as possible. A ML can be a read-only view that shares 12 data with an owner ML. The view can have 13 a different shape, count, and/or order of axes than its owner. 14 One way to construct a view is to call ml.slice 9 15 10 It is inspired by similar libraries such as 16 Methods that perform operation in place return "this" which allows for method chaining. 17 For example, 18 ml.permute(order).reshape(shape).transpose 19 If you want to perform operations on a view instead of the owner, call 20 ml.view.permute(order).reshape(shape).transpose 21 22 MultiList supports indexing, for example 23 element = ml[i,j,k] 24 ml[i,j,k] = element 25 There are plans to support syntactical multidimensional slicing, such as 26 ml2 = ml[a:b, c:d] 27 However, at the moment, you should use .slice instead 28 29 Inspiration for MultiList comes from similar libraries such as 11 30 Boost.MultiArray -- http://www.boost.org/doc/libs/1_50_0/libs/multi_array/doc/reference.html 12 31 Numpy.Array -- http://docs.scipy.org/doc/numpy/reference/arrays.ndarray.html 13 32 Ruby NArray -- http://narray.rubyforge.org/SPEC.en 33 MATLAB -- http://www.mathworks.com/products/matlab/ 14 34 15 Implementing more methods would for the most part not affect existing code 16 Some methods that would have changes are 35 For further discussion on MultiLists, see the forum 36 http://cobra-language.com/forums/viewtopic.php?f=4&t=974 37 """ 38 /# 39 --Notes-- 40 to-do: 41 (1) Implement syntactic multidimensional slicing as a property. 42 (3) Could potentially support ml[a:b:c, d:e:f] 43 44 Implementing more methods would for the most part not affect existing code. 45 The ability to write generic methods will open up a lot of possibilities 46 Some methods that would have changes, however, are 17 47 Reversing an axis -- http://docs.scipy.org/doc/numpy/reference/generated/numpy.flipud.html#numpy.flipud 18 Removing single length dimensions -- http://docs.scipy.org/doc/numpy/reference/generated/numpy.squeeze.html#numpy.squeeze48 Removing single length axes -- http://docs.scipy.org/doc/numpy/reference/generated/numpy.squeeze.html#numpy.squeeze 19 49 Roll -- http://docs.scipy.org/doc/numpy/reference/generated/numpy.roll.html#numpy.roll 20 """21 50 51 Another thing to consider is making a Matrix<of A> sublcass that is a 2-dim MultiList. 52 #/ 53 22 54 const minDimRank = 1 23 55 const maxDimRank = 10_000_000 24 56 const maxCount = 2_100_000_000 25 57 26 var _owner as MultiList<of T>?27 var _data as T[]58 var _owner as MultiList<of A>? 59 var _data as A[] 28 60 var _shape as IList<of int> 29 """ Shape is the size of each dimensionin the view """61 """ Shape is the size of each axis in the view """ 30 62 31 63 var _dimOrder as IList<of int> 32 64 """ 33 Maps the order of the dimensions stored internally to65 Maps the order of the axes stored internally to 34 66 the external permuted order 35 67 """ 36 68 var _inverseDimOrder as IList<of int> 37 69 var _strides as IList<of int> 38 70 """ 39 71 A stride is the number of places in _data separating 40 two adjacent elements of a particular dimension. 72 two adjacent elements of a particular axes. 73 Elements are stored in row major order. 74 for equation, see 75 http://en.wikipedia.org/wiki/Row-major_order#Generalization_to_higher_dimensions 41 76 A slice has the same strides as its owner 42 77 """ 43 44 78 var _ranges as IList<of Pair<of int>> 45 79 """ 46 For readonly multilists (aka views), _ranges is the range of elements per dimension80 For readonly multilists (aka views), _ranges is the range of elements per axis 47 81 included in the view. 48 82 """ 49 83 50 84 var _isReadOnly = false 51 85 var _isPermuted = false 52 86 var _isReferred = false 53 """ If a referrer is GC'ed, _isReferred will still be true"""87 """If a referrer is GC'ed, _isReferred will still be true""" 54 88 55 89 var _count = 1 56 90 … … 69 103 base.init 70 104 _shape = shape.clone 71 105 _computeFields 72 _dimOrder = for i in _numDims get i 73 _inverseDimOrder = _dimOrder.clone 74 _data = T[](_count) 106 _data = A[](_count) 75 107 76 108 cue init(shape as vari int) 77 109 body 78 110 .init(shape.toList) 79 111 80 cue init(shape as IList<of int>, data as T*) 112 cue init(shape as IList<of int>, data as A*) 113 """ Length of data can be less than the count but not more""" 81 114 body 82 115 .init(shape) 83 116 .fill(data) 84 117 85 cue init(o wner as MultiList<of T>, ranges as IList<of Pair<of int>>) is private118 cue init(original as MultiList<of A>, ranges as IList<of Pair<of int>>) is private 86 119 """ 87 120 The readonly view constructor returned from a slice 88 121 """ 89 122 require 90 ranges.count == o wner._numDims123 ranges.count == original._numDims 91 124 all for r in ranges.count get _ 92 125 ranges[r][0] >= 0 _ 93 126 and ranges[r][1] > ranges[r][0] _ 94 and ranges[r][1] - ranges[r][0] <= owner._shape[r] 127 and ranges[r][1] - ranges[r][0] <= original._shape[r] 128 ensure 129 not _owner.isReadOnly 95 130 body 96 131 base.init 97 _owner = o wner98 _data = o wner._data132 _owner = original.owner 133 _data = original._data 99 134 _isReadOnly = true 100 _isPermuted = o wner._isPermuted135 _isPermuted = original._isPermuted 101 136 _shape = for range in ranges get range[1] - range[0] 102 _dimOrder = o wner._dimOrder.clone103 _inverseDimOrder = o wner._inverseDimOrder.clone104 _strides = o wner._strides137 _dimOrder = original._dimOrder.clone 138 _inverseDimOrder = original._inverseDimOrder.clone 139 _strides = original._strides 105 140 _numDims = _shape.count 106 141 _ranges = ranges.clone 107 _count = o wner._count142 _count = original._count 108 143 _viewCount = 1 109 144 for s in _shape, _viewCount *= s 110 145 111 cue enumerate as T*146 cue enumerate as A* 112 147 for index in _generateIndices 113 148 yield this[index] 114 149 … … 126 161 127 162 get isReferred from var 128 163 129 get owner as MultiList<of T>? 164 get view as MultiList<of A> 165 """Syntactic placeholder for this[:] """ 166 return .slice 167 168 get owner as MultiList<of A>? 169 """ 170 In ml2 = ml1.slice, ml1 is the owner 171 and ml2 is a view 172 """ 130 173 return _owner ? this 131 174 132 pro [indices as vari int] as T 175 get toList as IList<of A> 176 return .enumerate.toList 177 178 pro [indices as vari int] as A 133 179 get 134 180 require 135 181 indices.length == .numDims … … 142 188 body 143 189 _data[_address(indices)] = value 144 190 145 def fill(start as int, data as T*) as MultiList<of T>191 def fill(start as int, data as A*) as MultiList<of A> 146 192 """ 147 193 Throw IndexOutOfRangeException if the length of data + start exceeds .count 148 194 """ … … 160 206 assert index > 0 161 207 return this 162 208 163 def fill(data as T*) as MultiList<of T>209 def fill(data as A*) as MultiList<of A> 164 210 return .fill(0, data) 165 211 166 def _computeStrides 212 213 def _computeFields 167 214 """ 168 Elements are stored in row major order.169 For equation, see170 http://en.wikipedia.org/wiki/Row-major_order#Generalization_to_higher_dimensions215 Computes/Recomputes _numDims, _strides, _count, ranges, 216 _dimOrder, and _inverseDimOrder for 217 non-sliced, non-permuted ML 171 218 """ 172 require not _isReadOnly 173 _strides = [1] 174 for s in _numDims -1 : 0 : -1 175 _strides.insert(0, _strides[0] * _shape[s]) 219 require 220 not .isPermuted 221 ensure 222 _count < .maxCount 223 _count >= 1 224 body 225 _numDims = _shape.count 176 226 177 def _computeRanges 178 """ 179 This computes ranges for owners where the range is the whole dimension 180 """ 181 require not _isReadOnly 182 _ranges = for s in _shape get Pair<of int>(0, s) 227 _strides = [1] 228 for s in _numDims -1 : 0 : -1 229 _strides.insert(0, _strides[0] * _shape[s]) 230 231 for s in _shape, _count *= s 232 #ranges - Each range is the whole axis 233 _ranges = for s in _shape get Pair<of int>(0, s) 234 _dimOrder = for i in _numDims get i 235 _inverseDimOrder = _dimOrder.clone 183 236 184 def _computeCount185 require not _isReadOnly186 for s in _shape, _count *= s187 188 def _computeFields189 _numDims = _shape.count190 _computeStrides191 _computeRanges192 _computeCount193 194 237 def _inversePermuteList(lst as IList<of int>) as IList<of int> 195 238 """ 196 Reorders the elements in a list according to _ dimOrder239 Reorders the elements in a list according to _inverseDimOrder 197 240 For example the indices the users gives are in permuted order 198 241 But the data is stored in the original order. 199 242 """ … … 225 268 if index < 0 or index >= _shape[i] 226 269 dim = _dimOrder[i] 227 270 throw IndexOutOfRangeException( 228 'Array index [index] is out of range 0 : [_shape[dim]] for dimension[dim].')271 'Array index [index] is out of range 0 : [_shape[dim]] for axis [dim].') 229 272 addr += (index + _ranges[i][0]) * _strides[i] 230 273 assert addr >= 0 and addr < _count 231 274 return addr … … 253 296 indices[pos] = index 254 297 yield indices 255 298 256 def clone as MultiList<of T> 257 return .clone(_shape) 299 def clone as MultiList<of A> 300 """ Returns a shallow copy """ 301 return MultiList<of A>(_shape, .enumerate) 258 302 259 def clone(shape as IList<of int>) as MultiList<of T> 260 """ Same as ml2 = MultiList<of T>(shape, ml1.enumerate) """ 261 return MultiList<of T>(shape, .enumerate) 262 263 def slice(ranges as vari Pair<of int>) as MultiList<of T> 303 def slice(ranges as vari Pair<of int>) as MultiList<of A> 264 304 """ 265 305 Returns a readonly view of the ML. 266 If ranges are provided for the first few dimensions,267 then the remaining ranges default to the whole dimension.306 If ranges are provided for the first few axes, 307 then the remaining ranges default to the whole axis. 268 308 ml.slice() 309 to-do: implement syntactic slicing as a property. 310 This method is a placeholder. 269 311 """ 270 312 _isReferred = true 271 313 completedRanges = _ranges.clone … … 276 318 lowerBound = completedRanges[r][0] + range[0] 277 319 upperBound = completedRanges[r][0] + range[1] 278 320 completedRanges[r] = Pair<of int>(lowerBound, upperBound) 279 return MultiList<of T>(this, completedRanges)321 return MultiList<of A>(this, completedRanges) 280 322 281 def permute(order as IList<of int>) as MultiList<of T>323 def permute(order as IList<of int>) as MultiList<of A> 282 324 """ 283 Permutes the dimensions in place325 Permutes the axes in place 284 326 """ 285 327 _isPermuted = true 286 328 _dimOrder = for o in order.count get _dimOrder[order[o]] 287 329 _inverseDimOrder = for d in _numDims get _dimOrder.indexOf(d) 288 330 return this 289 331 290 def transpose as MultiList<of T>291 """ Reverses the dimensions in place"""332 def transpose as MultiList<of A> 333 """ Reverses the order of axes in place""" 292 334 return .permute(for d in _numDims get _numDims - d - 1) 293 335 294 336 def toString as String is override … … 310 352 sb.append(String(c']', _numDims)) 311 353 return sb.toString 312 354 313 def reshape(shape as IList<of int>) as MultiList<of T> 314 """ Same as reshape(shape, false) """ 315 return .reshape(shape, false) 316 317 def reshape(shape as IList<of int>, unsafe as bool) as MultiList<of T> 355 def reshape(shape as IList<of int>) as MultiList<of A> 356 """ Same as reshape(shape, false, false) """ 357 return .reshape(shape, false, false) 358 359 def _truncate(stream as A*, length) as A* 360 count = 0 361 for e in stream 362 if count == length, break 363 count += 1 364 yield e 365 366 def reshape(shape as IList<of int>, noCopy as bool, unsafe as bool) as MultiList<of A> 318 367 """ 319 Reshapes the dimensionin place,368 Reshapes the axes in place, 320 369 unsafe == true will reshape the MultiList even if isReferred == true 321 Throws InvalidOperationException if size of new shape differs from 322 current shape. 370 and the count of the shape is different than .count 371 Will cause a data copy if necessary. 372 If a view is copied, it's owner will be nil and .isReadOnly will be false 323 373 """ 324 374 require 325 not .isReadOnly326 unsafe or not .isReferred327 375 all for s in shape get s >= .minDimRank and s <= .maxDimRank 328 376 body 329 # The order in which things are done in this method330 # is very important331 377 newCount = 1 332 378 for s in shape, newCount *= s 333 if newCount <> . count379 if newCount <> .owner.count and .isReferred and not unsafe 334 380 throw InvalidOperationException( 335 "New shape = [shape] with count [newCount] must have same count [.count] as previous shape [.shape]") 336 mustCopy = .isPermuted 337 if mustCopy 338 # Construct a ML with the new shape, 339 # and copy its fields 381 ".isReferred == true. To reshape, new shape = [shape] with count [newCount] " + _ 382 "must have same count [.count] as previous shape [.shape]. " + _ 383 "Call .reshape(shape, true) to bypass this restriction.") 384 else if newCount <> .owner.count and .isReadOnly 385 throw InvalidOperationException( 386 ".isReadOnly == true. To reshape, new shape = [shape] with count [newCount] " + _ 387 "must have same count [.owner.count] as owner with shape = [.owner.shape].") 388 else if newCount == .owner.count and not .isPermuted 389 _shape = shape 390 _computeFields 391 _isPermuted = false 392 else if noCopy 393 throw InvalidOperationException( 394 "To reshape to from [.shape] to [shape], " + _ 395 "data must be copied but noCopy == true. " + _ 396 "Call .reshape(shape, true, [unsafe]) to permit copy.") 397 else 398 # Construct a ML with the new shape, and copy its fields 340 399 # This procedure reuses a lot of code 341 copy = .clone(shape) 400 # The constructor expects a stream with size <= .count 401 copy = MultiList<of A>(shape, _truncate(.enumerate, newCount)) 342 402 _shape = shape 343 403 _numDims = _shape.count 344 404 _data = copy._data 345 405 _isPermuted = false 406 _owner = nil 346 407 _dimOrder = copy._dimOrder.clone 347 408 _inverseDimOrder = copy._inverseDimOrder.clone 348 409 _strides = copy._strides 349 410 _ranges = copy._ranges.clone 350 411 _count = copy._count 351 else352 _shape = shape353 _computeFields354 412 return this 355 413 356 414 def getHashCode as int is override … … 360 418 """ 361 419 throw InvalidOperationException() 362 420 363 def equals(m as MultiList<of T>) as bool421 def equals(m as MultiList<of A>) as bool 364 422 """ 365 423 Equal if shapes are the same, and 366 424 elements are in the same order. … … 369 427 if m.count <> .count, return false 370 428 if m._shape <> _shape, return false 371 429 for indices in _generateIndices 372 if not this[indices] ==m[indices], return false430 if this[indices] <> m[indices], return false 373 431 return true 374 432 375 433 def equals(obj as Object?) as bool is override 376 if obj inherits MultiList<of T>, return .equals(obj)434 if obj inherits MultiList<of A>, return .equals(obj) 377 435 return false 378 436 379 437 … … 421 479 422 480 test 423 481 # count, slice, permute, toString 424 ml = MultiList<of int>([3, 4,2], for i in 24 get i+3)482 ml = MultiList<of int>([3, 4, 2], for i in 24 get i+3) 425 483 assert ml.toString == _ 426 484 r"[[[3, 4], [5, 6], [7, 8], [9, 10]], [[11, 12], [13, 14], [15, 16], [17, 18]], [[19, 20], [21, 22], [23, 24], [25, 26]]]" 427 485 428 486 ml2 = ml.slice(@[Pair<of int>(0,2), Pair<of int>(1,3)]) 429 487 assert ml2.toString == r'[[[5, 6], [7, 8]], [[13, 14], [15, 16]]]' 430 488 431 ml2.permute([1, 0,2])489 ml2.permute([1, 0, 2]) 432 490 assert ml2.toString == r'[[[5, 6], [13, 14]], [[7, 8], [15, 16]]]' 433 491 assert ml2.count == 8 434 492 435 ml3 = ml.slice(@[Pair<of int>(0,2), Pair<of int>(1,3)]) 436 assert ml3.owner == ml 493 ml3 = ml.slice(@[Pair<of int>(0, 2), Pair<of int>(1, 3)]) 437 494 438 ml3.permute([0, 2,1])439 ml3.permute([2, 1,0])440 assert ml3.slice(Pair<of int>(0, 1), Pair<of int>(0,1)).toString == r'[[[5, 13]]]'441 assert ml3.slice(Pair<of int>(0, 1), Pair<of int>(0,1)).shape == [1,1,2]442 # 2012-07-31 Could not install Cobra without commenting out the following line, which failed 443 # issue is that installation builds Cobra.Core with -turbo 444 # see discussion at http://cobra-language.com/forums/viewtopic.php?f=4&t=974&p=4966#p4966 445 # expect AssertException, ml3[1,1,1] = 5 495 ml3.permute([0, 2, 1]) 496 ml3.permute([2, 1, 0]) 497 assert ml3.slice(Pair<of int>(0, 1), Pair<of int>(0, 1)).toString == r'[[[5, 13]]]' 498 assert ml3.slice(Pair<of int>(0, 1), Pair<of int>(0, 1)).shape == [1, 1, 2] 499 # 2012-07-31 Could not install Cobra without commenting out the following line, which failed 500 # issue is that installation builds Cobra.Core with -turbo 501 # see discussion at http://cobra-language.com/forums/viewtopic.php?f=4&t=974&p=4966#p4966 502 # expect AssertException, ml3[1,1,1] = 5 446 503 447 504 assert ml2.equals(ml2) 448 assert ml2.clone.equals(ml.slice(Pair<of int>(0, 2), Pair<of int>(1,3)).permute([1,0,2]))505 assert ml2.clone.equals(ml.slice(Pair<of int>(0, 2), Pair<of int>(1, 3)).permute([1, 0, 2])) 449 506 assert ml3.clone.equals(ml3) 450 507 assert not ml2.equals(ml3) 451 508 452 509 test 453 510 # fill 454 ml = MultiList<of int>(4, 3,2).fill(5, [1,2,3,4,5,6,7,8])511 ml = MultiList<of int>(4, 3, 2).fill(5, [1, 2, 3, 4, 5, 6, 7, 8]) 455 512 assert ml.toString == _ 456 513 r'[[[0, 0], [0, 0], [0, 1]], [[2, 3], [4, 5], [6, 7]], [[8, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]]]' 457 ml1 = MultiList<of Pair<of int>>([2, 2]).fill([Pair<of int>(0,2), Pair<of int>(1,3)])458 ml2 = MultiList<of Pair<of int>>([2, 2], [Pair<of int>(0,2), Pair<of int>(1,3)])514 ml1 = MultiList<of Pair<of int>>([2, 2]).fill([Pair<of int>(0, 2), Pair<of int>(1, 3)]) 515 ml2 = MultiList<of Pair<of int>>([2, 2], [Pair<of int>(0, 2), Pair<of int>(1, 3)]) 459 516 assert ml1.equals(ml2) 460 517 461 518 test 462 519 # clone, transpose 463 ml = MultiList<of int>([3, 4,2], for i in 24 get i+3)464 assert ml.clone.transpose.equals(ml.permute([2, 1,0]))520 ml = MultiList<of int>([3, 4, 2], for i in 24 get i+3) 521 assert ml.clone.transpose.equals(ml.permute([2, 1, 0])) 465 522 466 523 test 467 524 # owner 468 ml1 = MultiList<of int>([3,4,2], for i in 24 get i+3) 469 assert ml1.owner == ml1 470 ml3 = ml1.slice(@[Pair<of int>(0,2), Pair<of int>(1,3)]) 471 assert ml3.owner == ml1 525 # This test must check for physical equality 526 ml1 = MultiList<of int>([3, 4, 2], for i in 24 get i+3) 527 assert ml1.owner is ml1 528 ml3 = ml1.slice(@[Pair<of int>(0, 2), Pair<of int>(1, 3)]) 529 assert ml3.owner is ml1 530 ml4 = ml3.slice 531 assert ml4.owner is ml1 532 ml4 = ml4 #suppress warning 472 533 473 534 test 474 535 # reshape 475 ml = MultiList<of int>([3, 4,2], for i in 24 get i+3)476 ml.reshape([2, 4,3])536 ml = MultiList<of int>([3, 4, 2], for i in 24 get i+3) 537 ml.reshape([2, 4, 3]) 477 538 assert ml.toString == _ 478 539 r'[[[3, 4, 5], [6, 7, 8], [9, 10, 11], [12, 13, 14]], [[15, 16, 17], [18, 19, 20], [21, 22, 23], [24, 25, 26]]]' 479 540 480 ml2 = MultiList<of int>([3,4,2], for i in 24 get i+3) 481 expect InvalidOperationException, ml2.transpose.reshape([3,4]) 541 ml2 = MultiList<of int>([3, 4, 2], for i in 24 get i+3).transpose.reshape([3, 4]) 542 assert ml2.toString == _ 543 r'[[3, 11, 19, 5], [13, 21, 7, 15], [23, 9, 17, 25]]' 482 544 483 ml = MultiList<of int>([3,4,2], for i in 24 get i+3)484 assert ml .transpose.reshape([4,3,2]).toString == _545 ml3 = MultiList<of int>([3, 4, 2], for i in 24 get i+3) 546 assert ml3.transpose.reshape([4, 3, 2]).toString == _ 485 547 r'[[[3, 11], [19, 5], [13, 21]], [[7, 15], [23, 9], [17, 25]], [[4, 12], [20, 6], [14, 22]], [[8, 16], [24, 10], [18, 26]]]' 548 549 ml4 = MultiList<of int>([3, 4, 2], for i in 24 get i+3).view.reshape([2, 4, 3]) 550 assert ml4.toString == _ 551 r'[[[3, 4, 5], [6, 7, 8], [9, 10, 11], [12, 13, 14]], [[15, 16, 17], [18, 19, 20], [21, 22, 23], [24, 25, 26]]]' 552 553 expect InvalidOperationException, MultiList<of int>([3, 4, 2], for i in 24 get i+3).view.transpose.reshape([2, 4, 3], true, false) 554 555 expect InvalidOperationException, MultiList<of int>([3, 4, 2]).view.reshape([1, 2]) 556 ml5 = MultiList<of int>([3, 4, 2]) 557 ml6 = ml5.view #make .isReferred = true 558 ml6 = ml6 #suppresss warning 559 expect InvalidOperationException, ml5.reshape([1, 2])
