Question: Array Assignment

DonStauffer

@dc42 Question:

What does & doesn't "count" as "a copy of another array"? Is it pretty much any way of setting the array's values, even initialization, where the right side isn't a scalar? Or is it just a copy from a named array variable which can trigger this problem? For instance, could any of these lines trigger the problem later?

var A1= vector(2, vector(2, 0)) ; Presumably, array vector creates is copied to A1?
set var.A1[0] = {1, 2} ; Presumably, array constant gets copied to A1[0]?
set var.A1[0][0] = vector(2, 0) ; Presumably, array vector creates is copied to A1[0][0]?

If I'm understanding correctly, if I want an array with more dimensions than 4 or 5 and therefore can't subscript elements directly, there's no possible way to assign element values without making a copy. (see code)

; So, given:

var Inner = vector(2, vector(2, vector(2, vector(2, 0))))
var Outer= vector(2, vector(2, vector(2, vector(2, var.Inner))))
...

; Instead of

set var.Outer[0][0][0][0][0][0][0][0] = 5 ; 8 subscripts

; or

var A3 = var.Outer[0][0][0][0]
set var.A3[0][0][0][0] = 5 ; Dangerous, set using multi-subscript on copy
set var.Outer[0][0][0][0] = var.A3 ; (OK because Outer isn't a copy - or is it?)

; I'd have to do something like:

var Inner = vector(2, vector(2, vector(2, vector(2, 0))))
var Outer= vector(2, vector(2, vector(2, vector(2, var.Inner))))

; Presumably, Outer[n][n][n][n] elements all contain copies of Inner
...
var A3 = var.Outer[0][0][0][0] ; OK, assigning to A3, not to element of a copy

var A4 = var.A3[0] ; Now A4 contains a copy of var.Outer[0][0][0][0][0] (would be 5 subscripts)
var A5 = var.A4[0] ; Now A6 contains a copy of var.Outer[0][0][0][0][0][0] (would be 6 subscripts
var A6 = var.A5[0] ; Now A6 contains a copy of var.Outer[0][0][0][0][0][0][0] (would be 7 subscripts)

set var.A6[0] = 5 ; OK, only one subscript

set var.A5[0] = var.A6; OK, only one subscript
set var.A4[0] = var.A5; OK, only one subscript
set var.A3[0] = var.A4; OK, only one subscript

set var.Outer[0][0][0][0] = var.A3 ; OK? Or is Outer a copy of vector construct?
; ^ Important question above ^

gloomyandy

@DonStauffer The problem is triggered by any assignment that includes more than one subscript, it's that simple. So in your first example above line 3 may cause a problem, the others are probably ok.

I'm not really sure where you are going with all of this stuff to try and get more dimensions than the maximum. But you need to be aware that you can not write to a copy of the array and use that to modify the original. so for instance the moment that you do set var.A6[0] = 5 it will break the relationship between A6 and outer (by making a fresh copy of the values that you can access via A6[<n>] and will set the entry A6[0] to 5, it will not set Outer[0][0][0][0][0][0][0] to 5. If you understand the concept of "copy on write" basically all of the parts of an array are like that. Which in effect means that although you can probably construct an array with more than the max number of dimensions it will be in effect read only and even then you will need to jump through hoops to be able to read it.

You should probably try the build that DC42 has made available to you in this thread: https://forum.duet3d.com/topic/35769/error-meta-command-too-many-indices/14

It contains the fix that we have come up with for this problem.

gloomyandy

@gloomyandy Oh and as to that final

set var.Outer[0][0][0][0] = var.A3

It may be ok but it may not, if any of the existing component arrays that make up Var.outer have another variable that happens to reference them then you may be in trouble. So for instance if earlier in the program you had var aa = var.Outer[0][0] or something similar then that could trigger the problem.

Out of interest why do you need so many indices?

DonStauffer

@DonStauffer I didn't realize he had the build up already. That's fast work!

Where I'm going is simple: I don't want the global namespace cluttered, so I put all the global data I need for a project into a single global variable name. Were I to consider changing my project for this purpose right now, I'd probably wait, but I had already gone to the trouble of making all the changes when I discovered these unexpected challenges.

Sure, delayed writing is a performance trick. But I didn't expect to be able to take advantage of any link to the original. I'm not sure why it would be read only. That's why my code steps back through each subscript and assigns to the previous copy, until it gets back to the original array. It seems like that should work. Not efficient, but it's supposed to be a workaround.

Given a deep array, first, assign an element of the first dimension to a variable. Whether it's linked under the hood at this point doesn't matter. Now, assign an element of the first dimension of THAT variable to a second variable. Continue, until you are at the penultimate dimension. That variable gets the intended value assigned. Then, assign that to the same subscript of the prior variable that it came from. Then assign THAT variable to the same subscript of the prior variable that it came from. Etc., until you assign the first variable to the same subscript of the original array it came from. No link is needed to the original. Not an easy way to assign, but it seems like it makes it possible. I'm copying a branch node by node then reassembling the branch after assigning to the disassembled leaf node. Without the subscript bug, I'd only need to do this at level 4 or 5 without percolating all the way through the depth of the array. Much simpler.

DonStauffer

@gloomyandy OK, this being a deferred write issue really clarifies what's going on. I get it now.

DonStauffer

@gloomyandy OK, I got the update and so far it looks good. And to get a depth over 5, this seems to work fine:

;   BigArray Macro
;
;   Demonstrates how to effectively have array depths over 5 subscripts

;   Define an array with a depth of 7

var Inner = vector(2, vector(2, 0))
var Outer = vector(2, vector(2, vector(2, vector(2, vector(2, var.Inner)))))

;   Write access

var Inner1 = var.Outer[0][0][0][0][0] ; Copy part of branch
set var.Inner1[0][0] = 1 ; Not the original array. It is a copy of part of it
set var.Outer[0][0][0][0][0] = var.Inner1 ; Copy back whole copied branch part

;   Read access

var Inner2 = var.Outer[0][0][0][0][0] ; Copy part of branch
echo var.Inner2[0][0]

DonStauffer

@DonStauffer This hadn't occurred to me until the last day or so ...

Does my proposed >5 dimensional array concept produce a memory leak? I was assuming the language wouldn't permit a user to create a memory leak, but since I don't know anything about how memory is managed under the hood in RRF, I don't know.

Specifically, does assigning an array value to an element of a global array discard the memory which until then had been used for that element's array? If so, then doing that iteratively would use up all available memory (and then probably crash). So, is this problematic? Is this a memory leak, or does the "lost" memory get returned to the the heap?

global MyArray = {1, {2,3}}
...
set global.MyArray[2] = {4,5} ; New 2-element array allocated in global element, to hold {4,5}
; now the memory which was holding {2, 3} is lost
...
set global.MyArray[2] = {6,7}
; memory which was holding {4,5} is now lost

Likewise if the array comes from another variable rather than an "array value" in braces?

dc42

@DonStauffer the memory in question should get recycled to the help. If you have any evidence that there is a memory leak, please provide an example. The M122 report includes information on the state of the heap.

DonStauffer

@dc42 If I can get an isolated simple case I will. The reason I'm thinking memory leak is because the problem seems "elastic" if that makes any sense. For example, I put this into config.g:

global HOG = vector(250, 0)

The result was mild but definite: Reboots happened a little sooner. If I was simply low on memory it should have choked right away, since that increased global data by about 50%. But no matter what, it seems like things have to iterate at least a few times before the problem happens. Adding the HOG makes it fewer times before a crash. And I don't know of anything I could be doing that loses memory with each iteration, which is why I asked.

I've decided to bite the bullet and revamp the one global array so it fits in a 5D array. I intend to get rid of one dimension by stuffing all 3 RGB values into a single integer. Hopefully it won't take too much processing time. I was looking for bitwise operators, but didn't see any so I used multiplying/dividing by powers of 2 and truncating.

var Color = {1, 2, 255}

;   Stuff all 3 values into an integer

var c = var.Color[0] + var.Color[1] * pow(2, 10) + var.Color[2] * pow(2, 20)

;   Extract R, G & B values from the integer

var Color1 = {null, null, null}

set var.Color1[2] = floor(var.c / pow(2, 20))
set var.Color1[1] = floor(var.c / pow(2, 10)) - var.Color1[2] * pow(2, 10)
set var.Color1[0] = var.c - var.Color1[1] * pow(2, 10) - var.Color1[2] * pow(2, 20)

echo var.Color1

DonStauffer

@dc42 I was able to free up 112 integers from one of the global arrays by combining RGB values into single integers. The spontaneous rebooting stopped as far as I can tell. Through it all I've never again seen the array corruption, so I'm confident your fix solved that. Things are running fine now. Thanks for everything!