Any tips on debugging techniques for out of memory?

gloomyandy

@DonStauffer You are not the only user of memory. RRF will itself allocate more memory in certain situations, most of these allocations take place during system startup and configuration, but some may happen during normal operation of the printer, so it may well be that some of the movements that take place between your start script being run and a tool change have consumed more memory. Often such allocations will not be released.

DonStauffer

@gloomyandy That's helpful to know. So, other than "have a whole mess of available memory and hope it's enough", how does one know when there's enough available memory?

Hopefully the answer isn't on a job by job basis - "if it fails, you don't have enough". That would mean a long job would have to be attempted to find out if it will have enough memory.

DonStauffer

@gloomyandy You know, I'm starting to think it's not really ever getting to the tool change. It was crashing pretty far away from it, with a whole inside of the letter "D" to print. Then I was able to kill off 2 variables, and it got a little further - into that feature it never started before. Then it crashed. So all it's executing at the time of the crashes is a bunch of G1 commands, with an occasional G92. Just normal slicer stuff. I guess RFF could have another thread running that periodically does stuff asynchronously that uses memory. But this was in the midst of a long bunch of code like this - about 5 layers worth. Not where I'd expect memory problems to crop up. Not even my code. Simplify3D.

This is a bit confusing, because the actual job file has only 8 variables in it, all in the starting part. The macro called by daemon.g has one single variable. All variables are integers. daemon.g itself has 12 variables. Then there are my globals. I could look at cutting back daemon.g's variable use. But I'm not sure how much I need. It seems like I could get this short test print working and still not know if a bigger print job would crash.

Update: I got rid of 10 of the 12 variables used by daemon.g. This didn't change anything at all. It crashed at exactly the same place. I'm not sure there's anything else I can drop. The uncertainty is the problem. That would be a problem still even with a little more memory; I'd never know when there would be a crash. The odds would just be better.

gloomyandy

@DonStauffer In some situations RRF will need to allocate more memory to allow it to process moves. What does M122 report after starting a print and before it runs out of memory?

DonStauffer

@gloomyandy I could embed an M122 in the print file and find out. How far before the failure point, or do you want more than one? I can pinpoint where the failure happens in the file and insert the command before that.

DonStauffer

@gloomyandy Interesting development:

I noticed there's a whole part of the tree of one of my globals which only ever gets used in the job starting script. So at the end of that, I set the root of that branch to null, apparently thus freeing all of the 15 integers below it (and perhaps a bit of array overhead). This appears to have gotten me past the spot it rebooted before, and I've been through several tool changes since. Need some adjustment to the tool change - red plastic shows well on black! But it's running.

And the LEDs are performing basically as intended. Also probably a little fine tuning needed there, but actually, pretty decent.

This project has taken about 15 months of full time work, actually. I really, really wanted this to work.

DonStauffer

@gloomyandy

The LEDs did a nice job of graphing heating & cooling of tools & bed, leveling & probing the bed (with error values), and showing percent progress (2 digits!) through the print. Everything to do with tool 0 was green; tool 1 was blue. They had 3 intensities, for off, standby and active. The bed is yellow. Probing has its own colors.

The LEDs handled the transition from printing to tool change and back again well. The worst thing about them is a slight lag, maybe about a second or so. So you can look at the web interface and see 60%, but the LEDs still show 58%. Acceptable.

In the process I rewrote all my routines, so they still need fine tuning as far as retraction & priming, and of course, tool change ooze. I'll get there. I did it once before.

I think I know the object model by heart by now!

dc42

@DonStauffer how about getting your macro to also echo the amount of free memory at various points? It's available in the object model.

DonStauffer

@dc42 OK, about my "knowing the object model by heart" ... except that!

boards[0].freeRam?

dc42

@DonStauffer yes that's the one. It's the same as Never Used RAM in the M122 report.

DonStauffer

@dc42 Thanks. I'll look at it, though my immediate problem seems to be solved. But undoubtedly I'll run out of memory again at some point. But I'm pretty happy now.

dc42

@DonStauffer I'd be interested to know whether you have less or more free memory after running a print with 3.6 alpha compared to 3.5.2.

DonStauffer

@dc42 I'm a little afraid to install an alpha though. Is reverting easy?

dc42

@DonStauffer assuming you are running in standalone mode not SBC mode, reverting is very easy. To make it even easier you can install just the 3.6 Duet2CombinedFirmware.bin file and continue running DWC 3.5.2. You can ignore the warning about incompatible software versions in this case.

gloomyandy

@DonStauffer Just running M122 in the console should provide an idea of what the memory situation is before the print starts and after your startup code has run.

DonStauffer

@gloomyandy Echoing boards[0].freeRam at various places throughout a small, 15 minute, 8-layer print job with tool changes, it looks like this (no units given; bytes?):

Machine turn-on: 8544
Partway through job startup: 8520
Before purge in job startup: 4000
After purge: 920
Beginning of Layer 1: 920
Layer 2: 848
Layer 3: 848
Layer 4: 848
Layer 5: 848
Layer 5 before tool change: 464
Layer 6: 72
Layer 6 before tool change: 72
Layer 7: 72
Layer 7 before tool change: 72
Layer 8: 72
Layer 8 before tool change: 72
After job finished: 72

It seems like I'm grazing the treetops. Most of my global data is created from config.g, so I'm not really sure why my startup script uses so much memory compared to the job proper.

DonStauffer

@dc42 OK, I'll do that. I have the details on free memory echoing throughout a job and I recorded them. So I'll have something to compare it with.

DonStauffer

@dc42 Same GCode file as the free RAM I posted for RRF 3.5.2, but after installing 3.6 alpha 2:

After upgrade: 22624
I couldn't believe that so I did an M999 and looked again: 22600 (where are my 24 bytes?)
Partway through job startup: 22600
Before purge in job startup: 16840
After purge: 14472
Beginning of Layer 1: 14472
Layer 2: 14472
Layer 3: 14304
Layer 4: 14280
Layer 5: 14280
Layer 5 before tool change: 13512
Layer 6: 13096
Layer 6 before tool change: 13096
Layer 7: 13096
Layer 7 before tool change: 13096
Layer 8: 13096
Layer 8 before tool change: 13096
After job finished: 13096

What wizardry is this?!

oliof

@DonStauffer probably the work noted at https://github.com/Duet3D/RepRapFirmware/blob/bd8d9119d49ba425864f2280b005875b368746c0/src/Todo-next.txt#L25