Improving Macro Execution Time

robotsneversleep

I suspect I'm misusing macros, but I have a NeoPixel ring, and after discovering meta commands and daemon.g I may have gotten carried away
Also I'm trying my hardest to avoid using an SBC if I can.

Fundamentally, I'm trying to keep my NeoPixel control routine to under 500mS to prevent taking control away from the firmware for too long.
Here's a trial I've been running:

daemon.g:

while global.runDaemon
    M98 P"statusled/main.g"
    G4 P500

statusled/main.g

var time_b = state.upTime + (state.msUpTime / 1000)

while iterations < 1
    M98 P"statusled/effects/fade.g" D"I" C{global.STATUSLED_RED}
    M98 P"statusled/effects/fade.g" D"O" C{global.STATUSLED_RED}

var time_a = state.upTime + (state.msUpTime / 1000)
var time_d = var.time_a - var.time_b

if var.time_d > 0.5
    var str = "Warning: StatusLED took " ^ var.time_d ^ "s (> 0.5s) to run. Check SD read speed!"
    M118 P0 S{var.str} L1

fade.g:

var e = global.STATUSLED_STRIP_ID
var s = global.STATUSLED_NUM_LEDS
var d = global.STATUSLED_MAX_BRIGHTNESS / global.STATUSLED_FADE_DURATION
var b = param.D == "O" ? global.STATUSLED_MAX_BRIGHTNESS : 0x0

if param.D == "O"
    set var.d = -var.d

while iterations < global.STATUSLED_FADE_DURATION
    set var.b = var.b + var.d
    M150 E{var.e} P{floor(var.b)} S{var.s} R{param.C[0]} U{param.C[1]} B{param.C[2]} W{param.C[3]} F0
    G4 P1

fade.g just adjusts the brightness of the LED's every mS to fade in or out a given colour.
Depending on what I set STATUSLED_FADE_DURATION to, I can easily exceed 500mS, so it's configured at the moment to be very low e.g. 50mS.
This whole routine (all code above) takes around 0.43s to execute.

However, when doing some refactoring, I swapped out the M150 command in fade.g for a macro call called solid.g like so:
solid.g is just an M150 wrapper which updates all LED's to a single colour negating the need to pass the S param to M150.

M98 P"statusled/effects/solid.g" B{floor(var.b)} C{param.C}

This takes 1.19s. I was surprised until I realised the firmware is probably reading this file from SD and executing it on every loop iteration.
I'm using a SanDisk Ultra Class 10 A1 SD card with an advertised read speed of 98MB/s. The speed test gave me:

M122 P104 S1
Testing SD card write speed...
SD write speed for 1.0MByte file was 4.98MBytes/sec
Testing SD card read speed...
SD read speed for 1.0MByte file was 1.81MBytes/sec

From reading the Duet documentation, those speeds are actually pretty good, so I have some questions:

What limits the Duet in terms of SD card transfer speeds? Hardware?
Is the read buffer fixed at 512B (i.e. in hardware) or can it be increased in the firmware?
Is caching of macro's, either after reading from SD, or after parsing a possibility?
Apart from calling as few macros as possible, doing fancy fade effects etc, is there anything I can do to improve execution speed?
Should I really be using an SBC for this, and just passing the M150's to the firmware?

As I said at the beginning, I'm aware I'm using macro's in ways in which they were probably not intended, but there's so much potential...
Thanks for your time

gloomyandy

@robotsneversleep First of all doing this during a print is a really bad idea! Your Duet board is primarily intended to provide real time control of a printer or cnc system, not a light show. Trying to do both at the same time is an especially bad idea. As you have already discovered executing macros on a frequent basis will be hitting your SD card very hard and it is the same SD card that is providing the stream of gcode commands to control your printer.

But anyway if you must do this, then I suspect that the major overhead will be the opening of the files over and over again. But I've certainly not ever spent any time measuring it so it is only a guess. The FAT32 implementation has very little caching to hold the various control blocks associated with the file system so they are probably being constantly reloaded from the SD card. You might be able to increase the amount of RAM used for this purpose, but doing so is a little tricky (only certain parts of the address space can be used for this purpose on a Duet3 and RAM is in short supply on a Duet2), basically if you need to ask how to do it, you probably shouldn't. You may be able to reduce the overhead involved in this operation by placing your macros in the root directory of the SD card, but that is not something I've ever tried.

To answer some of your other questions. I doubt if making the read buffer larger would help as I doubt if reading the file is the issue. Anything is possible, but I can't imagine that caching of macros is very high on anyone's development plans. Memory on most of the control boards tends to be at a premium and this does not seem like a good use for it for most people. You could try an SBC that would certainly help in terms of opening and to some extent parsing the files. However a lot of the operations in a macro need to be sent to the main control board for evaluation/execution and I would not be surprised to find that overall it is slower.

If this sort of thing is really important you might want to consider using a second board to provide the LED control you are interested in, possibly something like an esp32 (which could then talk to the main printer control board either via http and the object model or perhaps using MQTT). There are probably lots of other options available.

Did I mention that I think running this sort of code on your main printer control board (especially during a print) is a bad idea?

infiniteloop

@robotsneversleep

I'm using macro's in ways in which they were probably not intended, but there's so much potential...

… for failure, you mean? Look, you try to set-up a real-time task for a fancy light show. There’s nothing wrong with that, except of…

The Duet’s main task is to run a print, which, by its very nature, has to be controlled in real time. This task, by itself, is already a computationally demanding job. Thus, daemon.g may not become dominant.
RRF’s macros are based on a scripting language, every single command has to be interpreted over and over again. So, execution is slow. Sure, you can speed up things by means like caching or pre-compilation techniques, but at the cost of large amounts of RAM, which tends to be a rare resource on MCUs.

I'm trying my hardest to avoid using an SBC if I can.

Who says that daemon.g is your only alternative option? Most programmable controllers on the market are capable to perform an elaborated light show, even the venerable Arduino UNO can do this (albeit not with ease), not to mention ESP32 or similar devices. So, use event scripts or daemon.g on the Duet to trigger complex light effects with transitions and all those bells and whistles you can imagine.

EDIT: @gloomyandy was ahead of me. That's perfectly fine, he's right.

dc42

@robotsneversleep as @gloomyandy says, one of the major overheads is access to the SD card. The FatFs filesystem used by RRF doesn't include a cache, just a single sector buffer per SD card. I have it on my list to implement a cache (see https://github.com/Duet3D/RepRapFirmware/issues/688) but this isn't high priority, and any improvement on Duet 2 is likely to be marginal because of lack of RAM.

Meanwhile you can keep the overhead to a minimum by:

Not calling macros more than you need to (every M98 command executed has to open the file)
Not using loops more than you need to (every 'while' command needs to re-read part of the file when an iteration finishes).

For example, your script contains this:

while iterations < 1
     M98 P"statusled/effects/fade.g" D"I" C{global.STATUSLED_RED}
     M98 P"statusled/effects/fade.g" D"O" C{global.STATUSLED_RED}

You can speed it up by removing the redundant while-clause and by expanding those two M98 calls to fade.g.

T3P3 created this issue in Duet3D/RepRapFirmware

Improving Macro Execution Time

open Add LRU caching to FatFS #688