3.5.1 - 'Error: Pop(): stack underflow on Aux'

gnydick

@T3P3Tony I do have PanelDue. It happens at different times, I don't remember exactly what the situation is. I will let you know next time.

gnydick

@T3P3Tony yep, it is when pausing a print.

omtek

Adding a new issue that I happened to notice:

After the error occurs, I am unable to see any files populated in the SD card section, or the Macro section on the PanelDue.
Additionally, and likely related, the 'Print Again' button on the PanelDue does nothing (I can hear the beep when I press it, but the printer doesn't repeat the last print). I am able to repeat the print in DWC using the "Status - Print Again" button.

After restarting the print using DWC, I went to pause the printer again. I pressed the pause button on the PanelDue, and nothing happened. I press pause again, and the printer paused. The PanelDue then gave a series of errors that don't show up in the DWC console (M25, because of the second pause command and M32 because of the "Print Again" button from before, I'm guessing).

After cancelling the print, I am able to browse the files in the SD card menu as well as my macros on the PanelDue.

M122 taken after the cancelled print.

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.1 (2024-04-19 14:41:25) running on Duet 3 Mini5plus WiFi (SBC mode)
Board ID: NTK2Z-J196U-D65J0-40KMN-1V03Z-ZJMGF
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 103232
Dynamic ram: 108224 of which 236 recycled
Never used RAM 25908, free system stack 118 words
Tasks: SBC(2,ready,75.6%,294) HEAT(3,nWait 6,5.2%,326) Move(4,nWait 6,88.3%,241) CanReceiv(6,nWait 1,9.0%,772) CanSender(5,nWait 7,5.1%,327) CanClock(7,delaying,1.1%,348) TMC(4,nWait 6,20.7%,68) MAIN(2,running,153.0%,580) IDLE(0,ready,0.2%,30) AIN(4,delaying,121.8%,256), total 480.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 181:48:17 ago, cause: software
Last software reset at 2024-05-02 23:14, reason: User, none spinning, available RAM 26452, slot 1
Software reset code 0x6013 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,2,0
MCU revision 3, ADC conversions started 654509289, completed 654509288, timed out 0, errs 0
MCU temperature: min 29.9, current 33.6, max 53.9
Supply voltage: min 24.2, current 24.4, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/9, heap memory allocated/used/recyclable 2048/2012/1860, gc cycles 3
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 2, write errors 0, ifcnt 167, reads 61817, writes 29, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 166, reads 61818, writes 29, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 168, reads 61817, writes 30, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 167, reads 61820, writes 28, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 1, write errors 0, ifcnt 166, reads 61817, writes 29, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 2, write errors 0, ifcnt 32, reads 61845, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 0, ifcnt 32, reads 61848, writes 0, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2024-05-10 13:02:24
Cache data hit count 4294967295
Slowest loop: 383.36ms; fastest: 0.10ms
=== Storage ===
Free file entries: 20
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 40, maxWait 356176223ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 13675, completed 13675, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 20], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC* is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 9346425, received 12112084, lost 0, errs 0, boc 0
Longest wait 2ms for reply type 6013, peak Tx sync delay 546, free buffers 26 (min 25), ts 2162545/2162545/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 5, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 818/818
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0 total, 0 by SBC, IAP RAM available 0x0d344
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server version 3.5.1 (2024-04-19 16:30:56, 32-bit)
HTTP+Executed:
> Executing M122
Aux:
Buffered code: M121
Buffered codes: 24 bytes total
Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 0
Full transfers per second: 38.46, max time between full transfers: 172.4ms, max pin wait times: 172.1ms/7.0ms
Codes per second: 1.94
Maximum length of RX/TX data transfers: 4499/7656

gnydick

@T3P3Tony I now experienced it after issuing T-1. Machine had been freshly rebooted. No pause was issued. The only unusual behavior was that T0 P0 was issued to load the tool.

gloomyandy

@gnydick So what exactly did you do to produce this situation?

reboot
T0 P1
T-1

Did you issue any other commands? How did you issue the commands (from the panelDue?). Is this error repeatable?

gnydick

@gloomyandy there were a couple of commands between the T0 P0 and T-1. I was basically just selecting the tool so I could load more filament.

I'm 99% sure it was all through PanelDue. I will try to reproduce it and let you know.

gloomyandy

@gnydick If you can get a set of commands that is reproducible and they are issued via the panelDue it might also be useful to check if the same set of commands produces an error if they are issued via the DWC console or perhaps via a USB connection.

omtek

@gloomyandy It seems like if the command from the PanelDue runs a predefined macro, then it's likely to generate the error message.

So far I've encountered the error when doing the following:

Pausing a print (runs pause.g)
Putting an active tool into standby (runs the toolchange macros)

What other commands besides those two trigger a predefined macro? The only other one I can think of is 'Cancel', but that won't show up unless the printer is paused.

oliof

@omtek any homing command (G28 X/Y/Z), as well as triggering a bed level (G32) or a bed mesh (G29). Main difference between these and pause/tool change is that pause and tool change may interact with the extruder for retract/unretracts?

omtek

@oliof You're right. I hadn't considered the homing macros. That's something else to try.

Getting the error when pausing during print seems to occur fairly consistently, as does putting a tool into standby. So far the printer appears to be unaffected, and the only additional error messages I'm getting in DWC appear to be when the PanelDue manages to correct itself and sends the commands that were queued up from the then non-working buttons.

droftarts

@omtek as this is on an SBC setup, @chrishamm could you have a look at this?

Ian

gnydick

@gloomyandy I issue those same commands from DWC all of the time and it hasn't happened.

omtek

Right now I can consistently get the error by doing the following:

Heating the tool to print temp (205C in this case)
Use the PanelDue to extrude 20mm of material.
Tap the tool icon on the PanelDue to put the tool into standby and deselect

I've been able to get a single 'Error: Pop(): stack underflow on Aux' with five out of five attempts.

If I perform the same actions in DWC, I don't get the error.

Additionally the error doesn't occur when doing the following:

Extruding in DWC and putting the tool into standby on the PanelDue.
Extruding on the PanelDue and putting the tool into standby in DWC

And aside from this error, the printer has been working great.

droftarts

@omtek how is the PanelDue wired to the Duet, ribbon cable or 4-wire cable? Is it particularly long?

You also have the PanelDue serial comms speed set higher than we normally suggest. From your config:

M575 P1 S1 B115200

Try M575 P1 S1 B57600

I suggest checking the wiring, crimps and connectors for good connections, and make sure they are not running close to motor or heater wires, which could be sources of interference.

Ian

omtek

@droftarts I'm using the 4-pin Molex KK cable. The original cable length was ~711mm long but I spent the morning re-crimping and shortening the length to ~400mm. I also updated the M575 line in my config.g per your suggestion. However, I'm still getting the single 'Error: Pop(): stack underflow on Aux' when performing the troubleshooting steps I outlined here.

droftarts

@omtek is there anyway you could setup in standalone mode (no RPi/SBC) and test if you have the same behaviour? That might help point out if it’s a firmware issue, or DSF. Is there anything else unique about your setup?

Ian

omtek

@droftarts My setup is nothing too exotic I hope: Duet 3 Mini 5+, Raspberry Pi 4b+ 4GB with external M.2 SSD (no SD card), RRF36 toolboard and a CannedERCF board (sitting on the canbus as passthrough, but no gcode configuration sent during this); all boards are running v3.5.1. I do have 5VEXT connected via an external power supply for a servo-powered filament cutter and the jumper set to disable the internal 5V.

I removed the jumper that disables the internal 5V power supply, but received the same error.
I copied my /sys folder to a microSD card flashed with image_2024-04-19-DuetPi-lite.zip, and I no longer get the Error: Pop(): stack underflow on Aux error using my troubleshooting steps. Several attempts over the last half-hour and the error has not occurred once.

My current setup was running the older ~~Jessie~~Buster image, and the only additional software I was running on the Pi was a uStreamer instance for the webcam (this was before Spyglass came out).

Is it worth trying out the new Bookworm image for troubleshooting now that it appears the error is specific to SBC mode?

droftarts

@omtek said in 3.5.1 - 'Error: Pop(): stack underflow on Aux':

Is it worth trying out the new Bookworm image for troubleshooting now that it appears the error is specific to SBC mode?

You can certainly give it a go, if you want! As it seems to be related to SBC mode, I'll ask @chrishamm to look at this.

Ian

omtek

@droftarts I used the same microSD card that I tested standalone mode to boot the raspberry pi and moved my configuration files over, however I still receive the Error: Pop(): stack underflow on Aux error when performing the troubleshooting steps.

Additionally, and probably out of the scope of this post, but I had issues booting my external M.2 drive when flashed with image_2024-04-19-DuetPi-lite.zip. The pi runs headless, but the green LED would flash 3x (missing start.elf iirc), and the M.2 drive activity would flash for a time, and then stop. I didn't feel like troubleshooting past that, but I was able to boot from the official Raspberry Pi OS (64-bit, lite) using the same drive and enclosure and following the instructions here I was able to get the repos added and DWC, DSF, et al installed. Unfortunately, the Error: Pop(): stack underflow on Aux error occurred using the official Raspberry Pi OS image and performing the troubleshooting steps.

chrishamm

@omtek I've got a bug fix ready for v3.5.2 which we will release sometime soon.