3.5.1 - 'Error: Pop(): stack underflow on Aux'

omtek

@T3P3Tony So far, it only seems to happen when using the PanelDue. Just now, I was able to pause and resume a print using DWC, and then again on the PanelDue and got the same behavior both times (on pause, the filament retracts, Z lifts, head moves to the "back" of the printer; on resume, the head returns to the previous position, Z lowers, 10mm of filament is extruded, and the print resumes as normal.)

I've been unable to recreate the initial unresponsiveness I encountered in my initial post and though the error is concerning, it doesn't appear to adversely affect the printers operation.

So far I've had the error occur the following ways:

Pressing the pause button on the PanelDue
Putting an at-temperature T0 into standby (turning the icon on the PanelDue from red to yellow, as seen here)

If I can get the error to occur again, is there any specific commands I should run or behavior to watch out for that would tease out more information?

T3P3Tony

@omtek that's a question for @dc42 I was just checking the obvious things this looks to be a PanelDue issue so I will move it to the PanelDue section

gnydick

@T3P3Tony I'm having this issue also, maybe it's related to why my tools are crashing??

T3P3Tony

@gnydick do you have a PanelDue and does it happen when you are pausing?

gnydick

@T3P3Tony I do have PanelDue. It happens at different times, I don't remember exactly what the situation is. I will let you know next time.

gnydick

@T3P3Tony yep, it is when pausing a print.

omtek

Adding a new issue that I happened to notice:

After the error occurs, I am unable to see any files populated in the SD card section, or the Macro section on the PanelDue.
Additionally, and likely related, the 'Print Again' button on the PanelDue does nothing (I can hear the beep when I press it, but the printer doesn't repeat the last print). I am able to repeat the print in DWC using the "Status - Print Again" button.

After restarting the print using DWC, I went to pause the printer again. I pressed the pause button on the PanelDue, and nothing happened. I press pause again, and the printer paused. The PanelDue then gave a series of errors that don't show up in the DWC console (M25, because of the second pause command and M32 because of the "Print Again" button from before, I'm guessing).

After cancelling the print, I am able to browse the files in the SD card menu as well as my macros on the PanelDue.

M122 taken after the cancelled print.

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.1 (2024-04-19 14:41:25) running on Duet 3 Mini5plus WiFi (SBC mode)
Board ID: NTK2Z-J196U-D65J0-40KMN-1V03Z-ZJMGF
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 103232
Dynamic ram: 108224 of which 236 recycled
Never used RAM 25908, free system stack 118 words
Tasks: SBC(2,ready,75.6%,294) HEAT(3,nWait 6,5.2%,326) Move(4,nWait 6,88.3%,241) CanReceiv(6,nWait 1,9.0%,772) CanSender(5,nWait 7,5.1%,327) CanClock(7,delaying,1.1%,348) TMC(4,nWait 6,20.7%,68) MAIN(2,running,153.0%,580) IDLE(0,ready,0.2%,30) AIN(4,delaying,121.8%,256), total 480.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 181:48:17 ago, cause: software
Last software reset at 2024-05-02 23:14, reason: User, none spinning, available RAM 26452, slot 1
Software reset code 0x6013 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,2,0
MCU revision 3, ADC conversions started 654509289, completed 654509288, timed out 0, errs 0
MCU temperature: min 29.9, current 33.6, max 53.9
Supply voltage: min 24.2, current 24.4, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/9, heap memory allocated/used/recyclable 2048/2012/1860, gc cycles 3
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 2, write errors 0, ifcnt 167, reads 61817, writes 29, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 166, reads 61818, writes 29, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 168, reads 61817, writes 30, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 167, reads 61820, writes 28, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 1, write errors 0, ifcnt 166, reads 61817, writes 29, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 2, write errors 0, ifcnt 32, reads 61845, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 0, ifcnt 32, reads 61848, writes 0, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2024-05-10 13:02:24
Cache data hit count 4294967295
Slowest loop: 383.36ms; fastest: 0.10ms
=== Storage ===
Free file entries: 20
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 40, maxWait 356176223ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 13675, completed 13675, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 20], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.1
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC* is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 9346425, received 12112084, lost 0, errs 0, boc 0
Longest wait 2ms for reply type 6013, peak Tx sync delay 546, free buffers 26 (min 25), ts 2162545/2162545/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 5, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 818/818
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0 total, 0 by SBC, IAP RAM available 0x0d344
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server version 3.5.1 (2024-04-19 16:30:56, 32-bit)
HTTP+Executed:
> Executing M122
Aux:
Buffered code: M121
Buffered codes: 24 bytes total
Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 0
Full transfers per second: 38.46, max time between full transfers: 172.4ms, max pin wait times: 172.1ms/7.0ms
Codes per second: 1.94
Maximum length of RX/TX data transfers: 4499/7656

gnydick

@T3P3Tony I now experienced it after issuing T-1. Machine had been freshly rebooted. No pause was issued. The only unusual behavior was that T0 P0 was issued to load the tool.

gloomyandy

@gnydick So what exactly did you do to produce this situation?

reboot
T0 P1
T-1

Did you issue any other commands? How did you issue the commands (from the panelDue?). Is this error repeatable?

gnydick

@gloomyandy there were a couple of commands between the T0 P0 and T-1. I was basically just selecting the tool so I could load more filament.

I'm 99% sure it was all through PanelDue. I will try to reproduce it and let you know.

gloomyandy

@gnydick If you can get a set of commands that is reproducible and they are issued via the panelDue it might also be useful to check if the same set of commands produces an error if they are issued via the DWC console or perhaps via a USB connection.

omtek

@gloomyandy It seems like if the command from the PanelDue runs a predefined macro, then it's likely to generate the error message.

So far I've encountered the error when doing the following:

Pausing a print (runs pause.g)
Putting an active tool into standby (runs the toolchange macros)

What other commands besides those two trigger a predefined macro? The only other one I can think of is 'Cancel', but that won't show up unless the printer is paused.

oliof

@omtek any homing command (G28 X/Y/Z), as well as triggering a bed level (G32) or a bed mesh (G29). Main difference between these and pause/tool change is that pause and tool change may interact with the extruder for retract/unretracts?

omtek

@oliof You're right. I hadn't considered the homing macros. That's something else to try.

Getting the error when pausing during print seems to occur fairly consistently, as does putting a tool into standby. So far the printer appears to be unaffected, and the only additional error messages I'm getting in DWC appear to be when the PanelDue manages to correct itself and sends the commands that were queued up from the then non-working buttons.

droftarts

@omtek as this is on an SBC setup, @chrishamm could you have a look at this?

Ian

gnydick

@gloomyandy I issue those same commands from DWC all of the time and it hasn't happened.

omtek

Right now I can consistently get the error by doing the following:

Heating the tool to print temp (205C in this case)
Use the PanelDue to extrude 20mm of material.
Tap the tool icon on the PanelDue to put the tool into standby and deselect

I've been able to get a single 'Error: Pop(): stack underflow on Aux' with five out of five attempts.

If I perform the same actions in DWC, I don't get the error.

Additionally the error doesn't occur when doing the following:

Extruding in DWC and putting the tool into standby on the PanelDue.
Extruding on the PanelDue and putting the tool into standby in DWC

And aside from this error, the printer has been working great.

droftarts

@omtek how is the PanelDue wired to the Duet, ribbon cable or 4-wire cable? Is it particularly long?

You also have the PanelDue serial comms speed set higher than we normally suggest. From your config:

M575 P1 S1 B115200

Try M575 P1 S1 B57600

I suggest checking the wiring, crimps and connectors for good connections, and make sure they are not running close to motor or heater wires, which could be sources of interference.

Ian

omtek

@droftarts I'm using the 4-pin Molex KK cable. The original cable length was ~711mm long but I spent the morning re-crimping and shortening the length to ~400mm. I also updated the M575 line in my config.g per your suggestion. However, I'm still getting the single 'Error: Pop(): stack underflow on Aux' when performing the troubleshooting steps I outlined here.

droftarts

@omtek is there anyway you could setup in standalone mode (no RPi/SBC) and test if you have the same behaviour? That might help point out if it’s a firmware issue, or DSF. Is there anything else unique about your setup?

Ian