Reboots/resets randomly - RRF 3.5.0-b4

[Leonard03]

Hello! I started to have the same symptoms with my board (https://forum.duet3d.com/topic/33857/duet-2-wifi-hardfault-invstate) and I wonder if this tyoe of reset has something to do with the powersupply itself. Can a quick drop in the powersupply output result in this type of crash? A drop that is long enogh to cause a drop in the 3v3 circuit of the processor?
Just an idea..

[NeoDue]

Just for the sake of completeness (the other two issues I posted here are more pressing to me) - I have those reboots as well, running 3.5.0rc1 on a Duet 3 6HC with a Duet 3 wifi module on the latest firmware. In addition to that, I sometimes see connection losses of both the webinterface and the PanelDue that can only be cured by a reset.

Thankfully they did not happen during a print as of now, but rather when the printer has been sitting idle for an extended period of time. That might be caused by the fact that it usually takes the printer about 10+ hours to reset as far as I have noticed by now - all prints I did so far were well finished by then.

When this happens again, I will add an M122 printout here. So far,, the only thing I noticed on previous cases was the recurring note "gcodes spinning" as reset reason in all those cases.

[Exerqtor]

"Glad" to see i'm not the only one having these crashes, must admit that.

I had another crash mid print roughly an hour ago

debug log:

power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-21 22:24:09 [warn] Date and time set at power up + 00:00:09

M122:

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1 (2023-08-31 16:16:56) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102836
Dynamic ram: 124928 of which 0 recycled
Never used RAM 10436, free system stack 136 words
Tasks: NETWORK(1,ready,108.3%,228) HEAT(3,nWait,0.2%,335) Move(4,nWait,0.0%,261) CanReceiv(6,nWait,0.3%,773) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,342) TMC(4,nWait,4.3%,74) MAIN(1,running,529.8%,704) IDLE(0,ready,1.7%,29) AIN(4,delaying,5.3%,264), total 649.8%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 01:52:48 ago, cause: software
Last software reset at 2023-10-21 22:24, reason: HardFault invState, Gcodes spinning, available RAM 4720, slot 0
Software reset code 0x4063 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001ae 00000002 200014ec 00000000 20032bfa 0009df2d 00000000 600f0000 00000000 00000000 00000000 00000000 200301d4 00000800 20035710 2002bf00 20018668 2002bd9d 20018668 2001e880 0002fedf 00000000 00000000 00000000 20012058 00000014 b5dd8a35
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 6768325, completed 6768325, timed out 0, errs 0
MCU temperature: min 37.4, current 41.5, max 48.6
Supply voltage: min 2.2, current 24.1, max 25.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/34, heap memory allocated/used/recyclable 2048/1572/1116, gc cycles 318
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 35, reads 28491, writes 35, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 35, reads 28491, writes 35, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 20, reads 28506, writes 20, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 19, reads 28506, writes 19, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 19, reads 28507, writes 19, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-10-22 00:16:47
Cache data hit count 4294967295
Slowest loop: 13.37ms; fastest: 0.12ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.0ms, write time 4.6ms, max retries 0
=== Move ===
DMs created 83, segments created 15, maxWait 5875833ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 7, on retry 6, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 25, completed 25, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000807
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 60937, received 138751, lost 0, boc 0
Longest wait 2ms for reply type 6034, peak Tx sync delay 1839, free buffers 26 (min 25), ts 33842/33840/0
Tx timeouts 0,0,1,0,0,0 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 8.41ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 42952
WiFi IP address 192.168.10.50
Signal strength -51dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

[Exerqtor]

And another crash mid print, this is starting to become a REALLY annoying problem!

power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [info] Event logging stopped
power up + 00:00:03 [info] Event logging started at level debug
power up + 00:00:03 [info] Running: Duet 3 Mini5plus WiFi: 3.5.0-rc.1 (2023-08-31 16:16:56)
power up + 00:00:03 [debug] Done!
power up + 00:00:03 [debug] RepRapFirmware for Duet 3 Mini 5+ is up and running.
power up + 00:00:04 [warn] WiFi module started
power up + 00:00:08 [warn] WiFi module is connected to access point RV32-IOT2G, IP address 192.168.10.50
2023-10-22 10:21:54 [warn] Date and time set at power up + 00:00:08

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.1 (2023-08-31 16:16:56) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 3 of 40 (32 max)
=== RTOS ===
Static ram: 102836
Dynamic ram: 124104 of which 0 recycled
Never used RAM 11620, free system stack 186 words
Tasks: NETWORK(1,ready,17.3%,213) HEAT(3,nWait,0.0%,335) Move(4,nWait,0.0%,352) CanReceiv(6,nWait,0.0%,796) CanSender(5,nWait,0.0%,337) CanClock(7,delaying,0.0%,342) TMC(4,nWait,0.7%,108) MAIN(1,running,80.9%,704) IDLE(0,ready,0.3%,29) AIN(4,delaying,0.8%,264), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:20:17 ago, cause: software
Last software reset at 2023-10-22 10:21, reason: HardFault invState, Expansion spinning, available RAM 7500, slot 1
Software reset code 0x4072 HFSR 0x40000000 CFSR 0x00020000 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20011fa8 Task NETW Freestk 482 ok
Stack: 000001ae 00000002 200014ec 00000000 20032c0a 0009df2d 00000000 600f0000 00000000 00000000 00000000 00000000 20031a2c 00000800 2002bf00 2002bf00 00000001 2002bd9d 20018668 2001e880 0002fedf 00000000 00000000 00000000 20012058 00000014 b5dd8a35
Error status: 0x00
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 1217157, completed 1217157, timed out 0, errs 0
MCU temperature: min 41.0, current 41.7, max 48.4
Supply voltage: min 2.2, current 24.1, max 25.2, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/33, heap memory allocated/used/recyclable 2048/2008/1572, gc cycles 56
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64030, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64029, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64029, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64029, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 13, reads 64030, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-10-22 10:42:02
Cache data hit count 2002338155
Slowest loop: 16.49ms; fastest: 0.13ms
=== Storage ===
Free file entries: 18
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 7.1ms, write time 4.4ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 S1" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 10966, received 24933, lost 0, boc 0
Longest wait 2ms for reply type 6053, peak Tx sync delay 736, free buffers 26 (min 25), ts 6086/6084/0
Tx timeouts 0,0,1,0,0,0 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 24.59ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 43128
WiFi IP address 192.168.10.50
Signal strength -46dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

Needless to say that the hypothesis of the core issue being a non Duet board was wrong at least.

[dc42]

@Exerqtor this thread is very long so I haven't time to re-read every message although I did search for relevant words, with no hits. So apologies if we have covered this before. However:

'invState' hard faults typically show nonsensical stack data, different every time. We suspect that these faults are caused by transients or ESD. Have you grounded your hot end and extruder metalwork to Duet ground, either directly or through a resistor? This will avoid the act of extruding filament causing static charge to build up, which is likely to flash cover to the thermistor connections.

[Exerqtor]

@dc42 Yeah it's getting pretty long, but starting a new/follow-up thread would potentially "spread" info so I guess keeping everything here still is the best way to go(?).

Uuhm I haven't done so, I'm running a LGX-lite and Revo-micro so there aren't really any obvious places for me to ground either of them.

If you have any ideas of good ways to ground those I'm all ears and will try to deploy it ASAP 😅

If ESD turns out being the issue, it's quite weird that it manifests after FW update though?

[Leonard03]

@Exerqtor here I cand give you an ideea about grounding. After I had almost the same issue about one year ago (?) as David said, I've grounded all seven steppers on by printer by a ring connector (if this is how is called) and three washers between sthe motors and the mounting points. The washers should be the same thickness as the ring of the connector. Not the most elegant way, but it does the job. All connections I've done to the ATX PSU and from there on is groundes by the power plug (seems that the duet2wifi is grounded with the PSU by the negative terminal).

Abount grounding the hotend, I'm running an E3D V6 gold edition with the PT100 and the DB but back then I've contacted E3D but they sayd that the heatsink of the hotend is not conductive, so the only grounding for the filament path is by the motor shaft being in contact with it

[Exerqtor]

@Leonard03 Thanks for weighing in, but i'm pretty sure this is the oposite in my case. Since the extruder stepper ain't in contact with the filament in any way. And the extruder motor aren't in contact with any metal in a regular LGX lite either. Everything on a Revo micro however is conductive

Also, everything is mounted in/on a Voron Stealthburner (printed ABS parts) , no metal conducting anything betwen the extruder and hotend that way either.

[Leonard03]

@Exerqtor that been said.. now I'm thinking about my setup.. I forgot that between the motor and the drive gears for a BMG is the big plastic gear
Seems like I'd grounded only the extruder motor, not the filament path in any way.. Great, another thing to think about

[Exerqtor]

@dc42

I've been looking a little bit into adding ground to the hotend, while I've figgured out a way i may do it I've also voiced the issue on other channels and gotten several concerns regarding it maybe causing a ground loop if it actually is ESD building up.

Some also mentioned (after reading through this post) that ESD don't really look like being the issue since the crashes happen just as often when the printer is idle with no ESD built up, which I didn't even consider when you mentioned it.

But I have to agree ESD buildup from the extruder/motion system being the culprit is debunked when it also happens in idle state.

[sebkritikel]

@Exerqtor said in Reboots/resets randomly - RRF 3.5.0-rc.1:

@dc42 Yeah it's getting pretty long, but starting a new/follow-up thread would potentially "spread" info so I guess keeping everything here still is the best way to go(?).

Uuhm I haven't done so, I'm running a LGX-lite and Revo-micro so there aren't really any obvious places for me to ground either of them.

If you have any ideas of good ways to ground those I'm all ears and will try to deploy it ASAP 😅

If ESD turns out being the issue, it's quite weird that it manifests after FW update though?

Don't have anything to add to directly help you (I hope it gets resolved soon!), but if you do make a new thread, I would list all the recommendations that were made to you, and then all the actions you took. Figure that would help cut out the fluff but detail every action that has been taken so far.

[Exerqtor]

Creating a new thread for this issue with RRF 3.5.0-rc1 as subject matter, therefore closing this thread as of 29th october 2023.