Duet mini 5+ freezes and shuts off heaters

[droftarts]

@michael-hathaway I just did a quick test on my Mini 5+ to see what a power cycle, M999 (reset) and reset button press reports on restart in M122.
M999:

=== Platform ===
Last reset 00:00:12 ago, cause: software
Last software reset at 2022-09-07 11:54, reason: User, GCodes spinning, available RAM 27680, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00

Reset button:

=== Platform ===
Last reset 00:00:13 ago, cause: reset button
Last software reset at 2022-09-07 11:54, reason: User, GCodes spinning, available RAM 27680, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00

Power cycle:

=== Platform ===
Last reset 00:00:22 ago, cause: power up
Last software reset at 2022-09-07 11:54, reason: User, GCodes spinning, available RAM 27680, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x00

I did the M999 first, and that is what the subsequent M122 'Last software reset...' is reporting. I notice that your M122 also reports the same time for 'Last software reset...' so I don't think the reset is coming from the firmware or gcode file.

I think if the firmware is reporting a reset button press, it probably actually is a reset button press or short. Is the printed case you have the Duet in new? My guess is that, as the board heats up, it's flexing slightly and somehow shorting out on the case. Maybe add some nylon washers under your mounting screws (ideally the board should be mounted using nylon washers), to lift the board up a little?

Ian

[Michael Hathaway]

@droftarts Thank you, I will try that. I will also order a new board just in case I have fried this one.

[droftarts]

@michael-hathaway Looking at the Mini 5+ schematic (https://raw.githubusercontent.com/Duet3D/Duet3-Mini5plus/main/v1.01/Duet3_5%2B_Wifi_schematic_v1.01.pdf), there are also reset pins on the 12864_EXP2 connector, SWD connector (just behind the reset button, but as it is unpopulated with a header, this shouldn't be a problem) and there's also a test point (on the back of the board, by the SWD connector pins, I think). I don't think you have anything plugged into the 12864 connector, but again check that it is not shorting out to anything, top or bottom.

Ian

[Michael Hathaway]

I ordered a new Duet mini 5 plus board with overnight postage, should have it tomorrow. I also ordered nylon washers as recommended.

This time when mounting in the petg enclosure, I will allow to board to float and not be fastened down so well.

I have found new Samsung endurance sdcards, which are designed to be powered on continuously. I am hopeful this increases reliability.

I appreciate everyone's help I have received and I will update with my progress as I proceed.

[Michael Hathaway]

Brand new Duet mini 5+ board, mounted loose on nylon washers.

Reset tripped, frozen mid print.

9/10/2022, 1:42:33 AM 	m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.1 (2022-06-01 21:06:56) running on Duet 3 Mini5plus Ethernet (standalone mode)
Board ID: 6MFQS-V096U-D65J0-40KMJ-JS03Z-R72JU
Used output buffers: 1 of 40 (15 max)
=== RTOS ===
Static ram: 103684
Dynamic ram: 105204 of which 0 recycled
Never used RAM 32824, free system stack 204 words
Tasks: NETWORK(ready,26.2%,246) ETHERNET(notifyWait,0.2%,568) HEAT(notifyWait,0.0%,358) Move(notifyWait,0.0%,363) CanReceiv(notifyWait,0.0%,942) CanSender(notifyWait,0.0%,372) CanClock(delaying,0.0%,339) TMC(notifyWait,0.7%,114) MAIN(running,71.9%,422) IDLE(ready,0.2%,29) AIN(delaying,0.8%,264), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:01:26 ago, cause: reset button
Last software reset details not available
Error status: 0x00
MCU revision 3, ADC conversions started 87093, completed 87092, timed out 0, errs 0
Step timer max interval 752
MCU temperature: min 60.2, current 60.2, max 62.8
Supply voltage: min 24.1, current 24.1, max 24.1, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 2, read errors 0, write errors 1, ifcnt 39, reads 4560, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 4, read errors 0, write errors 1, ifcnt 39, reads 4560, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 39, reads 4559, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 1, ifcnt 33, reads 4559, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 29, reads 4560, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2022-09-10 01:42:33
Cache data hit count 148434782
Slowest loop: 8.24ms; fastest: 0.13ms
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 1.0ms, write time 3.8ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== Filament sensors ===
Extruder 0 sensor: ok
=== CAN ===
Messages queued 783, received 0, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 18 (min 18), ts 436/0/0
Tx timeouts 0,0,435,0,0,346 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 4.93ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 2 of 8
- Ethernet -
State: active
Error counts: 0 0 0 0 0
Socket states: 2 2 2 2 2 0 0 0

[Michael Hathaway]

I have 3 boards now that do the same thing. One board ran great for 1.5 years and this happened.

[droftarts]

@michael-hathaway is it the same print, same Gcode file and stopping in the exact same place? I still don’t think anything else would cause M122 to report a manual reset except for a reset button press, or short on on of the other reset pins I mentioned in my earlier post.

Did you change anything just before this started happening? I can’t think that you’d have three Duets that have the same problem!

Ian

[Michael Hathaway]

@droftarts I didn't change anything else. I will try a few different prints and report back. I don't know the electronics well enough to know what would trigger a reset other than the button and those contacts which are not being touched. If a stepper was getting too hot, would that trigger a reset? Would a broken wire trigger a reset?

[Phaedrux]

Static build up maybe? Is the frame grounded? Is the PSU - side grounded to earth?

[droftarts]

@michael-hathaway do you have anything connected to the 12864 EXP2 header? Like a 12864 screen?

Ian

[Michael Hathaway]

@droftarts Hello, I do not have anything plugged into those headers. I am thankful all of you are working hard to help me on this. I was going to wait to share what I have found until I can confirm.

So far I have printed multiple prints which have all failed.

Before I go further, I want to add that I am printing high temperature PA66 CF in the 330C range. I have a custom built E3D copper/titanium air cooled head unit I created with dual fan 10.2cfm heat break cooling. Pt1000/50w cartridge all Slice engineering. Printing at these temperatures is a common daily practice.

After multiple prints failed, I installed a new pt1000. Result: print failed with an "emergency shutdown" listed in console.

Removed SDcard and installed new card with cloned programing. While heating hotend, I received a heat fault due to not heating fast enough. I found this interesting because the identical config was used with no heat fault before, only the sdcard was changed.

Ran M303 H1 S350 tuning, compared the results which changed and added to config.g. For safety measure, I ran for both the hotend and the bed.

For my next print, I changed out spools to lower temperature Nylon which is now at 280°C instead of 330°C.

Ran a small print which was successful. 1st in two weeks of struggling.

I am currently testing more prints at this lower temperature and then I will reload my expensive material back on to test. I have been working day and night and I will keep you updated with what I find.

[Michael Hathaway]

@phaedrux said in Duet mini 5+ freezes and shuts off heaters:

Static build up maybe? Is the frame grounded? Is the PSU - side grounded to earth?

Yes, all electrically grounded to an earth ground.

[Michael Hathaway]

Additional fail, just now.

Only this time, the heaters did not shut off. Printer just frozen in place.

9/10/2022, 3:30:58 PM 	m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.1 (2022-06-01 21:06:56) running on Duet 3 Mini5plus Ethernet (standalone mode)
Board ID: 6MFQS-V096U-D65J0-40KMJ-JS03Z-R72JU
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 103684
Dynamic ram: 105420 of which 0 recycled
Never used RAM 32152, free system stack 118 words
Tasks: NETWORK(ready,26.0%,178) ETHERNET(notifyWait,0.2%,562) HEAT(notifyWait,0.0%,326) Move(notifyWait,0.0%,265) CanReceiv(notifyWait,0.0%,942) CanSender(notifyWait,0.0%,348) CanClock(delaying,0.0%,331) TMC(notifyWait,0.7%,71) MAIN(running,72.2%,418) IDLE(ready,0.0%,29) AIN(delaying,0.8%,264), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 06:52:35 ago, cause: software
Last software reset at 2022-09-10 08:38, reason: User, GCodes spinning, available RAM 32752, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
MCU revision 3, ADC conversions started 24756664, completed 24756664, timed out 0, errs 0
Step timer max interval 751
MCU temperature: min 61.7, current 61.9, max 63.2
Supply voltage: min 23.8, current 24.1, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 41, reads 3363, writes 1, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 37, reads 3363, writes 1, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 2, read errors 0, write errors 0, ifcnt 41, reads 3364, writes 1, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 2, read errors 0, write errors 0, ifcnt 27, reads 3363, writes 1, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 19, reads 3364, writes 0, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2022-09-10 15:30:56
Cache data hit count 4294967295
Slowest loop: 5.15ms; fastest: 0.13ms
=== Storage ===
Free file entries: 9
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 1.0ms, write time 6.3ms, max retries 0
=== Move ===
DMs created 83, segments created 19, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 142236, completed 142236, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.4
Heater 1 is on, I-accum = 0.5
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== Filament sensors ===
Extruder 0 sensor: ok
=== CAN ===
Messages queued 576, received 0, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 18 (min 18), ts 320/0/0
Tx timeouts 0,0,320,0,0,256 last cancelled message type 4514 dest 127
=== Network ===
Slowest loop: 4.19ms; fastest: 0.04ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 2 of 8
- Ethernet -
State: active
Error counts: 0 0 0 0 0
Socket states: 5 2 2 2 2 0 0 0

[tmitch205]

@phaedrux I was going to ask the same thing about static considering it's sitting on carpet. I have seen Duet 2 Wifi's lose their wifi connectivity due to this until they are restarted.

[Michael Hathaway]

Just a small update. I switched from Nylon to PETG. 250C/83C hotend/bed temps. I didn't change anything but the print temperatures. I was able to make 6 small and large successful prints using both Cura 5 and MatterControl slicers. Prints were perfect.

Now, loaded Nylon back on, but kept the bed temp the same for testing. 281C/83C. I will test different parameters.

[droftarts]

@michael-hathaway are Nylon and CF more prone to static buildup? Perhaps grounding the nozzle would help. Make sure the extruder motor is grounded, too. I still have no idea how it’s triggering a ‘manual reset’, unless a sudden zap is enough to drag the reset signal to ground.

Ian

[Michael Hathaway]

The printer is grounded and there is no carpet, only concrete in workshop. If static were the culprit, it would not print at lower temperatures.

It was 45-46°C in my workshop last week and I think the higher printing temperatures over loaded the board. While it was triggering a reset, one time it also triggered an emergency stop on its own.

It is now 26°C in the shop and it is printing my lower temp Nylon fine. I will load the HTNCF next and see if fails or continues to print.

[droftarts]

@michael-hathaway I asked @dc42 if he thought high ambient temperature causing the Duet MCU to overheat would trigger a reset that looks like a manual reset. He said "I think that's unlikely, but not impossible." Without knowing how hot the Duet board is getting (map the on-board MCU temp sensor to see a graph) that's difficult to know. An MCU reset or shutdown would be triggered somewhere above 85°C.

I know you said "the printer is grounded", but specifically are the metal parts of the hotend grounded? I still think a static shock is more likely, and grounding the hot end would eliminate this. You said you swapped out the PT1000 for a new one and got a different failure, so possibly static charge in the the hot end has arced through that and caused a reset. High ambient temperature + high temperature nozzle (this would lower humidity, making static worse) + possibly CF and Nylon that produce more static charge than PETG might be enough to reset the MCU.

However, @chrishamm thinks that a static shock to the temperature inputs may cause a MemoryProtectionFault reset (it does on the 6HC, not sure about the Mini 5+), which would show up in the M122 report.

One other question: How did the old Mini 5+ fail? Could it be related? Maybe the MCU got zapped?

Ian

[Michael Hathaway]

@droftarts I ordered a ring terminal to fit the hot end for grounding, will be here tomorrow. It was extremely humid here and hot.

All the boards show the same failure. I have had trouble free performance for 1.5 years. I have had the mini board shut down before when the chamber temperature went over 50C. This is an unusual heat wave in California. If this keeps up, I think I will install air conditioning in the workshop.

The other Matterhacker guys prefer their Azteeg boards. But for me, I only use Duet.

[Michael Hathaway]

Added electrical ground wire from the hotend to the frame and to electrical earth ground. Printer still failed prints.

Pulled printer from workshop into air conditioned building. Printer has produced several 10+ hour prints.

For fun, I added a copper heat sink to the MCU. Temperature now reading a -6.7°C drop. Probably not worth doing, but if a few degrees means the difference between another stall and not, then I am ok with it. I can now be part of the "stuck a heat sink on my MCU" fan club. lol

I think that I will make a new case cover that brings ambient air from outside the heat chamber to the Duet and then draw it back out again. Shouldn't be too difficult, I think two 60mm fans would work well.