Duet 3 mini5+ nightly restarts 3.5.2

Chriss

I had a restart during printing after a long time again.

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.1 (2024-04-19 14:41:25) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: V9NWJ-R296U-D65J0-40KM6-4113Z-HM83B
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 103232
Dynamic ram: 123888 of which 0 recycled
Never used RAM 11440, free system stack 192 words
Tasks: NETWORK(1,ready,14.5%,220) HEAT(3,nWait 6,0.0%,332) Move(4,nWait 6,0.0%,341) CanReceiv(6,nWait 1,0.0%,797) CanSender(5,nWait 7,0.0%,336) CanClock(7,delaying,0.0%,348) TMC(4,nWait 6,1.4%,102) MAIN(1,running,83.1%,665) IDLE(0,ready,0.1%,30) AIN(4,delaying,0.9%,260), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 00:12:15 ago, cause: software
Last software reset at 2024-08-18 17:50, reason: HardFault imprec, Expansion spinning, available RAM 9284, slot 1
Software reset code 0x4072 HFSR 0x40000000 CFSR 0x00000400 ICSR 0x00000803 BFAR 0xe000ed38 SP 0x20012008 Task NETW Freestk 489 ok
Stack: 2002c5e0 200307c8 200014e8 00000000 200331ad 0002ff99 0002feb0 610f6000 2002c5c8 2002c5c8 00000001 2002c41e 200187fc 2001e9f8 0003011f 00000000 00000000 00000000 200120a0 00000014 00000000 00000002 ea400050 0801a8c0 08000161 00000001 000347e1
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 735652, completed 735650, timed out 0, errs 0
MCU temperature: min 39.1, current 39.3, max 53.2
Supply voltage: min 24.3, current 24.4, max 24.5, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/15, heap memory allocated/used/recyclable 2048/1408/1196, gc cycles 2
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 2, read errors 0, write errors 1, ifcnt 35, reads 1289, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 150, read errors 0, write errors 1, ifcnt 241, reads 1286, writes 16, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 138, read errors 0, write errors 1, ifcnt 231, reads 1286, writes 16, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 32, reads 1292, writes 10, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 2, read errors 0, write errors 1, ifcnt 35, reads 1289, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 5: standstill, SG min 0, read errors 0, write errors 1, ifcnt 35, reads 1289, writes 13, timeouts 0, DMA errors 0, CC errors 0
Driver 6: standstill, SG min 0, read errors 0, write errors 1, ifcnt 35, reads 1289, writes 13, timeouts 0, DMA errors 0, CC errors 0
Date/time: 2024-08-18 18:02:53
Cache data hit count 1378274719
Slowest loop: 11.12ms; fastest: 0.17ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 5.7ms, write time 1.8ms, max retries 0
=== Move ===
DMs created 83, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, max steps late 0, min interval 0, bad calcs 0, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000803
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 6636, received 14746, lost 0, errs 1, boc 0
Longest wait 2ms for reply type 6031, peak Tx sync delay 263, free buffers 26 (min 25), ts 3678/3677/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 11.55ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(1)
HTTP sessions: 3 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1.0
MAC address f0:08:d1:02:e6:75
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 40612
WiFi IP address 192.168.1.69
Signal strength -63dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 3 0 0 0 0 0 0 0

Btw: My bench board has no problem at all and is on since 8+ weeks. :?

Chriss

One more thing... The reboot happened again at 85-95% of the print. As it always does when it happens to my printer. Am I the only one with this observation?

oliof

@Chriss does it also happen when simulating a print?

Chriss

@oliof No idea to be honest. I saw it only when the printer was actually printing. (I grounded my hotend and my e-stepper some month ago for a other problem) I have only one printer with this firmware and I need it at the moment to finish some projects. And the problem is not easy to reproduce. It seems to me like it was wore at the beginning, like every morning a rebooted printer, but this is not the case anymore. I guess that this was a false observation but is not that I experiance a reboot at every print or every 2nd print.

I have the uneducated feeling that the failure happens only on higher prints too. I printed many pars which were below 100mm in Z. And two higher parts and the 2nd failed.

Not saying that any of them are causing the issue, I'm listing my observations only.

@droftarts Is there any kind of debug firmware which we can run to log the data to the serial interface? I would be more than happy to test more and capture as much as I can? This topic is getting older and older and I have the impression that we make no progress at all.
I have 4 of this boards currently, 1 or two of them should be from the first batch, the others are a bit younger. I'm not sure which one is in which printer. Would it make sense to check that? (I guess that my bench board is the youngest one.) I can place the bench board into the working printer if this make any sene? Just to see whether it happens in a real printer with the bench board not. The bench board looks like at so far:

=== Platform ===
Last reset 1667:38:29 ago, cause: power up
Last software reset details not available
Error status: 0x00

I have plenty of steppers and hotends and all the stuff in my lab too. I can build a "mock printer" with this parts and the board which runs into the issue if needed. Just ask for it.

droftarts

@Chriss I asked @dc42 if he'd made any progress on this issue. He said:

My previous investigations suggested that the crashing was caused by a hardware error in the MCU when a DMA transfer finished and the SPI peripheral was disabled. So not a software bug but a hardware defect that I need to find a workaround for.
It's possible that the RRF 3.6 builds are not affected by this issue, so worth getting the affected users to try 3.6.0-alpha.4 at https://www.dropbox.com/scl/fo/cckwiq91gn16hvl1zdjnp/AF0SMEtkVfiArSPeYaBDGPY?rlkey=kqkknk9q1kiq684u4s55ce8d4&dl=0

Could you try the 3.6.0-alpha.4 build?

Ian

Chriss

@droftarts said in Duet 3 mini5+ nightly restarts 3.5.2:

Could you try the 3.6.0-alpha.4 build?

Your wish is my command, but not before mid of the week. I'm installing a new hardware feature at the moment. I will do the upgrade asap. (Any new traps you want to tell me?)

Cheers, Chriss

droftarts

@Chriss There's a few things not fully working, check the release notes: https://github.com/Duet3D/RepRapFirmware/wiki/Changelog-RRF-3.x-Beta

Ian

Chriss

@droftarts said in Duet 3 mini5+ nightly restarts 3.5.2:

@Chriss There's a few things not fully working, check the release notes: https://github.com/Duet3D/RepRapFirmware/wiki/Changelog-RRF-3.x-Beta

I just upgraded the printer, including the DWC, some error message popped up in the DWC that some versions are not compatible. Does the DWC version look OK to you?

Cheers, Chriss

jay_s_uk

@Chriss it's the mismatch between DWC (alpha2) and the main firmware (alpha4).
I'm not sure there has been an alpha 4 release of DWC.
You can ignore the error as there's nothing breaking between those version mismatches

Chriss

@droftarts I used the same object I always had a problem with and it finished the first time successfully.
I'm not sure how much this will tell us, a lucky punch maybe? Problem solved?
The only problem I see now is the quality:

I'm not sure where this came from. I do not see that on objects which are not as high as this object is. What shall I do now? Bring back the old version and be unable to print objects which this dimension? Or drop the bench board in and go back to the latest stable version and test again to see whether the quality is better than?

Cheers, Chriss

droftarts

@Chriss You'll need to retune Pressure Advance with the 3.6 release. It should work much better!

Ian

Chriss

@droftarts I saw that in the release notes. My PA dial in process came out with very much the same value as it was before.
I will perform the test again with a material I know better, this grey filament (ABS+) is new for me, the temperature may not be ideal for this material too. Maybe to much rush this time.
But I'm confused how the gab at the seams piled up over the layer hight, I never experienced that before.

Phaedrux

@Chriss said in Duet 3 mini5+ nightly restarts 3.5.2:

But I'm confused how the gab at the seams piled up over the layer hight, I never experienced that before.

That is common when there is under extrusion on unretract at layer change.

Chriss

This post is deleted!

Chriss

@Phaedrux

The top line is without any PA. I have the feeling that I do not need any PA at all. Can that be possible with the alpha?

Or I print to slow.

Cheers, Chriss

Chriss

That is the result with no PA at all.

oliof

@Chriss what are your retraction values?

Chriss

@oliof I have to admit that I do not know that anymore at this print. I shift here some profiles in SuSi back and forth. The current print with 5.8mm and 35 and 25mm/s look far better.
I know that 5.8mm are far to much for a direct dive, but I feel a bit helpless so I just tried the Voron 2.4 standard profile and some others.

oliof

@Chriss try 2mm @40mm/sec, if that's not better, give it a quarter to a half millimeter extra unretract.

Chriss

@oliof I do not why but the "avoid crossing perimeter = true" solved it so far with the standard profile. Only the first layer looks a bit strange now:
file:///home/chriss/Downloads/3.6.0-alpha.4/Pics/IMG_0612.JPG
I can not stop thinking that there is something strange ongoing with PA, which is off with my setup at the moment. The top side is overextruded at the touchpoints with the walls. That is more than strange with ABS+ and a EM of 0.88.