Duet 6HC randomly crasing (again)

[Phaedrux]

Can you try this test firmware?

https://www.dropbox.com/scl/fo/9hum0skl7rew21go56q7z/h?rlkey=mh1wx7mi72oz0dmk86c7uf4un&dl=0

[agmeadows]

Here is the config.g

;Initialize variables
if !exists(global.materialtype)
	global materialtype = ""
if !exists(global.materialprint)
	global materialprint = ""
if !exists(global.materialstandby)
	global materialstandby = ""
if !exists(global.materialbed)
	global materialbed = ""
if !exists(global.machineheater)
	global machineheater = 0
if !exists(global.inFilamentError)	
	global inFilamentError = false

;General Setup - RailCore RRF Duet3
M552 P0.0.0.0 S1
G21								;Work in millimetres
G90								;Send absolute coordinates...
M83								;...but relative extruder moves

; Stepper configuration
M569 P0 S0 D3					;Drive 0 X / Rear
M569 P1 S1 D3					;Drive 1 Y / Front
M569 P2 S1 D3					;Drive 2 Z Front Left
M569 P3 S1 D3					;Drive 3 Z Rear Left
M569 P4 S1 D3					;Drive 4 Z Right
M569 P5 S1 D3					;Drive 5 Extruder

;Axis configuration
M669 K1							;CoreXY mode
M584 X0 Y1 Z2:3:4 E5			;Map X to drive 0 Y to drive 1, Z to drives 2, 3, 4, and E to drive 5

;Leadscrew locations
M671 X-13.6:-13.3:340 Y22:278:149 S7.5

M350 X16 Y16 Z16 E16 I1				;Set 16x microstepping for axes with interpolation
M906 X1500 Y1500 Z1200 E1200 I10	;Set motor currents (mA)

; Fully commissioned speeds.
M201 X3000 Y8000 Z100 E1500       ; Accelerations (mm/s^2)
M203 X24000 Y24000 Z800 E3600     ; Maximum speeds (mm/min)
M566 X1000 Y1000 Z100 E600 P1     ; Maximum jerk speeds mm/minute

M208 X290 Y290 Z275				;Set axis maxima and high homing switch positions (adjust to suit your machine)
M208 X0 Y0 Z0 S1				;Set axis minima and low homing switch positions (adjust to make X=0 and Y=0 the edges of the bed)
M92 X200 Y200 Z1600 E696		;Orbiter
;M92 X200 Y201 Z1600 E835    	;Bowden

;Mesh leveling parameters
M557 X1:300 Y38:290 P21:21

; End Stops
M574 X1 S1 P"io1.in"				;Map the X endstop to io1.in
M574 Y1 S1 P"io2.in"				;May the Y endstop to io2.in

; Filament Sensor 
M591 D0 P5 C"io5.in" R60:215 E3.0 S0 A1 L0.406 ;Duet3D laser sensor for extruder drive 0, connected to IO 5, tolerance 60 to 215%, 3mm comparison length, disabled
 
; Thermistors
M308 S2 P"temp3" Y"thermistor" A"chamber_temp" T100000 B3950 H0 L0			;Chamber thermistor
M308 S0 P"temp2" Y"thermistor" A"plate_heat" T100000 B3960 H0 L0			;Plate thermistor	
M308 S1 P"temp1" Y"thermistor" A"e0_heat" T100000 B4725 C7.06e-8 H0 L0		;E3D thermistor

;Define Heaters
M950 H0 C"out0" T0				;Bed heater is on out0
M950 H1 C"out1" T1				;Hotend heater is on out1
M950 H2 C"out2" T2				;Chamber heater is on out2

;Define Bed
M140 H0

;Define Chamber
M141 H2

M307 H0 R0.394 K0.158:0.000 D13.55 E1.35 S1.00 B0		;Bed Duet 3 configuration
M307 H1 R2.812 K0.374:0.000 D4.13 E1.35 S1.00 B0 V23.8  ;Phaetus Hotend Duet 3 configuration
M307 H2 R0.048 K0.134:0.000 D16.81 E1.35 S1.00 B0		;Chamber calibration
;M307 H2 R0.425 K0.744:0.000 D14.76 E1.35 S1.00 B0		
;old hotend value M307 H1 A310.1 C144.7 D2.6 S1.00 B0 		
;old bed value M307 H0 A90.0 C700.0 D10.0 S1.00 B0

M570 H0 T15							;Set hot end fault behaviour. 15C excursion before error
M143 H1 S300						;Set max hotend temperature	
M143 H0 S150						;Set max bed temp

; Fans
M950 F0 C"out6"	Q100				;Layer fan on  "out6" connector
M106 P0 S0 H-1 C"Part Fan"			;Layer fan						
M950 F1 C"out8"                     ;Hotend fan on "out8" connector without PWM
M106 P1 C"Hotend Fan" 	            ;M106 P1 S255 H1 T50 "Hotend Fan"	;enable thermostatic mode for hotend fan
M950 F2 C"out7" Q2500				;Heater cooling fan on  "out7" connector
M106 P2 S0 C"Cooling Fan"			;Heater cooling fan
M950 F3 C"out4"     		    	;Extruder fan on "out4" connector	
M106 P3 T40 H2 C"Extruder Fan"		;Extruder fan

; Accelerometer
M955 P0 C"spi.cs3+spi.cs2" ; all wires connected to temp DB connector

; Input Shaping
M593 P"mzv" F70

; Tool definitions
M563 P0 D0 H1 F0	                        ;Define tool 0 uses extruder 1, heater 1 and fan 1
G10 P0 S0 R0                        		;Set tool 0 operating and standby temperatures

; Pressure Advance
; M572 D0 S0.01 ; set extruder 0 pressure advance to 0.01 seconds

; Euclid
M558 P5 C"^io7.in" H3 F500:100 T24000 A6 S0.02

;Tip: A larger trigger height in G31 moves you CLOSER to the bed
G31 X1 Y37.5 Z3.20 P25 			;Nozzle-X set the probe offsets
;G31 X0 Y40 Z2.8 P25 			;0.25mm nozzle set the probe offsets

T0						;Select first hot end

As for the printer, it is a Railcore II 300ZL corexy. The hotend is a Phaetus Dragon with an Orbiter II extruder. LDO steppers all around with a 750w heat pad on a magnetic bed kinematically mounted. The Z probe is a Euclid.

Another thing I did today was verify the bed has ground to the frame. The firmware looks like it is for the Mini+. I have a 6HC will that firmware work on the 6HC as well?

Its so strange because everything worked fine for months after the initial problem of random reboots. Absolutely nothing has changed except the weather.

[droftarts]

@agmeadows said in Duet 6HC randomly crasing (again):

Absolutely nothing has changed except the weather.

That's actually significant. Humidity has a big effect on electrostatic buildup. I don't have any greater insights into how to further ground your machine, unfortunately.

Ian

[agmeadows]

@droftarts The long road to troubleshooting an intermittent problem is a long one. I am wondering if my chamber temp thermistor is collecting static. It is unshielded and very close to the hotend. I am not even sure if that is possible.

I have moved it away from the hotend for now to see if that helps.

[dc42]

@agmeadows this is just to let you know that I am tracking this and other reports of crashes on 6HC and Duet 3 Mini when running firmware 3.5.0-rc.1. The reports you posted may indicate a firmware problem. You could try reverting to 3.4.6 to see if the resets stop.

[agmeadows]

@dc42 Thank you for the heads up. I will continue to post crash logs as I receive them. I suspect mine is related to a hardware issue and static. I was on 3.4.5 and it was crashing with the same type of error.

Initial testing after moving the chamber thermistor is promising. I was able to do 2 successful prints that were 3 hours long.

Is there a way to ground the thermistors on the Duet side in order to protect the board? I'd rather wire something in if it all possible to eliminate that as an issue. the hotend thermistor has a metal enclosure and is grounded by the hotend. The other thermistor is bare with just the two leads going to it. I am not sure if it is acting as an antenna for static when it is near parts interacting with filament or what. I am really just shooting in the dark as this point.

[dc42]

@agmeadows to avoid static discharge to the thermistor input, it's only necessary to ensure that any parts close to the thermistor that might accumulate charge are connected to Duet ground one way or another. If the metal parts of the machine are connected to mains ground then it should only be necessary to connect the Duet VIN ground terminal to mains ground also.

If there is any risk of creating a ground loop, then instead of a direct connection to ground, connecting metal parts to Duet ground via a 100K resistor is sufficient to prevent static charge building up.

If no other source of Duet ground is available close to the thermistor, then a 100K resistor between the metal parts of the machine and the VSSA wire of the thermistor may be used instead. Be sure to use the thermistor wire that is connected to the VSSA pin on the TEMPx connector, because if you connect to the other one then the temperature reading could be affected.

[agmeadows]

Well, another crash. This is after 2 or 3 multi-hour prints finish successfully. The most recent print went for a little over an hour and then crashed. I am nearing my wits end and not sure what to do.

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.5.0-rc.1 (2023-08-31 16:19:24) running on Duet 3 MB6HC v1.01 (standalone mode)
Board ID: 08DJM-956BA-NA3TJ-6JTD8-3SJ6J-1UA2U
Used output buffers: 1 of 40 (36 max)
Error in macro line 97 while starting up: Accelerometer not found on specified port
=== RTOS ===
Static ram: 154852
Dynamic ram: 121320 of which 664 recycled
Never used RAM 66300, free system stack 210 words
Tasks: NETWORK(1,ready,39.0%,182) ETHERNET(5,nWait,0.1%,317) HEAT(3,nWait,0.0%,325) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,941) CanSender(5,nWait,0.0%,335) CanClock(7,delaying,0.0%,344) TMC(4,nWait,8.3%,59) MAIN(1,running,52.6%,137) IDLE(0,ready,0.0%,30), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:19:41 ago, cause: software
Last software reset at 2023-12-07 18:35, reason: MemoryProtectionFault mmarValid daccViol, Gcodes spinning, available RAM 65092, slot 0
Software reset code 0x4163 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x00400004 BFAR 0x0000198d SP 0x2045ff48 Task Move Freestk 64367 ok
Stack: 00001989 83b6454d 00001a88 00001989 00000000 0044684d 0044ab5c 61000027 00446809 4000c000 00000000 2042194c 204322b8 00000000 0044aa39 20459300 204322a4 00000007 00000000 2045f840 ffffffed 00000000 00f00000 e000e000 c0000000 00000000 00497f6b
Error status: 0x00
MCU temperature: min 29.1, current 29.2, max 32.6
Supply voltage: min 23.8, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0
Heap OK, handles allocated/used 99/8, heap memory allocated/used/recyclable 2048/156/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min n/a, mspos 904, reads 41806, writes 16 timeouts 0
Driver 1: standstill, SG min n/a, mspos 56, reads 41806, writes 16 timeouts 0
Driver 2: standstill, SG min n/a, mspos 664, reads 41806, writes 16 timeouts 0
Driver 3: standstill, SG min n/a, mspos 296, reads 41806, writes 16 timeouts 0
Driver 4: standstill, SG min n/a, mspos 504, reads 41807, writes 16 timeouts 0
Driver 5: standstill, SG min n/a, mspos 600, reads 41807, writes 16 timeouts 0
Date/time: 2023-12-07 18:55:04
Slowest loop: 6.27ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 4.2ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters 2 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x80000003
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
Extruder 0: no data received
=== CAN ===
Messages queued 10629, received 0, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 50), ts 5906/0/0
Tx timeouts 0,0,5905,0,0,4722 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 34.90ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 2 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

I unplugged the filaments sensor to see if that was causing issues. It still crashed. I have now removed everything not essential to printing.

[agmeadows]

Yet another crash

=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.5.0-rc.1 (2023-08-31 16:19:24) running on Duet 3 MB6HC v1.01 (standalone mode)
Board ID: 08DJM-956BA-NA3TJ-6JTD8-3SJ6J-1UA2U
Used output buffers: 1 of 40 (18 max)
Error in macro line 97 while starting up: Accelerometer not found on specified port
=== RTOS ===
Static ram: 154852
Dynamic ram: 120916 of which 240 recycled
Never used RAM 67128, free system stack 210 words
Tasks: NETWORK(1,ready,38.8%,157) ETHERNET(5,nWait,0.1%,321) HEAT(3,nWait,0.0%,325) Move(4,nWait,0.0%,340) CanReceiv(6,nWait,0.0%,941) CanSender(5,nWait,0.0%,335) CanClock(7,delaying,0.0%,344) TMC(4,nWait,8.4%,59) MAIN(1,running,52.6%,137) IDLE(0,ready,0.1%,30), total 100.0%
Owned mutexes:
=== Platform ===
Last reset 00:01:20 ago, cause: software
Last software reset at 2023-12-22 23:00, reason: MemoryProtectionFault mmarValid daccViol, Gcodes spinning, available RAM 65648, slot 0
Software reset code 0x4163 HFSR 0x00000000 CFSR 0x00000082 ICSR 0x00427804 BFAR 0x00000001 SP 0x2041bc20 Task MAIN Freestk 2212 ok
Stack: 20419934 20419934 00000001 00000000 00000000 00498559 00497de8 610f0000 004984df 20430030 ffffffff 20430140 2043019c 20419934 2042e0dc ffffffff 00000000 204309d8 0049866b 2041bccc 00000003 2042bb28 00495b6f 2042e0d8 0046db83 2042bb28 2042e0dc
Error status: 0x00
MCU temperature: min 31.8, current 32.1, max 32.3
Supply voltage: min 23.8, current 23.8, max 23.9, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.2, under voltage events: 0
Heap OK, handles allocated/used 99/8, heap memory allocated/used/recyclable 2048/156/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min n/a, mspos 872, reads 48401, writes 16 timeouts 0
Driver 1: standstill, SG min n/a, mspos 72, reads 48401, writes 16 timeouts 0
Driver 2: standstill, SG min n/a, mspos 792, reads 48401, writes 16 timeouts 0
Driver 3: standstill, SG min n/a, mspos 856, reads 48401, writes 16 timeouts 0
Driver 4: standstill, SG min n/a, mspos 600, reads 48401, writes 16 timeouts 0
Driver 5: standstill, SG min n/a, mspos 888, reads 48401, writes 16 timeouts 0
Date/time: 2023-12-22 23:01:31
Slowest loop: 5.98ms; fastest: 0.07ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 25.0MBytes/sec
SD card longest read time 3.9ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, segments created 0, maxWait 0ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
Moves shaped first try 0, on retry 0, too short 0, wrong shape 0, maybepossible 0
=== DDARing 0 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamber heaters 2 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x80000003
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
Extruder 0: no data received
=== CAN ===
Messages queued 720, received 0, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 50 (min 50), ts 401/0/0
Tx timeouts 0,0,400,0,0,318 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 8.77ms; fastest: 0.03ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
= Ethernet =
Interface state: active
Error counts: 0 0 0 1 0 0
Socket states: 5 2 2 2 2 0 0 0
=== Multicast handler ===
Responder is inactive, messages received 0, responses 0

I am tempted to ditch the Duet 6HC at this point. It is crashing constantly and I am wasting filament. I took the entire carriage apart and reassembled it being sure to check everything over. I have verified ground on the hotend and disconnected the chamber thermistor in case that was causing issues.

What else can I do short of replacing the entire board?

[shaunppl]

@agmeadows have you double checked the power supply feed and or the power supply output, where is the printer situated ?.
power spikes are another source of electrical interference that could be effecting the boards as a fluctuating input or ground can trip the boards to reset.

[Phaedrux]

3.5 final has released now, there has been a lot of fixes since rc1. Can you please update using the full release zip file?

https://github.com/Duet3D/RepRapFirmware/releases/tag/3.5.0