3.5.0-B4, having random CAN error timeouts

[dc42]

@engikeneer I've put a new build of the tool board firmware with more free RAM at https://www.dropbox.com/sh/c4aatfdsejjjab0/AACGuZ4qUBvZz49zxErkhmnWa?dl=0. Please try it to see if this out-of-memory issue is resolved.

[engikeneer]

@dc42 Thanks, that appears to have fixed it and the print completed fine and thge jerky motion I saw was gone. M122 and M122 B22 below if you're intersted.

The under extrusion remains but is much less severe. I got couple of Expansion board 22 stopped sending status (and board 21) during the print. It seems fine printing from T1 (board 21), so I am guessing that this is due to a CAN wiring issue or some interference on T0 (board 22) as that is the first board in the chain. I will make up some new CAN cables in the next week or so, but either way I am convinced it is not a Beta FW issue

M122

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-beta.4 (2023-06-08 23:40:14) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: QUS4M-U096U-D65J0-40KMD-3W03Z-ZK6H6
Used output buffers: 3 of 40 (40 max)
=== RTOS ===
Static ram: 102996
Dynamic ram: 123924 of which 216 recycled
Never used RAM 11152, free system stack 128 words
Tasks: NETWORK(2,nWait,49.4%,205) HEAT(3,nWait,0.1%,327) Move(4,nWait,8.4%,244) CanReceiv(6,nWait,0.3%,672) CanSender(5,nWait,0.1%,328) CanClock(7,delaying,0.0%,342) TMC(4,delaying,2.4%,74) MAIN(1,running,35.5%,698) IDLE(0,ready,0.8%,29) AIN(4,delaying,2.9%,266), total 100.0%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 04:35:03 ago, cause: software
Last software reset at 2023-07-01 09:00, reason: User, Gcodes spinning, available RAM 14832, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 16505513, completed 16505513, timed out 0, errs 0
MCU temperature: min 29.2, current 34.0, max 38.9
Supply voltage: min 24.4, current 24.6, max 24.7, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/312/312, gc cycles 0
Events: 2 queued, 2 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 1, ifcnt 57, reads 15760, writes 42, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 1, ifcnt 57, reads 15759, writes 42, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 1, ifcnt 33, reads 15786, writes 15, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 0, write errors 1, ifcnt 34, reads 15783, writes 18, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 1, ifcnt 36, reads 15780, writes 21, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-07-01 13:35:54
Cache data hit count 4294967295
Slowest loop: 999.94ms; fastest: 0.10ms
=== Storage ===
Free file entries: 20
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 10.3ms, write time 8.0ms, max retries 0
=== Move ===
DMs created 83, segments created 38, maxWait 218705ms, bed compensation in use: none, height map offset 0.000, ebfmin 0.00, ebfmax 0.00
no step interrupt scheduled
=== DDARing 0 ===
Scheduled moves 213781, completed 213781, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 3, 1], CDDA state -1
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0
Queue2 is idle in state(s) 0
Q0 segments left 0, axes/extruders owned 0x0000000
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== CAN ===
Messages queued 377421, received 659870, lost 0, boc 578
Longest wait 480ms for reply type 6029, peak Tx sync delay 47658, free buffers 18 (min 3), ts 82520/82158/0
Tx timeouts 63,0,89,0,0,53 last cancelled message type 30 dest 127
=== Network ===
Slowest loop: 215.44ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
= WiFi =
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
Firmware version 2.1beta4
MAC address f0:08:d1:02:ed:e4
Module reset reason: Power up, Vcc 3.36, flash size 2097152, free heap 39628
WiFi IP address 192.168.1.168
Signal strength -53dBm, channel 11, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

M122 B22

Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-beta.4+ (2023-06-30 15:40:06)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 2760, free system stack 88 words
Tasks: Move(3,nWait,1.5%,71) HEAT(2,nWait,1.7%,37) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.4%,75) CanClock(5,nWait,0.1%,66) ACCEL(3,nWait,0.0%,53) TMC(2,delaying,10.4%,57) MAIN(1,running,69.4%,322) IDLE(0,ready,0.0%,27) AIN(2,delaying,16.7%,114), total 100.0%
Last reset 04:34:51 ago, cause: software
Last software reset at 2023-06-27 09:59, reason: OutOfMemory, available RAM 4, slot 1
Software reset code 0x01c0 ICSR 0x00000000 SP 0x200054a8 Task Move Freestk 137 ok
Stack: 200057e0 000062df 200057f8 ae3a643e 36f08716 00009f99 200057f8 00007dd5 0001e926 3f62a79c 20005108 20007538 3d926a20 3d926a20 2dfcf290 200074c0 200074f4 00000000 470550b8 2ef478c0 2ef478c0 2000554c 20007538 20005108 00000000 00008c91 00000000
Driver 0: pos 0, 405.0 steps/mm, standstill, SG min 0, read errors 1, write errors 0, ifcnt 39, reads 28857, writes 2, timeouts 3, DMA errors 0, CC errors 0, failedOp 0x6f, steps req 0 done 11981257
Moves scheduled 220948, completed 220948, in progress 0, hiccups 2525, segs 51, step errors 0, maxPrep 1497, maxOverdue 764587, maxInc 239236, mcErrs 0, gcmErrs 0, ebfmin -1.00 max 1.00
Peak sync jitter -10/29, peak Rx sync delay 393, resyncs 3/0, no timer interrupt scheduled
VIN voltage: min 24.2, current 24.5, max 24.5
MCU temperature: min 32.4C, current 39.1C, max 51.3C
Last sensors broadcast 0x00000002 found 1 8 ticks ago, 0 ordering errs, loop time 1
CAN messages queued 325377, send timeouts 6, received 436534, lost 0, free buffers 18, min 17, error reg ff0000
Last cancelled message type 4514 dest 127
dup 6, oos 712/200/0/149, bm 0, wbm 0, rxMotionDelay 584, adv -763616/100219
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 0, other errors 0

[dc42]

@engikeneer thanks for confirming that the changes I made were sufficient for your print and machine configuration.

[Nuramori]

@dc42 I just had another out of memory occur related to the tool board - I have this patch applied (actually a more recent one I believe), and it ran out again.

[gloomyandy]

@Nuramori Please post the output for M122 B22 so we can check the build you have installed and the reset reason.