Error: Out of CAN buffers

[benecito]

On a machine using 7 1HCL boards I frequently get the below message during printing.

Any ideas what the source might be?

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.5 (2022-11-30 19:41:16) running on Duet 3 Mini5plus WiFi (SBC mode)
Board ID: ZS64B-PQ6KL-K65J0-409ND-21W1Z-RXX97
Used output buffers: 1 of 40 (39 max)
=== RTOS ===
Static ram: 103652
Dynamic ram: 101048 of which 0 recycled
Never used RAM 32836, free system stack 112 words
Tasks: SBC(resourceWait:,11.3%,459) HEAT(notifyWait,0.3%,320) Move(notifyWait,9.0%,183) CanReceiv(notifyWait,0.6%,772) CanSender(notifyWait,0.6%,326) CanClock(delaying,0.0%,339) TMC(notifyWait,3.7%,72) MAIN(running,70.1%,557) IDLE(ready,0.1%,30) AIN(delaying,4.4%,263), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 07:50:08 ago, cause: power up
Last software reset at 2023-08-11 15:11, reason: User, GCodes spinning, available RAM 33620, slot 1
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
MCU revision 3, ADC conversions started 28210249, completed 28210248, timed out 0, errs 0
Step timer max interval 1510
MCU temperature: min 22.9, current 43.9, max 43.9
Supply voltage: min 22.8, current 23.7, max 24.0, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/14, heap memory allocated/used/recyclable 2048/186/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: ok, SG min 0, read errors 0, write errors 0, ifcnt 17, reads 41805, writes 17, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 11, reads 41811, writes 11, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 41812, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 3: standstill, SG min 0, read errors 2, write errors 0, ifcnt 9, reads 41810, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 0, write errors 0, ifcnt 9, reads 41813, writes 9, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-08-17 15:46:17
Cache data hit count 4294967295
Slowest loop: 332.06ms; fastest: 0.10ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 55, maxWait 4442814ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 531743, completed 531736, hiccups 0, stepErrors 0, LaErrors 0, Underruns [7759, 0, 3], CDDA state 3
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.1
Heater 1 is on, I-accum = 0.4
=== GCodes ===
Segments left: 1
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet* is doing "G1 F7200" in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 2308026, received 1919412, lost 0, boc 0
Longest wait 8ms for reply type 6029, peak Tx sync delay 626, free buffers 18 (min 0), ts 141043/141042/0
Tx timeouts 0,0,0,0,0,0
=== SBC interface ===
Transfer state: 5, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 60674/60674
SPI underruns 0, overruns 0
State: 5, disconnects: 0, timeouts: 0 total, 0 by SBC, IAP RAM available 0x0f194
Buffer RX/TX: 892/1964-0, open files: 0
=== Duet Control Server ===
Duet Control Server v3.4.5
Telnet:
Buffered code: G1 F7200
Buffered code: G1 X272.364 Y222.364 E18059.75868
Buffered code: G1 X269.245 Y225.992 F15000
Buffered code: G1 F7200
Buffered code: G1 X314.012 Y270.759 E18062.04415
Buffered code: G1 X309.861 Y273.355 F15000
Buffered code: G1 F7200
Buffered code: G1 X266.647 Y230.141 E18064.25031
Buffered code: G1 Z52.85 F240
Buffered code: G1 X266.443 Y230.035 F15000
Buffered code: G1 F7200
Buffered code: G1 X266.532 Y229.863 E18064.25623
Buffered code: G1 X266.801 Y229.369 E18064.27348
Buffered code: G1 X267.078 Y228.879 E18064.29073
Buffered code: G1 X267.363 Y228.394 E18064.30797
Buffered code: G1 X267.657 Y227.914 E18064.32523
Buffered code: G1 X267.96 Y227.44 E18064.34247
Buffered code: G1 X268.27 Y226.971 E18064.35972
Buffered code: G1 X268.589 Y226.507 E18064.37697
Buffered code: G1 X268.916 Y226.049 E18064.39421
Buffered code: G1 X269.251 Y225.596 E18064.41146
Buffered code: G1 X269.593 Y225.15 E18064.42871
Buffered code: G1 X269.943 Y224.71 E18064.44596
Buffered code: G1 X270.301 Y224.275 E18064.46321
==> 1080 bytes
Code buffer space: 2132
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 1
Full transfers per second: 16.14, max time between full transfers: 81.2ms, max pin wait times: 142.6ms/19.6ms
Codes per second: 8.09
Maximum length of RX/TX data transfers: 8184/1352

[jay_s_uk]

@benecito 7 x 1HCL boards might be pushing the mini5+ beyond its limits. I know dc42 said 4 toolboards would be ok

[dc42]

@benecito said in Error: Out of CAN buffers:

=== CAN ===
Messages queued 2308026, received 1919412, lost 0, boc 0
Longest wait 8ms for reply type 6029, peak Tx sync delay 626, free buffers 18 (min 0), ts 141043/141042/0

From the above it does indeed appear that the Duet 3 Mini is running out of CAN buffers. I can increase the number available in RRF 3.5.0-rc.1. We didn't envisage that more than 4 expansion boards would be connected via CAN to a Duet 3 Mini.

[benecito]

@jay_s_uk @dc42 can you confirm this? Also can anyone advise if it would be better if we i.e. used a 6HC board?

[dc42]

@benecito a 6HC board (or 6XD board if you don't need to drive any stepper motors directly) has more than double the processing power of the Duet 3 Mini, so it might be a better solution.

I can provide you with an interim Duet 3 Mini build with the increased limits if you wish to do some performance tests.

[benecito]

@dc42 then we'll switch to the 6HC as we need the drivers for the extruder and see how it goes there.

[dc42]

@benecito ok. My offer of a preview build of the Duet 3 Mini firmware still stands. The current firmware supports up to 7 CAN-connected drivers but the assumption we made was that they would be across up to 4 CAN-connected boards. I have now increased the limit to 8 CAN-connected drivers across 8 expansion boards.