Connecting multiple duets to one SBC

chrishamm

@benecito That structure sounds reasonable to me. The service files come from /usr/share/systemd/system AFAIR.

benecito

@chrishamm quick update (and question)
We finally finished the test setup with actual printers connected. Unfortunately on pretty much any print one of the looses it's connection with a "Waiting for transfer ready pin" message.
Right now the wires are about 25cm long but well away from any other cables. We're trying to rearrange the setup to be able to use shorter cables but not sure if that's actually the issue.

Can it be somehow related to two SPI connections running at quite a bit of throughput on the same pi and thus some timing errors or so happening? In all cases the connection is back less than a second after.
Happy to hear any thoughts and maybe still get them running reliably!

benecito

@chrishamm Any thoughts on this?
We now tried to run the Duets over SBC but print over USB. This went well so far but haven't run any long term tests.
Therefore to me it sounds like a timing issue related to the transfer ready pin with the increased throughput of printing over SBC.
Can you confirm if the USB print does not go trough the SBC connection back and forth?

Still we would prefer to also print over SBC as we would have the whole ObjectModel available here.

chrishamm

@benecito Hard to say. Do you get any TfrRdy glitches in the M122 diagnostics? If yes, there is probably noise on the TfrRdy line causing the culprit. Try shielding the transfer cables and see if that makes a difference. If that doesn't work, consider trying out a different GPIO pin on the Pi side, perhaps even from a different gpiochip device if possible.

benecito

@chrishamm now I was able to reproduce this. It might be a little different this time as DWC showed as disconnected at first but then eventually came back to life.

That's the M122 diagnostics:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.4.5 (2022-11-30 19:41:16) running on Duet 3 Mini5plus WiFi (SBC mode)
Board ID: N5HL0-0P6KL-K65J0-409N2-M612Z-7NY58
Used output buffers: 1 of 40 (27 max)
=== RTOS ===
Static ram: 103652
Dynamic ram: 99256 of which 204 recycled
Never used RAM 34400, free system stack 118 words
Tasks: SBC(ready,13.3%,434) HEAT(notifyWait,0.5%,326) Move(notifyWait,2.1%,265) CanReceiv(notifyWait,0.0%,942) CanSender(notifyWait,0.0%,336) CanClock(delaying,0.2%,341) TMC(notifyWait,11.4%,106) MAIN(running,58.4%,557) IDLE(ready,0.1%,30) AIN(delaying,13.9%,263), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 23:39:07 ago, cause: power up
Last software reset at 2023-08-01 09:44, reason: User, GCodes spinning, available RAM 35960, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task SBC Freestk 0 n/a
Error status: 0x00
MCU revision 3, ADC conversions started 85147781, completed 85147780, timed out 0, errs 0
Step timer max interval 1490
MCU temperature: min 31.4, current 38.1, max 52.0
Supply voltage: min 23.4, current 23.8, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/8, heap memory allocated/used/recyclable 2048/106/0, gc cycles 0
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0, read errors 0, write errors 0, ifcnt 24, reads 24846, writes 24, timeouts 0, DMA errors 0, CC errors 0
Driver 1: standstill, SG min 0, read errors 0, write errors 0, ifcnt 24, reads 24846, writes 24, timeouts 0, DMA errors 0, CC errors 0
Driver 2: standstill, SG min 0, read errors 0, write errors 0, ifcnt 24, reads 24845, writes 24, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x6a
Driver 3: standstill, SG min 0, read errors 0, write errors 0, ifcnt 24, reads 24846, writes 24, timeouts 0, DMA errors 0, CC errors 0
Driver 4: standstill, SG min 0, read errors 1, write errors 0, ifcnt 24, reads 24845, writes 24, timeouts 0, DMA errors 0, CC errors 0
Driver 5: not present
Driver 6: not present
Date/time: 2023-08-09 08:35:44
Cache data hit count 4294967295
Slowest loop: 56.65ms; fastest: 0.09ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, segments created 56, maxWait 39571259ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 151579, completed 151579, hiccups 0, stepErrors 0, LaErrors 0, Underruns [3160, 0, 1], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet* is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty
=== CAN ===
Messages queued 766304, received 0, lost 0, boc 0
Longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 18 (min 18), ts 425738/0/0
Tx timeouts 0,0,425737,0,0,340565 last cancelled message type 4514 dest 127
=== SBC interface ===
Transfer state: 5, failed transfers: 0, checksum errors: 0
RX/TX seq numbers: 545/545
SPI underruns 0, overruns 0
State: 5, disconnects: 2, timeouts: 2 total, 2 by SBC, IAP RAM available 0x0f1bc
Buffer RX/TX: 0/0-0, open files: 0
=== Duet Control Server ===
Duet Control Server v3.4.5
Code buffer space: 4096
Configured SPI speed: 8000000Hz, TfrRdy pin glitches: 0
Full transfers per second: 38.41, max time between full transfers: 190.0ms, max pin wait times: 40.1ms/1.4ms
Codes per second: 2.74
Maximum length of RX/TX data transfers: 6836/368

And the end of the log file from the RepetierServer print should it be of any help:

Send:21:14:35.295: N160496 G1 X183.139 Y141.32 E14.18028
Send:21:14:35.295: N160497 G1 X183.746 Y141.494 E14.19944
Recv:21:14:35.338: ok
Send:21:14:35.338: N160498 G1 X184.369 Y141.6 E14.2186
Recv:21:14:35.607: ok
Send:21:14:35.607: N160499 G1 X185 Y141.636 E14.23776
Recv:21:14:35.693: ok
Send:21:14:35.693: N160500 G1 X185.631 Y141.6 E14.25692
Recv:21:14:35.968: ok
Send:21:14:35.968: N160501 G1 X186.254 Y141.494 E14.27608
Recv:21:14:36.003: ok
Send:21:14:36.003: N160502 G1 X186.861 Y141.32 E14.29524
Exec:21:14:38.589: @getip
Send:21:14:38.592: N160503 M117 192.168.187.49:3344
Recv:21:14:38.693: Lost connection to Duet (Timeout while waiting for transfer ready pin)
Recv:21:14:38.693: Connection to Duet established
Recv:21:14:38.696: start
Send:21:14:38.696: N1 M110
Exec:21:14:59.348: @call RHRead timer30 "null"
Mesg:21:15:09.696: Warning: Communication timeout - resetting communication buffer.
Mesg:21:15:09.696: This means that a expected firmware response was not seen within the expected time.
Mesg:21:15:09.696: The typical reason is a communication error and print should continue after the communication reset.
Mesg:21:15:09.696: Connection status: Buffered:8, Manual Commands: 30, Job Commands: 0
Mesg:21:15:09.696: Buffer used:8 Enforced free byte:17 lines stored:1
Offl:21:54:53.881: Ignored (offline):M119
Offl:21:54:53.881: Ignored (offline):M119
Exec:21:54:53.881: @syncAckState
Offl:21:54:53.881: Ignored (offline):M117 Finished
Offl:21:54:53.881: Ignored (offline):M105
Offl:21:54:53.881: Ignored (offline):M115
Offl:21:54:53.881: Ignored (offline):M114
Offl:21:54:53.881: Ignored (offline):T0

Since it has now been reproduced I'll look into some rewiring. I don't think it's noise though as the SBC wire bundle right now goes trough mid air to rule that out and we also tried shielded ethernet cables for that exact reason.

PrintMan

@benecito have you tried DuetPrintFarm? I'm waiting for their feature

benecito

@PrintMan what's DuetPrintFarm?

infiniteloop

@benecito said in Connecting multiple duets to one SBC:

what's DuetPrintFarm?

This: https://github.com/Duet3D/DuetPrintFarm

benecito

@infiniteloop Interesting. I'll take a look, but I think not what we are aiming for right now.

benecito

@chrishamm
Can you confirm that a USB print does not go trough the SBC connection back and forth?

chrishamm

@benecito I'm sorry, but I don't get what you mean by that.

benecito

@chrishamm no worries!

To narrow things down we originally wanted to go back to a usb connection but stopped in the middle.
The Duet was still connected via SBC (so no SD card in the duet) but also had a usb cable from the pi running RepetierServer (the external printing source) to the duet.
We noticed that this appears to work just fine and ran a few test prints.

So basically what we have now ist the duet connected via SBC to the pi and RepetierServer communicating with the duet via USB for status updates and printing (instead of the server being connected to the duet SBC socket on the pi). Obviously this would not make sense if the commands the duet receives over usb get sent back and forth trough the SBC connection, but as it has been going fine so far I assume they are not.

chrishamm

@benecito That's a pretty unusual setup but it should work. Certain commands are handled by the SBC (RRF sends them to the SBC for processing) but the majority of codes (G0/G1) are handled immediately by RRF.

benecito

@chrishamm Thanks! I know it's not the first option to chose, but at least it works for now. We'll still try to make an actual SBC setup work, but that's something for another day.