Duet 3HC Expansion looses Connection

developeralgo222

Got them powered up and syncing and then they started rapid flashing and lost synchronization again. so i unplugged all the extension boards and again started adding one by one, to see if any board or cable is the problem. I made all the connection cables and tested them for continuity myself to make sure they are ok .

As per documentation , Is the 2 CAN Bus Termination Jumpers(resistors) required only on the Last Expansion Board in the chain ? In my Case 6XD --> 3HC(1) --> 3HC2(2) --> 3HC(3 -- Has 2 CAN Bus jumpers)

This is getting to be a frustrating episode

droftarts

@developeralgo222 said in Duet 3HC Expansion looses Connection:

As per documentation , Is the 2 CAN Bus Termination Jumpers(resistors) required only on the Last Expansion Board in the chain ?

Yes, that should be correct. You should have termination at both ends of the bus.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

As per documentation , Is the 2 CAN Bus Termination Jumpers(resistors) required only on the Last Expansion Board in the chain ?

Yes, that should be correct. You should have termination at both ends of the bus.

Ian

i assume CAN Bus Termination Jumpers only on 3HC(3) but not on 6XD board ( i don't see any CAN Bus termination jumpers on 6XD ) in my case

droftarts

@developeralgo222 Yes, the 6XD should have CAN termination resistors built-in. See https://docs.duet3d.com/Duet3D_hardware/Duet_3_family/Duet_3_Mainboard_6XD_Hardware_Overview#can-fd-bus-expansion

There is a termination resistor fitted on each CAN bus, so normally this board must be at the end of the bus. There are drill-to-disconnect jumpers on the underside of the board that allow the termination resistor to be removed, however this is not required in normal operation.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 Yes, the 6XD should have CAN termination resistors built-in. See https://docs.duet3d.com/Duet3D_hardware/Duet_3_family/Duet_3_Mainboard_6XD_Hardware_Overview#can-fd-bus-expansion

There is a termination resistor fitted on each CAN bus, so normally this board must be at the end of the bus. There are drill-to-disconnect jumpers on the underside of the board that allow the termination resistor to be removed, however this is not required in normal operation.

Ian

Ok that's good. so it seems when i disconnect board 3 ( 3HC(3) ) still Boards (6XD, 3HC2(1) & 3HC(2) ) syncs even without a jumper on 3HC(2) but as soon as i connect board 3 (3HC(3) which has the jumpers it stops syncing.

A little troubleshooting :

6XD --> 3HC(1) ==> Syncs
6XD --> 3HC(1) ---> 3HC(2) with CAN Termination Jumpers) ==> Syncs
6XD --> 3HC(1) ---> 3HC(3) with CAN Termination Jumpers) ==> Syncs

Looses synchronization when i connect all 3 expansion boards ( 3HC(1) , 3HC(2) & 3HC(3) )

e.g 6XD --> 3HC(1) --> 3HC(2) with CAN Termination Jumpers) ==> Syncs



1/13/2024, 7:49:18 PM	M122 B2

1/13/2024, 7:49:18 PM	Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 184 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.1%,419) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:23:04 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21135, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21136, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21137, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -25/11, peak Rx sync delay 181, resyncs 9/0, no timer interrupt scheduled
VIN voltage: min 24.5, current 24.5, max 24.6
V12 voltage: min 12.3, current 12.4, max 12.4
MCU temperature: min 27.6C, current 36.0C, max 36.4C
Last sensors broadcast 0x00000000 found 0 203 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 7832, send timeouts 462, received 4244, lost 0, errs 4637716, boc 310, free buffers 38, min 38, error reg ff5f00
Last cancelled message type 4519 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0



1/13/2024, 7:49:11 PM	M122 B1

Diagnostics for board 1:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.0%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:22:57 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60248, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60248, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60249, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -17/13, peak Rx sync delay 183, resyncs 5/0, no timer interrupt scheduled
VIN voltage: min 24.4, current 24.4, max 24.5
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 24.6C, current 32.3C, max 32.6C
Last sensors broadcast 0x00000000 found 0 163 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 10108, send timeouts 133, received 6112, lost 0, errs 14212604, boc 0, free buffers 38, min 38, error reg ff6500
Last cancelled message type 4519 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

Not sure what might be the issue. I don't think its the cables anymore, it has to be the Duet 3 3HC boards or the RRF firmware ?

dc42

@developeralgo222 are you quite certain that you have identified the CAN termination jumper positions correctly? The CAN bus is quite tolerant and will often work with a termination resistor missing or with an extra termination resistor, but probably not with two extra resistors.

Other things to check:

Do the contacts in the CAN sockets of board 2 look OK? Does the soldering of those sockets on the underside of the PCB look OK?
Are all 3 boards set to different CAN addresses?
Does 6XD - board 2 - board 3 work?

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 are you quite certain that you have identified the CAN termination jumper positions correctly? The CAN bus is quite tolerant and will often work with a termination resistor missing or with an extra termination resistor, but probably not with two extra resistors.

Other things to check:

Do the contacts in the CAN sockets of board 2 look OK? Does the soldering of those sockets on the underside of the PCB look OK?

Are all 3 boards set to different CAN addresses?

Does 6XD - board 2 - board 3 work?

(1) Contacts in the CAN sockets of board 1,2 & 3 look ok . Yes contacts on board 2 look ok, The solder is good and has good contact and spread on the pin under side
(2) All three boards are set to CAN ID 1, 2 & 3 respectively
(3) Does 6XD - board 2 - board 3 work? NO - it does not sync . ( in this case only Board 3 has the Jumpers )

Sync tests : -- this is really strange

6XD --> 3HC(1 ) ==> Syncs without the CAN termination jumpers on 3HC1
6XD --> 3HC(2 ) ==> NO Sync W/O the CAN termination jumpers on 3HC2
6XD --> 3HC(3 ) ==> NO Sync W/O the CAN termination jumpers on 3HC3
6XD --> 3HC(1) ---> 3HC(2) ==> Syncs ---- No CAN termination jumpers on both 3HC1 & 3HC2
6XD --> 3HC(1) ---> 3HC(3) ==> Syncs ---- No CAN termination jumpers on 3HC1 but with CAN Termination Jumpers on 3HC2
6XD --> 3HC(2) ---> 3HC(3) ==> NO Sync -- No CAN termination jumpers on 3HC2 but CAN Termination Jumpers on 3HC3

NOTE: The only difference between 3HC1 , 3HC2 & 3HC3 is that 3HC2 have 3 x SMC Vacuum Pressure Sensors PWR & GND connected to same VIN & GND on the 3HC2 board & 3HC3 have another 3 x SMC Vacuum Pressure Sensors PWR & GND connected to same VIN & GND on the 3HC3 board as per @dc42 earlier suggestions as per the picture below. Do you think this might be the issue ?

dc42

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

i have tested both sockets on board 2

6XD --> Board 2 CAN_IN and Board 2 CAN_OUT --> Board 3 CAN_IN -- No Sync
6XD --> Board 2 CAN_OUT and Board 2 CAN_IN --> Board 3 CAN_IN -- No sync
6XD --> Board 2 CAN_OUT and Board 2 CAN_IN ---> Board 3 CAN_OUT -- No sync

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

i have tested both sockets on board 2

6XD --> Board 2 CAN_IN and Board 2 CAN_OUT --> Board 3 CAN_IN -- No Sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN --> Board 3 CAN_IN -- No sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN ---> Board 3 CAN_OUT -- No sync

i am not sure what might be the issue with CAN Bus connections on the Duet 3 Boards , i believe @dc42 is the designer or one of the designers of this boards if i am not wrong, please can you give some insights as to why 2 x 3HC are not syncing at all. i bought another 3HC(4) board 4 today and even that doesn't sync but with that it has RRF 3.4.1 so i need to update it to 3.5_rc2 and then test. But i am really getting frustrated with CAN Bus connections on this Duet 3 Boards. This are very simple straight forward CAN Bus connections and i should not be encountering such issues.

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

i have tested both sockets on board 2

6XD --> Board 2 CAN_IN and Board 2 CAN_OUT --> Board 3 CAN_IN -- No Sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN --> Board 3 CAN_IN -- No sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN ---> Board 3 CAN_OUT -- No sync

i am not sure what might be the issue with CAN Bus connections on the Duet 3 Boards , i believe @dc42 is the designer or one of the designers of this boards if i am not wrong, please can you give some insights as to why 2 x 3HC are not syncing at all. i bought another 3HC(4) board 4 today and even that doesn't sync but with that it has RRF 3.4.1 so i need to update it to 3.5_rc2 and then test. But i am really getting frustrated with CAN Bus connections on this Duet 3 Boards. This are very simple straight forward CAN Bus connections and i should not be encountering such issues.

This is the nightmare i am dealing with, disconnecting and reconnecting every 1 sec @Phaedrux @dc42 @droftarts

Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 4 reconnected

dc42

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

i have tested both sockets on board 2

6XD --> Board 2 CAN_IN and Board 2 CAN_OUT --> Board 3 CAN_IN -- No Sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN --> Board 3 CAN_IN -- No sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN ---> Board 3 CAN_OUT -- No sync

That's not what I asked you to test.

droftarts

@developeralgo222 to me, that looks like the boards are continually resetting, or one of them in the bus is, and affecting others. And that makes me think there is a short on the board, probably caused by something plugged into it.

Did you test the setup before installing in the machine and connecting everything? I think you’re going to need to disconnect everything from the boards, unmount them from the machine (in case the way you have mounted them is causing a short) and bench test with just CAN and power connected to either isolate the behaviour of the CAN error, or if it works correctly, to start looking elsewhere for what is causing the boards to possibly be resetting.

And yes, all boards need to be on the same firmware version.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 to me, that looks like the boards are continually resetting, or one of them in the bus is, and affecting others. And that makes me think there is a short on the board, probably caused by something plugged into it.

Did you test the setup before installing in the machine and connecting everything? I think you’re going to need to disconnect everything from the boards, unmount them from the machine (in case the way you have mounted them is causing a short) and bench test with just CAN and power connected to either isolate the behaviour of the CAN error, or if it works correctly, to start looking elsewhere for what is causing the boards to possibly be resetting.

And yes, all boards need to be on the same firmware version.

Ian

I disconnected everything from the Expansion boards. The only thing connected is power and CAN Bus cables. I also do design my own boards too but never used Duet 3 boards for any project of mine. This is the first time. Pretty sure about taking care to avoid shorts. Never imagined i would have such a hard time having Duet 3 Board working

oliof

@developeralgo222 and just to be sure, termination only happens on the last 3HC in the chain?

developeralgo222

@oliof said in Duet 3HC Expansion looses Connection:

@developeralgo222 and just to be sure, termination only happens on the last 3HC in the chain?

Yes

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 to me, that looks like the boards are continually resetting, or one of them in the bus is, and affecting others. And that makes me think there is a short on the board, probably caused by something plugged into it.

Did you test the setup before installing in the machine and connecting everything? I think you’re going to need to disconnect everything from the boards, unmount them from the machine (in case the way you have mounted them is causing a short) and bench test with just CAN and power connected to either isolate the behaviour of the CAN error, or if it works correctly, to start looking elsewhere for what is causing the boards to possibly be resetting.

And yes, all boards need to be on the same firmware version.

Ian

I disconnected everything from the Expansion boards. The only thing connected is power and CAN Bus cables

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

i have tested both sockets on board 2

6XD --> Board 2 CAN_IN and Board 2 CAN_OUT --> Board 3 CAN_IN -- No Sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN --> Board 3 CAN_IN -- No sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN ---> Board 3 CAN_OUT -- No sync

That's not what I asked you to test.

Still does not work

6XD --> Board 2 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

As indicated above i tested each individual expansion 3HC board( 4 boards in total ) with nothing connected to them only power.

6XD --> Board 1 (with CAN Jumpers) CAN_IN / CAN_OUT -- Sync
6XD --> Board 2 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync
6XD --> Board 3 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync
6XD --> Board 4 (with CAN Jumpers) CAN_IN / CAN_OUT -- No Sync

Then tested 2 expansion boards at a time

6XD --> Board 1 CAN_IN / CAN_OUT -- Board 2 (with CAN Jumpers) CAN_IN / CAN_OUT -----> Sync
6XD --> Board 1 CAN_IN / CAN_OUT -- Board 3 (with CAN Jumpers) CAN_IN / CAN_OUT -----> Sync
6XD --> Board 1 CAN_IN / CAN_OUT -- Board 4 (with CAN Jumpers) CAN_IN / CAN_OUT -----> No Sync

Then tested 3 expansion boards at a time

6XD --> Board 1 CAN_IN / CAN_OUT ---- Board 2 CAN_IN / CAN_OUT --- Board 3 (with CAN Jumpers) CAN_IN / CAN_OUT -----> No Sync
6XD --> Board 1 CAN_IN / CAN_OUT ---- Board 2 CAN_IN / CAN_OUT --- Board 4 (with CAN Jumpers) CAN_IN / CAN_OUT -----> No Sync
6XD --> Board 1 CAN_IN / CAN_OUT ---- Board 3 CAN_IN / CAN_OUT --- Board 4 (with CAN Jumpers) CAN_IN / CAN_OUT -----> No Sync

@dc42, @droftarts & @oliof I have completely dismantled the Duet 3 6XD and 4 x 3HC boards from the PnP machine to figure out why Duet Expansion boards are not synchronized

so far i am able to get them to synchronize partially with occasional CAN timeouts . As per @dc42 , "if your main board and 3HC boards use different power supplies, you must tie the negative outputs of the power supplies together ",

i have 2 identical x 24V 20A DC PSU, I am not sure how to tie the Negative outputs of the PSU together. i have read & researched a few options but they are too involved with a few risks for something this simple.

i have

6XD, 3HC1, 3HC2 on PSU1
3HC3, 3HC4 on PSU2

Terminal on both identical PSUs from same manufacture ( S-500-24 switching PSU)

PSU 1 & PSU2 both have

AC Live or Hot Wire
AC Neutral Wire
AC GND Wire
V+
V+
V+
V-
V-
V-

How do i connect the Negative outputs of the 2 PSUs safely without setting my hair on fire ? Do i simply connect a jumper cable between Negative(-Ve) terminal cable of PSU1 to the Negative(-Ve) terminal cable of PSU2 ? OR what is the safest way to do this and long-term?

dc42

@developeralgo222 said in Duet 3HC Expansion looses Connection:

PSU 1 & PSU2 both have

Live(+Ve)
Negative(-Ve)
GND
V+
V+
V+
V-
V-
V-

Live is not the same as +Ve. The terminal you have called Negative is actually Neutral.

You need to connect one of the V- terminals of one PSU to one of the V- terminals of the other.

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

PSU 1 & PSU2 both have

AC Liveor Hot
AC Neutral
AC GND
V+
V+
V+
V-
V-
V-

Live is not the same as +Ve. The terminal you have called Negative is actually Neutral.

You need to connect one of the V- terminals of one PSU to one of the V- terminals of the other.

@droftarts , @dc42 , Just to make sure , Is this what you are implying? Only one V- Terminal on PSU1 to another V- Terminal on PSU2 ?