Duet 3HC Expansion looses Connection

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I am wondering whether boards 2 and 3 are continually resetting. If you bypass board 1, can you get a M122 report from board 2 or board 3?

Does it make any difference if you change the order of the boards, or swap two of them over? For example, swap boards 1 and 3.

i have swapped the boards, Added each one by one but to no avail. The only way its able to Sync but disconnecting sometimes , is by having CAN jumpers on Board 2 & 3 ( 3HC expansion boards ) and No CAN jumper on Board 1 (3HC ). No sure why that would work somehow ?

The only difference between Board 1 & ( 2 or 3) is that board 2 & 3 have SMC NPN Vacuum Pressure Sensors connected to the IO pins and they are powered from the 3HC VIN & GND as per your earlier post suggestion and also Board 3 has CAN Termination Jumpers

dc42

@developeralgo222 if the boards are resetting once a second then I need a M122 report from one of them in order to determine why. So try powering up the system, let it run for a few seconds, then unplug some CAN cables to bypass one or two of the 3HC boards; then get a M122 report from the remaining one. It's electrically safe to connect and disconnected CAN cables while the boards are powered.

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 if the boards are resetting once a second then I need a M122 report from one of them in order to determine why. So try powering up the system, let it run for a few seconds, then unplug some CAN cables to bypass one or two of the 3HC boards; then get a M122 report from the remaining one. It's electrically safe to connect and disconnected CAN cables while the boards are powered.

console_3HC1_But_both_3HC2-n-3HC3-with-CANTermJumpers.txt

console_Only_3HC1-with-CANTermJumpers_ByPassing_Both_3HC2_n_3HC3.txt

console_3HC1_n_3HC3-with-CANTermJumpers_Bypassing-3HC2.txt

console_3HC2_n_3HC3-with-CANTermJumpers_ByPassing-3HC1.txt

console_Only_3HC2-with-CANTermJumpers_ByPassing_Both_3HC1_n_3HC3.txt

console_Only_3HC3-with-CANTermJumpers_ByPassing_Both_3HC1_n_3HC2.txt

console_3HC1_3HC2_3HC3-with-CANTermJumpers.txt

NOTE: when i CAN connect only Duet 6XD to

3HC1 only with CAN Termination Jumpers on it (CAN ID 1 ) --- this works and synchronization occurs
3HC2 only with CAN Termination Jumpers on it (CAN ID 2 ) --- NO synchronization
3HC3 only with CAN Termination Jumpers on it (CAN ID 3 ) --- NO synchronization
3HC1 & 3HC2 only with CAN Termination Jumpers on 3HC2 --- NO synchronization
3HC1 & 3HC3 only with CAN Termination Jumpers on 3HC3 --- NO synchronization
3HC2 & 3HC3 only with CAN Termination Jumpers on 3HC3 --- NO synchronization
3HC1 & 3HC2 & 3HC3 with CAN Termination Jumpers on both 3HC2 and 3HC3 --- this works and synchronization occurs but there is an occasional timeout on 3HC1 . This should not be working as per Duet's documentation ???????

NOTE:

Also Duet 6XD, 3HC1 , 3HC2 & 3HC3 All share the same 24V , 20A DC Power Supply.
The only difference between 3HC1, 3HC2 & 3HC3 is that 3HC2 & 3HC3 each have 3 x SMC NPN Vacuum Pressure Digital Sensors -ZSE30A-01-N-L (Not Z-Probe) connected to them and share the Power VIN & GND Terminal on the Board itself as per @dc42 earlier suggestion

developeralgo222

@developeralgo222 @dc42 can you respond to this issue ?

droftarts

@developeralgo222 I know this sounds rather basic, but have you swapped the CAN wires around? It sounds like the cable from 3HC1 to 3HC2 isn’t working, or only partially, or picking up interference. Check continuity, and that resistance is within specification. How long are the cable runs?

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 I know this sounds rather basic, but have you swapped the CAN wires around? It sounds like the cable from 3HC1 to 3HC2 isn’t working, or only partially, or picking up interference. Check continuity, and that resistance is within specification. How long are the cable runs?

Ian

I have checked the RJ11 cables and tested them with a Fluke professional cable tester to make sure they are ok. The lengths are :
6XD to 3HC1 —> 10 ft cable
3HC1 to 3HC2 & 3HC2 to 3HC3 —> 1ft cable

I am assuming that RJ11cable for CAN connection is supposed to be shielded Straight through cable ( ie wired at same pins positions on both ends ) as per the documentation . I was using high speed ADSL 2+ cable that were 10ft long , I tried also to shorten 2 of them to 1 ft but that did not solve the issue

developeralgo222

@developeralgo222 @dc42 & @droftarts Finally got this fixed . The issue was bad cable /connection from 6XD to Expansion Board 1 of 3HC . Never trust any cables from China , OMG when i decided to splice them , they are really crappy no shields and no proper insulations per pair. Buyer be aware.Have seen this in many chinese made products on Amazon.

droftarts

@developeralgo222 glad you got it fixed! It’s been bugging me what it could be if it wasn’t the cables!

Ian

developeralgo222

@droftarts

Got them powered up and syncing and then they started rapid flashing and lost synchronization again. so i unplugged all the extension boards and again started adding one by one, to see if any board or cable is the problem. I made all the connection cables and tested them for continuity myself to make sure they are ok .

As per documentation , Is the 2 CAN Bus Termination Jumpers(resistors) required only on the Last Expansion Board in the chain ? In my Case 6XD --> 3HC(1) --> 3HC2(2) --> 3HC(3 -- Has 2 CAN Bus jumpers)

This is getting to be a frustrating episode

droftarts

@developeralgo222 said in Duet 3HC Expansion looses Connection:

As per documentation , Is the 2 CAN Bus Termination Jumpers(resistors) required only on the Last Expansion Board in the chain ?

Yes, that should be correct. You should have termination at both ends of the bus.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

As per documentation , Is the 2 CAN Bus Termination Jumpers(resistors) required only on the Last Expansion Board in the chain ?

Yes, that should be correct. You should have termination at both ends of the bus.

Ian

i assume CAN Bus Termination Jumpers only on 3HC(3) but not on 6XD board ( i don't see any CAN Bus termination jumpers on 6XD ) in my case

droftarts

@developeralgo222 Yes, the 6XD should have CAN termination resistors built-in. See https://docs.duet3d.com/Duet3D_hardware/Duet_3_family/Duet_3_Mainboard_6XD_Hardware_Overview#can-fd-bus-expansion

There is a termination resistor fitted on each CAN bus, so normally this board must be at the end of the bus. There are drill-to-disconnect jumpers on the underside of the board that allow the termination resistor to be removed, however this is not required in normal operation.

Ian

developeralgo222

@droftarts said in Duet 3HC Expansion looses Connection:

@developeralgo222 Yes, the 6XD should have CAN termination resistors built-in. See https://docs.duet3d.com/Duet3D_hardware/Duet_3_family/Duet_3_Mainboard_6XD_Hardware_Overview#can-fd-bus-expansion

There is a termination resistor fitted on each CAN bus, so normally this board must be at the end of the bus. There are drill-to-disconnect jumpers on the underside of the board that allow the termination resistor to be removed, however this is not required in normal operation.

Ian

Ok that's good. so it seems when i disconnect board 3 ( 3HC(3) ) still Boards (6XD, 3HC2(1) & 3HC(2) ) syncs even without a jumper on 3HC(2) but as soon as i connect board 3 (3HC(3) which has the jumpers it stops syncing.

A little troubleshooting :

6XD --> 3HC(1) ==> Syncs
6XD --> 3HC(1) ---> 3HC(2) with CAN Termination Jumpers) ==> Syncs
6XD --> 3HC(1) ---> 3HC(3) with CAN Termination Jumpers) ==> Syncs

Looses synchronization when i connect all 3 expansion boards ( 3HC(1) , 3HC(2) & 3HC(3) )

e.g 6XD --> 3HC(1) --> 3HC(2) with CAN Termination Jumpers) ==> Syncs



1/13/2024, 7:49:18 PM	M122 B2

1/13/2024, 7:49:18 PM	Diagnostics for board 2:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 184 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.1%,419) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:23:04 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21135, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21136, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 21137, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -25/11, peak Rx sync delay 181, resyncs 9/0, no timer interrupt scheduled
VIN voltage: min 24.5, current 24.5, max 24.6
V12 voltage: min 12.3, current 12.4, max 12.4
MCU temperature: min 27.6C, current 36.0C, max 36.4C
Last sensors broadcast 0x00000000 found 0 203 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 7832, send timeouts 462, received 4244, lost 0, errs 4637716, boc 310, free buffers 38, min 38, error reg ff5f00
Last cancelled message type 4519 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0



1/13/2024, 7:49:11 PM	M122 B1

Diagnostics for board 1:
Duet EXP3HC rev 1.02 or later firmware version 3.5.0-rc.2 (2023-12-14 08:56:47)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 156976, free system stack 194 words
Tasks: Move(3,nWait,0.0%,182) HEAT(2,nWait,0.0%,120) CanAsync(5,nWait,0.0%,67) CanRecv(3,nWait,0.0%,80) CanClock(5,nWait,0.0%,70) TMC(2,nWait,5.8%,103) MAIN(1,running,93.0%,391) IDLE(0,ready,0.0%,39) AIN(2,delaying,1.1%,263), total 100.0%
Last reset 00:22:57 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60248, writes 11 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60248, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm, standstill, SG min 0, mspos 8, reads 60249, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, segs 0, step errors 0, maxLate 0 maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 0.00
Peak sync jitter -17/13, peak Rx sync delay 183, resyncs 5/0, no timer interrupt scheduled
VIN voltage: min 24.4, current 24.4, max 24.5
V12 voltage: min 12.4, current 12.4, max 12.4
MCU temperature: min 24.6C, current 32.3C, max 32.6C
Last sensors broadcast 0x00000000 found 0 163 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 10108, send timeouts 133, received 6112, lost 0, errs 14212604, boc 0, free buffers 38, min 38, error reg ff6500
Last cancelled message type 4519 dest 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

Not sure what might be the issue. I don't think its the cables anymore, it has to be the Duet 3 3HC boards or the RRF firmware ?

dc42

@developeralgo222 are you quite certain that you have identified the CAN termination jumper positions correctly? The CAN bus is quite tolerant and will often work with a termination resistor missing or with an extra termination resistor, but probably not with two extra resistors.

Other things to check:

Do the contacts in the CAN sockets of board 2 look OK? Does the soldering of those sockets on the underside of the PCB look OK?
Are all 3 boards set to different CAN addresses?
Does 6XD - board 2 - board 3 work?

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 are you quite certain that you have identified the CAN termination jumper positions correctly? The CAN bus is quite tolerant and will often work with a termination resistor missing or with an extra termination resistor, but probably not with two extra resistors.

Other things to check:

Do the contacts in the CAN sockets of board 2 look OK? Does the soldering of those sockets on the underside of the PCB look OK?

Are all 3 boards set to different CAN addresses?

Does 6XD - board 2 - board 3 work?

(1) Contacts in the CAN sockets of board 1,2 & 3 look ok . Yes contacts on board 2 look ok, The solder is good and has good contact and spread on the pin under side
(2) All three boards are set to CAN ID 1, 2 & 3 respectively
(3) Does 6XD - board 2 - board 3 work? NO - it does not sync . ( in this case only Board 3 has the Jumpers )

Sync tests : -- this is really strange

6XD --> 3HC(1 ) ==> Syncs without the CAN termination jumpers on 3HC1
6XD --> 3HC(2 ) ==> NO Sync W/O the CAN termination jumpers on 3HC2
6XD --> 3HC(3 ) ==> NO Sync W/O the CAN termination jumpers on 3HC3
6XD --> 3HC(1) ---> 3HC(2) ==> Syncs ---- No CAN termination jumpers on both 3HC1 & 3HC2
6XD --> 3HC(1) ---> 3HC(3) ==> Syncs ---- No CAN termination jumpers on 3HC1 but with CAN Termination Jumpers on 3HC2
6XD --> 3HC(2) ---> 3HC(3) ==> NO Sync -- No CAN termination jumpers on 3HC2 but CAN Termination Jumpers on 3HC3

NOTE: The only difference between 3HC1 , 3HC2 & 3HC3 is that 3HC2 have 3 x SMC Vacuum Pressure Sensors PWR & GND connected to same VIN & GND on the 3HC2 board & 3HC3 have another 3 x SMC Vacuum Pressure Sensors PWR & GND connected to same VIN & GND on the 3HC3 board as per @dc42 earlier suggestions as per the picture below. Do you think this might be the issue ?

dc42

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

developeralgo222

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

i have tested both sockets on board 2

6XD --> Board 2 CAN_IN and Board 2 CAN_OUT --> Board 3 CAN_IN -- No Sync
6XD --> Board 2 CAN_OUT and Board 2 CAN_IN --> Board 3 CAN_IN -- No sync
6XD --> Board 2 CAN_OUT and Board 2 CAN_IN ---> Board 3 CAN_OUT -- No sync

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

i have tested both sockets on board 2

6XD --> Board 2 CAN_IN and Board 2 CAN_OUT --> Board 3 CAN_IN -- No Sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN --> Board 3 CAN_IN -- No sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN ---> Board 3 CAN_OUT -- No sync

i am not sure what might be the issue with CAN Bus connections on the Duet 3 Boards , i believe @dc42 is the designer or one of the designers of this boards if i am not wrong, please can you give some insights as to why 2 x 3HC are not syncing at all. i bought another 3HC(4) board 4 today and even that doesn't sync but with that it has RRF 3.4.1 so i need to update it to 3.5_rc2 and then test. But i am really getting frustrated with CAN Bus connections on this Duet 3 Boards. This are very simple straight forward CAN Bus connections and i should not be encountering such issues.

developeralgo222

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

i have tested both sockets on board 2

6XD --> Board 2 CAN_IN and Board 2 CAN_OUT --> Board 3 CAN_IN -- No Sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN --> Board 3 CAN_IN -- No sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN ---> Board 3 CAN_OUT -- No sync

i am not sure what might be the issue with CAN Bus connections on the Duet 3 Boards , i believe @dc42 is the designer or one of the designers of this boards if i am not wrong, please can you give some insights as to why 2 x 3HC are not syncing at all. i bought another 3HC(4) board 4 today and even that doesn't sync but with that it has RRF 3.4.1 so i need to update it to 3.5_rc2 and then test. But i am really getting frustrated with CAN Bus connections on this Duet 3 Boards. This are very simple straight forward CAN Bus connections and i should not be encountering such issues.

This is the nightmare i am dealing with, disconnecting and reconnecting every 1 sec @Phaedrux @dc42 @droftarts

Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 4 reconnected
Error: Expansion board 3 reconnected
Error: Expansion board 2 reconnected
Error: Expansion board 4 reconnected

dc42

@developeralgo222 said in Duet 3HC Expansion looses Connection:

@dc42 said in Duet 3HC Expansion looses Connection:

@developeralgo222 I suspect a bad connection on the socket on board 2 that you are using to connect board 3 to it. Please try just connecting the 6XD and board 2 again but use the other board 2 connector this time, i.e. the one that you normally use to connect board 3 to it. The two CAN sockets are connected in parallel so it shouldn't matter which you use as input from the 6XD and which you use as output to the next board.

i have tested both sockets on board 2

6XD --> Board 2 CAN_IN and Board 2 CAN_OUT --> Board 3 CAN_IN -- No Sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN --> Board 3 CAN_IN -- No sync

6XD --> Board 2 CAN_OUT and Board 2 CAN_IN ---> Board 3 CAN_OUT -- No sync

That's not what I asked you to test.