CAN connection lost as soon as sending any command to 3HC

Yao

Everything worked fine during setup and installation of a new machine using DUET3 + 3HC + four 1XD boards.

Now every time I run anything related to the 3HC it stops functioning (Response timeout: CAN addr 1) when checking the board the light is no longer blinking in sync (more like randomly turning on or of or blinking a bit).
currently connected are,

1 limit switch and
1 stepper motor (V) to position a IR module connected to the main duet3 board.
-laser pointer 12v out6
-led 12v to out7
-led 12v to out8

30v power (from same power source as the main duet3)
3HC is last in line on CAN, before it are four 1XD boards (same power source) connected to a Leadshine driver for XYZU (different 70v power source)

Had it first wired the other way around: Duet3-3HC-4x1XD
The error started in that configuration.

The main DUET3 has 6 extruders, IR height sensor and 4 limit switches.

when resetting power, all boards blink properly. as soon as I send a command or move motor V on 3HC the board stops.

Had it working before, stopped after inserting a beep during moves. removing the beep did not solve the CAN error

this is the only code used on this board:

; homev.g
; called to home the V axis
;
; generated 24/02/2022
M42 P21 S1 ; turn on chamber light
M42 P22 S1 ; turn on WARNING light
M42 P20 S0.5 ; turn on laser line at 50%

;position IR module

G1 H1 V2 F2000 ; go back a few mm
G1 H1 V-50 F2000 ; move to V axis endstop and stop (first pass)
M300 S6000 P200 ; play sound
G1 H1 V5 F2000 ; go back a few mm
G1 H1 V-15 F2000 ; move to V axis endstop once more (second pass)
M300 S5000 P200 ; play sound
;G1 H1 V-15 F2000 ; move to V axis endstop once more (third pass)
;M300 S4000 P300 ; play sound
G92 V0 ; set V position to axis minimum
G4 S2 ; wait 2 sec

M42 P20 S0.4 ; turn brightness laser line to 40%
M42 P22 S0 ; turn off WARNING light

Any ideas what can be the underlying issue with the CAN connection loss?
btw. after a while (10min) the 3HC CAN blink is back in sync with the rest. As soon as I send it command it stops.
oh the 1XD boards maintain their connection and function.

Running firmware :
M115 b1: gave the following information:
Warning: Discarded std reply src=1 RID=31 exp 32 ""
Duet EXP3HC firmware version 3.3 (2021-06-15 16:12:41)

Yao

can a faulty stepper motor influence CAN error ?
Stepper motor attached is Nema8 running at 0.6A

chrishamm

@yao Please upgrade everything including the expansion boards to 3.4.1 and check if the problem persists.

Yao

@chrishamm did that after posting, No change.
Also made a new CAN wire, 2xtwisted pair for the 20cm to the main board and placed it first in the CAN line, no change
I don't understand .
After power reset the board is back in sync again.
After CAN error, red sync light stays on...

chrishamm

@yao Can you post the output of M122 B1 after the problem occurred? If you get no or an incomplete response, please share your full config (system files ZIP) and the G-code file that provoked the problem.

Yao

@chrishamm thanks, no gcode file, just the homev.g I showed above.

But wow, this, there is a new response, never seen this before... :

Error: M122: Response timeout: CAN addr 1, req type 6024, RID=66

M122 B1
Warning: Discarded std reply src=1 RID=64 exp 65 "Duet EXP3HC rev 1.01 or earlier firmware version 3.4.1 (2022�ZE ��[E ��Warning: Discarded std reply src=1 RID=64 exp 65 "-06-01 21:15:27)
Bootloader ID: SAME5x bootloader version 2.�ZE ��[E ��Warning: Discarded std reply src=1 RID=64 exp 65 "3 (2021-01-26b1)
All averaging filters OK"

after switching power on and of:

m122 b1
Diagnostics for board 1:
Duet EXP3HC rev 1.01 or earlier firmware version 3.4.1 (2022-06-01 21:15:27)
Bootloader ID: SAME5x bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 158900, free system stack 200 words
Tasks: Move(notifyWait,0.0%,160) HEAT(notifyWait,0.0%,108) CanAsync(notifyWait,0.0%,69) CanRecv(notifyWait,0.0%,82) CanClock(notifyWait,0.0%,71) TMC(notifyWait,7.3%,99) MAIN(running,91.3%,432) IDLE(ready,0.0%,40) AIN(delaying,1.3%,263), total 100.0%
Last reset 00:00:25 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 50.0 steps/mm,standstill, SG min 0, mspos 8, reads 33507, writes 16 timeouts 0, steps req 0 done 0
Driver 1: pos 0, 80.0 steps/mm,standstill, SG min 0, mspos 8, reads 33512, writes 11 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 80.0 steps/mm,standstill, SG min 0, mspos 8, reads 33513, writes 11 timeouts 0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0
Peak sync jitter -3/6, peak Rx sync delay 175, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 28.2, current 28.2, max 28.2
V12 voltage: min 12.1, current 12.1, max 12.2
MCU temperature: min 28.6C, current 28.6C, max 29.1C
Last sensors broadcast 0x00000000 found 0 54 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 242, send timeouts 0, received 149, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 0

Yao

@chrishamm config.g is very standard no difficult stuff in there.
it is basically a tiny stepper motor driving a lever up and down.

config.g

update:
when I run the homev.g script more than 3 times in a row the system hangs.

but when i run the homev.g once, then move the arm away from the end switch: G1 V10 and start the homev.g again, I can repeat this sequence as many times as I want without disrupting the CAN connection.

I do get a warning :
" Warning: Discarded std reply src=1 RID=53 exp 54""

20220609_203106_002_001_001_001_001.mp4

Yao

I think a found a workaround by never homing the V axis from an already homed position.
So after homing this axis it must travel away from the switch before any other move.

I dont know how, or why the CAN error initiates, but it is solved by not having the limit switch for V axis on the 3HC board io_0 header active when initiating a new homeV.g or equivalent.

Not sure this issue is solved, but its a workaround.

dc42

@yao have you checked that the main board and all expansion boards are running the same major firmware version? The usual reason for getting CAN error messages (other than while updating firmware ) is that the main boards and expansion boards are running different major firmware versions. For example, 3.3 is not compatible with 3.4.0, however 3.4.1 is compatible with 3.4.0.

Yao

@dc42 Hello! , thank you for your suggestion. Yes I updated all boards to the latest 3.4.1 right after posting the question, Much easier than anticipated and fully automatic! I followed the progress by the messages and the lights blinking in succession on all the boards.

With the current workaround I still get the occasional CAN bus error, but it does not totally shut down the board. i can continue operating the machine.
It would be better to have no message at all...

dc42

@yao what CAN messages do you get now? It's not unusual to get occasional CAN messages during updating firmware or when you reset individual boards, however you should not get any during normal use such as when printing.

CAN error messages when running different major firmware versions on the main board and expansion boards are common, because the CAN protocols often change between versions in order to support new features.

Yao

@dc42 this is the only message i see in DWC:
Error: Response timeout: CAN addr 1, req type 6037, RID=130

On the screen i get multiple CAN warnings and a few about endstop not reached etc etc.

I found that if the endstop for V axis is already open (pressed by arm) before any other move for that axis i receive a CAN error and endstop fail.
Board hangs, but with the new firmware, if I manually push the arm away from the endstop I can resume function and movements, CAN is reestablished?

Yao

@dc42 quick question i ran into while testing this homing;

Is the DWC "run mesh compensation (G29)" button a different underlying command to the 'run mesh button' from the touchscreen? I am getting different behavior between the two.
As if the DWC one does not use the bed.g but only mesh.g ?

what are the differences between the two?

When I adjust the mesh parameters in DWC and then use the touchscreen button, it does use the altered parameter (not the ones from the config.g) is this correct?
If I would copy the bed.g file and rename it mesh.g the DWC "run mesh" button would behave the same like the bed.g called from the touchscreen?

<<SOLVED>> it appears the situation is as stated above.

Yao

@dc42 I changed the connection to the switch from "1.io0.in" to the main board "io1.in"
The result was such erratic behaviour from the switch and board, I suspected it was EMI interferance, placed a ferrite ring around the switch cable near the board.
All issues are gone.
This switch is the only one that runs along a motor drive cable. All the other ones are 1pair twisted and schielded...
and cable is 5 meter long.. so, yeah, my mistake.

<<SOLVED>> i hope.

dc42

@yao said in CAN connection lost as soon as sending any command to 3HC:

Is the DWC "run mesh compensation (G29)" button a different underlying command to the 'run mesh button' from the touchscreen? I am getting different behavior between the two.

There is no standard button on PanelDue that runs G29. If you have a button that runs G29 then I guess you have a macro file that runs it. Change that macro to run G29 instead of G29 S0, then it will run sys/mesh.g if it exists.

Yao

@dc42 there is a button on right, (little golfed line icon?) Under the macro section. When pressed it will run the bed probing?
What command is it calling then?

Phaedrux

That button runs bed.g which would make sense as that's a good place to put G29 as well.

Yao

@phaedrux Thanks! I was wondering what the difference between the two were. I now just made a copy of bed.g and renamed it mesh.g to get identical behavior between the two ways to run the bed probing.