CAN connectivity: Duet 3 MB6HC to EXP3HC

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@dc42 Understood, I ended up updating to the 3.3beta firmware on both boards. It still requires a bit of patience to let the boards sync up.

How long does it take? If the fix is working then the LED on the expansion board should start flashing as soon as it received power (as evidenced by the other LEDs on the board), but may take about 2 seconds before it settles down to the slow flash indicating CAN sync. If the other LEDs on the expansion board light up immediately power is applied but the status LED still does not flash until many seconds later, then I need to revisit the fix.

kroybal

@dc42 Following a full update to 3.3beta seemed to be working at this point. It doesn't light up immediately but it flashes about two seconds later (w.r.t. the other LEDs) overall takes less than 10 seconds to sync with the MB. I cannot complain!

Cheers

Edit: I should have bit my tongue, I tried another time and the LED failed to turn on all together following power up. It has been about a minute and no light.

kroybal

@jrockland said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@kroybal hmm but why 15 ? If you have just a 6hc+3hc shouldnt it be address to 1 ? Or im missing something here

Addressing is arbitrary right? I certainly do not have tons of boards and channel 1 is free but I found it easiest to just flip them all to on in one motion instead of trying to be precise with flipping one switch.

Edit: Not totally arbitrary, but no right or wrong answer as long as its logical

kroybal

Very coarse statistics on my setup suggest about a 1/15 shot at lighting up upon startup, and 2 minutes light-up-and-sync time in the case of a startup failure.

Does this have to do with the crystal oscillators not powering up still or simply waiting for the signal from the boards to be in phase?

jrockland

@kroybal not a clue, it is why I was asking. I tough I was wrong doing it 1-2-3...

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@dc42 Following a full update to 3.3beta seemed to be working at this point. It doesn't light up immediately but it flashes about two seconds later (w.r.t. the other LEDs) overall takes less than 10 seconds to sync with the MB. I cannot complain!

Cheers

Edit: I should have bit my tongue, I tried another time and the LED failed to turn on all together following power up. It has been about a minute and no light.

Thanks for the update. I'll continue working on this and provide another update tomorrow.

dc42

@kroybal, I've put an updated 3.3beta Duet3Firmware-EXP3HC.bin file at https://www.dropbox.com/sh/wme9k0z86sytg33/AAAT6wrHp2eeJHK-dYoW1Um4a?dl=0. Please install it and see whether the delay from power up to when the LED on the 3HC starts flashing is eliminated.

kroybal

@dc42 I have installed the updated EXP3HC and it seems like the problem is still persisting.

Here is my M122 command for both boards

1/28/2021, 10:01:52 AM === Diagnostics ===

RepRapFirmware for Duet 3 MB6HC version 3.3beta running on Duet 3 MB6HC v1.01 or later (standalone mode)

Board ID: 08DJM-956L2-G43S8-6JKD8-3S06K-9B2GD

Used output buffers: 1 of 40 (13 max)

=== RTOS ===

Static ram: 149784

Dynamic ram: 92536 of which 0 recycled

Never used RAM 108352, free system stack 200 words

Tasks: NETWORK(ready,254) ETHERNET(blocked,111) HEAT(blocked,305) CanReceiv(blocked,909) CanSender(blocked,365) CanClock(blocked,334) TMC(blocked,52) MAIN(running,1120) IDLE(ready,20)

Owned mutexes: LwipCore(NETWORK) HTTP(MAIN)

=== Platform ===

Last reset 00:00:13 ago, cause: software

Last software reset at 2021-01-28 09:57, reason: User, GCodes spinning, available RAM 108312, slot 2

Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00400000 BFAR 0x00000000 SP 0x00000000 Task MAIN Freestk 0 n/a

Error status: 0x00

Aux0 errors 0,0,0

Aux1 errors 0,0,0

MCU temperature: min 38.6, current 40.3, max 40.5

Supply voltage: min 24.3, current 24.3, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes

12V rail voltage: min 10.8, current 12.6, max 12.8, under voltage events: 0

Driver 0: position 0, standstill, reads 10313, writes 14 timeouts 0, SG min/max 0/0

Driver 1: position 0, standstill, reads 10313, writes 14 timeouts 0, SG min/max 0/0

Driver 2: position 0, standstill, reads 10313, writes 14 timeouts 0, SG min/max 0/0

Driver 3: position 0, standstill, reads 10313, writes 14 timeouts 0, SG min/max 0/0

Driver 4: position 0, standstill, reads 10314, writes 14 timeouts 0, SG min/max 0/0

Driver 5: position 0, standstill, reads 10314, writes 14 timeouts 0, SG min/max 0/0

Date/time: 2021-01-28 10:01:53

Slowest loop: 4.11ms; fastest: 0.05ms

=== Storage ===

Free file entries: 10

SD card 0 detected, interface speed: 25.0MBytes/sec

SD card longest read time 2.3ms, write time 0.0ms, max retries 0

=== Move ===

DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000

=== MainDDARing ===

Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1

=== AuxDDARing ===

Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1

=== Heat ===

Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = 3 -1 -1 -1

Heater 0 is on, I-accum = 0.0

=== GCodes ===

Segments left: 0

Movement lock held by null

HTTP is ready with "M122 B0" in state(s) 0

Telnet is idle in state(s) 0

File is idle in state(s) 0

USB is idle in state(s) 0

Aux is idle in state(s) 0

Trigger is idle in state(s) 0

Queue is idle in state(s) 0

LCD is idle in state(s) 0

SBC is idle in state(s) 0

Daemon is idle in state(s) 0

Aux2 is idle in state(s) 0

Autopause is idle in state(s) 0

Code queue is empty.

=== Filament sensors ===

Extruder 0: pos 0.45, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0

Extruder 1: pos 3.66, errs: frame 0 parity 0 ovrun 0 pol 0 ovdue 0

=== CAN ===

Messages queued 74, send timeouts 0, received 20, lost 0, longest wait 1ms for reply type 6018, peak Tx sync delay 493, free buffers 48 (min 47)

=== Network ===

Slowest loop: 6.40ms; fastest: 0.02ms

Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions Telnet(0), 0 sessions

HTTP sessions: 1 of 8

Ethernet -

State: active

Error counts: 0 0 1 0 0

Socket states: 5 2 2 2 2 0 0 0

1/28/2021, 10:02:36 AM M122 B15

Diagnostics for board 15:

Duet EXP3HC firmware version 3.3beta (2021-01-28 09:11:08)

Bootloader ID: not available

Never used RAM 155236, free system stack 0 words

Move 160 HEAT 103 CanAsync 72 CanRecv 84 CanClock 74 TMC 64 MAIN 339 AIN 260

Last reset 00:00:13 ago, cause: software

Last software reset data not available

Driver 0: position 0, 80.9 steps/mm, standstill, reads 46408, writes 16 timeouts 0, SG min/max 0/0, steps req 0 done 0

Driver 1: position 0, 80.9 steps/mm, standstill, reads 46410, writes 16 timeouts 0, SG min/max 0/0, steps req 0 done 0

Driver 2: position 0, 80.0 steps/mm, standstill, reads 46419, writes 11 timeouts 0, SG min/max 0/0, steps req 0 done 0

Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0, maxPrep 0, maxOverdue 0, maxInc 0, mcErrs 0, gcmErrs 0

Peak sync jitter 12, peak Rx sync delay 174, resyncs 0, no step interrupt scheduled

VIN: 24.4V, V12: 12.2V

MCU temperature: min 43.6C, current 43.6C, max 43.9C

Ticks since heat task active 90, ADC conversions started 13832, completed 13831, timed out 0

Last sensors broadcast 0x00000000 found 0 94 ticks ago, loop time 0

CAN messages queued 33, send timeouts 0, received 136, lost 0, free buffers 36, min 36, error reg 110067

dup 0, oos 0, bm 0, wbm 0

kroybal

One difference I have noticed is that my EXP3HC board flashes without sync now and does not connect to the mainboard whatsoever even after waiting over five minutes at this point.

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

One difference I have noticed is that my EXP3HC board flashes without sync now and does not connect to the mainboard whatsoever even after waiting over five minutes at this point.

Please run M115 and M115 B1 and confirm that you see the following firmware versions and dates reported:

M115
FIRMWARE_NAME: RepRapFirmware for Duet 3 MB6HC FIRMWARE_VERSION: 3.3beta ELECTRONICS: Duet 3 MB6HC v0.6 or 1.0 FIRMWARE_DATE: 2021-01-21 09:00:34

M115 B1
Duet EXP3HC firmware version 3.3beta (2021-01-28 09:11:08)

(the M115 B1 command may work even though you don't have CAN sync).

Does the LED on the 3HC start flashing immediately when you power it up?
What board revision is your 3HC?

Thanks for your patience.

dc42

Hi @kroybal,

I guess you had better things to do over the weekend than work on this!

In the absence of feedback from you, I've taken a guess at what might be causing the loss of sync, and put new firmware binaries (both 6HC and 3HC) with today's build date at https://www.dropbox.com/sh/qr98k8fbkj5ue0k/AABPawUF99QVzDrheBQBDSxia?dl=0. Please try them and let me know whether they resolve the loss of sync.

Which country are you in?

kroybal

@dc42 I was absent from work Friday due to family reasons and do not normally work on weekends. I'll update the boards and report back to you.

@dc42 said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

What board revision is your 3HC?

For record, it is a V1.0 board

@dc42 said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

Which country are you in?

Central U.S.

kroybal

@dc42
I updated with the new binaries and confirmed the firmware date with an M115 command on both. They both reported back 2021-02-01.

Upon restarting the boards (via toggling the power switch) the 3HC was connected in under 10 seconds. I am a fan of repetition so I tried again and now its connection time is in the bracket of 30-45s following power-up.

Certainly an improvement, however, might I suggest a message to the console stating an established connection.

Another point I would like to note is that once the board is connected my endstops are not active until I press "STOP" on my pannel Due or emergency stop via the DWC and wait for a connection. Using a software reset does not cause the 3HC to lose connection.

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

Upon restarting the boards (via toggling the power switch) the 3HC was connected in under 10 seconds. I am a fan of repetition so I tried again and now its connection time is in the bracket of 30-45s following power-up.

Thanks. I'm sorry to hear that it is still taking a long time for the 3HC to connect. Does the LED on the 3HC start blinking rapidly as soon as it is powered up, but then take up to 45 sec before it blinks slowly to indicate CAN sync? Or does the LED remain off for up to 45 sec from power on; but having started blinking rapidly, it then establishes sync within a couple of seconds? Or something else?

The lack of functioning endstops suggests that during that period of up to 45 seconds, the EXP3HC was not receiving the commands to configure the endstop inputs in response to the M5784 commands in config.g.

I can authorise a warranty exchange of the 3HC if you wish. Unfortunately it will take some time for your board to reach me here in UK, so this won't help me to diagnose the cause in the short term.

kroybal

@dc42 said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

Thanks. I'm sorry to hear that it is still taking a long time for the 3HC to connect. Does the LED on the 3HC start blinking rapidly as soon as it is powered up, but then take up to 45 sec before it blinks slowly to indicate CAN sync? Or does the LED remain off for up to 45 sec from power on; but having started blinking rapidly, it then establishes sync within a couple of seconds? Or something else?

The light is not on for the 45 sec duration, then flickers for about 3 seconds before syncing up with the MB.

@dc42 said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

I can authorise a warranty exchange of the 3HC if you wish. Unfortunately it will take some time for your board to reach me here in UK, so this won't help me to diagnose the cause in the short term.

Sure thing, do you think this issue is unique to my board or is this issue common at this time? If it is not necessary, we could hold off on exchanging the board. If the problem does persist and/or is unique to my board, then we may have to go that route . So we can eliminate the possibility of this board slipping through a crack in QA and being unfixable for my purposes.

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

The light is not on for the 45 sec duration, then flickers for about 3 seconds before syncing up with the MB.

Thanks, that confirms that either the new firmware does not fix the slow crystal oscillator startup issue on your board, or the built-in on-chip RC oscillator is way out of calibration. If it's the oscillator startup time that's causing the problem then I should be able to fix it in firmware.

I would like to try one more firmware change before we swap your board. As you are US-based, I will work on this tonight, so that I should be able to send you updated firmware before the end of your day.

kroybal

@dc42 Sure thing, I will be leaving the lab today in 15 minutes so I will likely get back to you by tomorrow. Hope that is not too much of an inconvenience.

dc42

@kroybal said in CAN connectivity: Duet 3 MB6HC to EXP3HC:

@dc42 Sure thing, I will be leaving the lab today in 15 minutes so I will likely get back to you by tomorrow. Hope that is not too much of an inconvenience.

OK, no problem. Thanks for your patience.

dc42

I've updated just the EXP3HC firmware at https://www.dropbox.com/sh/qr98k8fbkj5ue0k/AABPawUF99QVzDrheBQBDSxia?dl=0. The M115 report for this version is:

Duet EXP3HC firmware version 3.3beta (2021-02-01 22:29:11)

Please install this firmware, verify the build time using M115, then check whether it solves the power-up issue. To be clear:

Once power is applied to the EXP3HC and its green 3.3V power LED illuminates, the red status LED should start flashing within less than a second, even if the CAN bus is not connected;
If the board is connected to a running Duet 3 main board via CAN running firmware 3.3beta, then the fast flashing should persist for one to two seconds before it changes to the 1Hz blink that indicates time sync with the main board has been achieved.

If this doesn't solve the problem, then please let me know who you purchased the board from so that we can arrange a replacement and a return shipping label.

kroybal

@dc42
IMG_2245-2.mp4

I verified the build time. Following the firmware update the reset worked my first go at it, the second go at it took about 10 seconds before the LED powered on.

The attached video is capturing the third and fourth attempt at resetting the board. I had to reduce quality and cut the video short but the fourth attempt was about 50 second connection, and I tried once more after that and it was immediate.

If firmware updates have an effect on the behavior of the LED/connection, could that be suggesting that the board has the potential to work? I have have seen an improvement, we started at the range of 10-30 minutes to connect and now we're down to the range of immediately to 50 seconds tops.