Duet3 6CH + 3CH expansion board - Missing steps.

evomotors

@hackinistrator said in Duet3 6CH + 3CH expansion board - Missing steps.:

@evomotors
are you 100% sure that its "data loss" causing the issue ?
it seems you set everything correctly (i'm far from being expert...) maybe the board ignores your current limit settings .
can you try running the same gcode (or simple high accel macro) while the motors disconnected from the load (remove belts) ?
i'm not sure its a comm speed issue . maybe the tboard simply ignores motor current settings .

I'm not 100% sure that it's caused by data loss. If I set my motors current limits correctly it should not ignore them, unless there is a bug in the firmware.

I prefer not to remove belts, but even if I do, what it is going to prove? How can you tell if gantry in the correct position if belts are disconnected?

But you maybe correct, it does feels like motors ether underpowered or pulses are irregular on these fast moves. Unfortunately there is no way for me to tell, I don't have oscilloscope on hands. But it should be really easy for @dc42 to test.

I'm still waiting for @dc42 to look at the videos and reply.

deckingman

@hackinistrator Have you watched the videos that @evomotors posted? If the board ignores the motor current, then it does so selectively at some seemingly random time after a print starts, then restores it again immediately after.

evomotors

I'm getting lonely here after proving existence of the issue.

@dc42 Do you want me to perform some additional tests?

Phaedrux

Thanks for your patience. He's aware and will respond ASAP.

evomotors

@Phaedrux said in Duet3 6CH + 3CH expansion board - Missing steps.:

Thanks for your patience. He's aware and will respond ASAP.

Oh thanks! The good news that I'm not alone here.

deckingman

@evomotors For want of anything better to say or suggest, have you tried commenting out the M593 (DAA) to see if that makes any difference? No reason for suggesting it other than it's something which modifies motion planning, which might get applied to main boards but not expansion boards. I don't know if that's even possible but testing without M593 would at least eliminate it as a potential cause. Just a thought if you've got nothing better to do while you are waiting...........

dc42

@evomotors, I believe at least some of these issues are caused by CAN clock sync jitter, meaning that the expansion board clocks don't run perfectly in sync either the main board. When I measured it, I found that it was higher than I expected, frequently reaching 300us and sometimes more under conditions of high step rates and short move segments. Fortunately we've had a plan to reduce this jitter ever since we designed Duet 3. I've been working on this all day, and the jitter is now reduced to 13us even with heavy traffic. There is the possibility to reduce it still further.

I have two other related issues to investigate before we release an early 3.3 beta.

hackinistrator

@deckingman sure , i watched the vid.
i'm not sure if the issue occurs at random time.
if you check the vid , you can see the printer completed pretty fast accel moves without issues while moving on XY plane .
the issue occurred when it was making fast diagonal move - single motor running .
what is the default motor current on those drivers anyways ?
maybe its a good idea to try and reduce motor current on exp. board to very low value and see if it has any effect .
there are no CAN errors so i don't see how it can be comm. error .

T3P3Tony

@hackinistrator there is some timing jitter (as @dc42 has pointed out) that is not currently reported in 3.2.

@evomotors thanks for all the work in detailing the problem, we are working on it!

evomotors

@dc42 @T3P3Tony
Thank you for your hard work. Waiting to test the changes.

dc42

I am pleased to report that we have made substantial progress on this issue. In addition to fixing the CAN clock jitter, we have identified and fixed an issue that could cause whole moves to be abandoned on CAN-connected expansion boards. We have only been able to reproduce this using high step rates and short moves, however the cause of the issue (floating point rounding error) could conceivably happen at lower step rates too.

I expect to provide unofficial 3.3beta versions of all firmwares tomorrow.

Luke'sLaboratory

@dc42 Holy crap! Big if true!!!!!! I can't wait to try out the firmware to see if it solves my issues!

evomotors

@dc42
Do you need a beta tester?

Veti

@evomotors said in Duet3 6CH + 3CH expansion board - Missing steps.:

Do you need a beta tester?

always welcome.

the firmware will be available for all here
https://github.com/Duet3D/RepRapFirmware/releases

evomotors

@dc42 Any updates? I guess there some roadblocks with other issues.

dc42

@evomotors, https://forum.duet3d.com/topic/20991/early-rrf-3-3beta-files-available

evomotors

@dc42 said in Duet3 6CH + 3CH expansion board - Missing steps.:

@evomotors, https://forum.duet3d.com/topic/20991/early-rrf-3-3beta-files-available

Issue still exists after 3.3beta install. Visually no difference.

This is after starting and canceling print:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.3beta running on Duet 3 MB6HC v1.01 or later (SBC mode)
Board ID: 08DJM-956L2-G43S8-6JKDL-3SJ6L-1802G
Used output buffers: 1 of 40 (15 max)
=== RTOS ===
Static ram: 149772
Dynamic ram: 64248 of which 104 recycled
Never used RAM 140548, free system stack 126 words
Tasks: Linux(ready,141) HEAT(blocked,299) CanReceiv(blocked,893) CanSender(blocked,346) CanClock(blocked,326) TMC(blocked,52) MAIN(running,613) IDLE(ready,19)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:20:29 ago, cause: power up
Last software reset details not available
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 23.3, current 35.4, max 35.6
Supply voltage: min 24.1, current 24.2, max 24.5, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.1, current 12.1, max 12.2, under voltage events: 0
Driver 0: position 71266, standstill, reads 15769, writes 24 timeouts 0, SG min/max 0/243
Driver 1: position 8428, standstill, reads 15769, writes 24 timeouts 0, SG min/max 0/248
Driver 2: position 1039, standstill, reads 15769, writes 24 timeouts 0, SG min/max 0/269
Driver 3: position 0, standstill, reads 15770, writes 24 timeouts 0, SG min/max 0/267
Driver 4: position 0, standstill, reads 15783, writes 11 timeouts 0, SG min/max 0/0
Driver 5: position 0, standstill, reads 15783, writes 11 timeouts 0, SG min/max 0/0
Date/time: 2021-01-16 15:50:32
Slowest loop: 178.46ms; fastest: 0.03ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 700263ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 2249, completed moves 2249, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 3], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
Heater 0 is on, I-accum = 0.4
Heater 1 is on, I-accum = 0.3
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 7096, send timeouts 0, received 38, lost 0, longest wait 1ms for reply type 6018, peak Tx sync delay 414, free buffers 48
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 1ms ago
RX/TX seq numbers: 44700/44700
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x2c478
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.2.0
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 31.34
Maximum length of RX/TX data transfers: 2656/1664

M122 B1
Diagnostics for board 1:
Duet EXP3HC firmware version 3.3beta (2021-01-16 08:18:18)
Bootloader ID: not available
Never used RAM 154492, free system stack 166 words
HEAT 92 CanAsync 92 CanRecv 86 TMC 30 MAIN 306 AIN 259
Last reset 00:21:14 ago, cause: power up
Last software reset data not available
Driver 0: position -343134, 160.0 steps/mm,  standstill, reads 45658, writes 24 timeouts 0, SG min/max 0/218, steps req 1681012 done 1337262
Driver 1: position -74772, 160.0 steps/mm,  standstill, reads 45661, writes 24 timeouts 0, SG min/max 0/349, steps req 1442638 done 1203360
Driver 2: position 5964, 172.0 steps/mm,  standstill, reads 45669, writes 20 timeouts 0, SG min/max 0/198, steps req 32691 done 32719
Moves scheduled 2198, completed 2198, in progress 0, hiccups 185, step errors 0
Peak sync jitter 14, peak Rx sync delay 183, resyncs 0, no step interrupt scheduled
VIN: 24.7V, V12: 12.3V
MCU temperature: min 43.8C, current 44.0C, max 44.2C
Ticks since heat task active 246, ADC conversions started 1274238, completed 1274236, timed out 0
Last sensors broadcast 0x00000000 found 0 0 ticks ago, loop time 0
CAN messages queued 58, send timeouts 0, received 13653, lost 0, free buffers 36, error reg 11004e

And this is after just after boot before printing:

m122 B1
Diagnostics for board 1:
Duet EXP3HC firmware version 3.3beta (2021-01-16 08:18:18)
Bootloader ID: not available
Never used RAM 154420, free system stack 200 words
HEAT 79 CanAsync 92 CanRecv 86 TMC 64 MAIN 298 AIN 259
Last reset 00:00:49 ago, cause: software
Last software reset data not available
Driver 0: position 0, 160.0 steps/mm,  standstill, reads 49081, writes 23 timeouts 0, SG min/max 0/0, steps req 0 done 0
Driver 1: position 0, 160.0 steps/mm,  standstill, reads 49083, writes 23 timeouts 0, SG min/max 0/0, steps req 0 done 0
Driver 2: position 0, 172.0 steps/mm,  standstill, reads 49087, writes 23 timeouts 0, SG min/max 0/0, steps req 0 done 0
Moves scheduled 0, completed 0, in progress 0, hiccups 0, step errors 0
Peak sync jitter 11, peak Rx sync delay 181, resyncs 1, no step interrupt scheduled
VIN: 24.7V, V12: 12.3V
MCU temperature: min 43.8C, current 43.8C, max 44.0C
Ticks since heat task active 236, ADC conversions started 49228, completed 49226, timed out 0
Last sensors broadcast 0x00000000 found 0 240 ticks ago, loop time 0
CAN messages queued 60, send timeouts 3, received 403, lost 0, free buffers 36, error reg 11004f

dc42

Thanks, from the M122 B1 report there's evidently still an issue with steps being lost on the 3HC.

Have you already posted your config.g and the GCode file somewhere?

evomotors

@dc42 said in Duet3 6CH + 3CH expansion board - Missing steps.:

Thanks, from the M122 B1 report there's evidently still an issue with steps being lost on the 3HC.

Have you already posted your config.g and the GCode file somewhere?

Yes, it is the same configuration and stl file as posted above in this thread

https://forum.duet3d.com/topic/20713/duet3-6ch-3ch-expansion-board-missing-steps/79?_=1610818681447

The config is where XY connected to expansion board

dc42

Thanks, I'll run those files on my bench system. I guess I should have done that earlier, rather than assuming that the bug that Tony and I reproduced was the one that affected your machine.