1LC - extruder randomly stopping/reversing - ≤3.5.0-rc1

Exerqtor

@KenW
Sucks to hear you also have problems, but it really does make me feel better to hear that it's not another "one off" issue, and it's something happening to others as well 😮‍💨

I've got alot of projects I hoped to get done by/in the comming holidays, but it's looking increasingly less and less likely 💩

It's luckily some bog standard ABS i've had to throw in the fuckit-bucket, but it's still annoying af.

A Former User

@Exerqtor That is the only printer running the beta firmware. It has to run it to use the RRF-36 toolboard. I had noticed your post but was ignoring them since I thought it was filament getting stuck in the heatbreak or possibly carbon fiber jamming the nozzle. I can take down and reassemble the EVA in minutes now with all the practice I have been getting.

Since mine is not acting up exactly like yours unless your extruder just stops for a bit then restarts with no error maybe it's another issue.
After I get back home in a few weeks I'll play around more with it,. Gave up today on the print I was trying to take with me.

Exerqtor

@KenW I also thought i had gotten a clog or something when it started happening again so i get your logic lol. It will be interesting seeing how it plays out when you get some time to look at it again.

I just reverted to 3.5.0b4 on the mobo & toolboard to see what happens at least, too be continued.

EDIT:
After reverting too 3.5.0b4 I was finally able to complete the print without issues.

I'm gonna try doing some more prints after work tomorrow to see if the "solution" holds, but it sure looks like the problem lies in 3.5.0-rc1/2.

Exerqtor

@dc42 I've ran over ballpark 24hr of prints on 2.5.0b4 now without any issues.

Weird that noone else reports about this.

A Former User

@Exerqtor Not home to look at anything but it might depend on the setup. I had another printer earlier that I also had a duet 3 mini 5+ with 3.5.0 r.1 and a RRF-36 mounted with 3.5.0 r1 firmware. That was an ender 3 pro, a simple bed slinger. I did multiple test prints on it before it was wrapped up as a gift and hauled up here. In 5 days it gets to live again and hopefully will still be problem free.

The printer giving me problems is a CoreXY with 3 Z steppers and a klicky so a bit more complex and different motion system. Also a Duet 3 mini 5_ with the other RRF-36 and the same firmware as the ender.

Exerqtor

@dc42 I tried installing 3.5.0-rc2 again, with a freshly formated SD card etc. to see if that might help (downloaded the binaries again also just in case) and it still shits the bed:

M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.5.0-rc.2 (2023-12-14 10:30:57) running on Duet 3 Mini5plus WiFi (standalone mode)
Board ID: XNHXF-HR6KL-K65J0-409N2-K9W1Z-RV2MZ
Used output buffers: 4 of 40 (40 max)
=== RTOS ===
Static ram: 102812
Dynamic ram: 128720 of which 0 recycled
Never used RAM 5196, free system stack 118 words
Tasks: NETWORK(1,ready,119.4%,195) HEAT(3,nWait,0.7%,328) Move(4,nWait,14.5%,239) CanReceiv(6,nWait,1.2%,774) CanSender(5,nWait,0.3%,327) CanClock(7,delaying,0.2%,350) TMC(4,nWait,17.4%,74) MAIN(1,running,53.1%,580) IDLE(0,ready,19.8%,29) AIN(4,delaying,20.2%,264), total 246.8%
Owned mutexes: WiFi(NETWORK)
=== Platform ===
Last reset 24:56:40 ago, cause: software
Last software reset at 2023-12-21 16:42, reason: User, Gcodes spinning, available RAM 12348, slot 0
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00446000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
MCU revision 3, ADC conversions started 89801376, completed 89801376, timed out 0, errs 0
MCU temperature: min 34.4, current 46.8, max 47.3
Supply voltage: min 8.0, current 23.9, max 24.7, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 198/28, heap memory allocated/used/recyclable 2048/532/136, gc cycles 15646
Events: 1 queued, 1 completed
Driver 0: ok, SG min 0, read errors 0, write errors 1, ifcnt 33, reads 7119, writes 123, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x72
Driver 1: ok, SG min 0, read errors 0, write errors 1, ifcnt 33, reads 7120, writes 123, timeouts 0, DMA errors 0, CC errors 0
Driver 2: ok, SG min 0, read errors 0, write errors 1, ifcnt 149, reads 7187, writes 56, timeouts 0, DMA errors 0, CC errors 0
Driver 3: ok, SG min 0, read errors 0, write errors 1, ifcnt 145, reads 7185, writes 57, timeouts 0, DMA errors 0, CC errors 0
Driver 4: ok, SG min 0, read errors 0, write errors 1, ifcnt 144, reads 7187, writes 55, timeouts 1, DMA errors 0, CC errors 0, failedOp 0x01
Driver 5: not present
Driver 6: not present
Date/time: 2023-12-22 17:38:55
Cache data hit count 4294967295
Slowest loop: 226.03ms; fastest: 0.09ms
=== Storage ===
Free file entries: 16
SD card 0 detected, interface speed: 22.5MBytes/sec
SD card longest read time 11.0ms, write time 62.9ms, max retries 0
=== Move ===
DMs created 83, segments created 43, maxWait 59621605ms, bed compensation in use: mesh, height map offset 0.000, max steps late 1, ebfmin 0.00, ebfmax 0.00
next step interrupt due in 37 ticks, enabled
Moves shaped first try 52954, on retry 20590, too short 71476, wrong shape 255081, maybepossible 11645
=== DDARing 0 ===
Scheduled moves 2684, completed 2644, hiccups 1184, stepErrors 0, LaErrors 1095, Underruns [0, 0, 4], CDDA state 3
=== DDARing 1 ===
Scheduled moves 0, completed 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.3
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Movement locks held by null, null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is doing "G1 X131.338 Y134.213 E.09057" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is doing "G4 P250" in state(s) 0 0, running macro
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
File2 is idle in state(s) 0, sync state 1
Queue2 is idle in state(s) 0
Q0 segments left 1, axes/extruders owned 0x0000807
Code queue 0 is empty
Q1 segments left 0, axes/extruders owned 0x0000000
Code queue 1 is empty
=== Filament sensors ===
check 0 clear 0
Extruder 0 sensor: no filament
=== CAN ===
Messages queued 1264763, received 1844452, lost 0, errs 1, boc 0
Longest wait 6ms for reply type 6013, peak Tx sync delay 276, free buffers 26 (min 24), ts 449002/449001/0
Tx timeouts 0,0,0,0,0,0
=== Network ===
Slowest loop: 211.13ms; fastest: 0.00ms
Responder states: MQTT(0) HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 1 of 8
=== WiFi ===
Interface state: active
Module is connected to access point 
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta6
MAC address c4:5b:be:ce:91:93
Module reset reason: Power up, Vcc 3.38, flash size 2097152, free heap 36292
WiFi IP address 192.168.10.x
Signal strength -52dBm, channel 6, mode 802.11n, reconnections 0
Clock register 00002001
Socket states: 0 0 0 0 0 0 0 0

M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-rc.2 (2023-12-14 08:58:51)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 2580, free system stack 89 words
Tasks: Move(3,nWait,2.9%,71) HEAT(2,nWait,7.9%,91) CanAsync(5,nWait,0.0%,55) CanRecv(3,nWait,1.0%,77) CanClock(5,nWait,0.4%,67) ACCEL(3,nWait,0.0%,53) TMC(2,nWait,70.2%,57) MAIN(1,running,53.1%,316) IDLE(0,ready,0.0%,27) AIN(2,delaying,111.1%,114), total 246.6%
Last reset 24:56:46 ago, cause: software
Last software reset data not available
Driver 0: pos 0, 568.8 steps/mm, ok, SG min 0, read errors 5, write errors 1, ifcnt 112, reads 29381, writes 30, timeouts 10, DMA errors 0, CC errors 0, failedOp 0x01, steps req 0 done 54758132
Moves scheduled 451072, completed 451072, in progress 0, hiccups 6856, segs 42, step errors 0, maxLate 2 maxPrep 759, maxOverdue 7315, maxInc 2784, mcErrs 0, gcmErrs 0, ebfmin -1.00 max inf
Peak sync jitter -2/12, peak Rx sync delay 280, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 8.3, current 24.1, max 24.6
MCU temperature: min 39.4C, current 73.3C, max 79.5C
Last sensors broadcast 0x00000012 found 2 138 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 1844535, send timeouts 0, received 1264877, lost 0, errs 1230, boc 0, free buffers 18, min 17, error reg 110000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 390, adv 34547/74657
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 6, contentions 0, other errors 0
=== Filament sensors ===
Interrupt 5726621 to 0us, poll 4 to 2681us

And it's still happening at random points.

droftarts

@Exerqtor I can’t remember if you have posted your config.g recently, but if not, could you post it again, please? If it calls macros, maybe a zip of you sys folder might be easier, to include them all. It’s just the Mini 5+ and a 1LC? What revision of each board? And what extruder type and motor? Does it still do it if you run a ‘dry’ job with no filament? Finally, can you post a Gcode file that provokes the retractions? Though I suspect it isn’t related to the Gcode itself.

Ian

Exerqtor

@droftarts I'm not by the printer now so I'll post everything later.

Yup it's just the Mini (1.02 WiFi) and 1LC (1.2) connected directly to eachother.

The extruder is a Bondtech LGX-lite with the stock motor they come with (running the Bondtech adviced current).. Uuhm I haven't tried a "dry" job, but since the stepper reverses the filament rather than extrude it when it happens i doubt it's load related, if thats what you were thinking of 😅

Regarding a Gcode file it's posted several through the thread here, but I've made a cloud folder with most of the config files and a job file in this post you can have a look at.

Doubt it's anything else than a firmware bug tbh. since 3.5.0-b4 don't act up like this at all. The symptoms and behaviour i'm experiencing is exactly the same ones that plagued 3.5.0-b3.

droftarts

@Exerqtor okay thanks. I don’t have an LGX, but I’ve probably got a motor similar enough. I might get enough time between Xmas and New Year to setup a bench test. I’ve also got a stepper motor analyser, which might be helpful.

Ian

Exerqtor

@droftarts Np, don't burn your xmas on this lol.
I just reverted to 3.5.0-b4 again so i'm good over the holidays ✌️

Wouldn't stepper specific issues have caused it to stall out rather than reverse btw (if that's what you're think about)? And again, since it don't happen on 3.5.0-b4 with the exact same config🤔

T3P3Tony

@Exerqtor reported here:

https://github.com/Duet3D/RepRapFirmware/issues/942

T3P3 created this issue in Duet3D/RepRapFirmware

closed 1LC - stepper randomly reversing - ≤3.5.0-r1 #942

Exerqtor

@T3P3Tony said in 1LC - stepper randomly reversing - ≤3.5.0-r1:

@Exerqtor reported here:

https://github.com/Duet3D/RepRapFirmware/issues/942

Hopefully this gets parched out right quick so I can update the fw and start using the StrideMax! 🤞

dc42

@Exerqtor I have reviewed the code that manages motor direction on the 1XD. At the start of every move, RRF sets the direction of the driver, regardless of whether it should already be correct. So the only way that I can see for the direction to reverse and remain reversed is if the direction control set by M569 has changed. This might happen in the following ways:

The memory that holds the direction value could have become corrupted. In this case, if M569 is run then it should report the incorrect direction value.
The board could have reset and the direction value changed back to the default. In this case, M569 would report the wrong value and M122 would report that a reset had occurred.
The main board has sent a M569 command to the tool board. This seems unlikely.

Therefore, please can you run the following test:

Upgrade your system to 3.5.0-rc.2
Start one of the prints that is likely to reverse direction
Once the print is going correctly, run M569 P121.0 and M122 B121 (if the tool board address is not 121 then change these commands appropriately)
Wait until the extruder reverses direction
Run M569 P121.0 and M122 B121 again

Then post all the results here.

Thanks - David

Exerqtor

@dc42 I will try it, but i don't think it's fully reversing either since it sure don't reverse/retract filament in the same rate that it would have extruded it under normal operation. If that makes sense?

It's experiencing it the same way as it did in the 3.5.0-beta2 bug.

I'll report back as soon as I get time to test it never the less.

dc42

@Exerqtor are you saying that when it is commanded to retract filament and is operating in reverse, it doesn't attempt to extrude filament instead?

Perhaps you should pause the print after it has started reversing, and see exactly how it does behave in response to extrusion commands?

Exerqtor

@dc42 What i meant is that when the issue happens, it don't behave like it simply reverses the extruder movement since it don't back the filament out at at the expected speed/feedrate.
I usually print at somewhere around 25mm³/s with a 0.4mm nozzle, so it's clearly visible when it works normally and one can see it pulling in filament. BUT when it starts acting up, it don't back out the filament at that rate at all.
It's more like it just stops doing the normal extrusions and only does retract moves (0.45mm FW retraction) but not the unretracts.
So if it was just reversing the stepper movement it would have unloaded the extruder within a minute as the job went on (have to do something like 35mm retraction to fully unload the extruder). But even after letting it go on with the problem active for 10-15minutes before pausing the print i only have to send one 20mm extrude command before it comes plastic out of the nozzle.

I've tried pausing the print(s) countless time when the issue happens, ans tried to run extrusion commands through PanelDue and those behave just as they should. But when i resue the print again it goes right back to the same mode of failure as before i paused.

Sorry for the messy atempt at an explenation 😮‍💨
I've changed the post title to better reflect whats going on.

dc42

@Exerqtor could it be that somehow the motor current has been reduced to a low value, so that the extruder motor is capable of retracting filament but not capable of overcoming nozzle back pressure?

It would be helpful if you could wait for the issue to happen, then pause the print and then run some tests on the extruder:

Verify that the nozzle temperature is correct stable, and was correct when the issue first occurred. Note, if you have a Revo hot end then on early ones the temperature sensing goes wrong after a while. The good ones have blue thermistor wires, the old ones were white.
Use M569 to check that the driver is in spread cycle mode, not stealthchop
Send M906 and M913 without parameters to check that RRF thinks it is using the correct motor current
Test extruding and retracting at slow and medium speeds, e.g. 1 to 5mm/sec.
Assuming it's not working properly try sending M906 Exxx (where xxx is the correct motor current) and M913 E100 to check whether this fixes it

Exerqtor

@dc42 said in 1LC - extruder randomly stopping/reversing - ≤3.5.0-r1:

@Exerqtor could it be that somehow the motor current has been reduced to a low value, so that the extruder motor is capable of retracting filament but not capable of overcoming nozzle back pressure?

I have no idea where how that could happen, but it sounds like a possible cause. Weird that it wouldn't happen on 3.5.0-b4 though.

It would be helpful if you could wait for the issue to happen, then pause the print and then run some tests on the extruder:

Verify that the nozzle temperature is correct stable, and was correct when the issue first occurred. Note, if you have a Revo hot end then on early ones the temperature sensing goes wrong after a while. The good ones have blue thermistor wires, the old ones were white.

I've got the "new" 60W heater (blue thermistor wires), so that shouldn't be an issue. I haven't seen any dips / fluctuations in the temperatures either (after I got the warranty replacement for the "white wire" variant or 60w from E3D, way before these extrusion issues started). Looking at the temperature chart in DWC everything is rock solid when the issue occurres

Use M569 to check that the driver is in spread cycle mode, not stealthchop
Send M906 and M913 without parameters to check that RRF thinks it is using the correct motor current

Test extruding and retracting at slow and medium speeds, e.g. 1 to 5mm/sec.

Assuming it's not working properly try sending M906 Exxx (where xxx is the correct motor current) and M913 E100 to check whether this fixes it

I will try all thay out some time this weekend! Haven't had the time during the week.

Exerqtor

I just had time to do the testing, updated to 3.5.0-rc2 on everything, sliced one of the "doomed" models again to verify that it wasn't slicer generated (Orca Slicer 1.9.0 stable has launched since last testing), and surely it fails just like it did a month ago 🤣

Here is what you asked for @dc42, first off M122 B121 & M569 P121.0 while laying down the first layer (not paused) with everything working like it should as a "baseline":

M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-rc.2 (2023-12-14 08:58:51)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 3252, free system stack 89 words
Tasks: Move(3,nWait,0.1%,75) HEAT(2,nWait,16.8%,97) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.5%,77) CanClock(5,nWait,0.8%,67) ACCEL(3,nWait,0.0%,53) TMC(2,nWait,136.3%,57) MAIN(1,running,616.0%,316) IDLE(0,ready,0.0%,27) AIN(2,delaying,215.9%,114), total 986.4%
Last reset 08:07:59 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 568.8 steps/mm, ok, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 24943, writes 12, timeouts 3, DMA errors 0, CC errors 0, failedOp 0x6f, steps req 0 done 106290
Moves scheduled 718, completed 716, in progress 1, hiccups 0, segs 14, step errors 0, maxLate 1 maxPrep 320, maxOverdue 1, maxInc 1, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 1.00
Peak sync jitter -2/10, peak Rx sync delay 218, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 23.3, current 24.1, max 24.4
MCU temperature: min 37.8C, current 55.1C, max 55.1C
Last sensors broadcast 0x00000012 found 2 166 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 600447, send timeouts 0, received 264383, lost 0, errs 0, boc 0, free buffers 18, min 18, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 332, adv 35999/109737
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 6, contentions 0, other errors 0
=== Filament sensors ===
Interrupt 5726621 to 0us, poll 4 to 1470us
Driver 0: ok

M569 P121.0
Driver 121.0 runs in reverse, active low enable, mode spreadCycle, ccr 0x00053, toff 3, tblank 0, hstart/hend/hdec 5/0/0, pos 448

And then we have M122 B121 & M569 P121.0 taken while the issue is occurring (not paused):

M122 B121
Diagnostics for board 121:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-rc.2 (2023-12-14 08:58:51)
Bootloader ID: SAMC21 bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 2724, free system stack 89 words
Tasks: Move(3,nWait,0.3%,71) HEAT(2,nWait,0.4%,97) CanAsync(5,nWait,0.0%,54) CanRecv(3,nWait,0.1%,77) CanClock(5,nWait,0.0%,67) ACCEL(3,nWait,0.0%,53) TMC(2,nWait,3.1%,57) MAIN(1,running,91.2%,316) IDLE(0,ready,0.0%,27) AIN(2,delaying,4.9%,114), total 100.0%
Last reset 08:11:58 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 568.8 steps/mm, standstill, SG min 0, read errors 0, write errors 0, ifcnt 12, reads 53546, writes 0, timeouts 0, DMA errors 0, CC errors 0, steps req 0 done 282853
Moves scheduled 2822, completed 2822, in progress 0, hiccups 3, segs 36, step errors 0, maxLate 1 maxPrep 833, maxOverdue 3250, maxInc 3250, mcErrs 0, gcmErrs 0, ebfmin -0.96 max inf
Peak sync jitter -2/10, peak Rx sync delay 226, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 23.9, current 24.0, max 24.3
MCU temperature: min 37.8C, current 56.7C, max 56.9C
Last sensors broadcast 0x00000012 found 2 25 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 4935, send timeouts 0, received 5351, lost 0, errs 0, boc 0, free buffers 18, min 18, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 350, adv 36140/74649
Accelerometer: LIS3DH, status: 00
Inductive sensor: not found
I2C bus errors 0, naks 0, contentions 0, other errors 0
=== Filament sensors ===
Interrupt 5726621 to 0us, poll 9 to 1550us
Driver 0: ok

M569 P121.0
Driver 121.0 runs in reverse, active low enable, mode spreadCycle, ccr 0x00053, toff 3, tblank 0, hstart/hend/hdec 5/0/0, pos 168

~~@Exerqtor could it be that somehow the motor current has been reduced to a low value, so that the extruder motor is capable of retracting filament but not capable of overcoming nozzle back pressure?~~

It would be helpful if you could wait for the issue to happen, then pause the print and then run some tests on the extruder:

Verify that the nozzle temperature is correct stable, and was correct when the issue first occurred. Note, if you have a Revo hot end then on early ones the temperature sensing goes wrong after a while. The good ones have blue thermistor wires, the old ones were white.

Screenshot of the temperature chart during warmup, working print & failing print:
alt text
So i would say that's sufficiently stable😄

~~Use M569 to check that the driver is in spread cycle mode, not stealthchop~~

~~Answered above.~~

Send M906 and M913 without parameters to check that RRF thinks it is using the correct motor current

Before pausing while issue is occuring:

M906
Motor current (mA) - X:1750, Y:1750, Z:850, E:650, idle factor 40%

M913
Motor current % of normal - X:100, Y:100, Z:100, E:100

After pausing while issue is occuring:

M906
Motor current (mA) - X:1750, Y:1750, Z:850, E:650, idle factor 40%

M913
Motor current % of normal - X:100, Y:100, Z:100, E:100

Test extruding and retracting at slow and medium speeds, e.g. 1 to 5mm/sec.

Tried extruding 100mm @ 1mm/sec through paneldue while paused : no issues
Tried extruding 100mm @ 5mm/sec through paneldue while paused : no issues

~~Assuming it's not working properly try sending M906 Exxx (where xxx is the correct motor current) and M913 E100 to check whether this fixes it~~

So yeah, that's that 😂

Exerqtor

@dc42 I suspect that didn't give you any more clues as to whats going on? 😓