many many hiccups (thousands)

[jens55]

@gloomyandy
A one hour test print resulted in 16654 hiccups

m122 b22
Diagnostics for board 22:
Duet TOOL1LC rev 1.1 or later firmware version 3.5.0-rc.3+ (2024-02-21 15:51:16)
Bootloader ID: SAMC21 bootloader version 2.3 (2021-01-26b1)
All averaging filters OK
Never used RAM 2976, free system stack 71 words
Tasks: Move(3,nWait 7,0.7%,83) HEAT(2,nWait 6,0.4%,91) CanAsync(5,nWait 4,0.0%,55) CanRecv(3,nWait 1,0.3%,71) CanClock(5,nWait 1,0.0%,59) ACCEL(3,nWait 6,0.0%,53) TMC(2,delaying,3.7%,53) MAIN(1,running,89.9%,315) IDLE(0,ready,0.0%,27) AIN(2,delaying,5.1%,112), total 100.0%
Owned mutexes:
Last reset 01:10:29 ago, cause: software
Last software reset time unknown, reason: AssertionFailed, available RAM 3392, slot 1
Software reset code 0x0120 ICSR 0x00000000 SP 0x2000415c Task Freestk 129 bad marker
Stack: 00000544 00022ffc 00019b65 20003134 00016cff 20003134 000163d1 20000ed0 00000000 00000001 00008275 200071c8 200071c8 200071e0 00000000 20000f50 00011647 000223b8 00022474 00021ac8 00019b05 200071c8 200071c8 20000f50 000083ed 200071d8 000009c7
Driver 0: pos 0, 830.0 steps/mm, standstill, SG min 0, read errors 3, write errors 1, ifcnt 42, reads 5046, writes 15, timeouts 7, DMA errors 0, CC errors 0, failedOp 0x6a, steps req 0 done 13112142
Moves scheduled 116998, completed 116998, in progress 0, hiccups 16654, segs 40, step errors 0, maxLate 0 maxPrep 490, maxOverdue 414, maxInc 72, mcErrs 0, gcmErrs 0, ebfmin 0.00 max 1.00
Peak sync jitter -1/4, peak Rx sync delay 265, resyncs 0/0, no timer interrupt scheduled
VIN voltage: min 20.3, current 24.4, max 24.5
MCU temperature: min 35.4C, current 52.9C, max 53.6C
Last sensors broadcast 0x00000004 found 1 105 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 86788, send timeouts 0, received 205816, lost 0, errs 1, boc 0, free buffers 18, min 17, error reg 20000
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 540, adv 35670/74677
Accelerometer: LIS3DH, status: 00
I2C bus errors 0, naks 3, contentions 0, other errors 0
=== Filament sensors ===
Interrupt 5726621 to 0us, poll 2 to 3188us
Driver 0: ok

[jens55]

I noticed this line in the m122 b22 output:

VIN voltage: min 20.3, current 24.4, max 24.5

I can't think of any reason why the minimum voltage was 20.3V and I am wondering if this could be the cause for hiccups or possibly the little temperature bumps I see. In the print that showed the low voltage, there was also a situation that nozzle temperature dropped for a while and also increased above the set point for a while which might have been caused by an input voltage fluctuation. This temperature change, unlike the little temperature bumps, was gradual and consistent with normal cool down and heat-up behaviour

Anyway, I am wondering if there is any way to display or keep track of the Vin voltage during a print to see if there is any correlation between input voltage and things that are happening. I also wonder if the hiccup count could be displayed continuously during a print.

BTW, neither the 6HC nor any of the tool boards that did not print anything show this substantial voltage drop (but I don't know if a1LC not being actively used would track this information). I will double check all the connections in the power lines that lead to the tool boards.

[gloomyandy]

@jens55 You might want to run M122 bxx (where xx is the board number you will be using during the print), before you start the print. That will clear the stats if you thn run the M122 again after the print you will see if the voltage has dropped over the print run. It may be that the voltage drop you are seeing was captured as the power supply came on line.

It looks like you have filament sensors attached to the toolboards, are these active at the moment? If so it might be worth trying a print with them disabled. The same goes for pressure advance and input shaping. Note this is just to try and identify what may be causing the hiccups, I'm not saying that these should not work.

[jens55]

@gloomyandy
The m122 b22 command was run before the test print and after.
Yes, I have filament sensors on the tool boards. I can't see how these would affect things since the status did not change during the print but I will pull the plug on the sensor, comment out input shaping in config.g and disable pressure advance. I will report back after my next print.
Thanks!

[gloomyandy]

@jens55 Monitoring the filament sensor will be adding to the load on the toolboard mcu, which may cause delays in processing the stepper commands. It's really just about trying to figure out possible causes at this point.

[jens55]

@gloomyandy

I ran the same model with three different tools. All filament sensors were disabled, no pressure advance was used and no input shaping was used.

M122 b23 - hiccups 6809, min Vin 23.1 max Vin 24.2
M122 b22 - hiccups 7561, min Vin 23.5 max Vin 24.5
M122 b21 - hiccups 776, min Vin 23.1, max Vin 24.1

No idea why the one tool had only a fraction of the hiccups as the other two (but still too many)

Edit:

b21 uses an LGX extruder, 22 and b23 both use a BMG clone extruder. The LGX runs at a considerably slower step rate.

m92
Steps/mm: X: 100.000, Y: 100.000, Z: 3200.000, U: 30.578, E: 404.800:830.000:830.000:830.000

[gloomyandy]

@jens55 Are those numbers for the hiccups for the same model that you used previously (when you had PA and the filament monitors enabled)? In the previous run above it was showing 16654 hiccups.

[jens55]

@gloomyandy
No, it was a much simpler model. I could try the other print to see if turning off the various features made a difference.
Oh, I also forgot to mention that when I ran the three tests, I did not feed filament (the printer did not know that there was no filament present)

The model with 16654 hiccups actually printed stuff so if you want me to run that previous print, I will run filament as well.

Edit: Thinking about it, I can't run the exact same model as I need to re-slice it and it no longer exists (it was a bunch of shims which I have the files for but there were multiple shims which I would need to put together and the model would then be slightly different)

The 16654 hiccups job was for 20? little pieces, the 6809/7561/776 hiccups model was a single larger object with a substantially simpler tool path.
Both jobs ran for roughly an hour.
I guess I could turn on everything I turned off and run the simpler job to see if it shows a significant difference in the hiccups while everything was turned back on.

[gloomyandy]

@jens55 It might be interesting to see what the hiccup count is if you print the simplified model with PA and filament monitor enabled. If the hiccups go up that may provide a clue as to what is going on.

But really at this stage I think we need @dc42 to take a look at this and see if he has any thoughts/comments.

[jens55]

@gloomyandy

I will run the simpler job with everything turned on and post the results.

[jens55]

@gloomyandy

Enabling all the previously turned off features resulted in a print with 7170 hiccups.

[dc42]

@jens55 please provide the GCode file you were printing that provoked this large number of hiccups. I see that you have already provided your config.g file.

[jens55]

@dc42
The file is too large (1.3 MB) for the forum. I do not have access to any sort of file storage/exchange service. I zipped the file but the forum does not permit .zip files. I renamed the .zip file as a .bin file and managed to upload it. Hopefully nothing was messed up in the process.
flip stop mounting block.bin

Please note that the issue is not limited to this particular file. An interesting side comment - I did a m122 b22 shortly after the start of one of the various test prints. The printer had just laid down a couple of loops of skirt and had just started the actual print and it already had something like 5 hiccups. I can't recall if I tried printing the same gcode with one of my Duet2WIFI based printers but those printers have never given me issues beyond a few hiccups at the most. I would be happy to run the same gcode on a Duet2 based printer if that would help.

[jens55]

@dc42, have you had a chance to look at the gcode file I provided?
Anything else I can try to figure out what causes the hiccups?

[jens55]

A related question: Are hiccups strictly related to stepper motor movements?
The reason I am asking is that I am currently about 1.5 hours into a 2 hr print job, I have roughly 64000(!!!!) hiccups on the 1LC and zero hiccups on the 6HC. All x/y/z movement in this printer is controlled by the 6HC and only the extruder movement is done from the tool board.
At the same time, the diagnostic on the 6HC shows a steady supply voltage (Vin) whereas the 1LC reports Vmin of around 7(!!!) volts and Vmax of around 30(!!!) volts. While I suppose the 7 volt figure could in theory be produced by a bad connection, the 30 volt reading is pure nonsense seeing that the 6HC is running off the same power supply and shows a steady voltage.

[oliof]

@jens55 can we have a photo of your 1LC including full view of the voltage wiring? Just curious whether this might be an issue with insufficient strain relief.

[jens55]

@oliof
I have taken a number of pictures of the tool that I am currently using but they are all too large to attach. I am trying to reduce the file sizes but no luck yet.
Everything is buttoned down pretty securely and remember that the errors happen with different tools. Since each tool has a different wiring path, I can't see how this could be a wiring issue from the tool distribution board to the individual tool. Of course nothing moves from the power supply to the tool distribution board. IIRC, the wiring is all highly flexible silicone wiring but again, if it was a wiring issue then it shouldn't occur on multiple tools.
With the 6HC showing a stable voltage, I don't understand how the tool board can see a 7 to 30V range for the input voltage.
I didn't take the 1LC out of it's mount to show it in detail. If you need that I can disassemble things and take another picture.

[jens55]

pictures:

[jens55]

more pictures:

[jens55]

Another tidbit of information I have discovered - the number of hiccups is directly proportional to the extruder step rate (or number of steps taken during a print).
I decided to try and print with a much lower step rate and a print that had previously generated thousands of hiccups ended up with 105 hiccups. The steps/mm for this extruder should have been 830 but I ran the print at 32 steps/mm.
I don't know what it means but this is the first time I have been able to correlate the hiccups to something.