Missing Steps - Cant Print SpreadCycle StealthChop tuning help

droftarts

Seems like we might, finally, have nailed this one. After testing all day today, @dc42 noticed the longest loop time in @carcamerarig's M122 reports was very long, often over 100ms. Looking through the M122 reports taken during printing in this thread, he noticed the SD card read times were very long, too, eg:

SD card longest read time 91.0ms, write time 5.2ms, max retries 0

By comparison, my loop time and SD card longest read, running the same code:

Slowest loop: 12.28ms; fastest: 0.10ms
...
SD card longest read time 7.9ms, write time 0.0ms, max retries 0

dc's current theory is that either the SD card is faulty, or the SD card interface is faulty. We're waiting for @carcamerarig to let us know if replacing the SD card helps.

Ian

dc42

To add to that, we also now have a firmware build that doesn't have these problems despite the excessively slow SD card read performance.

droftarts

All testing I was doing was with 0.9° motors (48mm long NEMA 17 motors, one from E3D, one from Wantai) with similar specs to LDO motors, running at x32 micro stepping with interpolation. These were quiet in stealthchop, and noisier in spreadcycle, but not noticeably noisier than 1.8° motors.

I kept jerk settings high at 900mm/m (15mm/s) with Marlin jerk policy, acceleration at 2000mm/s^2, speeds up to 18,000mm/m (300mm/s). Motor current was 1000mA on X and Y, though these are 1.7A motors.

With these setting (which are probably close to the limit my poor old machine can cope with!) I didn’t have any banging problems, though as noted in previous posts, we think these are actually related to an SD card problem.

I also didn’t have problems in hybrid mode, so long as changeover speed was low, but could replicate the “micro banging” @carcamerarig experienced in gyroid fill if set around 35mm/s, which is to be expected as this is the speed gyroid fill was being printed at, so X and Y axes speed up and slow down through the changeover speed on each gyroid curve.

One caveat is that I was using endstops for homing, not sensorless homing. We did get that working on @carcamerarig’s machine, I think reliably, though with 1.8 motors. I’ll test sensorless homing my setup when I can.

I’ll update the documentation with best practices for all of that we’ve learnt in this process!

Ian

dc42

All, we have tracked down this issue. It affects Duet 3 Mini boards running all released firmware versions when a heater output is configured to a low PWM frequency - for example, when the bed heater is set to run at 10Hz because it is driven via a SSR. Unfortunately, there is a synchronisation delay between the heater management task and the PWM port, which causes the heater task to hog the CPU for up to one cycle at the PWM frequency, and that occasionally causes the movement system to fail to finalise moves before they are due to be executed.

We already have Duet 3 Mini firmware with improved motion planning that works around this, and I expect to fix the underlying issue shortly. The fix will be in the 3.3beta3 release, which I hope to make available within the next few days. Meanwhile, for anyone not using SSRs to drive heaters, the workaround is to use a higher PWM frequency, e.g. 100Hz or 500Hz.

Special thanks to @carcamerarig, who has worked closely with us over the last few days to help us get to the bottom of this.

oc_geek

@dc42 good catch !

that might also be the issue behind slicer output .g files not completing the upload when the heater is turned on. They reach 100% but then start over again and again...
thanks

carcamerarig

Thanks to everyone involved in this, there are a few of you LOL made especially testing for some with my lack of knowledge of RRF, Gcode and 3d printing in general. I've learned a lot in a short space of time and glad I didn't quit on the mini, I'm especially impressed with the level of support from the Duet team to find the cause and then produce a number fixed fw / workarounds until resolved.
Looking forward to 3.3b3 in few days

ctilley79

@dc42 You owe @carcamerarig a bottle of Macallan lol.

Phaedrux

@carcamerarig Thanks for sticking with it.

Argo

@dc42

Oh that explains why I had trouble with slow downs at 10Hz and swiftly switched back to 100Hz because the slow downs were already obvious during the probing procedure.

https://forum.duet3d.com/topic/22142/slowdowns-pauses-pid-algorithm-and-pwm-freq?_=1619033412104

I already suspected it had something to do with internal cycles as Prusa explained that they also had issues with that.
https://blog.prusaprinters.org/dev-diary-2-how-we-made-the-heatbed-silent_30946/

dc42

@oc_geek said in Missing Steps - Cant Print SpreadCycle StealthChop tuning help:

@dc42 good catch !

that might also be the issue behind slicer output .g files not completing the upload when the heater is turned on. They reach 100% but then start over again and again...
thanks

Yes. I did some tests on the old firmware with 5Hz PWM frequency and found that it completely disrupted file uploading. Anyway, a 1-line change has fixed the underlying issue, so we're hoping to release 3.3beta3 tomorrow.

fulg

I was unable to contribute to the troubleshooting efforts due to a missing board, but now that I have one again I just wanted to confirm that 3.3b3 has also completely solved this issue for me.

Thank you @dc42 for sticking with the thread, which often wandered outside of the subject at hand

Cheers,
Ben.

carcamerarig

@fulg Well done, I think most issues have been resolved in this release which pretty cool eh. Not had a spare moment to print anything yet, hoping I will later tonight.

MartinNYHC

Just replaced a Duet 2 Wifi with a Mini 5 and upgraded to 3.3-rc1. Nothin else changed on the printer. Unfortunately it looks like i still have this issue. As described above after a "bang" there's a layer shift. Any ideas please?

; General preferences
G90                                                    ; send absolute coordinates...
M83                                                    ; ...but relative extruder moves
M550 P"Voron0"                                         ; set printer name
M669 K1                                                ; select CoreXY mode

; Drives
M569 P0.0 S1 D3 V0	                                   ; physical drive 0 goes forwards
M569 P0.1 S1 D3 V0
M569 P0.2 S0	                                      ; physical drive 2 goes forwards
M569 P0.3 S0                                            ; physical drive 3 goes forwards
M584 X0.0 Y0.1 Z0.2 E0.3                                       ; set drive mapping
M350 X16 Y16 Z16 E16 I1    	                           ; configure microstepping with interpolation

M92 X80.00 Y80.00 Z320.00 E711.97  		               ; set steps per mm
M566 X600.00 Y600.00 Z50.00 E2000.00    		       ; set maximum instantaneous speed changes (mm/min)
M203 X20000.00 Y20000.00 Z1000.00 E5000.00      	   ; set maximum speeds (mm/min)
M201 X2000.00 Y2000.00 Z200.00 E1000.00	               ; set accelerations (mm/s^2)
M906 X400 Y400 Z600 E400 I75	                       ; set motor currents (mA) and motor idle factor in per cent
M84 S120                                               ; Set idle timeout
M572 D0 S0.061										   ; Pressure advance


; Axis Limits
M208 X3 Y3 Z0 S1                                       ; set axis minima
M208 X118 Y118 Z110 S0                                 ; set axis maxima

; Endstops
M574 X2 S1 P"io1.in"                    				; configure active-high endstop for high end on X via pin io1.in
M574 Y2 S1 P"io2.in"                    				; configure active-high endstop for high end on Y via pin io2.in
M574 Z1 S1 P"io3.in"                    				; configure active-high endstop for high end on Z via pin io3.in

; Heaters
M308 S0 P"temp0" Y"thermistor" T100000 B4092       		; configure sensor 0 as PT1000 on pin temp0
M950 H0 C"out1" T0                      				; create bed heater output on out0 and map it to sensor 0
M307 H0 B0 S1.00                        				; disable bang-bang mode for the bed heater and set PWM limit
M140 H0                                 				; map heated bed to heater 0
M143 H0 S120                            				; set temperature limit for heater 0 to 120C
M308 S1 P"temp1" Y"thermistor" T100000 B3950        	; configure sensor 1 as PT1000 on pin temp1
M950 H1 C"out2" T1                      				; create nozzle heater output on out1 and map it to sensor 1
M307 H1 B0 S1.00                        				; disable bang-bang mode for heater  and set PWM limit
M143 H1 S280                            				; set temperature limit for heater 1
M308 S2 P"temp2" A"Chamber" Y"thermistor" T100000 B3950

; Fans
M950 F0 C"out3" Q500                    				; create fan 0 on pin out3 and set its frequency
M106 P0 C"Bed" S0 H-1                          				; set fan 0 value. Thermostatic control is turned off
M950 F1 C"out4" Q2000                    				; create fan 1 on pin out4 and set its frequency
M106 P1 C"Hotend" L0.70 X0.70 H1 T70 S0.70 B5           ; set fan 1 value. Thermostatic control is turned on
M950 F2 C"out5"							 			   ; create fan 2 on pin fan2
M106 P2 C"Chamber" S0 H-1								   ; set fan 2 name and values
M950 F3 C"out6"							 			   ; create fan 2 on pin fan2
M106 P3 C"Light" S1 H-1								   ; set fan 2 name and values

; Tools
M563 P0 D0 H1 F0                                       ; define tool 0
G10 P0 X0 Y0 Z0                                        ; set tool 0 axis offsets
G10 P0 R0 S0                                           ; set initial tool 0 active and standby temperatures to 0C

; Miscellaneous
M575 P1 S0 B57600                                      ; enable support for PanelDue
M501
T0                                                     ; select first tool

05/05/2021, 11:22:32 	M122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.3RC1 (2021-05-01 09:59:28) running on Duet 3 Mini5plus Ethernet (SBC mode)
Board ID: J4FFG-A196U-D65J0-40KM4-MG03Z-HG970
Used output buffers: 1 of 40 (40 max)
=== RTOS ===
Static ram: 102476
Dynamic ram: 92820 of which 0 recycled
Never used RAM 45552, free system stack 114 words
Tasks: SBC(resourceWait:,4.2%,294) HEAT(delaying,0.0%,320) Move(notifyWait,0.4%,276) CanReceiv(notifyWait,0.0%,943) CanSender(notifyWait,0.0%,357) CanClock(delaying,0.0%,341) TMC(notifyWait,0.7%,114) MAIN(running,93.9%,497) IDLE(ready,0.0%,29) AIN(delaying,0.8%,264), total 100.0%
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:39:02 ago, cause: power up
Last software reset at 2021-05-05 09:54, reason: User, none spinning, available RAM 51896, slot 1
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0400f000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a
Error status: 0x04
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU revision 3, ADC conversions started 2342835, completed 2342835, timed out 0, errs 0
Step timer max interval 1490
MCU temperature: min 37.7, current 43.2, max 43.7
Supply voltage: min 23.7, current 24.3, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/0, heap memory allocated/used/recyclable 2048/8/8, gc cycles 0
Driver 0: position 8800, standstill, SG min/max 0/436, read errors 0, write errors 0, ifcnt 20, reads 57528, writes 20, timeouts 0, DMA errors 0
Driver 1: position -7200, standstill, SG min/max 0/434, read errors 0, write errors 0, ifcnt 20, reads 57528, writes 20, timeouts 0, DMA errors 0
Driver 2: position 4224, standstill, SG min/max 0/10, read errors 0, write errors 0, ifcnt 18, reads 57530, writes 18, timeouts 0, DMA errors 0
Driver 3: position 0, standstill, SG min/max 0/44, read errors 0, write errors 0, ifcnt 16, reads 57531, writes 16, timeouts 0, DMA errors 0
Driver 4: position 0, standstill, SG min/max 0/0, read errors 0, write errors 0, ifcnt 9, reads 57539, writes 9, timeouts 0, DMA errors 0
Driver 5: position 0, assumed not present
Driver 6: position 0, assumed not present
Date/time: 2021-05-05 11:22:31
Cache data hit count 3725941281
Slowest loop: 158.94ms; fastest: 0.10ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 139363ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 26047, completed moves 26047, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 33], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 20985, send timeouts 20982, received 0, lost 0, longest wait 0ms for reply type 0, peak Tx sync delay 0, free buffers 17 (min 17)
Last cancelled message type 4514 dest 127
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 4ms ago
RX/TX seq numbers: 36438/36438
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x10c34
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.3-rc1
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 44.04
Codes per second: 19.47
Maximum length of RX/TX data transfers: 3460/1300

dc42

@martinnyhc I suspect that your high maximum speeds (20000 mm/min in M203) are not compatible with running permanently in stealthChop mode. stealthChop doesn't work when the driver supply voltage is too low to overcome the inductive and rotational back emf of the motor. You can use the back EMF calculator at https://www.reprapfirmware.org/ to get an estimate of the speed at which this will occur.

MartinNYHC

@dc42 Used the calculator and used some conserative settings:

M92 X80.00 Y80.00 Z320.00 E711.97  		               ; set steps per mm
M566 X600.00 Y600.00 Z50.00 E2000.00    		       ; set maximum instantaneous speed changes (mm/min)
M203 X8000.00 Y8000.00 Z1000.00 E5000.00 	     	   ; set maximum speeds (mm/min)
M201 X2000.00 Y2000.00 Z200.00 E1000.00	               ; set accelerations (mm/s^2)
M906 X400 Y400 Z600 E400 I75	                       ; set motor currents (mA) and motor idle factor in per cent
M84 S120                                               ; Set idle timeout
M572 D0 S0.061										   ; Pressure advance

The result is better in terms of the amount of layer shifts. But it still happened two times during the print. With the Duet 2 Wifi I never configured any Stealthchop/Spreadcycle settings but due to the other drivers on the Mini 5 I had to because steppers sounded terrible.

Is there anything else I can try? Thanks for your help in advance.

droftarts

@martinnyhc said in Missing Steps - Cant Print SpreadCycle StealthChop tuning help:

With the Duet 2 Wifi I never configured any Stealthchop/Spreadcycle settings

That's because the TMC2660 driver chips on Duet 2 WiFi only have spreadcycle.

Most noticeable for noise is spreadcycle at standstill. To test if your missed steps are an issue with stealthchop, you can set the drivers to spreadcycle only when moving, and stealthchop when the are stationary. To do this, set the V parameter in M569 for X and Y to a relatively high number, something like V2000, then send M569 P0 and check the 'tpwmthrs' response. When I do this, I get 'tpwmthrs 2000 (2.3 mm/sec)', but it depends on your microstepping and steps per mm. Adjust the V parameter until it is somewhere between 5 and 10mm/s. Use this speed as it's below your M566 instantaneous speed setting of X600 Y600 (which is 10mm/s).

Alternatively, switch it to spreadcycle for a test with M569 P# D2 (where # is the driver number).

Ian

MartinNYHC

@droftarts Already tried D2. The problem is that as soon as I use D2 or D3 without a V value or V higher than 0 both steppers are sound like they are going to explode in a few seconds. And beside of that i cannot get rid of the layer shifts completely.

droftarts

@martinnyhc What steppers are you using on X and Y?

Ian

MartinNYHC

@droftarts 14HS17-0504S

droftarts

@martinnyhc These? https://www.omc-stepperonline.com/nema-14-bipolar-1-8deg-23ncm-32-6oz-in-0-5a-7-5v-35x35x42mm-4-wires.html

The inductance and phase resistance strike me as being very high:
Phase Resistance: 15ohms
Inductance : 26mH ± 20%(1KHz)

High inductance is usually bad for high speed; see https://duet3d.dozuki.com/Wiki/Choosing_and_connecting_stepper_motors#Section_Inductance
When the bangs happen, is it during fast travel moves? But your Duet 2 coped fine with the same settings?

If yes, I think @dc42 will need to take a deeper look at this.

Ian