Unsolved Duet3 1XD STP/DIR output issues

[too]

Hello Guys,

I have a machine that uses 5 axes, X,Y,Z,E and U.
I used to have a Duet Ethernet with expansion board driving the machine, which worked without issues.

Now I am facing some issues with my upgraded machine:
I have a Duet3 6HC with 5 1XD expansion boards connected, one for each axis. The Step/Direction output is connected to a PLC system.
This is where it gets interesting: While X/Y/Z/E are moving, everything is fine. As soon as also the U axis starts moving, all other axes seem to be stopping for a short period of time (always exactly 30ms) and then continue to move smooth. Once the U axis stops its movement again, the same thing happens again.

If I trace the input signals on my PLC system, I see the the following:
(orange: calculated velocity of X axis, blue: position of X axis)

If I zoom into where the orange acceleration curve drops I can see a "plateau" in the blue position curve. This is also the same for all other currently moving axes:

Looking at the step/direction signals with a scope I see the following just when the short stop happens:

and then afterwards it comes back to nice looking signals:

Initially I thought the issue is because I was using v0.4 1XD expansion boards, so I changed them to v1.0 1XD boards, no change in the printer behavior.

I have tried to remove the E axis from the configuration to see if I am pushing the limits of the CAN bus, but that didn't change anything.
When I remove the U axis from the configuration I see no issues.

I have played around with acceleration and jerk of the U axis, without much success.

I have tried reducing the step/mm values for all 5 axes down to 100 step/mm without any impact.

Here is the relevant parts from my config.g:

; Drives
M569	P20.0	S1						; X forward on 20
M569	P21.0	S1						; Y forward on 21
M569	P22.0	S1						; Z forward on 22
M569	P23.0	S1						; E forward on 23
M569	P24.0	S1						; U forward on 24

M584	X20.0	Y21.0	Z22.0	E23.0	U24.0			; set drive mapping

M92	X100	Y100	Z100	E100	U50			; Set steps per mm
M566	X200 	Y200	Z200	E3000	U3000			; Set maximum instantaneous speed changes (mm/min)
M203	X60000	Y60000	Z1500	E60000	U36000			; Set maximum speeds (mm/min)
M201	X300	Y300	Z100	E500	U5000			; Set accelerations (mm/s^2)

Any ideas what could be going wrong?

Edit:
I have also updated all boards including expansion boards to the latest firmware:

Board: Duet 3 MB6HC (MB6HC)
DSF Version: 3.2.0-rc1
Firmware: RepRapFirmware for Duet 3 MB6HC 3.2-RC1 (2020-12-19)

m115 b24
Duet EXP1XD firmware version 3.2-RC1 (2020-12-19)
m115 b23
Duet EXP1XD firmware version 3.2-RC1 (2020-12-19)
m115 b22
Duet EXP1XD firmware version 3.2-RC1 (2020-12-19)
m115 b21
Duet EXP1XD firmware version 3.2-RC1 (2020-12-19)
m115 b20
Duet EXP1XD firmware version 3.2-RC1 (2020-12-19)

[jay_s_uk]

External drivers will more than likely need the timings set correctly otherwise strange things can happen, especially if moving lots of axis at once.
What external drivers are you using?

[dc42]

Does the problem only occur when all 5 axes and extruders are moving at the same time?

[too]

First of all, thanks for the quick responses

@jay_s_uk no, as long as I have the U axis removed from the configuration everything seems fine. If I add the U axis I start to get the issue.
I am connected to a PLC, not an external driver. The input is a NX-EC0142 card:
https://industrial.omron.eu/en/products/NX-EC0142

@dc42 no, it also seems to be occurring if some of the other axes are in standstill. I will do some data logging today to exactly pinpoint when it happens, but for now it looks like the U axis causes some kind of issue. Let me know if you need more info!

[too]

I have done some more tests and data traces in my PLC software, and it looks like this "interruption" happens not only when the U axis starts or stops, but when any axis starts or stops the movement:
(pink: Y position, green: U position, blue: X position, orange is the respective velocity)

I'll have access to a logic analyzer sometime later today, then I can make a trace of the actual signals coming out of the duet.

[Kulitorum]

@dc42 Any help here?

[dc42]

@Kulitorum, did you manage to get a logic analyser trace?

I recommend you use firmware 3.2RC2 if you are currently using an older version.

[Kulitorum]

Ok - we will update firmware on Monday and if problems percist, we will post a logic analyzer output.

Thank you.

[too]

Hey @dc42, all boards are now updated to the latest 3.2-RC2 firmware and show the same issues as before:
(Red: X velocity, Green: Y velocity, Blue: U velocity)

The Acceleration and Deceleration S-curve is pretty deformed...

I have also had a closer look at the M122 diagnostics report and found the following:

X (B20):	Moves scheduled 266, completed 266, in progress 0, hiccups 518
Y (B21):	Moves scheduled 403, completed 403, in progress 0, hiccups 3699

I don't exactly know what that means, but assume that the hiccups counter should be close to 0?

I have unfortunately ordered the wrong logic analyzer and will have to wait until tomorrow for the right one to arrive, so I will upload the results of the trace sometime tomorrow afternoon.

I am very sure that the issue lies somewhere in the duet 3 side of the machine. I am just guessing, but could it be something with the CAN bus speed? Please let me know if you need any more specific info or files and I will upload them here.

[T3P3Tony]

Hiccups indicate that you are attempting to drive the servos with a higher step rate than the 1XD can handle try a lower step rate to see if this resolves it.

You can see how the CAN bus is performing using the M122 on the Duet 3 mainboard and 1XDs to see if any packets are lost.

[dc42]

@too, did you resolve this?

[Kulitorum]

We replaced the Duet3 with a Duet2 and that runs fine, so we are delivering this machine with a Duet2 and then we can take it up again.

Kulitorum

[too]

Hello @dc42,

as Kulitorum mentioned, I have changed the Duet3 to a Duet2 Ethernet, with the latest 3.2 RC2 firmware.
I have added signal filtering/averaging options in the PLC and the Machine was running definitely smoother than with the Duet3, but still not satisfactory.
So I went ahead and changed the controller to a Duet2 with firmware 2.05, which is the version that all our previous machines are running, and the velocity curves look a lot better now:
(Red: X velocity, Green: Y velocity, Blue: U velocity, Orange: Extruder Velocity)

But that is only in some areas. Right after, the Duet outputs strange start-and-stop patterns instead of smooth movements:
(Red: X velocity, Green: Y velocity, Blue: U velocity, Orange: Extruder Velocity)

I have connected a scope to this and verified that the Duet is actually sending these waves, it is not an issue that is coming from the PLC.
The gcode file also looks like it has no slicing errors or corruptions that may cause this.
(printed file: 20210107_qatarcommissioning.gcode )

Changing the acceleration or jerk values is also not visibly influencing the movements of the machine.
(config.g with the values that were active during the testprint: config (8).g )

Can we set up a meeting during the day tomorrow to discuss this in person rather than through back and forth in the forum? We can even pay for that by the hour if necessary

@T3P3Tony I only saw dropped CAN packages in the M122 report of the Duet3 setup when I had step/mm values larger than I think 500. I usually had all axes set to 200 step/mm.

On the duet2 the M122 report gives me a hiccup counter over 1000 when it gets to this start-and-stop movement, even reducing all step/mm values to 20 for X/Y/U and 10 for Z/E doesn't help.

[Phaedrux]

@too said in Duet3 1XD STP/DIR output issues:

with the latest 3.2 RC2 firmware.

Latest is 3.2 final now by the way.

[T3P3Tony]

hi @Kulitorum @too I have had a look at the gcode and config and done some calculations on max step rate. the values are quite reasonable so surprised you are seeing hiccups. It would be good to discuss in person further, I will email you tomorrow morning.

In the mean time, at what point in the print are you seeing the weird outputs? Also do you see that with RRF 3.2?