Duet sometimes really slow? - I2C error or?

dc42

@gizmotronx5000 said in Duet sometimes really slow? - I2C error or?:

@dc42 I have noticed that the error will start for me sometimes after touching the printer. I do not have ground wires on the casings of my stepper motors right now. Occasionally I will get a static shock and the error occurs. (Can't confirm 100% that this is directly the cause, but it seems possible)

When I am working on the printer it is much more likely to occur than when I am just letting it run.

Thanks for the information. Static charge is very disruptive of electronic circuits. the question is whether it is causing:

An i2C protocol error, which it may be possible to recover from.
The i2C interface in the Duet main processor to lock up, which it may be possible to recover from by resetting the interface.
The i2C interface in the SX1509B to lock up, which can't be recovered from other than by a reset.

dc42

@tbs said in Duet sometimes really slow? - I2C error or?:

@dc42: You offered to send me a finished plug. Can you please send me the ready-made one? I can reproduce the error. In the last 6 prints over 2.5 days the error were occured every time. Than we can see of the plug works.
How can I send you my address?

Use the Chat facility to send me a message containing your address.

TBS

@dc42 said in Duet sometimes really slow? - I2C error or?:

@tbs said in Duet sometimes really slow? - I2C error or?:

@dc42: You offered to send me a finished plug. Can you please send me the ready-made one? I can reproduce the error. In the last 6 prints over 2.5 days the error were occured every time. Than we can see of the plug works.
How can I send you my address?

Use the Chat facility to send me a message containing your address.

@dc42: It is not possible for me to start a chat with you. Should I send you a Email to info@duet3d.com with my address?

dc42

@tbs, I'll email you.

Edgars Batna

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

deckingman

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Edgars Batna

@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Well, I wouldn't post if I wasn't dead sure.

I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551

I get the M122 as above plus slow movements. The machine should not resume printing like that...

deckingman

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Well, I wouldn't post if I wasn't dead sure.
I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551
I get the M122 as above plus slow movements. The machine should not resume printing like that...

No need to get stroppy. It's just that the rest of us get slow movement of all axes, not just Z so there is no way that for the rest of us, it could trash the machine in the way you describe. That's why I asked if you were getting the same problem because what you describe about the XY or E axes moving as normal is not something that anyone else has experienced.

Anyway, If you look at DC42's comment on 15th Jan he said quote:

"It appears that for those few DueX users who encounter these issues, something is disrupting the I2C communications between the two boards. Unfortunately the I2C protocol does not include error detection (other than an indication that no recipient accepted the data) or an error recovery protocol".

So I guess that means it's difficult for the firmware to know that the I2C communication has been disrupted and thus makes it difficult to implement any sort of fall back routine. Not my field of expertise though - just surmising.

The resistor fix detailed above might be what you need. Since I implemented it on my machine, I haven't had any reoccurrences of the problem so that's a promising sign (although it's only been about 3 weeks).

Having said that, all my axes are affected, not just selective ones as in your case. In my case, I have all 5 extruders connected to the Duex 5 and when I get the pauses, all the extruders stop as well as all the XYUV and Z motors.

Edgars Batna

@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:
@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Well, I wouldn't post if I wasn't dead sure.
I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551
I get the M122 as above plus slow movements. The machine should not resume printing like that...
No need to get stroppy. It's just that the rest of us get slow movement of all axes, not just Z so there is no way that for the rest of us, it could trash the machine in the way you describe. That's why I asked if you were getting the same problem because what you describe about the XY or E axes moving as normal is not something that anyone else has experienced.

Sorry, I'm surrounded by failing machines, so it's gloomy here.

All axes are affected for me. Maybe I wasn't clear enough, but it wasn't clear if just Z had stopped working entirely or was just pausing with the rest of the printer. The whole printer was pausing. The firmware shouldn't be tolerant of thousands of I2C errors, tho. After one thousand it should stop the machine.

deckingman

Sorry to resurrect this very old thread and I'm also sorry to have to report that I've had this I2C error occur again on me with the resistors fitted and which were showing so much promise.

This is the first time it has happened since I fitted the resistors on 23rd January as detailed a few posts up.

I don't how or if this can be of any relevance but the printer has been powered down and idle for a couple of weeks. This has been the sequence of events on a few occasions when I have experienced this I2C problem and whilst I can't see any reason why a period of inactivity could be relevant, it is too much of a coincidence to ignore. The only possible explanation I can think of, is that after a period of inactivity some sort of mechanical "sticksion" occurs so when the first move is attempted, the motors draw a higher than normal current and it is that which somehow triggers the I2C errors? Or maybe just pure coincidence is a more plausible explanation.

Anyway, the sequence of events leading up to this latest occurrence was as follows:
Printer turned on. A gcode file was uploaded.This file was then selected to print. My start gcode calls a macro which does the following.
Heat the bed to 40 degC, then when that temperature is reached, run homeall.g. It was during this homing that I noticed the problem of multi-second pauses between moves. XYU and V all homed as expected and it was while homing Z that the pauses started and continued for the rest of the macro, which basically moves the head to the rear of the bed and waits for all temperatures to stabilise.

I ran M122 which reported 5000+ I2C send and finish timeouts

As on every occasion in the past, cycling the power to the printer restored normality and as I write this, it is quite happily printing the part that I attempted earlier.

So it seems that the resistors may not be the answer - or at least the whole answer.

dc42

Thanks for the report. I'll try to find a way to provoke I2C errors on my bench setup.

fulg

I am having similar problem. Printer wiring is fine, no issue with the crimps either (which is the usual suspect). I am randomly getting this "motor phase A may be disconnected", not always from the same driver, and sometimes from multiple drivers at once (last occurrence was drivers 6 and 7 simultaneously on a Duex5).

When this happens (usually 3-4 hours into a print), there is a ~5 second pause between each motion and the occasional message about motor phase popping up. Naturally this ruins the print in progress. Rebooting the firmware (clicking the Emergency Stop button in DWC or the PanelDue) solves the issue for the next print.

I have attached my config and output from M122 after the latest occurrence.

config.g
diagnostics.txt

I am not sure if this line is interesting, I think I see an underrun error?

=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 1

Veti

I2C nak errors 0, send timeouts 511032, receive timeouts 0, finishTimeouts 511032

your wiring to the duex might be insufficient.

fulg

@veti Interesting, I did not notice this. The Duet/Duex combo were installed over a year ago and the wiring has never moved or changed. I am not aware of IDC connections degrading with time. As per the installation instructions, both boards share the same GND wire from the PSU.

The first time the error was reported on a Duet driver, not Duex (I did not think of grabbing a diagnostics log at that time).

I tried a 3-hour print yesterday in the same conditions and it completed without issues.

Veti

@fulg said in motor phase A may be disconnected reported by driver(s) 0 1 2:

, both boards share the same GND wire from the PSU.

whats the diameter of the GND wire?

fulg

@veti 20AWG silicone wire. I doubt this would have lasted over a year of printing without issues if it was the source of the problem. I2C does not go through that GND wire.

There is someone else on Discord with the same problem, he doesn't have the I2C timeouts but he does have the random failure reported from multiple drivers and the 5-second delays between motion.

fulg

This just happened again, this time I was sitting in front of the printer when it happened. First the pauses, then eventually (after a couple of minutes) the motor phases warning. This time it was phase B for drivers 1 and 8.

This time I managed to capture the M122 immediately as the problem happened (while the print was slowly progressing).

diagnostics.txt

Lots of timeout errors...
I2C nak errors 0, send timeouts 511032, receive timeouts 0, finishTimeouts 511032

I have also captured the behavior on video, before receiving my first motor phase warning:
https://www.youtube.com/watch?v=w1QNw_RJozM

Note that as I write this, the printer is again working fine after clicking the emergency stop button and starting another print. I did not even leave enough time for the bed to cool down from the print that failed.

wilriker

@fulg said in motor phase A may be disconnected reported by driver(s) 0 1 2:

I2C does not go through that GND wire.

No, it doesn't. But a degraded GND connection leads to I2C errors. It is recommended to use the thickest possible (12-14 AWG) solid core wire with the shortest possible length.

I2C issues are most certainly the cause for the slow/stop motion.

Also I have a hunch that since the "motor phase A/B disconnected" warning will only be issued if this state persists at least 500ms I2C timeout issues could confuse/influence the time-measurement. But that is more of a wild guess because I am everything but familiar how the code for I2C interacts with the rest of the code.

fulg

So I have changed my wiring from 20AWG to a single 16AWG pair (connected exactly like the documentation, a single pair from the PSU and short wires from the Duet to the Duex). The behavior has not changed, the wiring was not to blame.

Reproducing this is easy, leave the printer with the motors energized for 24-48 hours. Then eventually the I2C timeouts will take over and pause movements during a print, essentially ruining it. After that I get the random driver failures, which are likely caused by the I2C timeouts and not the root cause of the problem.

One thing that might be important, my printer uses 4 motors for Z, and all four are on the Duex. XYE are on the Duet, the rest are unused. So at any given time, 7 steppers are energized. Perhaps the recent rewrites in 2.02/2.03 have some unintended side effects?

The printer is not a new build, I have been printing for hundreds if not thousands of hours with RRF without problems. I will try going back to earlier RRF releases...

I have attached the new diagnostics output:
diagnostics.txt

You can find a complete copy of my config here.

deckingman

@fulg said in motor phase A may be disconnected reported by driver(s) 0 1 2:

.......................................The printer is not a new build, I have been printing for hundreds if not thousands of hours with RRF without problems. I will try going back to earlier RRF releases..............................

It seems that unfortunately you too are experiencing these I2C errors. There are quite a few threads about this, the latest one (apart from this one) is here https://forum.duet3d.com/topic/10313/printer-pausing-between-commands/24. See @dc42s post date 07 May 2019, 9:13 and try that.

From what I have been able to discover, the earliest thread regarding this issue is from around July 2018. There are no threads or posts reporting I2C errors prior to that date. Did you by any chance change the firmware around the time you started seeing these problems? If so, do you happen to know what version you were one before?