Duet sometimes really slow? - I2C error or?

deckingman

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Edgars Batna

@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Well, I wouldn't post if I wasn't dead sure.

I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551

I get the M122 as above plus slow movements. The machine should not resume printing like that...

deckingman

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Well, I wouldn't post if I wasn't dead sure.
I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551
I get the M122 as above plus slow movements. The machine should not resume printing like that...

No need to get stroppy. It's just that the rest of us get slow movement of all axes, not just Z so there is no way that for the rest of us, it could trash the machine in the way you describe. That's why I asked if you were getting the same problem because what you describe about the XY or E axes moving as normal is not something that anyone else has experienced.

Anyway, If you look at DC42's comment on 15th Jan he said quote:

"It appears that for those few DueX users who encounter these issues, something is disrupting the I2C communications between the two boards. Unfortunately the I2C protocol does not include error detection (other than an indication that no recipient accepted the data) or an error recovery protocol".

So I guess that means it's difficult for the firmware to know that the I2C communication has been disrupted and thus makes it difficult to implement any sort of fall back routine. Not my field of expertise though - just surmising.

The resistor fix detailed above might be what you need. Since I implemented it on my machine, I haven't had any reoccurrences of the problem so that's a promising sign (although it's only been about 3 weeks).

Having said that, all my axes are affected, not just selective ones as in your case. In my case, I have all 5 extruders connected to the Duex 5 and when I get the pauses, all the extruders stop as well as all the XYUV and Z motors.

Edgars Batna

@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:
@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Well, I wouldn't post if I wasn't dead sure.
I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551
I get the M122 as above plus slow movements. The machine should not resume printing like that...
No need to get stroppy. It's just that the rest of us get slow movement of all axes, not just Z so there is no way that for the rest of us, it could trash the machine in the way you describe. That's why I asked if you were getting the same problem because what you describe about the XY or E axes moving as normal is not something that anyone else has experienced.

Sorry, I'm surrounded by failing machines, so it's gloomy here.

All axes are affected for me. Maybe I wasn't clear enough, but it wasn't clear if just Z had stopped working entirely or was just pausing with the rest of the printer. The whole printer was pausing. The firmware shouldn't be tolerant of thousands of I2C errors, tho. After one thousand it should stop the machine.

deckingman

Sorry to resurrect this very old thread and I'm also sorry to have to report that I've had this I2C error occur again on me with the resistors fitted and which were showing so much promise.

This is the first time it has happened since I fitted the resistors on 23rd January as detailed a few posts up.

I don't how or if this can be of any relevance but the printer has been powered down and idle for a couple of weeks. This has been the sequence of events on a few occasions when I have experienced this I2C problem and whilst I can't see any reason why a period of inactivity could be relevant, it is too much of a coincidence to ignore. The only possible explanation I can think of, is that after a period of inactivity some sort of mechanical "sticksion" occurs so when the first move is attempted, the motors draw a higher than normal current and it is that which somehow triggers the I2C errors? Or maybe just pure coincidence is a more plausible explanation.

Anyway, the sequence of events leading up to this latest occurrence was as follows:
Printer turned on. A gcode file was uploaded.This file was then selected to print. My start gcode calls a macro which does the following.
Heat the bed to 40 degC, then when that temperature is reached, run homeall.g. It was during this homing that I noticed the problem of multi-second pauses between moves. XYU and V all homed as expected and it was while homing Z that the pauses started and continued for the rest of the macro, which basically moves the head to the rear of the bed and waits for all temperatures to stabilise.

I ran M122 which reported 5000+ I2C send and finish timeouts

As on every occasion in the past, cycling the power to the printer restored normality and as I write this, it is quite happily printing the part that I attempted earlier.

So it seems that the resistors may not be the answer - or at least the whole answer.

dc42

Thanks for the report. I'll try to find a way to provoke I2C errors on my bench setup.

fulg

I am having similar problem. Printer wiring is fine, no issue with the crimps either (which is the usual suspect). I am randomly getting this "motor phase A may be disconnected", not always from the same driver, and sometimes from multiple drivers at once (last occurrence was drivers 6 and 7 simultaneously on a Duex5).

When this happens (usually 3-4 hours into a print), there is a ~5 second pause between each motion and the occasional message about motor phase popping up. Naturally this ruins the print in progress. Rebooting the firmware (clicking the Emergency Stop button in DWC or the PanelDue) solves the issue for the next print.

I have attached my config and output from M122 after the latest occurrence.

config.g
diagnostics.txt

I am not sure if this line is interesting, I think I see an underrun error?

=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 1

Veti

I2C nak errors 0, send timeouts 511032, receive timeouts 0, finishTimeouts 511032

your wiring to the duex might be insufficient.

fulg

@veti Interesting, I did not notice this. The Duet/Duex combo were installed over a year ago and the wiring has never moved or changed. I am not aware of IDC connections degrading with time. As per the installation instructions, both boards share the same GND wire from the PSU.

The first time the error was reported on a Duet driver, not Duex (I did not think of grabbing a diagnostics log at that time).

I tried a 3-hour print yesterday in the same conditions and it completed without issues.

Veti

@fulg said in motor phase A may be disconnected reported by driver(s) 0 1 2:

, both boards share the same GND wire from the PSU.

whats the diameter of the GND wire?

fulg

@veti 20AWG silicone wire. I doubt this would have lasted over a year of printing without issues if it was the source of the problem. I2C does not go through that GND wire.

There is someone else on Discord with the same problem, he doesn't have the I2C timeouts but he does have the random failure reported from multiple drivers and the 5-second delays between motion.

fulg

This just happened again, this time I was sitting in front of the printer when it happened. First the pauses, then eventually (after a couple of minutes) the motor phases warning. This time it was phase B for drivers 1 and 8.

This time I managed to capture the M122 immediately as the problem happened (while the print was slowly progressing).

diagnostics.txt

Lots of timeout errors...
I2C nak errors 0, send timeouts 511032, receive timeouts 0, finishTimeouts 511032

I have also captured the behavior on video, before receiving my first motor phase warning:
https://www.youtube.com/watch?v=w1QNw_RJozM

Note that as I write this, the printer is again working fine after clicking the emergency stop button and starting another print. I did not even leave enough time for the bed to cool down from the print that failed.

wilriker

@fulg said in motor phase A may be disconnected reported by driver(s) 0 1 2:

I2C does not go through that GND wire.

No, it doesn't. But a degraded GND connection leads to I2C errors. It is recommended to use the thickest possible (12-14 AWG) solid core wire with the shortest possible length.

I2C issues are most certainly the cause for the slow/stop motion.

Also I have a hunch that since the "motor phase A/B disconnected" warning will only be issued if this state persists at least 500ms I2C timeout issues could confuse/influence the time-measurement. But that is more of a wild guess because I am everything but familiar how the code for I2C interacts with the rest of the code.

fulg

So I have changed my wiring from 20AWG to a single 16AWG pair (connected exactly like the documentation, a single pair from the PSU and short wires from the Duet to the Duex). The behavior has not changed, the wiring was not to blame.

Reproducing this is easy, leave the printer with the motors energized for 24-48 hours. Then eventually the I2C timeouts will take over and pause movements during a print, essentially ruining it. After that I get the random driver failures, which are likely caused by the I2C timeouts and not the root cause of the problem.

One thing that might be important, my printer uses 4 motors for Z, and all four are on the Duex. XYE are on the Duet, the rest are unused. So at any given time, 7 steppers are energized. Perhaps the recent rewrites in 2.02/2.03 have some unintended side effects?

The printer is not a new build, I have been printing for hundreds if not thousands of hours with RRF without problems. I will try going back to earlier RRF releases...

I have attached the new diagnostics output:
diagnostics.txt

You can find a complete copy of my config here.

deckingman

@fulg said in motor phase A may be disconnected reported by driver(s) 0 1 2:

.......................................The printer is not a new build, I have been printing for hundreds if not thousands of hours with RRF without problems. I will try going back to earlier RRF releases..............................

It seems that unfortunately you too are experiencing these I2C errors. There are quite a few threads about this, the latest one (apart from this one) is here https://forum.duet3d.com/topic/10313/printer-pausing-between-commands/24. See @dc42s post date 07 May 2019, 9:13 and try that.

From what I have been able to discover, the earliest thread regarding this issue is from around July 2018. There are no threads or posts reporting I2C errors prior to that date. Did you by any chance change the firmware around the time you started seeing these problems? If so, do you happen to know what version you were one before?

wilriker

@deckingman said in motor phase A may be disconnected reported by driver(s) 0 1 2:

From what I have been able to discover, the earliest thread regarding this issue is from around July 2018. There are no threads or posts reporting I2C errors prior to that date.

Now, that might be of importance. I checked and in July 2018 RRF 2.01beta1 was released. This means up until RRF 2.0 no one at least reported I2C issues and this at least hints to something change between 2.0 and 2.01beta1.
@dc42 You know your changes best. Is there anything in 2.01beta1 that persists until this day and also might be responsible for these issues?
Or was M122 simply only extended to include these numbers? Changelog lists a couple of modifications to M122 but non regarding I2C.

EDIT: found the discussion in the thread mentioned by @deckingman above and see that there are already cases with RRF 2.0.

deckingman

@wilriker Yes. As I stated in the other thread that I linked to, it seems that there were no reports of any of these issue which can be associated with I2C misbehaviour until around the same time that the firmware changed to RTOS. It could of course be pure coincidence but it's the only common denominator that I have been able to find across multiple users with multiple generations of boards. Mine are of the very first pre-production boards, both Duet Ethernet and Duex5, and which had no problems from when I got them in December 2016 until around July 2018. Other users who have reported similar problems have more recent hardware.

fulg

@deckingman Thank you for this, I was feeling quite alone! I have since downgraded to 2.01, as I know 2.02 is also affected by this (I was discussing with someone else on Discord with the same printer / same problem and they are using 2.02).

I tend to upgrade RRF only when non-beta/non-RC versions are available, I don't know why I upgraded to 2.03beta3. Last year I was turning off the motors after each print, it is only recently that I've started to keep the steppers powered all the time.

EDIT: 2.03beta3 not RC3, this is especially important to get this right now that RC1 actually exists.

deckingman

@fulg The earliest known report of similar issues was in this thread https://forum.duet3d.com/topic/6077/inconsistent-delays-during-homing-and-other-movements and the OP of that thread stated that he was using RRF 2.0. When I first reported my own experiences of these issues I was using 2.01 (RTOS) (2018-07-26b2). Of course, it may just be coincidence that there are no reports of any similar issues with firmware which pre-dates the 2.0 RTOS series.

fulg

@deckingman I found the other threads later and noticed the problem also occurs in 2.0. I had hopes when seeing the I2C resistors hack but sadly you later confirmed it does not fix this problem.

I will leave my 2.01 setup running for now until the next occurrence (to validate Duex detection with M115 after a soft reset, as asked in another thread), and then downgrade again to 1.21.

The repro is easy for me, I just leave the steppers powered all the time. As I said before my setup is somewhat unique in that I have 4 Z motors on the Duex that are constantly moving, most printers are not built like that. Sometimes I get the error within a few hours but so far I've always reproduced it with 1 or 2 days of having the steppers powered-but-idling.

I have never seen the problem occur when turning off the steppers after each print (even if the printer itself stayed on), although I suppose it could have happened mid-print and I was just lucky until now.