Duet sometimes really slow? - I2C error or?

deckingman

Sorry to resurrect this very old thread and I'm also sorry to have to report that I've had this I2C error occur again on me with the resistors fitted and which were showing so much promise.

This is the first time it has happened since I fitted the resistors on 23rd January as detailed a few posts up.

I don't how or if this can be of any relevance but the printer has been powered down and idle for a couple of weeks. This has been the sequence of events on a few occasions when I have experienced this I2C problem and whilst I can't see any reason why a period of inactivity could be relevant, it is too much of a coincidence to ignore. The only possible explanation I can think of, is that after a period of inactivity some sort of mechanical "sticksion" occurs so when the first move is attempted, the motors draw a higher than normal current and it is that which somehow triggers the I2C errors? Or maybe just pure coincidence is a more plausible explanation.

Anyway, the sequence of events leading up to this latest occurrence was as follows:
Printer turned on. A gcode file was uploaded.This file was then selected to print. My start gcode calls a macro which does the following.
Heat the bed to 40 degC, then when that temperature is reached, run homeall.g. It was during this homing that I noticed the problem of multi-second pauses between moves. XYU and V all homed as expected and it was while homing Z that the pauses started and continued for the rest of the macro, which basically moves the head to the rear of the bed and waits for all temperatures to stabilise.

I ran M122 which reported 5000+ I2C send and finish timeouts

As on every occasion in the past, cycling the power to the printer restored normality and as I write this, it is quite happily printing the part that I attempted earlier.

So it seems that the resistors may not be the answer - or at least the whole answer.

dc42

Thanks for the report. I'll try to find a way to provoke I2C errors on my bench setup.

fulg

I am having similar problem. Printer wiring is fine, no issue with the crimps either (which is the usual suspect). I am randomly getting this "motor phase A may be disconnected", not always from the same driver, and sometimes from multiple drivers at once (last occurrence was drivers 6 and 7 simultaneously on a Duex5).

When this happens (usually 3-4 hours into a print), there is a ~5 second pause between each motion and the occasional message about motor phase popping up. Naturally this ruins the print in progress. Rebooting the firmware (clicking the Emergency Stop button in DWC or the PanelDue) solves the issue for the next print.

I have attached my config and output from M122 after the latest occurrence.

config.g
diagnostics.txt

I am not sure if this line is interesting, I think I see an underrun error?

=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 1

Veti

I2C nak errors 0, send timeouts 511032, receive timeouts 0, finishTimeouts 511032

your wiring to the duex might be insufficient.

fulg

@veti Interesting, I did not notice this. The Duet/Duex combo were installed over a year ago and the wiring has never moved or changed. I am not aware of IDC connections degrading with time. As per the installation instructions, both boards share the same GND wire from the PSU.

The first time the error was reported on a Duet driver, not Duex (I did not think of grabbing a diagnostics log at that time).

I tried a 3-hour print yesterday in the same conditions and it completed without issues.

Veti

@fulg said in motor phase A may be disconnected reported by driver(s) 0 1 2:

, both boards share the same GND wire from the PSU.

whats the diameter of the GND wire?

fulg

@veti 20AWG silicone wire. I doubt this would have lasted over a year of printing without issues if it was the source of the problem. I2C does not go through that GND wire.

There is someone else on Discord with the same problem, he doesn't have the I2C timeouts but he does have the random failure reported from multiple drivers and the 5-second delays between motion.

fulg

This just happened again, this time I was sitting in front of the printer when it happened. First the pauses, then eventually (after a couple of minutes) the motor phases warning. This time it was phase B for drivers 1 and 8.

This time I managed to capture the M122 immediately as the problem happened (while the print was slowly progressing).

diagnostics.txt

Lots of timeout errors...
I2C nak errors 0, send timeouts 511032, receive timeouts 0, finishTimeouts 511032

I have also captured the behavior on video, before receiving my first motor phase warning:
https://www.youtube.com/watch?v=w1QNw_RJozM

Note that as I write this, the printer is again working fine after clicking the emergency stop button and starting another print. I did not even leave enough time for the bed to cool down from the print that failed.

wilriker

@fulg said in motor phase A may be disconnected reported by driver(s) 0 1 2:

I2C does not go through that GND wire.

No, it doesn't. But a degraded GND connection leads to I2C errors. It is recommended to use the thickest possible (12-14 AWG) solid core wire with the shortest possible length.

I2C issues are most certainly the cause for the slow/stop motion.

Also I have a hunch that since the "motor phase A/B disconnected" warning will only be issued if this state persists at least 500ms I2C timeout issues could confuse/influence the time-measurement. But that is more of a wild guess because I am everything but familiar how the code for I2C interacts with the rest of the code.

fulg

So I have changed my wiring from 20AWG to a single 16AWG pair (connected exactly like the documentation, a single pair from the PSU and short wires from the Duet to the Duex). The behavior has not changed, the wiring was not to blame.

Reproducing this is easy, leave the printer with the motors energized for 24-48 hours. Then eventually the I2C timeouts will take over and pause movements during a print, essentially ruining it. After that I get the random driver failures, which are likely caused by the I2C timeouts and not the root cause of the problem.

One thing that might be important, my printer uses 4 motors for Z, and all four are on the Duex. XYE are on the Duet, the rest are unused. So at any given time, 7 steppers are energized. Perhaps the recent rewrites in 2.02/2.03 have some unintended side effects?

The printer is not a new build, I have been printing for hundreds if not thousands of hours with RRF without problems. I will try going back to earlier RRF releases...

I have attached the new diagnostics output:
diagnostics.txt

You can find a complete copy of my config here.

deckingman

@fulg said in motor phase A may be disconnected reported by driver(s) 0 1 2:

.......................................The printer is not a new build, I have been printing for hundreds if not thousands of hours with RRF without problems. I will try going back to earlier RRF releases..............................

It seems that unfortunately you too are experiencing these I2C errors. There are quite a few threads about this, the latest one (apart from this one) is here https://forum.duet3d.com/topic/10313/printer-pausing-between-commands/24. See @dc42s post date 07 May 2019, 9:13 and try that.

From what I have been able to discover, the earliest thread regarding this issue is from around July 2018. There are no threads or posts reporting I2C errors prior to that date. Did you by any chance change the firmware around the time you started seeing these problems? If so, do you happen to know what version you were one before?

wilriker

@deckingman said in motor phase A may be disconnected reported by driver(s) 0 1 2:

From what I have been able to discover, the earliest thread regarding this issue is from around July 2018. There are no threads or posts reporting I2C errors prior to that date.

Now, that might be of importance. I checked and in July 2018 RRF 2.01beta1 was released. This means up until RRF 2.0 no one at least reported I2C issues and this at least hints to something change between 2.0 and 2.01beta1.
@dc42 You know your changes best. Is there anything in 2.01beta1 that persists until this day and also might be responsible for these issues?
Or was M122 simply only extended to include these numbers? Changelog lists a couple of modifications to M122 but non regarding I2C.

EDIT: found the discussion in the thread mentioned by @deckingman above and see that there are already cases with RRF 2.0.

deckingman

@wilriker Yes. As I stated in the other thread that I linked to, it seems that there were no reports of any of these issue which can be associated with I2C misbehaviour until around the same time that the firmware changed to RTOS. It could of course be pure coincidence but it's the only common denominator that I have been able to find across multiple users with multiple generations of boards. Mine are of the very first pre-production boards, both Duet Ethernet and Duex5, and which had no problems from when I got them in December 2016 until around July 2018. Other users who have reported similar problems have more recent hardware.

fulg

@deckingman Thank you for this, I was feeling quite alone! I have since downgraded to 2.01, as I know 2.02 is also affected by this (I was discussing with someone else on Discord with the same printer / same problem and they are using 2.02).

I tend to upgrade RRF only when non-beta/non-RC versions are available, I don't know why I upgraded to 2.03beta3. Last year I was turning off the motors after each print, it is only recently that I've started to keep the steppers powered all the time.

EDIT: 2.03beta3 not RC3, this is especially important to get this right now that RC1 actually exists.

deckingman

@fulg The earliest known report of similar issues was in this thread https://forum.duet3d.com/topic/6077/inconsistent-delays-during-homing-and-other-movements and the OP of that thread stated that he was using RRF 2.0. When I first reported my own experiences of these issues I was using 2.01 (RTOS) (2018-07-26b2). Of course, it may just be coincidence that there are no reports of any similar issues with firmware which pre-dates the 2.0 RTOS series.

fulg

@deckingman I found the other threads later and noticed the problem also occurs in 2.0. I had hopes when seeing the I2C resistors hack but sadly you later confirmed it does not fix this problem.

I will leave my 2.01 setup running for now until the next occurrence (to validate Duex detection with M115 after a soft reset, as asked in another thread), and then downgrade again to 1.21.

The repro is easy for me, I just leave the steppers powered all the time. As I said before my setup is somewhat unique in that I have 4 Z motors on the Duex that are constantly moving, most printers are not built like that. Sometimes I get the error within a few hours but so far I've always reproduced it with 1 or 2 days of having the steppers powered-but-idling.

I have never seen the problem occur when turning off the steppers after each print (even if the printer itself stayed on), although I suppose it could have happened mid-print and I was just lucky until now.

deckingman

@fulg For me, it's a bit different. I've only had one or two print failures, neither of which was more than a few of hours long. Usually, I get the failure quite soon after powering up the printer during the first homing moves or shortly after. On many occasions that has been after the printer has been ~~idle~~ pwered down for a few days. I have had the issue during homing, cycled the power, then homed again and done a 30 plus hour print without further problems. The way it manifests itself is long pauses between moves but not reports of motor phase errors.
It's this randomness, that makes it so hard to track down. Many users report different problems. Some have the Duex5 board well populated, others have only a few things connected. The only thing in common between users is the I2C errors.

deckingman

In case anyone is following this thread but not one of the other I2C error related threads, I can now confirm that running M199 to reset the firmware fixes the issue so no need to cycle the power. At least that's how it is for me.

I can also confirm that when running M115 immediately after M999, it reports that the Duex5 is present. Since I had the error and ran M199 to restart the firmware, without powering down, I've re-homed the printer and it is now happily printing away (at least for now).

dc42

@deckingman said in motor phase A may be disconnected reported by driver(s) 0 1 2:

In case anyone is following this thread but not one of the other I2C error related threads, I can now confirm that running M199 to reset the firmware fixes the issue so no need to cycle the power. At least that's how it is for me.

I can also confirm that when running M115 immediately after M999, it reports that the Duex5 is present. Since I had the error and ran M199 to restart the firmware, without powering down, I've re-homed the printer and it is now happily printing away (at least for now).

Thanks for confirming this. Now all I need to do is find a way to reproduce the problem so that I can test possible fixes.

deckingman

@dc42 said in motor phase A may be disconnected reported by driver(s) 0 1 2:

Thanks for confirming this. Now all I need to do is find a way to reproduce the problem so that I can test possible fixes.

Best of luck with that !!. I certainly can't find a way a reproduce it - I wish I could. It seems to me like it's just something that randomly happens. But I'm more than happy to try anything you can think of to provoke it.

BTW, the print I started after restarting the firmware using M999 was a tad under 4 hours and completed without any further I2C errors or problems.