Duet sometimes really slow? - I2C error or?

dc42

@deckingman said in Duet sometimes really slow? - I2C error or?:

@dc42,

David. As I'm waiting for some filament to arrive, the machine is just sitting here idle. Is there anything I can do to attempt to provoke the misbehaviour? I was thinking along the lines of writing a little script that would generate a gcode file that simply toggled one of the duex5 fans on and off. Or may send multiple M260 commands. Is there any point in me doing that?

Yes, you could try that.

GizmotronX5000

@dc42

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.02(RTOS) running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DDM-9FAM2-LW4SD-6J9F2-3S46R-K2XBW
Used output buffers: 1 of 20 (14 max)
=== RTOS ===
Static ram: 25524
Dynamic ram: 98292 of which 0 recycled
Exception stack ram used: 384
Never used ram: 6872
Tasks: NETWORK(ready,544) HEAT(blocked,1232) MAIN(running,3812) IDLE(ready,200)
Owned mutexes:
=== Platform ===
Last reset 00:16:58 ago, cause: power up
Last software reset at 2019-01-23 14:44, reason: User, spinning module GCodes, available RAM 6776 bytes (slot 3)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 0
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 15.9ms, max retries 0
MCU temperature: min 31.3, current 31.5, max 31.9
Supply voltage: min 12.0, current 12.3, max 12.4, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max 37/510
Driver 1: standstill, SG min/max 70/544
Driver 2: standstill, SG min/max 63/552
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max 0/462
Driver 7: standstill, SG min/max 0/0
Driver 8: standstill, SG min/max 0/0
Driver 9: standstill, SG min/max not available
Date/time: 2019-01-23 15:17:17
Cache data hit count 4294967295
Slowest loop: 59.17ms; fastest: 29.15ms
I2C nak errors 0, send timeouts 63402, receive timeouts 0, finishTimeouts 63402
=== Move ===
Hiccups: 0, StepErrors: 0, LaErrors: 0, FreeDm: 240, MinFreeDm: 237, MaxWait: 181633ms, Underruns: 0, 0
Scheduled moves: 21, completed moves: 21
Bed compensation in use: none
Bed probe heights: 0.000 0.000 0.000 0.000 0.000
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.0
Heater 1 is on, I-accum = 0.0
=== GCodes ===
Segments left: 0
Stack records: 1 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 63.08ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
Interface state 5, link 100Mbps full duplex
The Duex5 endstops do not work as I had previously thought. The endstops on the Duet work, but not the Duex5.
M122 right after pressing the endstops a few times (most lines removed):
M122
Date/time: 2019-01-23 15:18:05
Cache data hit count 4294967295
Slowest loop: 59.10ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 4797, receive timeouts 0, finishTimeouts 4797
No fans are actually connected to the Duex5, so I can't test this. I can configure a dummy fan to the Duex5 for next time though. All fans on the Duet work fine.
M122 (after messing around with duet fans)
Date/time: 2019-01-23 15:22:28
Cache data hit count 4294967295
Slowest loop: 59.11ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551
No changes. Just sending M122 every few seconds.

M122
Date/time: 2019-01-23 15:23:31
Cache data hit count 4294967295
Slowest loop: 59.11ms; fastest: 29.47ms
I2C nak errors 0, send timeouts 10480, receive timeouts 0, finishTimeouts 10480

M122
Date/time: 2019-01-23 15:24:00
Cache data hit count 4294967295
Slowest loop: 59.10ms; fastest: 29.46ms
I2C nak errors 0, send timeouts 4900, receive timeouts 0, finishTimeouts 4900

M122
Date/time: 2019-01-23 15:24:42
Cache data hit count 4294967295
Slowest loop: 59.10ms; fastest: 29.47ms
I2C nak errors 0, send timeouts 6885, receive timeouts 0, finishTimeouts 6885

I'm assuming the error count starts over each time I send M122. But the number is constantly changing. The lowest I saw was 195 from sending M122 back to back as fast as I could manually from the console. Looks like about 160-170 errors per second on average (164 seems to be the usual) regardless of what commands I send or which endstops I press.

Edit:
I let it idle in this state for about 15 minutes. The average was 166.2 errors per second. It's very consistent.

deckingman

I can add a bit to what @GizmotronX5000 has come with in that I do have fans connected to the Duex5 and that they did not respond to any commands while the machine was in error state (neither did the duex endstops).

dc42

Thanks, you have both confirmed that the DueX5 isn't responding to I2C traffic at all, and the Duet is repeatedly trying (and failing) to communicate with it.

TBS

@dc42: Are there any news about this error?
When does a new release candidate appear?

Thanks for answering.

dc42

For now my recommendation to anyone having this problem is:

Make sure that the wiring of the ground side of the VIN terminals on the Duet and DueX follows our recommendations.
Keep stepper motor cables and other sources of noise at least a few cm away from the ribbon cable connecting the Duet and DueX.
Try adding the extra pullup resistors as I describe earlier in this thread.

The problem I have is that I don't know what triggers this issue.

deckingman

Thus far, I haven't had the problem manifest itself since installing the pull-up resistors. But it's early days yet as the problem only occurs rarely for me. I'll report back in a month or so, unless the probelm manifests itself again. Fingers crossed it won't............

TBS

@dc42 said in Duet sometimes really slow? - I2C error or?:

PS - in testing the I2C communication at high speeds, I found it helpful to reduce the values of the I2C pullup resistors on the DueX5. The simplest way to do this is to connect a resistor with a value between 1.3K and 2.2k between the TWC and +3.3 pins of connector J13 on the DueX5, and another between TWD and +3.3, like this:

If you don't have the means to make this yourself, we may be able to send you a ready-made one. If you do make one yourself, take great care not to short the 3.3 and 5V pins together.

@dc42: You offered to send me a finished plug. Can you please send me the ready-made one? I can reproduce the error. In the last 6 prints over 2.5 days the error were occured every time. Than we can see of the plug works.
How can I send you my address?

Thanks.

GizmotronX5000

@dc42 I have noticed that the error will start for me sometimes after touching the printer. I do not have ground wires on the casings of my stepper motors right now. Occasionally I will get a static shock and the error occurs. (Can't confirm 100% that this is directly the cause, but it seems possible)

When I am working on the printer it is much more likely to occur than when I am just letting it run.

dc42

@gizmotronx5000 said in Duet sometimes really slow? - I2C error or?:

@dc42 I have noticed that the error will start for me sometimes after touching the printer. I do not have ground wires on the casings of my stepper motors right now. Occasionally I will get a static shock and the error occurs. (Can't confirm 100% that this is directly the cause, but it seems possible)

When I am working on the printer it is much more likely to occur than when I am just letting it run.

Thanks for the information. Static charge is very disruptive of electronic circuits. the question is whether it is causing:

An i2C protocol error, which it may be possible to recover from.
The i2C interface in the Duet main processor to lock up, which it may be possible to recover from by resetting the interface.
The i2C interface in the SX1509B to lock up, which can't be recovered from other than by a reset.

dc42

@tbs said in Duet sometimes really slow? - I2C error or?:

@dc42: You offered to send me a finished plug. Can you please send me the ready-made one? I can reproduce the error. In the last 6 prints over 2.5 days the error were occured every time. Than we can see of the plug works.
How can I send you my address?

Use the Chat facility to send me a message containing your address.

TBS

@dc42 said in Duet sometimes really slow? - I2C error or?:

@tbs said in Duet sometimes really slow? - I2C error or?:

@dc42: You offered to send me a finished plug. Can you please send me the ready-made one? I can reproduce the error. In the last 6 prints over 2.5 days the error were occured every time. Than we can see of the plug works.
How can I send you my address?

Use the Chat facility to send me a message containing your address.

@dc42: It is not possible for me to start a chat with you. Should I send you a Email to info@duet3d.com with my address?

dc42

@tbs, I'll email you.

Edgars Batna

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

deckingman

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Edgars Batna

@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Well, I wouldn't post if I wasn't dead sure.

I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551

I get the M122 as above plus slow movements. The machine should not resume printing like that...

deckingman

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Well, I wouldn't post if I wasn't dead sure.
I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551
I get the M122 as above plus slow movements. The machine should not resume printing like that...

No need to get stroppy. It's just that the rest of us get slow movement of all axes, not just Z so there is no way that for the rest of us, it could trash the machine in the way you describe. That's why I asked if you were getting the same problem because what you describe about the XY or E axes moving as normal is not something that anyone else has experienced.

Anyway, If you look at DC42's comment on 15th Jan he said quote:

"It appears that for those few DueX users who encounter these issues, something is disrupting the I2C communications between the two boards. Unfortunately the I2C protocol does not include error detection (other than an indication that no recipient accepted the data) or an error recovery protocol".

So I guess that means it's difficult for the firmware to know that the I2C communication has been disrupted and thus makes it difficult to implement any sort of fall back routine. Not my field of expertise though - just surmising.

The resistor fix detailed above might be what you need. Since I implemented it on my machine, I haven't had any reoccurrences of the problem so that's a promising sign (although it's only been about 3 weeks).

Having said that, all my axes are affected, not just selective ones as in your case. In my case, I have all 5 extruders connected to the Duex 5 and when I get the pauses, all the extruders stop as well as all the XYUV and Z motors.

Edgars Batna

@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:
@deckingman said in Duet sometimes really slow? - I2C error or?:

@edgars-batna said in Duet sometimes really slow? - I2C error or?:

I'm also having this problem, but it's super-rare, perhaps once or twice a year.

Are stepper drivers on the DueX5 also affected? All 4 Z motors are connected on the DueX5 and I didn't wait long enough to see if they are affected...

Shouldn't the Duet have a fallback routine if the errors persist for longer than a few seconds? It should do the usual power fail routine and reset itself. I can imagine my printer being trashed because XY would hit the blobs really hard as it still extruding material, but just not lifting Z or extruding in place.

Are you sure you are experiencing the same problem? If so, there is no danger of your printer being trashed. The problem being discussed in this thread is that it appears to be a disruption of the I2C communications between the Duet main board and the Duex 5 expansion board. When this occurs, all moves slow down with noticeable pauses between moves.

What you describe about XY and E moving as normal but Z not moving is a different problem to that which we are experiencing in this thread, so I guess you must have a different issue.

Well, I wouldn't post if I wasn't dead sure.
I2C nak errors 0, send timeouts 17551, receive timeouts 0, finishTimeouts 17551
I get the M122 as above plus slow movements. The machine should not resume printing like that...
No need to get stroppy. It's just that the rest of us get slow movement of all axes, not just Z so there is no way that for the rest of us, it could trash the machine in the way you describe. That's why I asked if you were getting the same problem because what you describe about the XY or E axes moving as normal is not something that anyone else has experienced.

Sorry, I'm surrounded by failing machines, so it's gloomy here.

All axes are affected for me. Maybe I wasn't clear enough, but it wasn't clear if just Z had stopped working entirely or was just pausing with the rest of the printer. The whole printer was pausing. The firmware shouldn't be tolerant of thousands of I2C errors, tho. After one thousand it should stop the machine.

deckingman

Sorry to resurrect this very old thread and I'm also sorry to have to report that I've had this I2C error occur again on me with the resistors fitted and which were showing so much promise.

This is the first time it has happened since I fitted the resistors on 23rd January as detailed a few posts up.

I don't how or if this can be of any relevance but the printer has been powered down and idle for a couple of weeks. This has been the sequence of events on a few occasions when I have experienced this I2C problem and whilst I can't see any reason why a period of inactivity could be relevant, it is too much of a coincidence to ignore. The only possible explanation I can think of, is that after a period of inactivity some sort of mechanical "sticksion" occurs so when the first move is attempted, the motors draw a higher than normal current and it is that which somehow triggers the I2C errors? Or maybe just pure coincidence is a more plausible explanation.

Anyway, the sequence of events leading up to this latest occurrence was as follows:
Printer turned on. A gcode file was uploaded.This file was then selected to print. My start gcode calls a macro which does the following.
Heat the bed to 40 degC, then when that temperature is reached, run homeall.g. It was during this homing that I noticed the problem of multi-second pauses between moves. XYU and V all homed as expected and it was while homing Z that the pauses started and continued for the rest of the macro, which basically moves the head to the rear of the bed and waits for all temperatures to stabilise.

I ran M122 which reported 5000+ I2C send and finish timeouts

As on every occasion in the past, cycling the power to the printer restored normality and as I write this, it is quite happily printing the part that I attempted earlier.

So it seems that the resistors may not be the answer - or at least the whole answer.

dc42

Thanks for the report. I'll try to find a way to provoke I2C errors on my bench setup.