Duet sometimes really slow? - I2C error or?

dc42

The length of the power cables shouldn't matter much as long as there is a very short and thick wire between the ground (negative) terminals of the VIN connectors of the Duet and the DueX5, to make sure that the stepper motor currents don't cause the grounds of the two boards to be at different potentials.

It appears that for those few DueX users who encounter these issues, something is disrupting the I2C communications between the two boards. Unfortunately the I2C protocol does not include error detection (other than an indication that no recipient accepted the data) or an error recovery protocol. It could be ground noise causing the disruption; or coupling between the I2C clock and data wires and sources of noise such as stepper motor cables; or something else could be causing the SX1509B chip on the DueX to stop responding.

Surgikill

@dc42 I meant the power cables from the Duet to Duex5. A Low ESR capcitor connected to the power block on each side would probably do wonders to help. I know that in my drones it does quite a bit to smooth out all the noise, and I get significantly more due to the nature of brushless motors. Might be worth a shot.

dc42

@surgikill said in Duet sometimes really slow? - I2C error or?:

@dc42 I meant the power cables from the Duet to Duex5. A Low ESR capcitor connected to the power block on each side would probably do wonders to help. I know that in my drones it does quite a bit to smooth out all the noise, and I get significantly more due to the nature of brushless motors. Might be worth a shot.

That might be worth a try; however the DueX5 already has five 100uF capacitors and ceramic capacitors between VIN and ground, and the Duet has even more.

Surgikill

@dc42 said in Duet sometimes really slow? - I2C error or?:

@surgikill said in Duet sometimes really slow? - I2C error or?:

@dc42 I meant the power cables from the Duet to Duex5. A Low ESR capcitor connected to the power block on each side would probably do wonders to help. I know that in my drones it does quite a bit to smooth out all the noise, and I get significantly more due to the nature of brushless motors. Might be worth a shot.

That might be worth a try; however the DueX5 already has five 100uF capacitors and ceramic capacitors between VIN and ground, and the Duet has even more.

Are they surface mount and low ESR? You can also get a phenomenom known as ringing. I would recommend trying it, I'll give it a shot if I have an issue. I use the Rubycon 2500 uF electrolytic low ESR caps. Might help.

TBS

Here are pictures from my duet setup.

@deckingman and @dc42: Solution 1 (Update):
I will update to version 2.02 final today.

@deckingman and @Martin1454: Solution 2 (resistor):
I would be very happy if you would test the solution with the resistor on your Duet. Unfortunately, I do not know enough about this topic here.

@deckingman: Solution 3 (ribbon cable too close to other cables):
All stepper motor cables were already at least 5 cm away from the ribbon cable. All cables from the heater were at least 6 cm away. The only exception is the heater 2. As can be seen in the picture. The socket of the Duex5 for the heater 2 is relatively close to the ribbon cable. But there is no contact with the ribbon cable. All cables are tied together (see pictures).
Note: I had a 72 hour 3D print without heat and without extrusion. Only the movements. There were no problems here. That could have been coincidence. It could also point to the heater.

@Surgikill: Solution 4 (cable twists):
The cables between the boards were not twisted (See picture)

@dc42: Solution 5 (stepper motor):
The stepper motor cables do not touch any of the two boards. But the housings of the 3-axis stepper motors are grounded to the floor. Could that lead to the problem?

@Surgikill: Solution 6 (capcitor):
Unfortunately, I am not a specialist for Low ESR capcitor.

Many thanks for your help.

Martin1454

@dc42 said in Duet sometimes really slow? - I2C error or?:

It appears that for those few DueX users who encounter these issues, something is disrupting the I2C communications between the two boards. Unfortunately the I2C protocol does not include error detection (other than an indication that no recipient accepted the data) or an error recovery protocol. It could be ground noise causing the disruption; or coupling between the I2C clock and data wires and sources of noise such as stepper motor cables; or something else could be causing the SX1509B chip on the DueX to stop responding.

One thing I have been thinking about - When the discussed error occurs, only 1 in maybe a 100 commands gets sent sucessfully through the I2C interface, and the rest is lost/ tries again, and that's why it is so slow.

It seems like the duex gets stuck in this "mode" where no messages gets accepted, but just rebooting the duet makes the interface work perfect - So it seems like the interface controller gets stuck in some bad mode and accumulates the errors -

Would it be possible (in future update) to check if I2C errors are accumulating, and in case of - try to reset / reinitiate the I2C interface in the SW of the duet? I know there is no connection to the reset pin of the SX1509B, but mabye by reinitiating the I2C interface of the duet, it could get out of the mode?

dc42

@martin1454 said in Duet sometimes really slow? - I2C error or?:

One thing I have been thinking about - When the discussed error occurs, only 1 in maybe a 100 commands gets sent sucessfully through the I2C interface, and the rest is lost/ tries again, and that's why it is so slow.

I was under the impression that when the problem occurs, no I2C commands get through at all. Do you have any evidence that any I2C commands are getting through? I2C commands are needed to set the fan RPM and to read the endstops. They are not needed to drive the motors or the heaters.

I think the reason it is slow is that the processor is spending too much time trying to send data through the I2C interface and/or servicing the interrupt from the DueX5 that indicates that an endstop input has been changed.

Would it be possible (in future update) to check if I2C errors are accumulating, and in case of - try to reset / reinitiate the I2C interface in the SW of the duet? I know there is no connection to the reset pin of the SX1509B, but mabye by reinitiating the I2C interface of the duet, it could get out of the mode?

I have it on my list to see whether something along those lines could be done, but unfortunately the I2C specification assumes that errors do not occur and makes no provision for recovering from a protocol error. If the SX1509B gets into a state in which it does not recognise a start bit followed by its own address, it may be that nothing except a hardware reset can get it out of that state. I can try sending several stop bits first.

dragonn

maybe at boot the SX1509B needs a lite bit more time too 'initialize' before sending the first I2C command? Just my idea since most of the time the problem as I read happens only at boot.

Martin1454

@dc42 said in Duet sometimes really slow? - I2C error or?:

@martin1454 said in Duet sometimes really slow? - I2C error or?:

One thing I have been thinking about - When the discussed error occurs, only 1 in maybe a 100 commands gets sent sucessfully through the I2C interface, and the rest is lost/ tries again, and that's why it is so slow.

I was under the impression that when the problem occurs, no I2C commands get through at all. Do you have any evidence that any I2C commands are getting through? I2C commands are needed to set the fan RPM and to read the endstops. They are not needed to drive the motors or the heaters.

Yeah I just remembered that the motors don't use the I2C interface, so yeah there is a possibility there gets no communication through - disregard my previous message.

EDIT: it looks like there is a "RESET" signal on the large ribbon cable - If there will be a duet wifi 0.9, mabye connect the reset to the SX1509B reset pin so it would be possible for the MCU to reset it.

GizmotronX5000

I was under the impression that when the problem occurs, no I2C commands get through at all. Do you have any evidence that any I2C commands are getting through? I2C commands are needed to set the fan RPM and to read the endstops. They are not needed to drive the motors or the heaters.

When I get this error, everything still works, including Duex5 endstops for homing. I haven't tried changing fan RPM during this error though.

dc42

@gizmotronx5000 said in Duet sometimes really slow? - I2C error or?:

I was under the impression that when the problem occurs, no I2C commands get through at all. Do you have any evidence that any I2C commands are getting through? I2C commands are needed to set the fan RPM and to read the endstops. They are not needed to drive the motors or the heaters.

When I get this error, everything still works, including Duex5 endstops for homing. I haven't tried changing fan RPM during this error though.

That's very interesting, because if there was a total breakdown in I2C communication then the endstops should stop working. Next time it happens, please can you pause the print and then:

Send M122 to get the I2C error counts.
Toggle the DueX5 endstops and check whether the Duet can see the changes.
Send M122 to get the error counts again. [When you toggle a DueX5 endstop, it sends an interrupt to the Duet, which uses I2C to read the new states.]
Send some M106 commands to alter the state of fans/LEDs connected to the DueX5 fan outputs, and see if that works.
Send M122 and get the I2C error counts again.
Also send M122 a few more times, to see if when the machine is idle and no endstops are changing, the I2C error count goes on increasing.

GizmotronX5000

@dc42 Can do. It doesn't happen often, but the next time it occurs I'll be sure to reply here with that information.

TBS

Here is my short update:
I have updated to 2.02. The error has recurred. This time again at the start of printing.

dc42

@tbs said in Duet sometimes really slow? - I2C error or?:

Here is my short update:
I have updated to 2.02. The error has recurred. This time again at the start of printing.

Thanks, please see my post addressed to @GizmotronX5000 for useful (to me) tests you can do when this problem occurs.

deckingman

I've just had this slow down happen mid print - first time for me - it's only ever happened at first boot until now,. The only thing I have not yet done is DC42s suggestion of fitting resistors which was mentioned in another thread.

The manifestation is that the printer carries out one or 2 moves, then pauses for several seconds before continuing. M122 shows I2C nak errors 0, send timeouts 34803, receive timeouts 0, finish timeouts 34803. Also reported were:

"Warning: motor phase A may be disconnected reported by driver(s) 5 7" (Never seen that before).

I have the full M122 report if there is any other info that might be needed.

I've sent M106 Pn H-1 to a few of the fans connected to the Duex5 then tried a few M106 Pnn Snnn commands with no reaction on the fans themselves. So, I guess that means a total breakdown in I2C comms?

I've taken a video of what happens with the printer. I've cancelled the print but I'll leave it in this state for a few hours in case anyone wants me to do any tests or provide any more information.

dc42

@deckingman, thanks for keeping your machine on. Please see my response to @GizmotronX500 a few posts back and run the tests there.

deckingman

@dc42 said in Duet sometimes really slow? - I2C error or?:

@deckingman, thanks for keeping your machine on. Please see my response to @GizmotronX500 a few posts back and run the tests there.

Yes tried that. As I said, the fans are all configured as thermostatic so I tried sending M106 Pn H-1 first, then a few M106 Pnn Snnn commands and nothing happened as far as the fans were concerned.

I don't have any Duex end stops configured so I can't test them. That's not strictly true. I do have micro switches connected to E2 and E3 but no axes configured to use those switches....... Just tried toggling those switches but the status on machine properties page remains unchanged.

So I'm guess that I2C comms have completely stopped in my case. M122 now reports I2C send and finish timeouts as 1861859.

I'll leave the printer powered up for a while yet.

dc42

@deckingman, thanks. What were the results of #3 and #6? I'd like to know whether the I2C error count increases (a) continuously, (b) [if not continuously] when you try to change the speed of a DueX fan, (c) [if not continuously] when you toggle the state of a DueX endstop switch.

deckingman

@dc42 said in Duet sometimes really slow? - I2C error or?:

@deckingman, thanks. What were the results of #3 and #6? I'd like to know whether the I2C error count increases (a) continuously, (b) [if not continuously] when you try to change the speed of a DueX fan, (c) [if not continuously] when you toggle the state of a DueX endstop switch.

Damm. Sorry I didn't pay enough attention to what you were asking. The only info I have is that when I did the first M122 during the print, the send timeouts were 34803 and some time later, they were 1861859 which isn't much use to you. I turned the printer off before I went to bed last night and of course, the bloody thing is behaving itself again now.

I'll pay more attention next time it happens.

dc42

@deckingman, if they went up that much then I suspect they were continuously incrementing.

What I suspect happened is that the interrupt signal from the SX1509B on the DueX went high, indicating a change in an endstop input signal (real or imagined). That causes the Duet main processor to try to read the SX1509B input registers, which will also make the interrupt signal go away. But if I2C communication with the SX1509B has broken down, the interrupt won't go away and the Duet will keep trying to read it - which accounts for the slowdown.