Duet sometimes really slow? - I2C error or?

dc42

It's a standard 2x25 IDE ribbon cable connector at each end, 2.5mm pitch pin spacing. The cable I use is 150mm long and I suggest you don't make it much longer than that.

You could try re-clamping the existing ribbon cable connectors in a vice first. Use cardboard to protect the plastic from the jaws.

TBS

Hello,

I replaced the ribbon cable. I have a third time replaced the cables from the powersupply to Duet and Duex5 (11 cm, maximum cable thickness which fits into the Duet socket)

A few longer prints have worked with it. A few days ago, the error unfortunately occurred again.

Procedure:
It was printed simultaneously with Driver 5 and 6 (M567).
Duration of the print about 1 day.
Everything worked here.
After that, the printer has been waiting for 5 hours without turning off.
At the beginning of the next print, the error has occurred again.

Here are the M122 traces:

All 3 traces files have different error messages and show different values (for example in the category: Driver 5: ok, SG min / max not available, Driver 6: ok, SG min / max not available or LaErrors: 694)

How can I solve the problem now?
Should I buy a new Duex5?
Would it be a workaround to generate a power load on all two boards (always turn on all heaters)?
Maybe the cables to the extruders are too long with 2m (E0 E3D Super Whopper Motor, E1-E3 E3D High Torque Motor)?

I have already spent 300 euros on filament for misprints because of this error.

I am urgently asking for help.

deckingman

@tbs I had similar I2C problems but in my case it only happens once every 3 months or so. David (DC42) told me to upgrade to 2.02 final with the comment that the I2C part had been completely re-written between RC4 that I was using and the final release. I note that you are on RC6 but I have been unable to find anything to say when exactly the I2C code was re-written so it could be after RC6. Suggest you upgrade to 2.02 final (DC will tell you to in any case) then report back if you still have problems.

dc42

@TBS, please upgrade to the 2.02 release as @deckingman suggests. However, the major rewrite of the I2C subsystem happened at version 2.02RC6, so in all honesty I don't expect this to change things.

It appears that either something is disrupting the I2C communication between the Duet and the DueX5, or the communication is being lost for some other reason. If we can eliminate wiring issues, then I'll recommend that we try replacing your DueX5.

Please post a photo or photos of your Due/DueX5 setup, that show clearly the power wiring and the ribbon cable.

dc42

PS - in testing the I2C communication at high speeds, I found it helpful to reduce the values of the I2C pullup resistors on the DueX5. The simplest way to do this is to connect a resistor with a value between 1.3K and 2.2k between the TWC and +3.3 pins of connector J13 on the DueX5, and another between TWD and +3.3, like this:

alt text

If you don't have the means to make this yourself, we may be able to send you a ready-made one. If you do make one yourself, take great care not to short the 3.3 and 5V pins together.

deckingman

@dc42 Thanks for that. I'll give it a go on mine. I only rarely get the problem so can't yet say if upgrading from RC4 to final has fixed it for me. I may as well try this mod as well. I think I'll make up a lead that will plug in. Then I can insulate the leads and test it for shorts before I plug it in.

Martin1454

@dc42 Will try to add some resistors to mine - I get the problem about once every 20-ish start - Way fewer than before the new SW, but will see if that gets rid of the last errors. - The 1:20 error is not that much of an problem, but I fear that it will happen midt print (It did in the old SW, but haven't had any longer prints yet with the new sw)

deckingman

@martin1454 If it's any consolation, I've never ever had it happen mid print - only when first powering up. That's even with 30 + hour prints.

dc42

One more suggestion: don't run any stepper motor or heater cables right next to the ribbon cable.

Surgikill

@dc42 Could it just be crosstalk? What if he twisted the power cables going from the Duet to the DueX? I'm looking at a similar setup with ~150mm cables and I'm concerned I'll be having similar issues.

dc42

The length of the power cables shouldn't matter much as long as there is a very short and thick wire between the ground (negative) terminals of the VIN connectors of the Duet and the DueX5, to make sure that the stepper motor currents don't cause the grounds of the two boards to be at different potentials.

It appears that for those few DueX users who encounter these issues, something is disrupting the I2C communications between the two boards. Unfortunately the I2C protocol does not include error detection (other than an indication that no recipient accepted the data) or an error recovery protocol. It could be ground noise causing the disruption; or coupling between the I2C clock and data wires and sources of noise such as stepper motor cables; or something else could be causing the SX1509B chip on the DueX to stop responding.

Surgikill

@dc42 I meant the power cables from the Duet to Duex5. A Low ESR capcitor connected to the power block on each side would probably do wonders to help. I know that in my drones it does quite a bit to smooth out all the noise, and I get significantly more due to the nature of brushless motors. Might be worth a shot.

dc42

@surgikill said in Duet sometimes really slow? - I2C error or?:

@dc42 I meant the power cables from the Duet to Duex5. A Low ESR capcitor connected to the power block on each side would probably do wonders to help. I know that in my drones it does quite a bit to smooth out all the noise, and I get significantly more due to the nature of brushless motors. Might be worth a shot.

That might be worth a try; however the DueX5 already has five 100uF capacitors and ceramic capacitors between VIN and ground, and the Duet has even more.

Surgikill

@dc42 said in Duet sometimes really slow? - I2C error or?:

@surgikill said in Duet sometimes really slow? - I2C error or?:

@dc42 I meant the power cables from the Duet to Duex5. A Low ESR capcitor connected to the power block on each side would probably do wonders to help. I know that in my drones it does quite a bit to smooth out all the noise, and I get significantly more due to the nature of brushless motors. Might be worth a shot.

That might be worth a try; however the DueX5 already has five 100uF capacitors and ceramic capacitors between VIN and ground, and the Duet has even more.

Are they surface mount and low ESR? You can also get a phenomenom known as ringing. I would recommend trying it, I'll give it a shot if I have an issue. I use the Rubycon 2500 uF electrolytic low ESR caps. Might help.

TBS

Here are pictures from my duet setup.

@deckingman and @dc42: Solution 1 (Update):
I will update to version 2.02 final today.

@deckingman and @Martin1454: Solution 2 (resistor):
I would be very happy if you would test the solution with the resistor on your Duet. Unfortunately, I do not know enough about this topic here.

@deckingman: Solution 3 (ribbon cable too close to other cables):
All stepper motor cables were already at least 5 cm away from the ribbon cable. All cables from the heater were at least 6 cm away. The only exception is the heater 2. As can be seen in the picture. The socket of the Duex5 for the heater 2 is relatively close to the ribbon cable. But there is no contact with the ribbon cable. All cables are tied together (see pictures).
Note: I had a 72 hour 3D print without heat and without extrusion. Only the movements. There were no problems here. That could have been coincidence. It could also point to the heater.

@Surgikill: Solution 4 (cable twists):
The cables between the boards were not twisted (See picture)

@dc42: Solution 5 (stepper motor):
The stepper motor cables do not touch any of the two boards. But the housings of the 3-axis stepper motors are grounded to the floor. Could that lead to the problem?

@Surgikill: Solution 6 (capcitor):
Unfortunately, I am not a specialist for Low ESR capcitor.

Many thanks for your help.

Martin1454

@dc42 said in Duet sometimes really slow? - I2C error or?:

It appears that for those few DueX users who encounter these issues, something is disrupting the I2C communications between the two boards. Unfortunately the I2C protocol does not include error detection (other than an indication that no recipient accepted the data) or an error recovery protocol. It could be ground noise causing the disruption; or coupling between the I2C clock and data wires and sources of noise such as stepper motor cables; or something else could be causing the SX1509B chip on the DueX to stop responding.

One thing I have been thinking about - When the discussed error occurs, only 1 in maybe a 100 commands gets sent sucessfully through the I2C interface, and the rest is lost/ tries again, and that's why it is so slow.

It seems like the duex gets stuck in this "mode" where no messages gets accepted, but just rebooting the duet makes the interface work perfect - So it seems like the interface controller gets stuck in some bad mode and accumulates the errors -

Would it be possible (in future update) to check if I2C errors are accumulating, and in case of - try to reset / reinitiate the I2C interface in the SW of the duet? I know there is no connection to the reset pin of the SX1509B, but mabye by reinitiating the I2C interface of the duet, it could get out of the mode?

dc42

@martin1454 said in Duet sometimes really slow? - I2C error or?:

One thing I have been thinking about - When the discussed error occurs, only 1 in maybe a 100 commands gets sent sucessfully through the I2C interface, and the rest is lost/ tries again, and that's why it is so slow.

I was under the impression that when the problem occurs, no I2C commands get through at all. Do you have any evidence that any I2C commands are getting through? I2C commands are needed to set the fan RPM and to read the endstops. They are not needed to drive the motors or the heaters.

I think the reason it is slow is that the processor is spending too much time trying to send data through the I2C interface and/or servicing the interrupt from the DueX5 that indicates that an endstop input has been changed.

Would it be possible (in future update) to check if I2C errors are accumulating, and in case of - try to reset / reinitiate the I2C interface in the SW of the duet? I know there is no connection to the reset pin of the SX1509B, but mabye by reinitiating the I2C interface of the duet, it could get out of the mode?

I have it on my list to see whether something along those lines could be done, but unfortunately the I2C specification assumes that errors do not occur and makes no provision for recovering from a protocol error. If the SX1509B gets into a state in which it does not recognise a start bit followed by its own address, it may be that nothing except a hardware reset can get it out of that state. I can try sending several stop bits first.

dragonn

maybe at boot the SX1509B needs a lite bit more time too 'initialize' before sending the first I2C command? Just my idea since most of the time the problem as I read happens only at boot.

Martin1454

@dc42 said in Duet sometimes really slow? - I2C error or?:

@martin1454 said in Duet sometimes really slow? - I2C error or?:

One thing I have been thinking about - When the discussed error occurs, only 1 in maybe a 100 commands gets sent sucessfully through the I2C interface, and the rest is lost/ tries again, and that's why it is so slow.

I was under the impression that when the problem occurs, no I2C commands get through at all. Do you have any evidence that any I2C commands are getting through? I2C commands are needed to set the fan RPM and to read the endstops. They are not needed to drive the motors or the heaters.

Yeah I just remembered that the motors don't use the I2C interface, so yeah there is a possibility there gets no communication through - disregard my previous message.

EDIT: it looks like there is a "RESET" signal on the large ribbon cable - If there will be a duet wifi 0.9, mabye connect the reset to the SX1509B reset pin so it would be possible for the MCU to reset it.

GizmotronX5000

I was under the impression that when the problem occurs, no I2C commands get through at all. Do you have any evidence that any I2C commands are getting through? I2C commands are needed to set the fan RPM and to read the endstops. They are not needed to drive the motors or the heaters.

When I get this error, everything still works, including Duex5 endstops for homing. I haven't tried changing fan RPM during this error though.