Duet sometimes really slow? - I2C error or?

kazolar

@dc42 I didn't get a chance to run my tests last night, will try tonight. I can generally get it to start going slowly during the calibration process of leveling out the nozzles. A lot of repetitive moves and tool changes positioning each tool at various points on the bed and checking for z equivalency. It's manual, and fairly time consuming, hence I don't know of a way to automate an i2c issue I was hitting.

deckingman

@dc42 said in Duet sometimes really slow? - I2C error or?:

@deckingman, thanks for the update.

To be clear, what I would like to establish is:

Whether the original fault is fixed (with or without a nonzero i2c reset coun). The original fault is that i2c communication breaks down. The symptom of the printer going slow was a side effect, which I don't expect to be present in the new release even if i2c comms does break down. To test whether i2c communication is working, try changing the speed of a fan connected to the DueX.

Whether or not the new code that fetches the states of the DueX endstop inputs is reliable. So after a long print, please use M119 or the Machine Properties page of the old DWC to read the endstop states as you operate the switches.

Errr, that might be tad difficult.

The fans that are connected to the Duex are all thermostatic and used to cool the steppers. However I can get round that because I just changed the mounts to aluminium ones so I won't need to cool the motors. I deliberately left all the fans, thermistors and wiring as was in the interest of consistency so it'll be an easy thing to either play around with the thermostatic thresholds or disable thermostatic mode and manually change fan speeds.

End stops might be a bit more challenging. The first 2 (E2 and E3) on the Duex5 are connected to the upper gantry. These only get mapped to X and Y when homing so don't normally show up, and with this firmware I can't map them to X and Y either because M574 is non-functional. The second two (E4 and E5) are used with axes maxima and trigger emergency stop using M574 (which then of course requires a firmware reset).

I don't mind temporarily changing my config to test but please advise what and how.

Cheers

kazolar

@dc42 I'm getting somewhere -- prior to this release it took a lot of work to get it to break. Now I try to home axis W and it causes the board to reset -- no error -- so I'm going to replace that wire. I see it does toggle properly when I trigger the end stop, but it will not home. Axis A homes, so that's the next one on i2c bus, so it must be interference or something in this specific wire run. Will update after replacing it

EDIT: replaced the wire -- now homes. Will do the calibration tomorrow night. This looks like interference from stepper wiring, I had this end stop wire in the same nylon sleeve as stepper wires, I've had this problem with other wires before, and sleeving the stepper wires by themselves apart from the end stop solved it. Will post an update tomorrow how calibration goes -- I didn't catch any i2c errors during this troubleshooting, but something did trigger duet to reboot.

fulg

I have not been able to reproduce the issue with 2.01 since the last time I posted (without touching the hardware), it seems I was just really lucky to get it (or unlucky based on your point of view...). I used to get it a lot and very easily, not so much now.

Since then I tried the 2.03RC2 release, and haven't seen an I2C reset yet. Still keeping an eye on it. I don't have any endstops on the Duex but all my fans are on it due to the 12V regulator.

@deckingman I believe if the Duex communication fails, the fans on the Duex will simply never turn off, even when below the temperature threshold (this is what happened for me, the hotend fan was stuck on even at room temp). So you should be able to confirm this without changing your config.

deckingman

@fulg said in Duet sometimes really slow? - I2C error or?:
.......................

@deckingman I believe if the Duex communication fails, the fans on the Duex will simply never turn off, even when below the temperature threshold (this is what happened for me, the hotend fan was stuck on even at room temp). So you should be able to confirm this without changing your config.

Yes that's fine but what I can't test is David's second point which is quote:

"Whether or not the new code that fetches the states of the DueX endstop inputs is reliable."

Because, the implemented changes are based on a version of firmware that does not allow me to use those end stops, (at least in the way that my machine is configured) because support to map end stops to axes has been withdrawn.

@DC42 David. Is there any way that we could have a version of firmware to test which has both the changes you have made to the I2C code, and which also reinstates the M574 mapping of end stops to axes? I am concerned that essentially two things have changed. Firstly, you have made changes to the I2C related code but secondly, I am no longer to use the exact same test sequence which had proven to provoke the issue on 4 consecutive occasions. So as my test methodology is no longer the same (because I cannot use the Duex end stops in the same way), then it reduce the level of confidence one might have that the changes to the code are an effective to solution to this somewhat elusive problem.

dc42

Ian, you can test the status of the endstops on the DueX by creating additional axes and making them visible. If you create 3 axes then the third one will use the E2 endstop input on the Due X. When that axis is visible, the state of the E2 endstop will show up in the Machine Properties page of the old DWC, and in the M119 response.

kazolar

I am almost positive at this point these issues have to do with end stop wires having interference on them from the stepper wiring. It doesn't appear to bother regular end stops, but does mess with the i2c end stops, to such an extent that it kills the entire i2c connection. Outside of isolating stepper wiring, is there anything else we can do? Why is this an issue for i2c end stops only?

dc42

@kazolar said in Duet sometimes really slow? - I2C error or?:

I am almost positive at this point these issues have to do with end stop wires having interference on them from the stepper wiring. It doesn't appear to bother regular end stops, but does mess with the i2c end stops, to such an extent that it kills the entire i2c connection. Outside of isolating stepper wiring, is there anything else we can do?

Use normally-closed endstop switches. Normally-open ones are susceptible to capacitively-coupled interference when they are not triggered. Normally-closed ones are instead susceptible to inductive interference, which is easier to control.
Use twisted pair wires to connect each endstop to the DueX, which minimises pick-up of inductive interference.

Why is this an issue for i2c end stops only?

The Duet has additional filtering on the endstop inputs.

kazolar

@dc42 I use silicone wire for moving axis, so twisting them doesn't work particularly well, that's where I'm seeing issues. Stationary end stops, I can easily isolate and eliminate all interference. I have 2 moving gantries, I have 2 end stops on each, primary x gantry has both end stops connected to the duet, no issues on it, the 2nd gantry has end stops going to the duex5, that's where I've had issues, originally on the A axis, now on the W axis. I've isolated both runs in their own nylon shielding, and I'm hoping that solves the problem. At least it solved it for the mean time and I was able to home. I have been using normally closed end stops for some time. Even with the wire which got all wrapped up around the stepper wire, end stop was reacting, and m119 was reporting it properly, but clearly there was significant enough intereference on the line that with the new firmware it wouldn't home, trigger a full reboot, interestingly enough it homed fine in the previous version, so if you added an error condition, I must have hit it. Just wondering if there is some filtering we can add to the endstop input itself to mitigate the issue.

dc42

@kazolar said in Duet sometimes really slow? - I2C error or?:

Just wondering if there is some filtering we can add to the endstop input itself to mitigate the issue.

You could try a 100 or 200 ohm resistor in series with the wire to the STP pin of the endstop connector, and a 10nF capacitor between the STP and GND pins.

If I was certain that the problem was interference on the endstop wires, then we could add filtering to the next revision of the DueX. However, I'm fairly sure that some users have reported getting this i2C comms failure even with no endstops connected to the DueX.

kazolar

There appears to be 3 vectors, end stops, thermistors and fans. I have not had any issues with fans. May just be related to my fan choices. I have had to isolate every endstop going to the duex5, as well as every thermistor going to the duex5. Obviously big wire bundles which include stepper wires are an interference nightmare. I'll run through a calibration routine tonight, as I level out each tool to each other, previously that was a guaranteed way to trigger i2c timeout issues. I may add the caps and filters proactively even if I things are working fine.

DocTrucker

I had issues with interference and false trips on a Duet Ethernet with Duex5 when I used a common for the switches to save wiring two cores back to the board for each switch. Swapped back to two cores for each switch and the interference went.

deckingman

@kazolar said in Duet sometimes really slow? - I2C error or?:

I am almost positive at this point these issues have to do with end stop wires having interference on them from the stepper wiring. ....................................

That might be true in your case but for me, I didn't have any I2C issues from when I installed the board in December 2016 until July(ish) 2018. As far as I can recall, I didn't make any hardware or wiring changes at that time. Also, there are no other threads related to I2C errors which pre-date July 2018 and that date is coincidental with the introduction of RTOS 2.0 firmware.

kazolar

Yes, I didn't have these problems with the 1.x versions, there was an i2c rewrite in 2.0, I couldn't go back to a 2.0 version which predates the rewrite easily as I have some custom changes I need to apply and I need to compile the firmware myself. What appears to happen is the i2c rewrite brought those hardware issues to the surface, because i can't blame the code if replacing the end stop wire fixes a problem. Just because it homed fine before the code change, simply says to me that there was hardware (wiring issue) and the new code is just exposing it. I'd rather shake out the wiring bugs now than have them bite me during a 100 hour print.

deckingman

@dc42 said in Duet sometimes really slow? - I2C error or?:

Ian, you can test the status of the endstops on the DueX by creating additional axes and making them visible. If you create 3 axes then the third one will use the E2 endstop input on the Due X. When that axis is visible, the state of the E2 endstop will show up in the Machine Properties page of the old DWC, and in the M119 response.

Yes I thought about that but maybe you don't remember the problems I had when I tried it before - this thread https://forum.duet3d.com/topic/6677/corexyuvwa-3rd-gantry-homing-solved/. I could only get it to work with the introduction of 2.03 beta2 when you introduced mapping of end stops to axes which is not possible in this RC candidate.

I'll see if a can cobble together some sort of configuration that'll make the end stops visible but as I said before, I won't be able to run the exact same sequence that was proven to provoke the problem unless I can map those additional end stops to the XY axes.

Just out of curiosity, why was that facility withdrawn and can you confirm that it'll be available in 3.0?

Cheers

dc42

@deckingman said in Duet sometimes really slow? - I2C error or?:

Just out of curiosity, why was that facility withdrawn and can you confirm that it'll be available in 3.0?

It messed up the M585 command, which is needed by CNC users, and there was no easy fix. Endstop inputs can be mapped in firmware 3.0, see https://duet3d.dozuki.com/Wiki/RepRapFirmware_3_overview.

deckingman

@dc42

David,

I just ran "the sequence" as far as I am able (everything apart from homing the upper gantry with mapped endstops). After which I changed (lowered) the temperature threshold for one of the thermostatic fans connected to the Duex5, to force the fan to come on which it did. Then I toggled the E2 end stop whilst observing the status on the machine properties page and that all worked as normal. After which I ran M122 and noted the following:

I2C nak errors 0, send timeouts 1, receive timeouts 0, finishTimeouts 1, resets 0

Is that what you would expect to see?

Cheers

dc42

@deckingman said in Duet sometimes really slow? - I2C error or?:

@dc42

David,

I just ran "the sequence" as far as I am able (everything apart from homing the upper gantry with mapped endstops). After which I changed (lowered) the temperature threshold for one of the thermostatic fans connected to the Duex5, to force the fan to come on which it did. Then I toggled the E2 end stop whilst observing the status on the machine properties page and that all worked as normal. After which I ran M122 and noted the following:

I2C nak errors 0, send timeouts 1, receive timeouts 0, finishTimeouts 1, resets 0

Is that what you would expect to see?

Cheers

Thanks for testing. I was expecting to see 1 reset as well, because it evidently had one I2C transaction error that it recovered from. I will check the code that counts resets.

EDIT: I omitted to increment the reset count at the proper place. I will fix this in the next RC or final release..

deckingman

@dc42 Cool. Do you want to carry on testing?

kazolar

It appears I triggered a condition, and it appears to recover.
M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.03RC2-YG2 running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DGM-9T6BU-FG3S0-7JTD4-3S06K-1A4ZD
Used output buffers: 3 of 24 (18 max)
=== RTOS ===
Static ram: 25676
Dynamic ram: 96832 of which 0 recycled
Exception stack ram used: 396
Never used ram: 8168
Tasks: NETWORK(ready,648) HEAT(blocked,1232) DUEX(suspended,164) MAIN(running,1688) IDLE(ready,156)
Owned mutexes:
=== Platform ===
Last reset 00:32:18 ago, cause: power up
Last software reset time unknown, reason: User, spinning module GCodes, available RAM 8196 bytes (slot 0)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 8
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 0.0ms, max retries 0
MCU temperature: min 26.6, current 26.7, max 27.7
Supply voltage: min 24.3, current 24.6, max 24.7, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max 0/93
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max 0/242
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max 83/432
Driver 7: standstill, SG min/max 39/449
Driver 8: standstill, SG min/max 0/413
Driver 9: standstill, SG min/max 29/422
Date/time: 2019-05-18 16:08:09
Cache data hit count 4294967295
Slowest loop: 30.12ms; fastest: 0.08ms
I2C nak errors 1, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 169, MinFreeDm: 163, MaxWait: 70775ms
Bed compensation in use: none
Bed probe heights: -0.004 0.002 -0.006 0.002 -0.044
=== DDARing ===
Scheduled moves: 434, completed moves: 434, StepErrors: 0, LaErrors: 0, Underruns: 0, 0
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.2
=== GCodes ===
Segments left: 0
Stack records: 2 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 6.51ms; fastest: 0.03ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 1 of 8
Interface state 5, link 100Mbps full duplex