Duet 2 underruns high loop times stuttering
-
@dc42 so we're back the i2c mess. I looked through some other posts regarding i2c -- and I saw some related to not running wires next to or along the ribbon connecting the 2 boards...agh...that's the change that I did make when I added extra cable chains, I changed some wiring paths inside the enclosure. God help me, i2c with duet+duex5 is so glitchy. I re-routed cables around the ribbon, best I could, it's kinda tricky now that the case isn't really designed to do that, but -- OK, I found ways. I just started a print -- same one that failed 2 times yesterday before somehow magically working to completion overnight. Well I am running it now, and all loop times are low -- 3-4ms -- it's early in the print, but that looks promising. So it's not just the heavy grounding wire, but the ribbon needs to be clear of anything -- that might appear to be the issue here -- not defective hardware, or cold solder or firmware -- but cross talk on that ribbon. I'm using some insulated ribbons for my PC -- PCIEX extension ribbons -- they're expensive -- about $30 per, but they have a lot of protection against this kind of cross talk, shielding and such, is that something that is worth considering -- I am going to design a new case to split the duet and duex5 apart from each other the way it's now -- they're one on top of each other (as I believe it's intended) but that leads to some odd wiring runs that make it very difficult to avoid the ribbon.
I will do a triple head print later tonight -- see if that works without a reboot -- so far too far into this duplication print with very low loop times to consider a possible failure. -
@dc42 question -- as it seems (too early to conclude) - that i2c is to blame for the issues -- why is i2c not showing any errors or timeouts in m122 reports -- freeing up the ribbon from interference from heater and other wires appears (for now) to have solved the issue -- but how come all this i2c interference -- slow loop times and no i2c errors... would have been too obvious and easy to investigate that path if there were some - poor SD cards got blamed and they appear to have been faultless.
-
well that was it -- I just did a bunch of starts and stops of a triple headed print -- trying to re-acquire my z offset and I did not reset anything and the current print is running with normal loop times, this would have inevitably triggered high loop times previously. Good to know no physical defect on anything -- just that ribbon cable must be treated like it's a newborn -- I didn't even think about it when I rerouted all the wires to the 2nd cable chain how they were connecting to where they had to connect to. I am still confused why i2c never showed any errors or timeouts or anything -- just increased the loop time tremendously. Thank you for your patience in sorting this out -- I do wish this ribbon came shielded -- I'd gladly pay extra for a shielded 50 pin cable -- I've been searching for one -- honestly at this point $100 for a cable that would not be bothered by things around it would be a bargain.
-
Glad we found the smoking gun.
Doing some googling it would appear that there is shielded ribbon cable available from digikey and mouser, and in certain other hobbies it looks like it's often DIYed with metallic tape.
-
I've thought about doing that, but I know how the PC shielded PCIEX extensions are made -- but I have tried that with similar cheap PCIEX cables -- putting HVAC tape around it and then putting regular insulation so that aluminum tape wouldn't short anything, and it didn't make the cable perform any better. I had purchased some inexpensive similar IDE style extensions which claimed to support PCIE-4X, and PCIE-4X devices would not work with them, 1X would -- Doing the DIY trick of insulting etc, did not help. Getting a cable for 4x the money that was already shielded worked. So I'd have to hunt around for the proper cable that's premade 50 pin and is the right spacing.
-
this can be marked solved -- I don't know how to.
-
You have to convert it into a question and then mark it as solved. I don't know where that's done but the option should be available to you somewhere.
-
Solved!
Hey I don't suppose you could post a short video of the triplicate printing in action?
-
@Phaedrux sure: not sure how long I'll keep it on my dropbox:
https://www.dropbox.com/s/2fjgf55myqh7rrt/2020-04-27 15-39-26.mp4?dl=0
After covid is over I'll release a full build vlog -
@kazolar some hints about 50pin cable
Normally one whole side of that 50pin connector should be ground and in that case fancy pre-made cables would do miracles. For 50pin a scsi cable would be perfect, but they require whole side to be ground. I guess it was too expensive and too unnecesary to use 2 50pin connectors instead of one to make this possible. Maybe isolating i2c on the connector? (I don't see where the SDA/SCK pins are on the 50pin ?)
Anyhow, the thing that solved my behind many times in the industry is COPPER TAPE. One would assume aluminium tape would work too but no, in many cases where aluminium tape did nothing copper tape saved the day. The one I use mostly is some noname 0.5mm sticky copper tape on a roll. You make sure your cable from connector to connector is totally shielded with copper tape. Do not assume there is connection between two strips of copper tape if they are just glued on top of each other so if you use 2 narrower strips to go top/bottom instead spiraling tape around the cable before attaching to cable solder the two stripes to each other. When you cover the cable with copper tape attach (solder) a ground wire to only one point on the tape and connect to ground on main board. That's about it.
The connectors are on the edge of the board so there's not much that this copper tape around cable can short, but still, good practice is to now add insulation over the copper tape (big heatshrink is best, simple electrical insulation tape works ok, but I'v known to just paint the cable with thick coat of pu
-
@kazolar said in Duet 2 underruns high loop times stuttering:
https://www.dropbox.com/s/2fjgf55myqh7rrt/2020-04-27 15-39-26.mp4?dl=0
Mesmerizing.
-
@kazolar, I'm glad you solved it. FWIW I suspect the issue was noise pickup on the interrupt line from the DueX to the Duet. Every time the DueX signals an interrupt, the Duet does an I2C transaction to ask what input has changed. The interrupt line for the chip on the DueX is open-drain and we use a 1K pullup resistor.
Another possibility is that you had a bad connection in the cable on this pin, and moving the ribbon cable has fixed it at least temporarily.
If I'm correct then an I2C transaction count would have been abnormally high. The commit in the v3.01-dev branch of RRF that adds this to the diagnostics is the change to DueXn.cpp in commit 9ead98ce7dc8a4fe1454a7003b14fee1aab51941. But I haven't tested it yet.
I'll check that we enable the deglitch filter on the input used to receive the interrupt signal.
-
@dc42 I did not actually touch the ribbon did not re-seat it -- i just moved everything away from the ribbon, kinda of a hack job now -- but ribbon is freed up. I have to design a new case for this cable arrangement -- I've seen examples of boards sorta facing up sorta next to each other -- I can sort out cooling for them by adding extra 40mm fans to blow across them -- but that lengthens the ground wire run -- would that configuration be worse for i2c transaction -- it would completely eliminate the possibility of any interference to the ribbon.
-
What cables were running over the ribbon cable? Stepper motor cables would be the worst. I doubt that any other sort of cable would be a serious problem, unless you are using very high PWM frequencies on heaters or fans.
Rather than change the layout, it's probably simpler to shield the ribbon cable from the other cables, using aluminium or copper tape, or aluminum foil, or thin sheet metal. It should be grounded, e.g. to a ground pin on the 50-way connector or to the ground pin of an endstop connector. Make sure there is no possibility of it coming into contact with components or connectors on the Duet. One option would be a thin plastic sheet or card with copper tape on top.
Here's an example: https://www.youtube.com/watch?v=8QgT8eiEgFU.
-
@dc42 I had no stepper wires crossing the ribbon, but I did have fans and heaters -- several of them and thermistors ( but I doubt they matter) and several optical end stops -- a bunch of stuff. I actually been thinking of doing a case redesign (after covid -- still printing shields -- not sure when normal manufacturing will catch up) -- so I tried to move 2.05.1 again, and i hit an underrun, maybe I changed too many 32 bit ints to 64 bit -- I was checking vs the build that was working -- I had changed very few. I'll try to start 2.05.1 from scratch -- just take the changes I have in 2.05 that is working and apply them only to 2.05.1 -- the other thing I hadn't noticed is I had power going into the board and going through the loop of the ribbon, so I moved that away. My take on the ribbon is -- I'd rather not mess with it by wrapping up in stuff, either I can find a shielded ribbon that I can buy, or i redesign the case so that basically the side where the ribbon is has not wire access holes. How bad would it be to have the boards next to each other and a longer ground wire -- It will still be a 14 gauge ground wire, just longer.
-
@kazolar said in Duet 2 underruns high loop times stuttering:
How bad would it be to have the boards next to each other and a longer ground wire -- It will still be a 14 gauge ground wire, just longer.
That should be OK. Lots of users do it that way, so do E3D tool changers.
-
@dc42 ok -- I fixed my 2.05.1 -- now it's running on that version. I'll look at 3.0x for i2c count for debug -- see if I can add that in there -- I just don't know what are good or bad numbers -- what's considered high?
-
@kazolar said in Duet 2 underruns high loop times stuttering:
@dc42 ok -- I fixed my 2.05.1 -- now it's running on that version. I'll look at 3.0x for i2c count for debug -- see if I can add that in there -- I just don't know what are good or bad numbers -- what's considered high?
You should get a small number of i2C transactions recorded whenever an endstop or GPIO input on the DueX changes state. That's all. There are additional i2C transactions when DueX fan speeds change, but those are not counted in the stats.
