Duet 2 underruns high loop times stuttering
-
duplication mode -- underruns. This worked before, not now:
11:12:00 PMM122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 2.05 running on Duet Ethernet 1.02 or later + DueX5
Board ID: 08DGM-9T6BU-FG3S0-7JTD4-3S06K-1A4ZD
Used output buffers: 1 of 24 (21 max)
=== RTOS ===
Static ram: 25708
Dynamic ram: 96332 of which 0 recycled
Exception stack ram used: 448
Never used ram: 8584
Tasks: NETWORK(ready,616) HEAT(blocked,1136) DUEX(blocked,164) MAIN(running,1668) IDLE(ready,156)
Owned mutexes: I2C(DUEX)
=== Platform ===
Last reset 00:44:07 ago, cause: software
Last software reset at 2020-04-25 22:27, reason: User, spinning module GCodes, available RAM 8560 bytes (slot 3)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 0
Free file entries: 9
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest block write time: 16.5ms, max retries 0
MCU temperature: min 24.1, current 26.0, max 26.6
Supply voltage: min 24.2, current 24.5, max 25.0, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max 0/333
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max 0/242
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max 21/422
Driver 7: standstill, SG min/max 144/451
Driver 8: standstill, SG min/max 72/422
Driver 9: standstill, SG min/max 45/438
Date/time: 2020-04-25 23:11:57
Cache data hit count 4294967295
Slowest loop: 215.62ms; fastest: 0.08ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Move ===
Hiccups: 0, FreeDm: 157, MinFreeDm: 105, MaxWait: 574595ms
Bed compensation in use: none, comp offset 0.000
=== DDARing ===
Scheduled moves: 1335, completed moves: 1304, StepErrors: 0, LaErrors: 0, Underruns: 0, 31
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 1.0
Heater 1 is on, I-accum = 0.3
Heater 2 is on, I-accum = 0.4
=== GCodes ===
Segments left: 0
Stack records: 2 allocated, 0 in use
Movement lock held by null
http is idle in state(s) 0
telnet is idle in state(s) 0
file is idle in state(s) 0
serial is idle in state(s) 0
aux is idle in state(s) 0
daemon is idle in state(s) 0
queue is idle in state(s) 0
autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 267.21ms; fastest: 0.06ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0) Telnet(0)
HTTP sessions: 2 of 8
Interface state 5, link 100Mbps full duplex -
@dc42 at this point I tried a duplication print which worked fine last week even when I had trouble with triplicate -- and this print is now causing underruns and stuttering. I ask again -- is the board under warranty. I did the same thing that has worked in the past, and it didn't I'm not sure I can get this print to work at all now. I formatted the card, copied the files over and trying it again.
-
@dc42 I had trouble getting all the ear relievers to stay put -- so I slowed down the first layer to 15mm/sec -- after 2 hours that froze up with an underrun -- that's an idex print. Now I wanted to see if it underruns sooner, and tried speeding it up and at 30mm/sec it seems to be sticking OK -- so I ran that -- and that print is giving me much lower loop times. I will wait for another 10 minutes, but this has a shot of working -- is a super slow first layer a problem? I know I've used that technique before with no issues.
-
@kazolar said in Duet 2 underruns high loop times stuttering:
Question -- it's highly likely the problem is 2 fold in 2.05 -- and most likely it's fixed in 2.05.1 -- but the same reason that now results in cold power on or idle - to require 40 minutes of printing to print properly is the reason I'm having this problem.
There was an important bug fix in 2.05.1 (a 1-byte buffer overflow) and the consequences of that bug are unknown. You should definitely use a firmware build that includes that bug fix. It was in file OutputBuffer.cpp.
I can literally power a warm system off for a few minutes -- power it up -- hit print and have it work...cold, I get underruns -- my sense is that's an indication of a hardware fault. Anything software is reproducible every time doing the same thing -- hardware tends to act funny if there is cold solder joint somewhere and magically works when warm.
Tasks: NETWORK(ready,616) HEAT(blocked,1136) DUEX(blocked,164) MAIN(running,1668) IDLE(ready,156)
Owned mutexes: I2C(DUEX)That's significant. It means that the Duet was communicating with the DueX when you ran M122. Were you doing anything that might cause that? For example, changing the speed of a fan connected to the DueX; or toggling an endstop or other switch connected to the DueX?
If it is a hardware problem then I think a likely cause is a poor solder joint between the SX1509B chip on the DueX board and the PCB. We've seen trouble with that before. That could cause spurious input transitions on the pin with the bad joint, leading to extra I2C traffic to read the changed input, leading in turn to increased loop times and underruns.
Do you have any normally-open switches connected to endstop inputs on the DueX5, or to GPIO pins on the DueX5?
I will add a counter in RRF3 to count the number of I2C transactions and display a transactions/minute count in the diagnostics. As you are building your own firmware, you could add a similar count in your RRF2 build.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 this print is only 2 extruders, but the all z axis motors are on duex5, one of the fans used by the 2 extruder is on duex5. That's it. The 2nd extruder has some minimal stuff overflowed to duex5.
Yes I am building my own firmwareCan you give me the code pointer to add the counter and I'll do that.
I'll switch over 2.05.1 in the process.
if you're saying it's the duex5, it would make some sense. There was one instance when I just rerouted a wire going to it, just moved it for better management, didn't disturb anything, just unplugged an end stop and plugged it back in the on power up, duet refused to boot. It kept cycling. I disconnected everything from the duex5, duet booted up. Then I plugged it all back in, one at at time and everything worked again. So then it sounds like I need a new duex5. I've had i2c issues with it before. Seems my duex5 might have been glitchy for a while, and the times the z axis motors got singed could have caused some issues with it.
-
@dc42 I don't use normally open, you guys don't recommend normally open, and it makes sense not use it. I have 3 normally closed switches and the rest are optical powered switches which also act normally closed.
The duex5 is older is also purchased from filastruder in May 2017. Is that under warranty, or do I need to order a replacement. Kinda sucks cause I just had that board out when I replaced the 4 z axis connectors. I'm not that proficient in microsoldering to try to reflow an smd chip. -
I just want to clarify something:
You have the 5 stepper drivers on the Duet board, are they all being used?
Then you have 5 more stepper drivers on the duex, are they all being used?
And you also have 4 external drivers, driving Y, U, E0 and E1.
It seems that may be more motors than the firmware can handle? I thought 11 or 12 was the most RRF2 could manage. If you are using all of these, that's 14. Can you clarify?
-
@bot yes. dc42 helped me add 2 more. It's not in the official firmware because is 2 extra ops which would normally not be used by anyone else with a duet 2. It has worked for the last 3+ years perfectly fine. There is a way to reuse the pt100 pins for on duet and duex5. Same way that the LCD pins are being reused for 2 extra steppers.
-
@kazolar Ahh, gotcha. Thanks.
-
@dc42 so we're back the i2c mess. I looked through some other posts regarding i2c -- and I saw some related to not running wires next to or along the ribbon connecting the 2 boards...agh...that's the change that I did make when I added extra cable chains, I changed some wiring paths inside the enclosure. God help me, i2c with duet+duex5 is so glitchy. I re-routed cables around the ribbon, best I could, it's kinda tricky now that the case isn't really designed to do that, but -- OK, I found ways. I just started a print -- same one that failed 2 times yesterday before somehow magically working to completion overnight. Well I am running it now, and all loop times are low -- 3-4ms -- it's early in the print, but that looks promising. So it's not just the heavy grounding wire, but the ribbon needs to be clear of anything -- that might appear to be the issue here -- not defective hardware, or cold solder or firmware -- but cross talk on that ribbon. I'm using some insulated ribbons for my PC -- PCIEX extension ribbons -- they're expensive -- about $30 per, but they have a lot of protection against this kind of cross talk, shielding and such, is that something that is worth considering -- I am going to design a new case to split the duet and duex5 apart from each other the way it's now -- they're one on top of each other (as I believe it's intended) but that leads to some odd wiring runs that make it very difficult to avoid the ribbon.
I will do a triple head print later tonight -- see if that works without a reboot -- so far too far into this duplication print with very low loop times to consider a possible failure. -
@dc42 question -- as it seems (too early to conclude) - that i2c is to blame for the issues -- why is i2c not showing any errors or timeouts in m122 reports -- freeing up the ribbon from interference from heater and other wires appears (for now) to have solved the issue -- but how come all this i2c interference -- slow loop times and no i2c errors... would have been too obvious and easy to investigate that path if there were some - poor SD cards got blamed and they appear to have been faultless.
-
well that was it -- I just did a bunch of starts and stops of a triple headed print -- trying to re-acquire my z offset and I did not reset anything and the current print is running with normal loop times, this would have inevitably triggered high loop times previously. Good to know no physical defect on anything -- just that ribbon cable must be treated like it's a newborn -- I didn't even think about it when I rerouted all the wires to the 2nd cable chain how they were connecting to where they had to connect to. I am still confused why i2c never showed any errors or timeouts or anything -- just increased the loop time tremendously. Thank you for your patience in sorting this out -- I do wish this ribbon came shielded -- I'd gladly pay extra for a shielded 50 pin cable -- I've been searching for one -- honestly at this point $100 for a cable that would not be bothered by things around it would be a bargain.
-
Glad we found the smoking gun.
Doing some googling it would appear that there is shielded ribbon cable available from digikey and mouser, and in certain other hobbies it looks like it's often DIYed with metallic tape.
-
I've thought about doing that, but I know how the PC shielded PCIEX extensions are made -- but I have tried that with similar cheap PCIEX cables -- putting HVAC tape around it and then putting regular insulation so that aluminum tape wouldn't short anything, and it didn't make the cable perform any better. I had purchased some inexpensive similar IDE style extensions which claimed to support PCIE-4X, and PCIE-4X devices would not work with them, 1X would -- Doing the DIY trick of insulting etc, did not help. Getting a cable for 4x the money that was already shielded worked. So I'd have to hunt around for the proper cable that's premade 50 pin and is the right spacing.
-
this can be marked solved -- I don't know how to.
-
You have to convert it into a question and then mark it as solved. I don't know where that's done but the option should be available to you somewhere.
-
Solved!
Hey I don't suppose you could post a short video of the triplicate printing in action?
-
@Phaedrux sure: not sure how long I'll keep it on my dropbox:
https://www.dropbox.com/s/2fjgf55myqh7rrt/2020-04-27 15-39-26.mp4?dl=0
After covid is over I'll release a full build vlog -
@kazolar some hints about 50pin cable
Normally one whole side of that 50pin connector should be ground and in that case fancy pre-made cables would do miracles. For 50pin a scsi cable would be perfect, but they require whole side to be ground. I guess it was too expensive and too unnecesary to use 2 50pin connectors instead of one to make this possible. Maybe isolating i2c on the connector? (I don't see where the SDA/SCK pins are on the 50pin ?)
Anyhow, the thing that solved my behind many times in the industry is COPPER TAPE. One would assume aluminium tape would work too but no, in many cases where aluminium tape did nothing copper tape saved the day. The one I use mostly is some noname 0.5mm sticky copper tape on a roll. You make sure your cable from connector to connector is totally shielded with copper tape. Do not assume there is connection between two strips of copper tape if they are just glued on top of each other so if you use 2 narrower strips to go top/bottom instead spiraling tape around the cable before attaching to cable solder the two stripes to each other. When you cover the cable with copper tape attach (solder) a ground wire to only one point on the tape and connect to ground on main board. That's about it.
The connectors are on the edge of the board so there's not much that this copper tape around cable can short, but still, good practice is to now add insulation over the copper tape (big heatshrink is best, simple electrical insulation tape works ok, but I'v known to just paint the cable with thick coat of pu
-
@kazolar said in Duet 2 underruns high loop times stuttering:
https://www.dropbox.com/s/2fjgf55myqh7rrt/2020-04-27 15-39-26.mp4?dl=0
Mesmerizing.
