Duet2 connection interrupted: Let's get to the bottom of this
-
@phaedrux said in Duet2 connection interrupted: Let's get to the bottom of this:
@nxt-1 Would you be able to collect a wireshark trace?
1.pcapng.c
2.pcapng.c
3FullDataset.pcapng.c
Here you go, the .c extension needs to be removed abviously, the forum doesn't allow from .pcapng files The first two datasets were started when I notices warning popping up in dwc, so they are probabily missing the first pieces of vital information. The 3rd set contains everything from start to finish.Note: all these traces were without the new v1.26 debug firmware as sugested by @chrishamm. I will install that now and report back.
-
@chrishamm said in Duet2 connection interrupted: Let's get to the bottom of this:
@nxt-1 Yes, the part where the DWC loses connection to the Duet should suffice. In other interest there is now v1.26 with extra debugging available. If you want to give it a try and a connection drop occurs, please send
M111 P14 S1
followed byM122
and share the full output here. It may help us further to isolate the underlying problem.As requested: as soon as a connection interupted warning poped, I did a
M11 P14 S1
followed byM122
.7/30/2021, 11:16:00 AM M122 === Diagnostics === RepRapFirmware for Duet 2 WiFi/Ethernet version 3.3 (2021-06-15 21:44:54) running on Duet WiFi 1.0 or 1.01 Board ID: 08D6M-91AST-L23S4-7JTD0-3S86K-1NX1K Used output buffers: 3 of 24 (24 max) === RTOS === Static ram: 23876 Dynamic ram: 74668 of which 24 recycled Never used RAM 15848, free system stack 94 words Tasks: NETWORK(ready,91.8%,143) HEAT(delaying,0.9%,330) Move(notifyWait,2.4%,301) MAIN(running,4.5%,128) IDLE(ready,0.4%,29), total 100.0% Owned mutexes: WiFi(NETWORK) === Platform === Last reset 116:49:31 ago, cause: power up Last software reset at 2021-07-25 14:11, reason: User, GCodes spinning, available RAM 15848, slot 2 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a Error status: 0x14 Aux0 errors 0,0,0 Step timer max interval 0 MCU temperature: min 30.9, current 36.0, max 36.9 Supply voltage: min 24.1, current 24.2, max 24.5, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0 Driver 0: position 91505, standstill, SG min/max not available Driver 1: position 73958, standstill, SG min/max not available Driver 2: position 73795, standstill, SG min/max not available Driver 3: position 0, standstill, SG min/max not available Driver 4: position 0, standstill, SG min/max not available Driver 5: position 0 Driver 6: position 0 Driver 7: position 0 Driver 8: position 0 Driver 9: position 0 Driver 10: position 0 Driver 11: position 0 Date/time: 2021-07-30 11:15:41 Cache data hit count 4294967295 Slowest loop: 88.64ms; fastest: 0.15ms I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0 === Storage === Free file entries: 10 SD card 0 detected, interface speed: 20.0MBytes/sec SD card longest read time 1.5ms, write time 87.0ms, max retries 0 === Move === DMs created 83, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000 === MainDDARing === Scheduled moves 68595, completed moves 68595, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 Daemon is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 1927.81ms; fastest: 0.00ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions HTTP sessions: 1 of 8 - WiFi - Network state is active WiFi module is connected to access point Failed messages: pending 0, notready 0, noresp 0 WiFi firmware version 1.26-D WiFi MAC address a0:20:a6:16:ec:ed WiFi Vcc 3.36, reset reason Turned on by main processor WiFi flash size 4194304, free heap 24360 WiFi IP address 192.168.178.30 WiFi signal strength -50dBm, mode 802.11n, reconnections 0, sleep mode modem Clock register 00002002 Socket states: 0 0 0 0 0 0 0 0
7/30/2021, 11:15:47 AM M111 P14 S1 Debugging enabled for modules: WiFi(14 - 0xffffffff) Debugging disabled for modules: Platform(0) Network(1) Webserver(2) GCodes(3) Move(4) Heat(5) DDA(6) Roland(7) Scanner(8) PrintMonitor(9) Storage(10) PortControl(11) DuetExpansion(12) FilamentSensors(13) Display(15) LinuxInterface(16) CAN(17)
7/30/2021, 11:15:36 AM Connection established 7/30/2021, 11:15:31 AM Connection interrupted, attempting to reconnect... HTTP request timed out
EDIT: After a few more disconnects, another
M122
incase it is usefulM122 === Diagnostics === RepRapFirmware for Duet 2 WiFi/Ethernet version 3.3 (2021-06-15 21:44:54) running on Duet WiFi 1.0 or 1.01 Board ID: 08D6M-91AST-L23S4-7JTD0-3S86K-1NX1K Used output buffers: 3 of 24 (24 max) === RTOS === Static ram: 23876 Dynamic ram: 74668 of which 24 recycled Never used RAM 15848, free system stack 94 words Tasks: NETWORK(ready,13.8%,143) HEAT(delaying,0.0%,330) Move(notifyWait,0.1%,301) MAIN(running,86.1%,128) IDLE(ready,0.0%,29), total 100.0% Owned mutexes: WiFi(NETWORK) === Platform === Last reset 117:01:45 ago, cause: power up Last software reset at 2021-07-25 14:11, reason: User, GCodes spinning, available RAM 15848, slot 2 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a Error status: 0x14 Aux0 errors 0,0,0 Step timer max interval 0 MCU temperature: min 35.6, current 36.3, max 37.0 Supply voltage: min 24.1, current 24.2, max 24.5, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0 Driver 0: position 91505, standstill, SG min/max not available Driver 1: position 73958, standstill, SG min/max not available Driver 2: position 73795, standstill, SG min/max not available Driver 3: position 0, standstill, SG min/max not available Driver 4: position 0, standstill, SG min/max not available Driver 5: position 0 Driver 6: position 0 Driver 7: position 0 Driver 8: position 0 Driver 9: position 0 Driver 10: position 0 Driver 11: position 0 Date/time: 2021-07-30 11:27:55 Cache data hit count 4294967295 Slowest loop: 5.81ms; fastest: 0.17ms I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0 === Storage === Free file entries: 10 SD card 0 detected, interface speed: 20.0MBytes/sec SD card longest read time 1.3ms, write time 0.0ms, max retries 0 === Move === DMs created 83, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000 === MainDDARing === Scheduled moves 68595, completed moves 68595, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 Daemon is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 203.84ms; fastest: 0.08ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions HTTP sessions: 1 of 8 - WiFi - Network state is active WiFi module is connected to access point Failed messages: pending 0, notready 0, noresp 1 WiFi firmware version 1.26-D WiFi MAC address a0:20:a6:16:ec:ed WiFi Vcc 3.36, reset reason Turned on by main processor WiFi flash size 4194304, free heap 24360 WiFi IP address 192.168.178.30 WiFi signal strength -51dBm, mode 802.11n, reconnections 0, sleep mode modem Clock register 00002002 Socket states: 0 0 0 0 0 0 0 0
-
@nxt-1 I'm sorry I forgot to mention it but I'll need the M122 output from a USB console, else the useful stats from the debug build aren't printed. I'll have a look at your Wireshark dumps nevertheless.
PS: Your Wireshark dumps show that the WiFi module sometimes doesn't seem to receive incoming packets. I suspect we could confirm this by looking at the LWIP stats reported over USB.
-
Here you go, two more M122 dumps, taken from my laptop connected via USB.
M122<CR> === Diagnostics === RepRapFirmware for Duet 2 WiFi/Ethernet version 3.3 (2021-06-15 21:44:54) running on Duet WiFi 1.0 or 1.01 Board ID: 08D6M-91AST-L23S4-7JTD0-3S86K-1NX1K Used output buffers: 1 of 24 (20 max) === RTOS === Static ram: 23876 Dynamic ram: 74668 of which 24 recycled Never used RAM 15848, free system stack 134 words Tasks: NETWORK(ready,78.8%,244) HEAT(delaying,0.6%,330) Move(notifyWait,1.5%,303) MAIN(running,18.9%,443) IDLE(ready,0.2%,29), total 100.0% Owned mutexes: WiFi(NETWORK) USB(MAIN) === Platform === Last reset 13:24:33 ago, cause: software Last software reset at 2021-07-31 22:17, reason: User, GCodes spinning, available RAM 15848, slot 0 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a Error status: 0x00 Aux0 errors 0,0,0 Step timer max interval 0 MCU temperature: min 29.4, current 29.7, max 42.3 Supply voltage: min 23.8, current 24.2, max 25.0, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0 Driver 0: position 195079, standstill, SG min/max 0/447 Driver 1: position 195010, standstill, SG min/max 0/464 Driver 2: position 194953, standstill, SG min/max not available Driver 3: position 0, standstill, SG min/max 0/479 Driver 4: position 0, standstill, SG min/max not available Driver 5: position 0 Driver 6: position 0 Driver 7: position 0 Driver 8: position 0 Driver 9: position 0 Driver 10: position 0 Driver 11: position 0 Date/time: 2021-08-01 11:41:44 Cache data hit count 4294967295 Slowest loop: 6.87ms; fastest: 0.17ms I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0 === Storage === Free file entries: 10 SD card 0 detected, interface speed: 20.0MBytes/sec SD card longest read time 1.3ms, write time 0.0ms, max retries 0 === Move === DMs created 83, maxWait 44916359ms, bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 4, completed moves 4, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 Heater 1 is on, I-accum = 0.0 === GCodes === Segments left: 0 Movement lock held by null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is ready with "M122" in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 Daemon is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 17.09ms; fastest: 0.00ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions HTTP sessions: 1 of 8 - WiFi - Network state is active WiFi module is connected to access point Failed messages: pending 0, notready 0, noresp 0 WiFi firmware version 1.26-D WiFi MAC address a0:20:a6:16:ec:ed WiFi Vcc 3.36, reset reason Turned on by main processor WiFi flash size 4194304, free heap 22944 WiFi IP address 192.168.178.30 WiFi signal strength -52dBm, mode 802.11n, reconnections 0, sleep mode modem Clock register 00002002 Socket states: 0 0 0 0 0 0 0 0 ok WiFi: Conns: 0:closePending 80, 59705, 192.168.178.27, 1:free, 2:free, 3:free, 4:free, 5:free, 6:free, 7:free WiFi: IP xmit: 44326 recv: 9906 fw: 0 drop: 848 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 0 WiFi: TCP xmit: 62257 recv: 24835 fw: 0 drop: 95 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 95 opterr: 0 err: 0 cachehit: 22789 WiFi: MEM UDP_PCB avail: 6 used: 4 max: 4 err: 0 WiFi: MEM TCP_PCB avail: 8 used: 1 max: 8 err: 0 WiFi: MEM TCP_PCB_LISTEN avail: 4 used: 1 max: 1 err: 0 WiFi: MEM TCP_SEG avail: 16 u<DEL>MEM PBUF_REF/ROM avail: 10 used: 0 max: 0 err: 0 WiFi: MEM PBUF_POOL avail: 4 used: 0 max: 0 err: 0
M122 === Diagnostics === RepRapFirmware for Duet 2 WiFi/Ethernet version 3.3 (2021-06-15 21:44:54) running on Duet WiFi 1.0 or 1.01 Board ID: 08D6M-91AST-L23S4-7JTD0-3S86K-1NX1K Used output buffers: 1 of 24 (20 max) === RTOS === Static ram: 23876 Dynamic ram: 74668 of which 24 recycled Never used RAM 15848, free system stack 134 words Tasks: NETWORK(ready,14.2%,243) HEAT(delaying,0.0%,330) Move(notifyWait,0.1%,303) MAIN(running,85.6%,423) IDLE(ready,0.0%,29), total 100.0% Owned mutexes: WiFi(NETWORK) USB(MAIN) === Platform === Last reset 13:39:20 ago, cause: software Last software reset at 2021-07-31 22:17, reason: User, GCodes spinning, available RAM 15848, slot 0 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a Error status: 0x00 Aux0 errors 0,0,0 Step timer max interval 0 MCU temperature: min 29.4, current 30.1, max 30.8 Supply voltage: min 24.1, current 24.2, max 24.5, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0 Driver 0: position 195079, standstill, SG min/max not available Driver 1: position 195010, standstill, SG min/max not available Driver 2: position 194953, standstill, SG min/max not available Driver 3: position 0, standstill, SG min/max not available Driver 4: position 0, standstill, SG min/max not available Driver 5: position 0 Driver 6: position 0 Driver 7: position 0 Driver 8: position 0 Driver 9: position 0 Driver 10: position 0 Driver 11: position 0 Date/time: 2021-08-01 11:56:31 Cache data hit count 4294967295 Slowest loop: 14.32ms; fastest: 0.17ms I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0 === Storage === Free file entries: 10 SD card 0 detected, interface speed: 20.0MBytes/sec SD card longest read time 1.3ms, write time 0.0ms, max retries 0 === Move === DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 4, completed moves 4, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1 Heater 1 is on, I-accum = 0.0 === GCodes === Segments left: 0 Movement lock held by null HTTP is idle in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is ready with "M122" in state(s) 0 Aux is idle in state(s) 0 Trigger is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 Daemon is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === Network === Slowest loop: 200.67ms; fastest: 0.08ms Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions HTTP sessions: 1 of 8 - WiFi - Network state is active WiFi module is connected to access point Failed messages: pending 0, notready 0, noresp 1 WiFi firmware version 1.26-D WiFi MAC address a0:20:a6:16:ec:ed WiFi Vcc 3.36, reset reason Turned on by main processor WiFi flash size 4194304, free heap 26096 WiFi IP address 192.168.178.30 WiFi signal strength -54dBm, mode 802.11n, reconnections 0, sleep mode modem Clock register 00002002 Socket states: 0 0 0 0 0 0 0 0 ok WiFi: Conns: 0:free, 1:free, 2:free, 3:free, 4:free, 5:free, 6:free, 7:free WiFi: IP xmit: 57117 recv: 25483 fw: 0 drop: 864 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 0 opterr: 0 err: 0 cachehit: 0 WiFi: TCP xmit: 7106 recv: 39465 fw: 0 drop: 95 chkerr: 0 lenerr: 0 memerr: 0 rterr: 0 proterr: 95 opterr: 0 err: 0 cachehit: 36781 WiFi: MEM UDP_PCB avail: 6 used: 4 max: 4 err: 0 WiFi: MEM TCP_PCB avail: 8 used: 4 max: 8 err: 0 WiFi: MEM TCP_PCB_LISTEN avail: 4 used: 1 max: 1 err: 0 WiFi: MEM TCP_SEG avail: 16 used: 0 max: 14 err: 0 WiFi: MEM ARP_QUEU<DEL>MEM PBUF_REF/ROM avail: 10 used: 0 max: 0 err: 0
-
I wonder if this might be a DHCP issue. Is your Duet configured for DHCP or static IP address? If DHCP, try configuring it with a static IP (that is outside the DHCP lease list range). Aternatively, try changing the lease period expiration on the DHCP server (probably the router) to zero (or whatever = never).
-
@ajdtreyd Man I would have never guessed to look in that direct, good thinking on your part. Although I must admit I have never known DHCP to cause intermittent issues. And I doubt it is the case here. My Duet is indeed in DHCP mode, with a preferred IP set at the DHCP-server (the router in my case), with a default lease time of 10 days. To give some perspective, the issues I described happend multiple times a day, variying amounts but I'd guestimate around 10-15 times/day.
I will try if setting a proper static ip will make any difference just to be sure.
-
@ajdtreyd As promissed, I just tried with a static ip. No change sadly. One more thing to cross off the list I guess
-
Another interesting and possibly related experience:
Most of yesterday, I had substantial problems downloading files from Cura to my Duet2wifi. When you select to print via Cura, it asks you to confirm the file name and then you see a progress window for the download. Most of the time this goes very fast. Well, yesterday the progress would go in little chunks. Maybe 2% at a time, sometimes more, sometimes less, sometimes it would just sit there for a second or ten before moving again.
To me this looks like interference of some kind but short of a spectrum analyser, I have no means of tracing the problem. I am 99.9999% sure this has nothing to do with anything in my household but I have no idea what kind of crap comes in from the outside. I am not sure how the neigborhood would react to me wrapping the house in aluminum foil but I suspect they would not be pleased.Today it is business as usual with a very fast file transfer.
The funny thing is that the printer is in the basement, thick concrete walls and what not so presumably a substantial amount of reduction in any interfering signals. The network access point is about 30 feet away but I have had it sitting right next to the printer with no difference in the issue.
-
@jens55 I feel you, I can't really justify one but I'd love to have capable spectrum analyzer as well. My 200MHz scope fft isn't going to cut it when looking for 2,4 GHz noise . Although I must admit that I doubt this an interference issue, seeing how constant it is. My connection issues happen 24/7 for random lengths of time. Also, and I did have them before I moved as well, eliminating some HAM neighbour going wild on the spectrum
@chrishamm or @Phaedrux Did you manage to figure out anything from the wireshark traces / debug dumps?
-
I'm not sure if @chrishamm has had a chance to view your debug dumps yet.
How handy are you with a soldering iron? Handy enough to replace the wifi module confidently?
-
@phaedrux said in Duet2 connection interrupted: Let's get to the bottom of this:
How handy are you with a soldering iron? Handy enough to replace the wifi module confidently?
I do some sodering every now and then and do have a hot air station available as well. I believe I should be able to get it off.
-
One more question/suggestion; Have you tried reflashing the wifi module ( M997 S1 )?
Back when 3.3 was in pre-release I was doing a bit of version hopping and occasionally the wifi module would give me some erratic connection problems. Sometimes it would take a couple of poweroff/on-flash-poweroff/on cycles to get the wifi stable again, but this has worked for me. Dunno if that will do the trick, but it takes about 5min and no soldering skills to try. -
@ajdtreyd said in Duet2 connection interrupted: Let's get to the bottom of this:
One more question/suggestion; Have you tried reflashing the wifi module ( M997 S1 )?
Back when 3.3 was in pre-release I was doing a bit of version hopping and occasionally the wifi module would give me some erratic connection problems. Sometimes it would take a couple of poweroff/on-flash-poweroff/on cycles to get the wifi stable again, but this has worked for me. Dunno if that will do the trick, but it takes about 5min and no soldering skills to try.I just tried a
M997 S1
and a couple of power cycles, but no relief. -
Sorry about that. Sounds like if the dumps don't provide insight it's gonna be a soldering exercise.
-
@nxt-1 side note, but if you do replace the WiFi module, put some heat protection around the other components, especially the SD slot.
I did mine a while back and it was pretty easy. However, I didn't protect the SD slot and slightly melted some of the internals. It still worked, but no longer clicked in so I had to tape the SD card in until I got round to replacing it.
A bit of kapton tape over it should do -
@nxt-1 My question is this, is the blue LED off, solid on, or blinking during the times your Duet board is not pingable? That would at least tell if the board is connected to the access point or wifi router or hub.