Duet2 wifi stuck repeating Wifi Error.
-
So... I'm preparing to leave for BSDCan. Printer has been idle for about 24h. I was in the area and saw this:
https://nextcloud.towernet.ca/s/FNtRo4MyJHiG7PR
So then I checked the console...
WiFi: I (142798071) wifi:bcn_timout,ap_probe_send_start WiFi: I (142818052) wifi:state: 5 -> 0 (6c0) ESP reported status change Error: WiFi module reported: Lost connection, auto reconnecting ESP reported status change WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(179): wifi evt: WIFI_EVENT id: 5 ESP reported status change WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(199): disconnect reason: 6 WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(469): Lost connection to AP WiFi: I (142820117) wifi:state: 0 -> 2 (b0) WiFi: I (142820123) wifi:state: 2 -> 3 (0) WiFi: I (142820130) wifi:state: 3 -> 5 (10) WiFi: I (142820174) wifi:connected with ac.DaveG.ca, aid = 1, channel 6, HT20, bssid = b4:fb:e4:d7:be:cc WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(179): wifi evt: WIFI_EVENT id: 4 ESP reported status change Error: WiFi module reported: Auto reconnect succeeded ESP reported status change WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/SocketServer.cpp(179): wifi evt: IP_EVENT id: 0
followed by...
WiFi: I (190597586) wifi:bcn_timout,ap_probe_send_start
Now... this wasn't a repeating error, but I'm reporting it anyways.
-
@zBeeble Seems like the disconnect was detected at the lower levels of the ESP WiFi stack. The upper layers did what it's supposed to and attempted a reconnect. I don't think this was what causes the repeated SPI timeout error - but noted nontheless.
-
Would you guys expected this debug wifi to be significantly slower?
-
@zBeeble You mean in terms of like upload/download speed? Debug builds would be 'unoptimized' so yes.
Also, by the way, did the repeated "SPI timeout" issue not appear till now? -
@rechrtb said in Duet2 wifi stuck repeating Wifi Error.:
@zBeeble You mean in terms of like upload/download speed? Debug builds would be 'unoptimized' so yes.
Also, by the way, did the repeated "SPI timeout" issue not appear till now?I didn't notice it until now, but I can't answer with certainty.
-
OK. So ... printers been idle for a week or two --- serial console connected to system that stays online. Nothing to note.
However, was just printing something and the web UI had a few seconds where it was complaining no connection. This is certainly something I noticed before the larger problems.
No activity on the serial console, tho. Should I be entering something to turn that up?
-
@zBeeble Nothing beyond turning on the debug messages for network and WiFi. If I'm correct, you turned off the network debug messages though since it was too noisy? Also, please make sure it's logging to file so we don't miss anything.
-
@rechrtb said in Duet2 wifi stuck repeating Wifi Error.:
@zBeeble Nothing beyond turning on the debug messages for network and WiFi. If I'm correct, you turned off the network debug messages though since it was too noisy? Also, please make sure it's logging to file so we don't miss anything.
Yeah... the networking was generating a line every so often ... printer on or not.
But the machine is connected to the home server ... which is always on. I'm using minicom ... which is set to log all traffic
-
I generally have at least 2 computers (so I expect that to be 4 connections from time-to-time)... go this just now:
WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns
... but can have as many as 3 or 4. Is this number tunable?
This doesn't seem to be my problem, tho... as it should recover.
-
@zBeeble From the https://docs.duet3d.com/en/User_manual/Troubleshooting/WiFi_troubleshooting page, two things that might be relevant:
-
In Duet Web Control, in the Settings > Machine specific > Communication section, check that "Maximum number of AJAX retries" is at least 3.
-
Is the connection stable when the printer is idle, but unreliable when printing? If so then there may be too few CPU cycles to service the network interface, because of an excessive step pulse rate.
- Check the M122 report after a disconnection during a print, or after completing the print, and look at the number of 'hiccups' ('MaxReps' in old firmware versions) figure in the 'DDARing' or 'Move' section of the diagnostics.
- This value should be kept below about 50. If it is higher, reduce either microstepping (M350) or maximum speed (M203).
- Please note, hiccups and MaxReps is reset when you run M122 so only the value the first time you run M122 after a disconnection or completion of a print is significant.
Edit: Sorry, the above is old info, and MaxReps isn't reported in the current firmware M122 report. I've asked @dc42 how this information is reported now.
Edit: Updated the above with info from dc42.Ian
-
-
@zBeeble said in Duet2 wifi stuck repeating Wifi Error.:
as generating a line every so often ... printer on or not.
But the machine is connected to the home server ... which is always on. I'm using minicom ... which is set to log all traffic
Reply
Those "already 4 conns" should be normal, especially when there are multiple clients with the web interface open.
-
So... been printing OK... for awhile. Just started acting up today again. Oddly, the serial output is "chunky" .... and incomplete. One recent query:
m122 === Diagnostics === RepRapFirmware for Duet 2 WiFi/Ethernet version 3.5.1 (2024-04-19 14:40:46) running on Duet WiFi 1.02 or later Board ID: 08DDM-9FAM2-LW4SD-6J9F8-3SN6J-T3ZRY Used output buffers: 1 of 26 (26 max) === RTOS === Static ram: 23256 Dynamic ram: 72928 of which 12 recycled Never used RAM 13540, free system stack 110 words Tasks: NETWORK(1,ready,15.7%,175) HEAT(3,nWait 5,0.1%,307) Move(4,nWait 5,2.9%,298) MAIN(1,running,81.3%,711) IDLE(0,ready,0.0%, 29), total 100.0% Owned mutexes: WiFi(NETWORK) USB(MAIN) === Platform === Last reset 10:03:58 ago, cause: power up Last software reset at 2024-07-29 10:46, reason: User, Gcodes spinning, available RAM 17140, slot 2 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a Error status: 0x04 Aux0 errors 0,0,0 MCU temperature: min 35.4, current 35.9, max 36.9 Supply voltage: min 11.0, current 12.2, max 12.3, under voltage events: 0, over voltage events: 0, power good: yes Heap OK, handles allocated/used 99/6, heap memory allocated/used/recyclable 2048/932/816, gc cycles 1 Events: 0 queued, 0 completed Driver 0: ok, SG min 0 Driver 1: ok, SG min 0 Driver 2: ok, SG min 0 Driver 3: ok, SG min 0 Driver 4: standstill, SG min n/a Driver 5: Driver 6: Driver 7: Driver 8: Driver 9: Driver 10: Driver 11: Date/time: 2024-07-30 01:04:58 Cache data hit count 4294967295 Slowest loop: 11.10ms; fastest: 0.20ms I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0 === Storage === Free file entries: 9 SD card 0 detected, interface speed: 20.0MBytes/sec SD card longest read time 1.5ms, write time 0.0ms, max retries 0 === Move === DMs created 83, segments created 23, maxWait 448ms, bed compensation in use: mesh, height map offset -0.031, max steps late 0, m in interval 0, bad calcs 0, ebfmin 0.00, ebfmax 1.00 next step interrupt due in 20 ticks, disab
-
Notably, on one output not shown, "slowest loop" was around 146ms.
-
I've turned on the debugging for 14 and 1 (wifi and network). Lots of chatter, but the printer is definitely acting up. The serial is no longer chunky, but at one point, I disconnected the serial ... which would reset the driver on the server.... and probably the duet, too.
m111 and chatter look like:
Debugging enabled for modules: Network(1 - 0xffffffff) WiFi(14 - 0xffffffff) Debugging disabled for modules: Platform(0) Webserver(2) Gcodes(3) Move(4) Heat(5) Kinematics(6) InputShaping(7) unused(8) Print Monitor(9) Storage(10) PortControl(11) DuetExpansion(12) FilamentSensors(13) Display(15) SbcInterface(16) CAN(17) Expansion(18) New conn on socket 0 for local port 80 Found responder Received 367 bytes New conn on socket 1 for local port 80 Found responder Received 329 bytes New conn on socket 0 for local port 80 Found responder New conn on socket 1 for local port 80 Found responder Received 329 bytes Received 367 bytes New conn on socket 0 for local port 80
-
-
Have you tried updating to 3.5.2 yet? Or the 3.6 alpha release? I think the beta is upcoming as well.
-
The printer was idle, so I shut off the constant network chatter. wifi debug was still on (as it's fairly quiet). During the idle day, it said this:
Debugging enabled for modules: WiFi(14 - 0xffffffff) Debugging disabled for modules: Platform(0) Network(1) Webserver(2) Gcodes(3) Move(4) Heat(5) Kinematics(6) InputShaping(7) unus ed(8) PrintMonitor(9) Storage(10) PortControl(11) DuetExpansion(12) FilamentSensors(13) Display(15) SbcInterface(16) CAN(17) Exp ansion(18) WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns
Now.... 2 computers look at the web server and the web server is on an RFC 1918 address behind NAT ... so not general internet exposed.
-
@Phaedrux said in Duet2 wifi stuck repeating Wifi Error.:
Have you tried updating to 3.5.2 yet? Or the 3.6 alpha release? I think the beta is upcoming as well.
I'm a little bit upgrade adverse ... you can convince me, but equally, I don't believe this is upgrade related.
-
OOB ... looking at your debug message ... you could probably significantly cut binary size by cutting /home/renz/Projects from the path of all debug messages --- ie: cut to the root of the project.
-
OK. Now it's full-on inaccessible.
Lots of these lines scrolling on the console.
ResponseTimeout, pending=1 ResponseTimeout, pending=1 ResponseTimeout, pending=1 WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns ResponseTimeout, pending=1 WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns ResponseTimeout, pending=1 WiFi: /home/renz/Projects/Duet3D/WiFiSocketServerRTOS/src/Connection.cpp(578): refused conn on port 80 already 4 conns
-
And just to be clear, two local computers are pointing at it and no network traffic (as expected) is coming from the router for it.