Duet2 stops talking to the network randomly

jens55

This issue has started popping up recently. It's not the common 'the Duet is randomly loosing internet connection' thing. Whenever I used to have the connectivity issues before it was always a temporary thing that would fix itself with time. The errors I am getting now do not self correct. I can't even tell my router to force a re-connect. There is zero communication with the network. The only thing that fixes this problem is a power cycle and then it's all back to normal.
There is no effect on the job currently running - it carries on to completion (I think/hope). I am about 50% into a 10 hour print and have lost connectivity but the printer is still happily doing it's thing.
Anybody ever run into that issue?
It is very odd that I can't force a reconnect from my router. That tends to fix most connectivity issues.
I can't check which version of RRF or DWC I am currently running but I believe it is the latest beta or RC.

Edit: I think that I will try talking to the controller via USB when the current print s finished (before the reboot). Maybe I can see what is happening. Besides the M122 command, is there something else I should try that would shed more light on the wifi end of things?

jens55

Here is my M122 result:
Note that the diagnostic says that wifi is connected yet I can't talk to the board with wifi
Pinging the ip gets me 'destination host unreachable
Forcing a reconnect (Ubiquity router) gets me a "Please connect this client to an SSID broadcasting the selected network for the changes to be reflected."
The router shows the Duet being connected to an access point

=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.5.0-rc.1 (2023-08-31 16:16:15) running on Duet WiFi 1.02 or later + DueX5
Board ID: 0JD0M-9P6B2-NJ4S4-6J9DA-3SD6R-KV12L
Used output buffers: 1 of 26 (26 max)
=== RTOS ===
Static ram: 23076
Dynamic ram: 76476 of which 0 recycled
Never used RAM 9824, free system stack 110 words
Tasks: NETWORK(1,ready,88.1%,212) HEAT(3,nWait,3.8%,308) Move(4,nWait,116.9%,261) DUEX(5,nWait,0.0%,26) MAIN(1,running,39.7%,716) IDLE(0,ready,2.0%,29), total 250.4%
Owned mutexes: USB(MAIN)
=== Platform ===
Last reset 47:55:53 ago, cause: power up
Last software reset at 2023-09-22 09:12, reason: User, Gcodes spinning, available RAM 11504, slot 2
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f000 BFAR 0xe000ed38 SP 0x00000000 Task MAIN Freestk 0 n/a
Error status: 0x04
MCU temperature: min 23.6, current 24.9, max 35.5
Supply voltage: min 23.9, current 24.2, max 24.5, under voltage events: 0, over voltage events: 0, power good: yes
Heap OK, handles allocated/used 99/2, heap memory allocated/used/recyclable 2048/232/188, gc cycles 1
Events: 0 queued, 0 completed
Driver 0: standstill, SG min 0
Driver 1: standstill, SG min 0
Driver 2: standstill, SG min n/a
Driver 3: standstill, SG min 0
Driver 4: standstill, SG min n/a
Driver 5: standstill, SG min 0
Driver 6: standstill, SG min 0
Driver 7: standstill, SG min n/a
Driver 8: standstill, SG min n/a
Driver 9: standstill, SG min n/a
Driver 10:
Driver 11:
Date/time: 2023-09-24 10:14:52
Cache data hit count 4294967295
Slowest loop: 549.89ms; fastest: 0.10ms
I2C nak errors 0, send timeouts 1, receive timeouts 0, finishTimeouts 1, resets 1
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 52.7ms, write time 99.3ms, max retries 0
=== Move ===
DMs created 83, segments created 38, maxWait 18920947ms, bed compensation in use: mesh, height map offset 0.000, ebfmin -1.00, ebfmax 1.00
no step interrupt scheduled
Moves shaped first try 56319, on retry 30975, too short 356808, wrong shape 2003805, maybepossible 166552
=== DDARing 0 ===
Scheduled moves 1010489, completed 1010489, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 11], CDDA state -1
=== Heat ===
Bed heaters 0 -1 -1 -1, chamber heaters -1 -1 -1 -1, ordering errs 0
Heater 0 is on, I-accum = 0.1
Heater 1 is on, I-accum = 0.6
=== GCodes ===
Movement locks held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is ready with "m122" in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Q0 segments left 0
Code queue 0 is empty
=== Filament sensors ===
Extruder 0 sensor: ok
=== DueX ===
Read count 1, 0.00 reads/min
=== Network ===
Slowest loop: 393.30ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 0 of 8
=== WiFi ===
Interface state: active
Module is connected to access point
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 2.1beta4
MAC address e0:98:06:22:c9:0c
Module reset reason: Turned on by main processor, Vcc 3.41, flash size 2097152, free heap 29668
WiFi IP address 192.168.1.162
Signal strength -40dBm, channel 1, mode 802.11n, reconnections 0
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0
ok

gloomyandy

@jens55 How is your WiFi network set up, is it a single router or do you have multiple access points in some sort of mesh? In general the esp8266 (used on the Duet boards) does not seem to play well with some "sophisticated" WiFi setups that try to use things like "band steering" and the like, so if you can it might be worth trying to disable any such options you may have. If this problem has only started happening since you installed 2.1beta4 of the WiFi firmware you may want to try switching back to 1.27 to see if that makes the problem go away.

jens55

@gloomyandy, thanks for your thoughts. Yes, I have several access points and you could call the setup 'sophisticated' but I have many devices that use the same wifi module (a lot of esp8266's and what not) without problems. I had forgotten that I had upgraded to the new wifi firmware so going to the old version sounds like the thing to do.
Since the issue is intermittent, I won't be able to report back if it is fixed but if the issue pops up again with the old firmware I will post a follow-up.

jens55

ok, stuid question - do I need to downgrade with the file
DuetWiFiServer.bin or
DuetWiFiModule_32S3.bin
Is there an easy way to see which firmware version is in which file ?

I have not had great luck upgrading firmware - the last two upgrades resulted in trashed systems so I would rather not just try and upload any file without knowing what exactly I am uploading.

Edit: both files have a wildly different byte count between ver 3.4.6 and 3.5.0-rc1 ... do I maybe need to replace both?

gloomyandy

@jens55 You only need to install the older wifi firmware. The file you need is here: https://github.com/Duet3D/DuetWiFiSocketServer/releases/tag/1.27 The file you need to install is DuetWiFiServer.bin

jens55

@gloomyandy <sigh> she be dead Jim ... and I can't find the tri-corder

Anyway, uploaded the old firmware and this is what the wifi section of M122 gives me:
=== WiFi ===
Interface state: idle
Module is idle
Failed messages: pending 0, notrdy 0, noresp 0
Firmware version 1.27
MAC address e0:98:06:22:c9:0c
Module reset reason: Turned on by main processor, Vcc 3.39, flash size 4194304, free heap 30592
Clock register 00002002
Socket states: 0 0 0 0 0 0 0 0
ok

From that I gather that the firmware was uploaded successfully but it is not connected to anything

Did :
m552 s-1 - system reports wifi module stopped
m552 s0 - wifi module started
m558 s"*" - ok
m558 s"network name" p"password" module reports 'ok'

did an M122 and still not connected ....

edit:
M997 s1 uploads the firmware ok but still no connection
A raspberry pi sitting right next to the Duet has no issue with wifi so I am assuming there is plenty of signal from the access point that's maybe 8 ft away.

jens55

I have figured out what was going on. The moral of the story is to NEVER EVER change multiple things at a time. As it turned out, while the particular printer was down, I looked at the wireless connection and noticed the printer was always connected to an access point that was further away. I decided to lock the printer to the closest access point. For some reason or other which is still to be determined, the printer is not able to talk to the closer access point and since it was locked to it, it could not connect to the network at all.

dc42

@jens55 I'm glad you solved it.

jens55

@dc42, it gets stranger .... turns out that if I force the more distant access point off line, the Duet will happily talk to the closer AP. I am writing this off as a quirk of Ubiquity software or hardware. I am also writing myself a note not to try and force the access point to do anything it doesn't want to. No more locked AP for me. Only time will tell if the firmware downgrade fixed the original issue.

dc42

@jens55 do you have multiple access points with the same SSID, operating on different WiFi channels? If so, please see https://forum.duet3d.com/topic/33673/duet-3-6hc-wifi-module-chooses-wrong-access-point/3.

jens55

@dc42, Yes, I have multiple access points with the same SSID. Your scenario of starting at the lowest channel and picking the first AP it can connect to fits the situation perfectly except when the Duet was locked to the closer AP and decided to not connect at all.
It is actually hilarious that all this started with my reply to @NeoDue where I suggested locking the printer to the closer AP .... which I then did myself which resulted in a full day of drama.
I just finished downgrading the wifi module firmware so the M537.1 and M537.2 commands are not available to me but I would suggest the outcome would be exactly like what @NeoDue has posted in the referenced thread.
I have made piece with the idea that locking the Duet to a specific AP is bad karma. Everything is working and I am not touching a thing lest it all goes south again!