Wifi 2.1beta6 from 3.5.0-rc.2/3 still disconnecting

droftarts

@Chriss said in Wifi 2.1beta6 from 3.5.0-rc.2/3 still disconnecting:

Do you want to tell us how you have fixed it? Is it by doing a full reset of the WiFi module after a connection lost? Or was there a real issue with the board firmware? (I'm just curious because the disconnects are not new, the not recovery was new.)

From a conversation we had with @rechrtb

Ok, I think I may have found a fix to the issue. The reason I say 'may' is because as you might imagine, this issue seems to be very intermittent - I have only been able to reproduce it that one time last week.
But in trying to debug this problem, I inserted a bunch of debug printf's that I was able to get the same symptoms namely:

board seems to disconnect and never reconnect again - unless module is disabled and re-enabled

module led is still on, but board is unpingable

repeating "responsebusy" and "bad recv status size"

Ok, so the issue I found is that one of the tasks block indefinitely on https://github.com/Duet3D/WiFiSocketServerRTOS/blob/dev/src/Connection.cpp#L694.
Inserting the printf's must've slowed things down enough that simulated the connectionQueue to be backed up. There is supposed to be a task that consumes events from this queue, but since this callback occurs on the lwip task - if that consumer task calls and lwip function, it might also lock up.
Increasing the queue size seems to have alleviated the issue. I have set the size to MaxConnections * 3 , since there are three types of connection events that can be enqueued in connectionQueue :
Accept, Close and Terminate.
That said, it is probably still needed to verify if this is the issue @Chriss encountered. Since they have multiple boards, I'll probably advise them to load this firmware onto one of the boards, while the other retains the current firmware - to see if the 'fixed' version has reduced occurrences of the disconnects.
Though long term, I'll probably think of potentially better ways to refactor this part of the code.

The relevant fix is here: https://github.com/Duet3D/WiFiSocketServerRTOS/commit/0f8bdc18f2968ee357cdb09d1319590abb7cdd08

Ian

0

rechrtb committed to Duet3D/WiFiSocketServerRTOS

Tentative disconnection fix

rechrtb

@Chriss Hello, as in @droftarts response, I might have managed to re-create the instance in which the WiFi module firmware locks up - which is consistent with the symptoms you displayed. With the fix, I wasn't able to recreate the lock up anymore 'artificially'. Now, we wait to confirm maybe we can't recreate the lock-up 'naturally'.

As it is a very intermittent issue, it's hard to say when we say the issue is fixed. It has to be long term test, but when the beta6 board encounters the issue and beta7 still does not, I think it can be a good sign.

Chriss

@rechrtb @droftarts

Thanks for the information, I appreciate that very much.
I was very busy during the last days so I skipped to tell you: "Yes, it still works" every day.

What I can tell so far is that the error is gone. I was printing a lot with the beta7 board in since the upgrade (many under 30minutes prints) and the WiFi connection was very stable. I would vote with a "hump up" and would say that the problem is gone.

I work with my beta6 board at the moment a lot (IDEX setup is a pain) and I saw the problem here twice since yesterday evening.

I guess you guys want to close the case now and release beta7 officially. Thank you very much for your good support, I felt very compy during the process. And I'm more than happy that it was not a stupid wrong config on my site this time.

Cheers, Chriss

droftarts

@Chriss I'm not quite sure why this thread is in the STM category. I'll move it to the beta firmware category.

Ian

Chriss

@droftarts Hahaha... I started it in the Beta, somebody moved it to here.

droftarts

@Chriss Hmm... not as far as I can see! Anyway, I'm glad it's working well again now. It was difficult to track down! I'll mark the thread as solved. Hopefully @rechrtb will do the beta7 version as a release soon.

Ian

Chriss

@droftarts Maybe my bad than. I only remember the one of my threads where moved, maybe a other one.

Thanks you very much! Glad that we found it and it is stable now and all of us can concentrate on other things.

jensus11

@rechrtb Do you give me the 2.1 beta7, please? I have terrible issues with my connection. That is so bad, that I have to restart the printer on every time.

dc42

@jensus11 here's a copy. DuetWiFiServer_beta7.bin

jensus11

Thanks, for the first time it looks really better.