Connectivity issues with 6HC
-
Hi,
As @Phaedrux recommended to me, I'm making a thread about my connectivity issues with the 6HC.
I'm having random connection losses just like @e4d described in this thread.
Having multiple people connected to the machine hinders the connectivity performance a lot.
mDNS isn't working properly also. The duet.local/ only works on Windows weirdly and multicast discovery in RRF3.4.4 changed nothing to this behavior.I'm currently running a custom version of RRF3.4.4 and DWC3.4.0 (That I'm updating to 3.4.4 but it seems that there is not that much changes).
Running in SBC mode fixes all issues so I'm thinking it's a hardware limitation. I just can't go with an SBC for sourcing issues and I'm having a lot of bugs related to meta commands (that I rely on heavily).
I could try connecting the Duet directly to the computer but I'm trying to have the networking working correctly as my printers could be delivered to clients.
Is there any way to diagnose the connectivity errors with internal log or something external like Wireshark?
Thanks in advance,
Hugues
-
@Hugsy said in Connectivity issues with 6HC:
Wireshark?
Grabbing a wireshark capture on a PC experiencing the issue would be a good start.
-
@Phaedrux So, I see a lot of TCP errors such as :
- TCP Spurious retransmission
- TCP Out-of-order
- TCP Dup ACK
- [RST,ACK]
10.0.1.235 is the Duet
I can also see some HTTP notifications from the Duet board such as "HTTP/1.1 503 Service Unavailable"
I'm on a separate router that doesn't have internet, with only one computer with 5 windows of DWC wired in.
If you want I can send you a dump of it.
-
@Hugsy please can you upload the Wireshark trace as an attachment.
Also how many clients are you trying to connect at once and do you see this issue with only a single client attached to the 6HC?
-
@T3P3Tony I'm sending you the Wireshark trace via PM as there might be confidential data in it.
I have about 3/4 clients, just the lab operators and be basically.
I also have this console capture from the chrome console inspect tool.
-
@Hugsy thanks. please do try with just one connection to see if that stops the crashes so we can be sure its due to the number of connections.
Also It would be helpful if you could load a non customised version of RRF and DWC (3.4.5) and see if the issue persists.
-
@T3P3Tony Okay I'll try loading some non-customized versions of RRF and DWC and see if it helps. Before we customized DWC we still had this issue but we always ran a custom RRF.
-
@T3P3Tony I still have the same issues with non-customized versions RRF3.4.4 and DWC3.4.4, I'll upgrade to 3.4.5 now.
With only one tab of DWC everything seems to be fine, but when I add a second one the problems starts to appear.
Crashes are random but one good way to crash an upload on tab 1 of DWC is to reload any other tab. -
@T3P3Tony RRF3.4.5 and DWC3.4.5 are less problematic, I can still get it to crash with multiple tabs open but way more than on 3.4.4 (for both RRF and DWC). I'll look into the commits to see what changed and merge them into our branches. Still no idea of what is going on.
-
@Hugsy thanks for the update. Please try and capture the issue when occu4ring with 3.4.5 on wireshark, also if possible connect a USB temrinal and send M122 after a crash.
-
@T3P3Tony Hi, on Wireshark I see a few "Destination unreachable (Network unreachable)" over ICMP protocol. This happens when I have multiple (6) DWC tabs open, as usual some tabs have their connection interrupted when of them is refreshed. This is with RRF3.4.5 and DWC3.4.5 non-customized.
Funny enough, I merged DWC3.4.5 into our custom DWC branch and the performance is similar between custom and non-custom DWC. I'm thinking it's more related to software.
I'll try some diagnostics via usb, M122 B0 and a M111 P1 S1.
-
@Hugsy thanks, and yes the diagnostics will be helpful. One thing that I note is that you have a maximum of 8 http sessions so you may be hitting that limit, M122 will let you know, although you only get 1 session per IP client address.
-
@T3P3Tony Oh wait, that's interesting. We're not hitting the 8 http sessions limit (I did quite some tests with that setting in the firmware), but you're telling me a physical client can have as many tabs as he wants and it will only generate 1 http session ?
Is it a hardware limitation of the ethernet chip or SoC on the 6HC to be limited on the number of sessions ? -
@Hugsy do not have more than one client from 1 IP address or else you will get odd behaviour with replies being consumed by one and thus not showing up in the other.
The session limit is due to memory limitations on the Duet MCU.
-
@T3P3Tony Okay, we can work with a 1 tab per client. Although we observed that 2 different IP clients can consume each other replies already, and while I did tests with one client that has multiple tabs, multiple clients with one tab each lead to the same problems. Do you want a multi-Wireshark session with different clients ?
-
@Hugsy before grabbing more wireshark essions I would like to see what @chrishamm thinks.
-
@Hugsy said in Connectivity issues with 6HC:
I just can't go with an SBC for sourcing issues and I'm having a lot of bugs related to meta commands (that I rely on heavily).
I use a lot of meta commands myself in SBC mode and I'm not aware of any bugs. Can you elaborate?
I'll have a look at your Wireshark dumps when I find some time.
-
@chrishamm Hi Chris,
It was at RRF3.3,
if I recall correctly it was the syntax on some commands that changed the motors current, it was a long time ago and I think these commands were changed (in our macros) so it's not relevant anymore.Actually I found the issue by looking into it, we rely on a lot of global variables in the machine and their support started at RRF3.4 on SBC mode, so it's not a problem anymore. -
@Hugsy I've had a chance to check out your Wireshark dumps and I believe the Duet is overloaded by network requests. For a similar reason we don't recommend putting your Duet on the public internet. Here some ideas how you can reduce the number of simultaneous requests:
- Use only one browser window (DWC instance) per client PC
- Increease the AJAX request interval in Settings -> Machine
- Upgrade your setup to SBC mode (it doesn't have to be a Pi although that's the easiest way)
We're increasing the number of maximum TCP connections in the underlying IP stack in the next firmware version, perhaps that will resolve the spurious disconnects as well.
