Issue with USB->Serial on Duet 3 Mini 5+

[RYANPDX]

I am having an issue with the USB Serial port being locked up on the Duet 3 Mini 5+ after rebooting or freshly booting the computer, with the USB plug plugged in.

Setup:

• Duet 3 Mini 5+
  • Powered by 24V Power Supply
  • No motors connected.
  • Only three thermocouple inputs and three outputs are connected.
• Debian 11 Linux
  • For purposes of this discussion, attempted both 5.10.x and 6.0.X kernels.
  • Attempted multiple USB ports and separate busses.

Error:
When freshly booting the system with the USB cable plugged in, the serial-port is hung.

If I press the reset button after booting - the terminal becomes responsive.

If I leave the USB cable unplugged and boot the system, then plug in the USB cable - the system is responsive.

To continue troubleshooting, I attached a RS232->TTL adapter to /dev/ttyS0 and changed my config.g to use the additional serial port. (For good measure I forced the USB Serial Baud / Mode)

After setting that up and plugging in the adapter and restarting the machine, you can see that the board itself is working. The web-interface responds, and the second serial port responds as well. However, the USB Serial port is still hung up when the cable is plugged in during boot-up.

1. Connect to USB Serial
2. Observe timestamps. System hung briefly. No response.
3. Connect to hardware serial.
4. Observe responses are coming back. Indicates board is running.
5. Try to connect to USB Serial again - system completely hung. Unable to close terminal.
6. Mash keyboard in frustration.

Walked over to Duet and hit the reset button. Got an error message about writing to tty device. Finally disconnected and closed terminal.

Opened up the port again to see if it was responding after hitting the reset button:

For reference, I have tried the normal things like different cables, boards, etc. etc. etc.

I would also say that this exact same setup works flawlessly on the same hardware (PC) on the Duet 6HC. The USB Serial port does not hang on boot, and is used extensively. I am attempting to replicate that setup to a smaller machine using the Mini 5+

I also found the following strangeness with the USB init in dmseg:

Thank you all in advance, and sorry this is so hard to follow! Please let me know if I can explain it in an easier way, or if any supplemental information is needed.

[dc42]

@RYANPDX I have asked one of my colleagues who runs Linux to do some testing. Meanwhile I have tested the USB port using the YAT terminal emulator running on Windows and established the following:

• The Mini5+ will receive and transmit via the USB port regardless of the state of the virtual RTS and DTR signals set by the host, and regardless of whether the USB connection provides 5V power
• YAT reports that the virtual CTS and DSR signals are not active. Therefore, if your Linux TTY driver or terminal emulator application is configured to require either CTS or DSR before it will transmit or receive, that could cause a problem. Please check the configuration of the driver and terminal emulator you are using.

As all Duet boards using a genuine USB device interface (not a USB-to-serial converter as many older 3D printer control boards did), flow control is intrinsic to the channel and the virtual RTS/CTR/DSR/DTR signals serve no purpose.

Please also confirm that you have tried this with more than one Duet 3 Mini board, to rule out a hardware issue with a particular board.

If you have access to a Windows PC or laptop, you could try connecting to that PC and running YAT, to see whether the issue is specific to using a Linux host.

Finally, please read https://docs.duet3d.com/en/User_manual/Overview/USB_ground_loops and take appropriate precautions against mains ground noise causing problems.

HTH David

[chrishamm]

@RYANPDX A few notes:

1. Please make sure your Duet is on the latest RepRapFirmware version 3.4.4.
2. Setting the USB baud rate in config.g isn't needed. It is overridden by tio anyway when you launch it (and probably by potential Python scripts, too). If you want to use a different baud rate. use the -b option instead (defaults to 115200 in tio).
3. To get proper line outputs in tio, use the INLCRNL mode: tio -e -m INLCRNL /dev/ttyACM0. On Debian Buster, the "local echo" function doesn't work, but it may be different on Debian Bullseye. I haven't tried it there yet.
4. I can confirm that something writes into the Duet's input buffer when both the Duet and PC/SBC are powered on at the same time and it does not happen again when tio is closed and opened again. I am investigating why.

[chrishamm]

@RYANPDX Re my last point: It's actually a flaw in the Linux kernel that causes start-up messages to be echoed back to the Duet. In detail, the cdc_acm module enables TTY + echo modes by default for all ACM/serial devices and you have no way to disable this behaviour unless you recompile the Linux kernel (see here for further details).

I have not had any issues with the USB console actually hanging up my system. Please confirm if the problem persists with RRF v3.4.4.

[RYANPDX]

@dc42 said in Issue with USB->Serial on Duet 3 Mini 5+:

As all Duet boards using a genuine USB device interface (not a USB-to-serial converter as many older 3D printer control boards did), flow control is intrinsic to the channel and the virtual RTS/CTR/DSR/DTR signals serve no purpose.

Very good information, thank you!

@dc42 said in Issue with USB->Serial on Duet 3 Mini 5+:

Please also confirm that you have tried this with more than one Duet 3 Mini board, to rule out a hardware issue with a particular board.

I have confirmed this on three separate boards. I have not yet tested with my 6HC boards but will try and squeeze that in soon. (I don't expect there to be a problem on the 6HC, as we use them in this same configuration.)

@dc42 said in Issue with USB->Serial on Duet 3 Mini 5+:

against mains ground noise causing problems

Confirmed there are no ground loops or leakages.

@chrishamm said in Issue with USB->Serial on Duet 3 Mini 5+:

Please make sure your Duet is on the latest RepRapFirmware version 3.4.4.

I reproduced the issue on both 3.3 and 3.4.4.

@chrishamm said in Issue with USB->Serial on Duet 3 Mini 5+:

Setting the USB baud rate in config.g isn't needed. It is overridden by tio anyway when you launch it (and probably by potential Python scripts, too). If you want to use a different baud rate. use the -b option instead (defaults to 115200 in tio).

Took that out of my configuration, I think it was one of those desperation measures where I was throwing everything at the wall to see what stuck.

@chrishamm said in Issue with USB->Serial on Duet 3 Mini 5+:

To get proper line outputs in tio, use the INLCRNL mode: tio -e -m INLCRNL /dev/ttyACM0.

@chrishamm said in Issue with USB->Serial on Duet 3 Mini 5+:

you have no way to disable this behaviour unless you recompile the Linux kernel (see here for further details).

@chrishamm said in Issue with USB->Serial on Duet 3 Mini 5+:

Please confirm if the problem persists with RRF v3.4.4.

Confirmed in 3.4.4

[RYANPDX]

@chrishamm said in Issue with USB->Serial on Duet 3 Mini 5+:

(see here for further details

Any idea if it would be possible to hack in the USB-IDs of a device with the DISABLE_ECHO flag already in it? Or should I start looking at recompiling with the hack in the driver side?

[chrishamm]

@RYANPDX The easiest way to work-around it would be to open the TTY and to discard all the stuff that's echoed back at first (possibly after the first NL). If you want to have this solved for good, you'll have to modify the cdc_acm driver for our USB IDs (VID 0x1d50, PIDs 0x60ec - 0x60ef) and submit a PR to the Linux kernel to exclude them from TTY mode (aka DISABLE_ECHO) by default. TBH I don't understand why the cdc_acm module doesn't offer any params for this anyway.

[chrishamm]

@RYANPDX Regarding the USB hang issue, I'd really like to reproduce and fix it, but I've been unable to provoke the same problem on a Raspberry Pi 4B. Can you tell me more about the machine running Debian 11?

PS: I just managed to reproduce it, I am looking into it.

[RYANPDX]

@chrishamm I do not believe we have this same issue on the Duet 3 6HC boards, only the Mini 5+. Would it be useful to you for me to double-check that assumption?

[dc42]

@RYANPDX yes it woud be useful.

[chrishamm]

@RYANPDX I'm happy to report that I've got a fix for both problems ready. With my latest version no more messages are written to USB before the device is actually opened and the reboot problem is fixed, too.

The MB6HC does not seem to be affected from either problem.

[RYANPDX]

@chrishamm said in Issue with USB->Serial on Duet 3 Mini 5+:

I'm happy to report that I've got a fix for both problems ready. With my latest version no more messages are written to USB before the device is actually opened and the reboot problem is fixed, too.

Amazing news! Sorry for the delay between responses as always, I'm very appreciative!

@chrishamm do you have an expected release date? Or is there any way I could get my hands on it a bit early?

[chrishamm]

@RYANPDX You can try out 3.5-b1 to get this fix. I don't know yet when we will release 3.4.6.