Solved Duet 3 Mini 5+ VIN under-voltage issues in SBC mode

[droftarts]

@fletcher @paulhew We’ve been testing various new firmwares from @dc42 over the last week, but there’s still an issue. The trouble with this bug is that it can take 24hrs+ to happen! As far as we are aware, it does not happen in Standalone mode, so rather than waste filament on failed prints and raise the level of frustration, disconnect the SBC.

Ian

[Fletcher]

@droftarts Overnight, I had two more events:

I've tried two different PSUs and replaced 100% of the 24v wiring (before finding this thread), so I'm fairly certain it's not a power supply or wiring issue. Only remaining thing I could swap out is the 120V from the power entry module to the PSU, but this really feels like the same issue.

Good news is, I've got a build that seems to reliably generate failures every 12 hours, if you need more beta testers for firmware!

Due to the way my space is laid out, I'd need a wifi bridge (or to re-configure my SBC as a wifi bridge, I suppose) to use the Mini5 Ethernet without the SBC, so I'm down to test firmware for RRF.

[ukwebb]

I too ( as previously mentioned ) was having this issue , and to test the undervolt side I've swapped back to standalone mode, but kept everything else the same, so the pi4 is still running,its still running Duet interface ( although it cant connect! ), its still running the 7" touch hdmi screen and C270 webcam etc etc etc. The only difference is the onboard SD card, and disconnected the GPIO cable.
2 weeks have now passed and not a single undervolt issue.

[dc42]

@ukwebb said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

I too ( as previously mentioned ) was having this issue , and to test the undervolt side I've swapped back to standalone mode, but kept everything else the same, so the pi4 is still running,its still running Duet interface ( although it cant connect! ), its still running the 7" touch hdmi screen and C270 webcam etc etc etc. The only difference is the onboard SD card, and disconnected the GPIO cable.
2 weeks have now passed and not a single undervolt issue.

Thanks, this confirms what we suspected, that it only happens in SBC mode.

[dc42]

I have identified and (I believe) fixed an issue that was contributing to this problem. For anyone who would like to try it, there is a new firmware build at https://www.dropbox.com/sh/b65da04w1gl94nf/AACLpPslfDiWDlJN5KDdPydAa?dl=0.

[PaulHew]

@dc42 Thanks as always for your hard work.
Have loaded and will let you know..

[M-Pixel]

+1 over here. Duet 3 Mini 5+ in SBC mode; SBC is powered by a step-down which is in turn powered by the same 48V supply that powers the Duet, so it's not possible for the Duet to lose power without the SBC also losing power. Occurrence seems completely random - I haven't observed a single common variable between the occurrences (apart from the fact that they all occur on the same printer, of course).

I've been contending with the issue by figuring out approximately where the print stopped, then manually editing the gcode file to pick up where it left off.

I'm loading the April 7th firmware now, and will post diagnostics if the issue occurs again.

[Fletcher]

@dc42 Thank you! I loaded your 7APR firmware at 10:33 AM local time today and left the board sitting idle while I worked. It's now 18:50 with zero VIN under-voltage events, whereas I had at least one during this time-frame each of the last two days (with the machine sitting idle). I'm going to let the machine sit for another 24 hours before I try printing, but, signs are positive!

[PaulHew]

Morning. After loading the firmware last night, I was greeted by this this morning. Machine was idle.

E-Stopped the board and has stopped the VSSA faults.

Have some tuning to do this morning, so will report back.

Regards,
Paul.

[dc42]

@paulhew, please run M115 to confirm the firmware build date/time.

[PaulHew]

@dc42

m115
FIRMWARE_NAME: RepRapFirmware for Duet 3 Mini 5+ FIRMWARE_VERSION: 3.3beta2+1 ELECTRONICS: Duet 3 Mini5plus Ethernet FIRMWARE_DATE: 2021-04-07 13:33:17

Regards,

Paul

[PaulHew]

@paulhew Just had these pop up

They are new!

edit: and it did all of the movements and heating but nothing on the bed.
No blockage. Going back to 3.2.2

edit.edit: was I supposed to upgrade the 1LC firmware also?

Downloaded 3.2.2, upload to board, now I get this.

M997 S0
Error: M997: Failed to find IAP file /opt/dsf/sd/firmware/Duet3_SBCiap32_Mini5plus.bin

P.

[dc42]

@paulhew said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

edit.edit: was I supposed to upgrade the 1LC firmware also?

Yes, if you run 3.3 on the main board then you need 3.3 on expansion and tool boards too.

[PaulHew]

@dc42

[dc42]

@paulhew, thanks for your reports.

I believe I have identified an issue with the DMA controller in the microcontroller, which means that although we are able to detect the onset of the VSSA fault and associated high VIN reading, our attempts to recover from it fail. I have raised a support case with Microchip in the hope that they can provide a workaround, and I await a response.

[PaulHew]

@dc42

[droftarts]

@paulhew We've been hunting this bug down for the last couple of weeks. The VSSA errors are part of the issue; you'll probably see that the readings for VIN, Z probe and thermistors are all wildly out, too. What's happening is that the DMA channels that hold the values for temperatures, VIN, VSSA, VREF and probe get moved around, so the value for temp0 reads the value of VIN, temp1 reads the probe, temp2 reads temp0, VREF reads temp1, VSSA reads temp2, VIN reads VREF, and probe read VSSA. Naturally, the numbers are incorrect, and trigger the errors.

Unfortunately, this looks to be an error in the microcontroller itself, which is why @dc42 is contacting Microchip for help. For now, we don't need any more error reports.

As far as we are aware, this problem only affects the Duet in SBC mode; if you run in standalone, it shouldn't be an issue.

If anyone is running in SBC mode and NOT getting these issues, it would be useful to know your setup: firmware version, Duet 3 Mini 5+ board version, SBC version. I get these errors with a very basic test bench setup, and when the board is idle: Duet 3 Mini 5+ v0.5 on firmware 3.3beta (all of them), 12V PSU, Raspberry Pi 4B (official RPi 5V PSU), DWC 3.2.2, 5x motors connected, resistors in place of thermistors (just to give a constant reading).

Ian

[PaulHew]

@droftarts Ian, I am aware DC has been working hard on resolving this issue, I try and be as informative as possible, hence the screenshots etc.

I think I have found a new error with the latest firmware.
I have M557 defined in my config.g.
Using the latest FW, if I use Dashboard - Compensation & Calibration - Define area, it is blank.
If I set it it loses the info If I check again.

My build is Meanwell 24v 300w PSU Duet 3 Mini 5 +, 1LC toolboard, BLT, 2 fans, 50w heater, slice thermistor on1LC.
1 small strip of LEDs running out of OUT1 on Mini, PanelDue.

I need to get back to 3.2.2 but am struggling to find info on how to go back.

[droftarts]

@paulhew I was just explaining where the problem is, as we understand it now. As such, the screenshots and even M122 reports don't tell us anything extra; they show the result, but not the cause! We've been using M122 P1007 [memory address] to peek at values in memory to work out what's happening, which is not very user-friendly; it crashes the Duet if you get it wrong, then you have to wait for the error to happen again!

The DWC issue you are seeing because you're using DWC 3.2.2 with RRF 3.3beta. You need the new DWC that recognises the new OM values in 3.3beta. At the moment, an updated DWC has not yet been released for SBC mode. There is a DWC 3.3beta for standalone, though.

I'll reply on your other thread about downgrading.

Ian

[Fletcher]

@droftarts

Edit: After about 36 hours of uptime, I got the VSSA fault error too..

I have yet to experience issues on my idling machine. I'll probably try a print tonight and see how it goes.

When I initially installed the upgrade, my extruder thermistor (which is on toolboard TEMP0) was reporting something absurd like 2000C, but turning my 24v PSU off and then on again fixed that. It's been sitting idle with no errors for the last 22 hours.

Setup: 120v wall power, 24V PSU, Raspberry Pi 3B+ with the Canakit PSU, Duet Web Control 3.3.0-b2, 5x motors, live thermistors for bed/hotend. Bed is a 12v bed connected via a MOSFET board, and it's connected to one of the GPIOs rather than the big high-current bed connector. M115 as follows from up-thread:

4/8/2021, 8:21:47 AM 	m115 B121
Duet TOOL1LC firmware version 3.3beta2+1 (2021-04-07 10:51:19)
4/8/2021, 8:21:41 AM 	M115
FIRMWARE_NAME: RepRapFirmware for Duet 3 Mini 5+ FIRMWARE_VERSION: 3.3beta2+1 ELECTRONICS: Duet 3 Mini5plus Ethernet FIRMWARE_DATE: 2021-04-07 13:33:17