Duet 3 Mini 5+ VIN under-voltage issues in SBC mode
-
Just had another VSSA.
12/04/2021, 21:12:04 Error: VSSA fault, check thermistor wiring 12/04/2021, 21:12:01 m122 === Diagnostics === RepRapFirmware for Duet 3 Mini 5+ version 3.3beta2+1 (2021-04-12 08:36:30) running on Duet 3 Mini5plus Ethernet (SBC mode) Board ID: A45XG-F396U-D65J0-40KML-1F03Z-H03XP Used output buffers: 4 of 40 (20 max) === RTOS === Static ram: 100072 Dynamic ram: 91932 of which 0 recycled Never used RAM 51700, free system stack 172 words Tasks: Linux(resourceWait,127) HEAT(delaying,322) CanReceiv(notifyWait,774) CanSender(notifyWait,371) CanClock(delaying,340) TMC(notifyWait,99) MAIN(running,484) IDLE(ready,20) AIN(notifyWait,259) Owned mutexes: HTTP(MAIN) === Platform === Last reset 09:07:01 ago, cause: software Last software reset at 2021-04-12 12:04, reason: User, GCodes spinning, available RAM 51700, slot 1 Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a Error status: 0x00 Aux0 errors 0,0,0 Aux1 errors 0,0,0 MCU revision 3, ADC conversions started 39529293, completed 39529292, timed out 0, errs 1 tnd=0 rnd=0 wrd=0 wtd=1 war=1 Supply voltage: min 0.0, current 0.1, max 24.0, under voltage events: 1, over voltage events: 0, power good: no Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0 Driver 0: position 0, standstill, SG min/max 2/2, read errors 0, write errors 1, ifcnt 61, reads 12224, writes 11, timeouts 20730, DMA errors 0, failedOp 0x6f Driver 1: position 0, standstill, SG min/max 2/2, read errors 0, write errors 1, ifcnt 61, reads 32954, writes 11, timeouts 0, DMA errors 0 Driver 2: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 63, reads 32953, writes 11, timeouts 0, DMA errors 0 Driver 3: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 63, reads 32953, writes 11, timeouts 0, DMA errors 0 Driver 4: position 0, standstill, SG min/max 2/2, read errors 0, write errors 1, ifcnt 63, reads 32954, writes 11, timeouts 0, DMA errors 0 Driver 5: position 0, assumed not present Driver 6: position 0, assumed not present Date/time: 2021-04-12 21:12:00 Cache data hit count 4294967295 Slowest loop: 36.07ms; fastest: 0.10ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 0.0MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1, chamberHeaters = -1 -1 Heater 1 is on, I-accum = 0.0 === GCodes === Segments left: 0 Movement lock held by null HTTP* is doing "M122" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === CAN === Messages queued 295399, send timeouts 0, received 393841, lost 0, longest wait 2ms for reply type 6049, peak Tx sync delay 384, free buffers 17 (min 16) === SBC interface === State: 4, failed transfers: 0 Last transfer: 4ms ago RX/TX seq numbers: 54864/54864 SPI underruns 0, overruns 0 Number of disconnects: 0, IAP RAM available 0x118a8 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.2.2 Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 35.62 Maximum length of RX/TX data transfers: 3268/604 12/04/2021, 21:12:00 Error: VSSA fault, check thermistor wiring -
Thanks, everyone. There are two clear patterns emerging.
I am testing a new firmware build overnight, so maybe I will have news in the morning.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 Did MicroChip get back to you. Or is this all you and the teams hard work.....Just to say thank you really ..
Paul
-
@dc42 Just out of interest, what is the critical bit in our logs that assist you DC42?
-
Undervotage.... Printer was idle.
13/04/2021, 09:18:15 m122 === Diagnostics === RepRapFirmware for Duet 3 Mini 5+ version 3.3beta2+1 (2021-04-12 08:36:30) running on Duet 3 Mini5plus Ethernet (SBC mode) Board ID: A45XG-F396U-D65J0-40KML-1F03Z-H03XP Used output buffers: 1 of 40 (23 max) === RTOS === Static ram: 100072 Dynamic ram: 91932 of which 0 recycled Never used RAM 51700, free system stack 172 words Tasks: Linux(resourceWait,119) HEAT(delaying,322) CanReceiv(notifyWait,774) CanSender(notifyWait,371) CanClock(delaying,340) TMC(notifyWait,99) MAIN(running,484) IDLE(ready,20) AIN(notifyWait,259) Owned mutexes: HTTP(MAIN) === Platform === Last reset 12:02:11 ago, cause: software Last software reset at 2021-04-12 21:16, reason: User, none spinning, available RAM 51700, slot 2 Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a Error status: 0x00 Aux0 errors 0,0,0 Aux1 errors 0,0,0 MCU revision 3, ADC conversions started 51519632, completed 51519631, timed out 0, errs 3 tnd=0 rnd=0 wrd=0 wtd=3 war=3 Supply voltage: min 0.0, current 23.9, max 24.0, under voltage events: 1, over voltage events: 0, power good: yes Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0 Driver 0: position 0, standstill, SG min/max 2/2, read errors 0, write errors 1, ifcnt 82, reads 28663, writes 20, timeouts 51838, DMA errors 0, failedOp 0x6f Driver 1: position 0, standstill, SG min/max 2/2, read errors 0, write errors 1, ifcnt 82, reads 14925, writes 20, timeouts 40, DMA errors 0, failedOp 0x06 Driver 2: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 84, reads 14965, writes 20, timeouts 0, DMA errors 0 Driver 3: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 84, reads 14963, writes 20, timeouts 0, DMA errors 0 Driver 4: position 0, standstill, SG min/max 2/2, read errors 0, write errors 1, ifcnt 84, reads 14965, writes 20, timeouts 0, DMA errors 0 Driver 5: position 0, assumed not present Driver 6: position 0, assumed not present Date/time: 2021-04-13 09:18:16 Cache data hit count 4294967295 Slowest loop: 63.01ms; fastest: 0.10ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 0.0MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1, chamberHeaters = -1 -1 Heater 1 is on, I-accum = 0.0 === GCodes === Segments left: 0 Movement lock held by null HTTP* is doing "M122" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === CAN === Messages queued 389993, send timeouts 0, received 519973, lost 0, longest wait 2ms for reply type 6049, peak Tx sync delay 500, free buffers 17 (min 16) === SBC interface === State: 4, failed transfers: 0 Last transfer: 4ms ago RX/TX seq numbers: 36429/36429 SPI underruns 0, overruns 0 Number of disconnects: 0, IAP RAM available 0x118a8 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.2.2 Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 35.63 Maximum length of RX/TX data transfers: 3284/604 12/04/2021, 23:04:48 Warning: VIN under-voltage event (0.0V) -
@dc42 Another VSSA fault chain on the latest firmware:
4/13/2021, 7:41:44 AM Error: VSSA fault, check thermistor wiring 4/13/2021, 7:41:44 AM m122 === Diagnostics === RepRapFirmware for Duet 3 Mini 5+ version 3.3beta2+1 (2021-04-12 08:36:30) running on Duet 3 Mini5plus Ethernet (SBC mode) Board ID: FDQG2-Q296U-D65J0-40KMG-1K03Z-HXFTZ Used output buffers: 1 of 40 (12 max) === RTOS === Static ram: 100072 Dynamic ram: 91764 of which 64 recycled Never used RAM 51804, free system stack 172 words Tasks: Linux(ready,137) HEAT(delaying,322) CanReceiv(notifyWait,774) CanSender(notifyWait,371) CanClock(delaying,340) TMC(notifyWait,99) MAIN(running,498) IDLE(ready,20) AIN(notifyWait,260) Owned mutexes: HTTP(MAIN) === Platform === Last reset 23:27:06 ago, cause: software Last software reset at 2021-04-12 16:13, reason: User, none spinning, available RAM 51856, slot 1 Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a Error status: 0x00 Aux0 errors 0,0,0 Aux1 errors 0,0,0 MCU revision 3, ADC conversions started 100481305, completed 100481304, timed out 0, errs 1 tnd=0 rnd=0 wrd=0 wtd=1 war=1 Supply voltage: min 0.0, current 0.1, max 24.4, under voltage events: 1, over voltage events: 0, power good: no Heap OK, handles allocated/used 0/0, heap memory allocated/used/recyclable 0/0/0, gc cycles 0 Driver 0: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 218, reads 4487, writes 11, timeouts 0, DMA errors 0 Driver 1: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 220, reads 4487, writes 11, timeouts 0, DMA errors 0 Driver 2: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 192, reads 4489, writes 9, timeouts 0, DMA errors 0 Driver 3: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 218, reads 4486, writes 11, timeouts 0, DMA errors 0 Driver 4: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 218, reads 4487, writes 11, timeouts 0, DMA errors 0 Driver 5: position 0, assumed not present Driver 6: position 0, assumed not present Date/time: 2021-04-13 15:41:41 Cache data hit count 4294967295 Slowest loop: 57.18ms; fastest: 0.08ms === Storage === Free file entries: 10 SD card 0 not detected, interface speed: 0.0MBytes/sec SD card longest read time 0.0ms, write time 0.0ms, max retries 0 === Move === DMs created 83, maxWait 0ms, bed compensation in use: none, comp offset 0.000 === MainDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === AuxDDARing === Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1 === Heat === Bed heaters = 0 -1, chamberHeaters = -1 -1 === GCodes === Segments left: 0 Movement lock held by null HTTP* is doing "M122" in state(s) 0 Telnet is idle in state(s) 0 File is idle in state(s) 0 USB is idle in state(s) 0 Aux is idle in state(s) 0 Trigger* is idle in state(s) 0 Queue is idle in state(s) 0 LCD is idle in state(s) 0 SBC is idle in state(s) 0 Daemon is idle in state(s) 0 Aux2 is idle in state(s) 0 Autopause is idle in state(s) 0 Code queue is empty. === CAN === Messages queued 759838, send timeouts 0, received 1013078, lost 0, longest wait 2ms for reply type 6049, peak Tx sync delay 354, free buffers 17 (min 16) === SBC interface === State: 4, failed transfers: 0 Last transfer: 4ms ago RX/TX seq numbers: 14877/14877 SPI underruns 0, overruns 0 Number of disconnects: 0, IAP RAM available 0x118a8 Buffer RX/TX: 0/0-0 === Duet Control Server === Duet Control Server v3.3-b2 Code buffer space: 4096 Configured SPI speed: 8000000 Hz Full transfers per second: 35.89 Codes per second: 0.00 Maximum length of RX/TX data transfers: 3044/804 4/13/2021, 7:41:39 AM Error: VSSA fault, check thermistor wiringIt was stable for almost 24 hours, which is positive change, at least! Thank you for all the work you're putting in on this issue.
-
All, please try the new firmware at https://www.dropbox.com/sh/0mwfe22wepk5efh/AAB6Lp0rwBPjVK2R4SxbKhLya?dl=0. M115 reports it as 3.2beta2+2.
This version avoids using the ADC DMA sequencer completely, so I am hopeful that it will solve the problem. I was worried that the additional ADC interrupt load would be excessive, but with careful code optimisation I have got it down to 0.5% of total CPU time.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 Looks like this version (and probably the other recent ones) breaks duetcontrolserver.
With 3.3b2 the logs are nice and quiet, with 3.3b2+2 the duetcontrolserver log is a fast rolling spam like this:
Apr 13 20:58:38 ender3 DuetControlServer[1585]: [error] Failed to merge JSON due to internal error: {"key":"","flags":"d99fn","result":{"boards":[{"mcuTemp":{},"vIn":{"current":0.4}}],"fans":[],"heat":{"heaters":[]},"inputs":[{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"},{"feedRate":50.0,"lineNumber":0,"state":"idle"}],"job":{"build":null,"duration":null,"filePosition":0,"layer":null,"layerTime":null,"pauseDuration":null,"timesLeft":{"filament":null,"file":null,"slicer":null},"warmUpDuration":null},"move":{"axes":[{"machinePosition":0,"userPosition":0},{"machinePosition":0,"userPosition":0},{"machinePosition":0,"userPosition":0}],"currentMove":{"acceleration":0,"deceleration":0,"laserPwm":null,"requestedSpeed":0,"topSpeed":0},"extruders":[],"virtualEPos":0},"sensors":{"analog":[],"endstops":[null,null,null],"filamentMonitors":[],"gpIn":[],"probes":[]},"seqs":{"boards":0,"directories":0,"fans":0,"global":0,"heat":0,"inputs":77,"job":1,"move":2,"network":2,"reply":0,"scanner":1,"sensors":0,"spindles":0,"state":0,"tools":0,"volChanges":[0,0],"volumes":0},"spindles":[{"current":0,"state":"unconfigured"},{"current":0,"state":"unconfigured"}],"state":{"atxPower":null,"currentTool":-1,"gpOut":[],"laserPwm":null,"msUpTime":101,"nextTool":-1,"previousTool":-1,"status":"off","time":"2021-04-13T20:58:38","upTime":83},"tools":[]},"next":0} Apr 13 20:58:38 ender3 DuetControlServer[1585]: System.InvalidOperationException: The requested operation requires an element of type 'Number', but the target element has type 'Array'. Apr 13 20:58:38 ender3 DuetControlServer[1585]: at System.Text.Json.JsonDocument.TryGetValue(Int32 index, Int32& value) Apr 13 20:58:38 ender3 DuetControlServer[1585]: at System.Text.Json.JsonElement.GetInt32() Apr 13 20:58:38 ender3 DuetControlServer[1585]: at DuetControlServer.Model.Updater.Run() in /home/christian/Duet3D/DuetSoftwareFramework/src/DuetControlServer/Model/Updater.cs:line 157root@ender3 ~ # dpkg -l duetcontrolserver Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=================-============-============-============================================ ii duetcontrolserver 3.3-b2 armhf Control server application for Duet 3 seriesI was hoping to do another batch of small sequential prints with the new firmware
-
FIRMWARE_NAME: RepRapFirmware for Duet 3 Mini 5+ FIRMWARE_VERSION: 3.3beta2+1 ELECTRONICS: Duet 3 Mini5plus WiFi FIRMWARE_DATE: 2021-04-09 14:40:21This is the first version that causes the DCS log spam
-
Actually this is a bit weird. I just went through the remaining versions in sequence and 3.3b2+2 isn't spamming the logs.
-
@serbitar, I've updated the binary at https://www.dropbox.com/sh/0mwfe22wepk5efh/AAB6Lp0rwBPjVK2R4SxbKhLya?dl=0 to avoid DCS reporting those errors. The build time reported by M115 is now 21:29:10. The previous version was 18:55.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:
With the 18:55 version I had a print paused over night and the printer didn't have any VSSA or VIN troubles.
Print has now been resumed and will upgrade again later today after it finishes. -
Thanks. I am 99% certain that my rewrite of the ADC driver has fixed this issue, so I will mark it as solved. Feel free to revert that if the problem recurs.
For others who have experienced this issue, thanks for your patience. The fix is in the Duet3Firmware_Mini5plus.uf2 file at https://www.dropbox.com/sh/0mwfe22wepk5efh/AAB6Lp0rwBPjVK2R4SxbKhLya?dl=0. The 3.3beta3 release will include this fix.
Duet WiFi hardware designer and firmware engineer
Please do not ask me for Duet support via PM or email, use the forum
http://www.escher3d.com, https://miscsolutions.wordpress.com -
@dc42 Yep, looks good. No issues in the last 24h.
Many thanks for sorting this out! -
@dc42 Running a print successfully as we speak, after 24hr with no issues. Thank you for your work sorting this out!
-
Many thanks to everyone who helped resolve this issue, for your patience, persistence and providing vital testing and feedback. I hope you didn’t waste too much filament!
Ian
