Duet 3 Mini 5+ VIN under-voltage issues in SBC mode

dc42

@Serbitar said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

I should have said, that when i run this in standalone mode I don't see the under-voltage warnings.

Thanks for that information, it's very useful. I have one other report of spurious under-voltage readings from a Duet 3 Mini, also in SBC mode.

Can you provide a photo showing the Duet, the Pi and the wiring?

Serbitar

Its all a bit of a mess due to the investigations... Power supply and controller are on opposite sides of the printer currently. Approx 50cm cable length in the main power cables.

Terminals on the psu from centre outwards are:
Controller, 10000uF capacitor, LED lighting.

Capacitor was added as part of the investigations when i thought I was possibly suffering from mains failures. Ideally i would put it at the controller end of the cable run. LED lighting was not connected until late this afternoon (after the log messages i shared earlier).

Let me know if these aren't quite what you are looking for, I can probably shuffle things around tomorrow to give more context if necessary.

Thanks for your help!

Serbitar

I have been printing smaller parts for the last few days (<4h) and no power issues until running a longer job over night (24h).

Woke up to find the job was paused during a nozzle swap.

Mar 16 06:30:13 ender3 DuetControlServer[820]: [info] Print auto-paused due to low voltage
Mar 16 06:30:13 ender3 DuetControlServer[820]: [warn] VIN under-voltage event (0.0V)

SBC however didn't loose power

root@smalldeprinter ~ # uptime
 08:53:25 up 1 day, 11 min,  1 user,  load average: 0.10, 0.08, 0.08

Nor did anything else in the house.

Better photo including all the power wiring:

The smaller power supply at the top of the photo is currently completely disconnected.

Serbitar

Another occurrence over night.
Is there anything I can do to contribute to investigation? If not I'm going to go back to standalone mode.

m122
=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.2.2 running on Duet 3 Mini5plus WiFi (SBC mode)
Board ID: W4ZW4-V296U-D65J0-40KM0-LR03Z-H6928
Used output buffers: 1 of 40 (30 max)
=== RTOS ===
Static ram: 98732
Dynamic ram: 100676 of which 60 recycled
Never used RAM 45748, free system stack 126 words
Tasks: Linux(ready,83) HEAT(blocked,297) CanReceiv(blocked,947) CanSender(blocked,358) CanClock(blocked,362) TMC(blocked,106) MAIN(running,380) IDLE(ready,20) AIN(blocked,262)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 09:35:39 ago, cause: software
Last software reset at 2021-03-16 23:02, reason: User, none spinning, available RAM 46164, slot 2
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
Supply voltage: min 0.0, current 24.1, max 25.3, under voltage events: 505, over voltage events: 0, power good: yes
Driver 0: position -1155, standstill, SG min/max 0/0, read errors 0, write errors 23, ifcnt 243, reads 41867, writes 4545, timeouts 7, DMA errors 0, failedOp 0x01
Driver 1: position -2187, standstill, SG min/max 0/44, read errors 0, write errors 23, ifcnt 79, reads 41848, writes 4545, timeouts 2, DMA errors 0, failedOp 0x71
Driver 2: position 9535, standstill, SG min/max 0/66, read errors 1, write errors 23, ifcnt 79, reads 41831, writes 4545, timeouts 3, DMA errors 0, failedOp 0x6a
Driver 3: position 0, standstill, SG min/max 0/46, read errors 0, write errors 20, ifcnt 242, reads 41574, writes 4546, timeouts 5, DMA errors 0, failedOp 0x72
Driver 4: position 0, standstill, SG min/max 0/298, read errors 2, write errors 22, ifcnt 57, reads 41858, writes 4546, timeouts 5, DMA errors 0, failedOp 0x71
Driver 5: position 0, standstill, SG min/max 0/2, read errors 1, write errors 16, ifcnt 242, reads 41847, writes 4545, timeouts 3, DMA errors 0, failedOp 0x80
Driver 6: position 0, standstill, SG min/max 0/2, read errors 0, write errors 16, ifcnt 242, reads 41630, writes 4545, timeouts 30, DMA errors 0, failedOp 0x6f
Date/time: 2021-03-17 08:38:39
Cache data hit count 4294967295
Slowest loop: 4.77ms; fastest: 0.13ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 0ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 108660, completed moves 108660, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.0
Heater 2 is on, I-accum = 0.4
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is doing "G1 X29.985001 Y-4.522000 E0.322590" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Filament sensors ===
Extruder 0 sensor: ok
Extruder 1 sensor: ok
=== CAN ===
Messages queued 194722, send timeouts 194722, received 0, lost 0, longest wait 0ms for reply type 0, free buffers 16
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 5ms ago
RX/TX seq numbers: 23696/23696
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x10eec
Buffer RX/TX: 1464/2272-0
=== Duet Control Server ===
Duet Control Server v3.2.2
File:
Buffered code: G1 X29.985 Y-4.522 E0.32259
Buffered code: G1 X29.967 Y-5.211 F9000.000
Buffered code: G1 F1800.000
Buffered code: G1 X29.940 Y-5.511 E0.00858
Buffered code: G1 X29.914 Y-5.812 E0.00763
Buffered code: G1 X29.892 Y-6.011 E0.00444
Buffered code: G1 X29.871 Y-6.211 E0.00373
Buffered code: G1 X29.849 Y-6.410 E0.00302
Buffered code: G1 X29.827 Y-6.609 E0.00232
Buffered code: G1 X29.805 Y-6.809 E0.00161
Buffered code: G10 ; retract
Buffered code: G1 X29.967 Y5.211 F9000.000
Buffered code: G11 ; unretract
Buffered code: G1 F1800.000
Buffered code: G1 X29.940 Y5.511 E0.00858
Buffered code: G1 X29.914 Y5.812 E0.00763
Buffered code: G1 X29.892 Y6.011 E0.00444
Buffered code: G1 X29.871 Y6.211 E0.00373
Buffered code: G1 X29.849 Y6.410 E0.00302
==> 800 bytes
Code buffer space: 1824
Configured SPI speed: 8000000 Hz
Full transfers per second: 36.74
Maximum length of RX/TX data transfers: 3777/1612
File /opt/dsf/sd/gcodes/stripyegg1_0.2mm_PLA_ENDER3_9h33m.gcode is selected, processing

Log looks a bit different this time! I added m122 to the power fail actions.

M911 S21 R23 P"M122 M913 X0 Y0; G10 P0 R0 S0 G10 P1 R0 S0 G91 M83 G1 Z3 E-5 F10000" ; set voltage thresholds and actions to run on power loss

2021-03-16 23:03:28 Error: G0/G1: insufficient axes homed
2021-03-17 02:38:04 === Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.2.2 running on Duet 3 Mini5plus WiFi (SBC mode)
Board ID: W4ZW4-V296U-D65J0-40KM0-LR03Z-H6928
Used output buffers: 2 of 40 (13 max)
=== RTOS ===
Static ram: 98732
Dynamic ram: 100676 of which 60 recycled
Never used RAM 45748, free system stack 126 words
Tasks: Linux(ready,99) HEAT(blocked,297) CanReceiv(blocked,947) CanSender(blocked,358) CanClock(blocked,362) TMC(blocked,106) MAIN(running,388) IDLE(ready,20) AIN(blocked,262)
Owned mutexes: Autopause(MAIN)
=== Platform ===
Last reset 03:35:03 ago, cause: software
Last software reset at 2021-03-16 23:02, reason: User, none spinning, available RAM 46164, slot 2
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x00000000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
Supply voltage: min 0.0, current 20.9, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: position -1155, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 139, reads 11001, writes 12, timeouts 8, DMA errors 0, failedOp 0x72
Driver 1: position -2187, standstill, SG min/max 0/242, read errors 0, write errors 1, ifcnt 201, reads 10987, writes 18, timeouts 16, DMA errors 0, failedOp 0x01
Driver 2: position 9535, standstill, SG min/max 0/250, read errors 0, write errors 1, ifcnt 187, reads 10991, writes 20, timeouts 10, DMA errors 0, failedOp 0x06
Driver 3: position 0, ok, SG min/max 0/356, read errors 0, write errors 1, ifcnt 229, reads 10914, writes 21, timeouts 86, DMA errors 0, failedOp 0x71
Driver 4: position 0, ok, SG min/max 0/346, read errors 0, write errors 1, ifcnt 235, reads 10862, writes 21, timeouts 138, DMA errors 0, failedOp 0x6a
Driver 5: position 0, ok, SG min/max 0/100, read errors 0, write errors 1, ifcnt 246, reads 10990, writes 23, timeouts 8, DMA errors 0, failedOp 0x01
Driver 6: position 0, ok, SG min/max 0/110, read errors 0, write errors 1, ifcnt 246, reads 8032, writes 23, timeouts 2966, DMA errors 0, failedOp 0x01
Date/time: 2021-03-17 02:38:03
Cache data hit count 4294967295
Slowest loop: 163.49ms; fastest: 0.08ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 50255ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 108660, completed moves 108660, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 2], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = -1 -1
Heater 0 is on, I-accum = 0.0
Heater 2 is on, I-accum = 0.6
=== GCodes ===
Segments left: 0
Movement lock held by Autopause
HTTP* is idle in state(s) 0
Telnet is idle in state(s) 0
File* is doing "G1 X29.985001 Y-4.522000 E0.322590" in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is ready with "M122 M913 X0 Y0" in state(s) 64
Code queue is empty.
=== Filament sensors ===
Extruder 0 sensor: ok
Extruder 1 sensor: ok
=== CAN ===
Messages queued 116136, send timeouts 116133, received 0, lost 0, longest wait 0ms for reply type 0, free buffers 16
=== SBC interface ===
State: 0, failed transfers: 0
Last transfer: 17ms ago
RX/TX seq numbers: 50168/50169
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x10eec
Buffer RX/TX: 1464/2104-0
Print auto-paused due to low voltage
2021-03-17 02:38:07 Warning: VIN under-voltage event (9.3V)
2021-03-17 02:38:11 Warning: VIN under-voltage event (7.1V)
2021-03-17 02:38:16 Warning: VIN under-voltage event (7.9V)
2021-03-17 02:38:20 Warning: VIN under-voltage event (3.8V)
2021-03-17 02:38:24 Warning: VIN under-voltage event (5.1V)
2021-03-17 02:38:29 Warning: VIN under-voltage event (5.1V)
2021-03-17 02:38:33 Warning: VIN under-voltage event (5.1V)
2021-03-17 02:38:38 Warning: VIN under-voltage event (5.1V)
2021-03-17 02:38:42 Warning: VIN under-voltage event (5.2V)
2021-03-17 02:38:46 Warning: VIN under-voltage event (5.2V)
2021-03-17 02:38:51 Warning: VIN under-voltage event (5.2V)
2021-03-17 02:38:55 Warning: VIN under-voltage event (5.2V)
2021-03-17 02:39:00 Warning: VIN under-voltage event (5.2V)
2021-03-17 02:39:04 Warning: VIN under-voltage event (5.2V)
2021-03-17 02:39:09 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:13 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:18 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:22 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:27 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:31 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:36 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:40 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:45 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:49 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:54 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:39:58 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:40:03 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:40:07 Warning: VIN under-voltage event (5.3V)
2021-03-17 02:40:12 Warning: VIN under-voltage event (5.4V)
2021-03-17 02:40:16 Warning: VIN under-voltage event (5.4V)
2021-03-17 03:01:26 Warning: VIN under-voltage event (0.0V)
2021-03-17 06:28:49 Warning: VIN under-voltage event (0.0V)

Phaedrux

@Serbitar Thanks for the report.

In SBC mode you could try enabling more detailed logging.

https://duet3d.dozuki.com/Wiki/Getting_Started_With_Duet_3#Section_Monitoring_optional

carcamerarig

I also had one this evening why performing home all
3.3b2 sbc mini 5

dc42

@carcamerarig, thanks for the additional data point.

My best guess at this point is that the DMA for the SPI transfers between the Pi and the Duet is somehow interfering with the DMA used by the ADC. Do any of you experiencing this problem see short-lived temperature jumps too?

Serbitar

@dc42 by the time i'm back at the printer, usually several hours have past since it stopped so the temperature charts have gone past.

I have a few prints to complete in standalone mode and then i will enable the extra monitoring. Is there any way to save logs of the temp values too?

dc42

Until I have pinned down the exact cause, it might be worth trying a lower SPI frequency. The frequency is set in the DCS configuration file on the Pi.

dc42

PS - if anyone who has experienced this has their machine idle, please can you run some simulated prints, to see whether they can trigger this issue. The increased SPI traffic rate from the Pi during a simulation might make it happen more often. OTOH it could turn out that step interrupts also need to happen for this issue to occur.

WILD9

@dc42 Happy to try, how would this manifest during a simulation, just console messages showing low voltage detected? In which case I will make sure I have 24Vin active during simulation.

dc42

@WILD9 said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

@dc42 Happy to try, how would this manifest during a simulation, just console messages showing low voltage detected? In which case I will make sure I have 24Vin active during simulation.

I expect the indications of the problem to be:

A console report of an under-voltage event, with the reported VIN being 0.0
If you run M122 afterwards then it will report that one or more under-voltage events occurred and the minimum recorded VIN will be 0.0. Note, a subsequent M122 will only report under-voltage events and the minimum VIN that happened since the previous M122.

You will need to have VIN power applied throughout.

carcamerarig

@dc42 said in Duet 3 Mini 5+ VIN under-voltage issues in SBC mode:

PS - if anyone who has experienced this has their machine idle, please can you run some simulated prints, to see whether they can trigger this issue. The increased SPI traffic rate from the Pi during a simulation might make it happen more often. OTOH it could turn out that step interrupts also need to happen for this issue to occur.

I’ve been spending quite a few hours daily of stop start printing while I’m still tuning and configuring (mostly learning) and have only seen the one which was at a home all, the effect was a crash into the left x carriage carrier, I’m unsure if the error was before during or after the crash.
I’ll be doing more of the same later if you want me to test or log anything, I’ll need some instructions though if I’m to alter some code.

Serbitar

@dc42 Is 4MHz low enough?

lodger

Is it worth running some heaters duing a simulation. To give the PSU some load.
During a print my VIN can jump up and down a few points of a volt. But during a sim . She is bang on 24.2v and dont move.

WILD9

6ish hours of simulation so far with no issues

droftarts

@WILD9 I ran simulations for 8 hours last night, with no issues. Still trying to track this issue down.

Ian

lodger

I have come down this morning to find a stalled print. Here is my M122,

m122

=== Diagnostics ===
RepRapFirmware for Duet 3 Mini 5+ version 3.2.2 running on Duet 3 Mini5plus WiFi (SBC mode)
Board ID: XJDX9-1296U-D65J0-40KMJ-4K03Z-7J6X6
Used output buffers: 1 of 40 (18 max)
=== RTOS ===
Static ram: 98732
Dynamic ram: 95844 of which 64 recycled
Never used RAM 50576, free system stack 122 words
Tasks: Linux(ready,99) HEAT(blocked,297) CanReceiv(blocked,947) CanSender(blocked,358) CanClock(blocked,362) TMC(blocked,75) MAIN(running,476) IDLE(ready,20) AIN(blocked,264)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 17:19:34 ago, cause: software
Last software reset at 2021-03-19 15:58, reason: User, none spinning, available RAM 50808, slot 1
Software reset code 0x0012 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0400f000 BFAR 0xe000ed38 SP 0x00000000 Task Linu Freestk 0 n/a
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
Supply voltage: min 0.0, current 24.2, max 24.3, under voltage events: 2, over voltage events: 0, power good: yes
Driver 0: position 16355, standstill, SG min/max 0/428, read errors 0, write errors 1, ifcnt 223, reads 13005, writes 43, timeouts 17110, DMA errors 0, failedOp 0x71
Driver 1: position 2458, standstill, SG min/max 0/410, read errors 0, write errors 1, ifcnt 223, reads 30107, writes 43, timeouts 7, DMA errors 0, failedOp 0x6f
Driver 2: position 63048, standstill, SG min/max 0/44, read errors 0, write errors 1, ifcnt 189, reads 29631, writes 32, timeouts 494, DMA errors 0, failedOp 0x71
Driver 3: position 0, standstill, SG min/max 0/30, read errors 0, write errors 1, ifcnt 188, reads 29889, writes 32, timeouts 235, DMA errors 0, failedOp 0x01
Driver 4: position 0, standstill, SG min/max 0/0, read errors 0, write errors 1, ifcnt 155, reads 30130, writes 27, timeouts 0, DMA errors 0
Driver 5: position 0, assumed not present
Driver 6: position 0, assumed not present
Date/time: 2021-03-20 09:18:05
Cache data hit count 4294967295
Slowest loop: 178.10ms; fastest: 0.08ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 0.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 83, maxWait 11368610ms, bed compensation in use: mesh, comp offset 0.000
=== MainDDARing ===
Scheduled moves 190766, completed moves 190766, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = 0 -1, chamberHeaters = 2 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File* is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue* is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 561364, send timeouts 561361, received 0, lost 0, longest wait 0ms for reply type 0, free buffers 15
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 3ms ago
RX/TX seq numbers: 42328/42328
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x10eec
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.2.2
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 36.40
Maximum length of RX/TX data transfers: 5108/1664.

dc42

@lodger, was that a print or a simulation?

Please can you try the firmware at https://www.dropbox.com/sh/6203euyyobllic5/AABTHIbn7F_maK9p-oy7vq3Ea?dl=0 with the same file to see if it happens again.

lodger

@dc42 Hi,

Yes it was a print. I will just put the kettle on and update and print. It was a 10 hour print so i may not get back to you till tomo.

Edit:: I have updated and run a Similator of the same file. Without problems !
Just running the print now .

Solved Duet 3 Mini 5+ VIN under-voltage issues in SBC mode