6HC+2x 3HC+SBC. 3.2 Crashes at high print speed.

Phaedrux

Now that they are all on 3.2 hopefully that will show an improvement.

If it does crash again get an M122 afterwards and setup this monitoring on the pi.

https://duet3d.dozuki.com/Wiki/Getting_Started_With_Duet_3#Section_Monitoring_optional

jrockland

@Phaedrux
I did but they didn't gave me any info, so I did it again and recorded the whole thing.
video here (1.30 min)
the video is not publish publicly, you can only see it from the link. I didn't want people to have the wrong idea.

you can see at 1:20 everything just *freeze as the whole system just quit.
when I reboot I have no error message or anything, it just boot as usual.

Phaedrux

And when it boots up again, gather a M122 so we can see what the last reset reason was.

jrockland

@Phaedrux
m122
=== Diagnostics ===
RepRapFirmware for Duet 3 MB6HC version 3.2 running on Duet 3 MB6HC v1.01 or later (SBC mode)
Board ID: 08DJM-956L2-G43S4-6J1F8-3SD6T-9V56D
Used output buffers: 1 of 40 (16 max)
=== RTOS ===
Static ram: 149788
Dynamic ram: 63132 of which 56 recycled
Never used RAM 145856, free system stack 200 words
Tasks: Linux(ready,107) HEAT(blocked,302) CanReceiv(blocked,848) CanSender(blocked,371) CanClock(blocked,352) TMC(blocked,53) MAIN(running,1189) IDLE(ready,19)
Owned mutexes: HTTP(MAIN)
=== Platform ===
Last reset 00:04:03 ago, cause: power up
Last software reset at 2021-01-22 13:19, reason: HardFault undefInstr, Platform spinning, available RAM 145856, slot 2
Software reset code 0x4060 HFSR 0x40000000 CFSR 0x00010000 ICSR 0x0440f803 BFAR 0x00000000 SP 0x2041fca8 Task MAIN Freestk 1720 ok
Stack: 0008eddf 00000000 204180d8 20420d58 000000af 00462ac7 00462ac6 610f0000 a5a5a5a5 a5a5a5a5 00000000 0046bd7f 00000001 fffc0000 204193df 00000000 2041dbcc a5a5a5a5 1a28e11f a5a5a5a5 a5a5a5a5 a5a5a5a5 a5a5a5a5 004170a1 204276c0 00462ac3 00000000
Error status: 0x00
Aux0 errors 0,0,0
Aux1 errors 0,0,0
MCU temperature: min 18.5, current 34.4, max 34.5
Supply voltage: min 24.2, current 24.2, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
12V rail voltage: min 12.0, current 12.1, max 12.1, under voltage events: 0
Driver 0: position 0, standstill, reads 51221, writes 14 timeouts 0, SG min/max 0/0
Driver 1: position 0, standstill, reads 51221, writes 14 timeouts 0, SG min/max 0/0
Driver 2: position 0, standstill, reads 51221, writes 14 timeouts 0, SG min/max 0/0
Driver 3: position 0, standstill, reads 51222, writes 14 timeouts 0, SG min/max 0/0
Driver 4: position 0, standstill, reads 51222, writes 14 timeouts 0, SG min/max 0/0
Driver 5: position 0, standstill, reads 51222, writes 14 timeouts 0, SG min/max 0/0
Date/time: 2021-01-22 14:04:38
Slowest loop: 0.47ms; fastest: 0.06ms
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 37.5MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
DMs created 125, maxWait 0ms, bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== AuxDDARing ===
Scheduled moves 0, completed moves 0, hiccups 0, stepErrors 0, LaErrors 0, Underruns [0, 0, 0], CDDA state -1
=== Heat ===
Bed heaters = -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP* is doing "M122" in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger* is idle in state(s) 0
Queue is idle in state(s) 0
LCD is idle in state(s) 0
SBC is idle in state(s) 0
Daemon is idle in state(s) 0
Aux2 is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== CAN ===
Messages queued 860, send timeouts 0, received 18, lost 0, longest wait 1ms for reply type 6018, free buffers 48
=== SBC interface ===
State: 4, failed transfers: 0
Last transfer: 1ms ago
RX/TX seq numbers: 7514/7514
SPI underruns 0, overruns 0
Number of disconnects: 0, IAP RAM available 0x2c8a8
Buffer RX/TX: 0/0-0
=== Duet Control Server ===
Duet Control Server v3.2.0
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 2.71
Maximum length of RX/TX data transfers: 4102/904

jrockland

@Phaedrux
I think we have our issue here: Last software reset at 2021-01-22 13:19, reason: HardFault undefInstr, Platform spinning, available RAM 145856, slot 2.
Slot 2 mean the second 3HC in line right ?
Im having some issue with him, loosing connection (red led start flashing fast). I didnt bother as it does the exact same job as the first 3HC the print was going anyway.

Phaedrux

I've flagged this for review. Will just need to wait for time zones before anyone can take a look.

jrockland

@Phaedrux
Im having that fault every 10 min now while trying normal print speed..:
HardFault undefInstr, Platform spinning, available RAM 145856, slot 2
since I updated the boards.. Ill just shut it down for now.

no worry I know we are not in the same time zones. thank for the help

Phaedrux

If you disconnect that tool board entirely does the problem go with it?

jrockland

@Phaedrux
seem like it.
as per my config.g file do you think I can move X-Y to the main board and the 4 Z to the extension boards ? (older version wouldn't accept my Z sensor on extension boards, and also the sensor will not be on the same board as all the Z (only 3 ports, 4 motors)..

But that would solve the communication issue as Z barely receive anything compare to x and y..

dc42

@jrockland, there have been fixes to CAN communication at high movement rates in 3.3beta. But I don't think any of them accounts for the reset you are seeing. I will analyse the reset data.

dc42

I've looked at the reset data. It returned from a function call, but then complained that the instruction following the call wasn't valid - whereas it certainly is in the firmware. So either the flash memory or the cache memory has returned the wrong data. The most likely cause I can think of for this happening is a power brownout.

jrockland

@dc42 so we are looking at the boards power controller failing at high rate ? as for the power supply they are stable +/- 0.15v during the whole process, and under 30% load.

what can we do to stabilize the power distribution on the board ?

jrockland

@dc42 my friend is bringing over is marlin boards, as we still want to see stability vs speed of the machine. we will be using the same power supply for comparison.

Also I now need to downgrade the boards as since I updated they are not useable anymore..

jrockland

@dc42 might actually be power distribution to the sbc.. I know they are unstable and I wouldn't trust a raspberry pi to be anywhere near a functional tool.
I will look into it.

jrockland

@jrockland
negative, same problem with the sbc self powered.

jrockland

@dc42 power distribution is stable: 120.4 at source, 24.1 at the power supply tested while printing and when it failed again. going back to version 3.1 as I dont remember having that issue then.

Phaedrux

@jrockland said in 6HC+2x 3HC+SBC. 3.2 Crashes at high print speed.:

--Why 2 power supply : at first I tough I was running out of amps when everything was kicking together.. actually helped when I added the second one.

Is it possible the first power supply was having issues browning out?

jrockland

@Phaedrux I tested both while running and they are stable and way underload.
I added a second power supply because of that exact issue where I taught I was running out of power.

We also tried with my friend marlin's boards (using the same power supply) and we have no issue. I can push it twice as fast without any power loss.

I will be flashing those board with klipper and see if it is firmware related, if it is the board I guess I will have to learn how to program marlin and go with their board.

jrockland

@Phaedrux when we pulled out the marlin board we realised the main difference was the grounding; the exp boards dont have any external grounding connection.

I added grounding using io0 gnd on each ext board and it fixed the issue of the main board shutting down.

I still have issue with the ext board loosing connection for a sec under high load (not high power, just when they receive lots of commands: If I do a giant square it work good but a giant circle make them fail).
But I think moving Z motors to the ext board and x/y back to the main should solve that.

dc42

@jrockland said in 6HC+2x 3HC+SBC. 3.2 Crashes at high print speed.:

@Phaedrux when we pulled out the marlin board we realised the main difference was the grounding; the exp boards dont have any external grounding connection.

I added grounding using io0 gnd on each ext board and it fixed the issue of the main board shutting down.

I still have issue with the ext board loosing connection for a sec under high load (not high power, just when they receive lots of commands: If I do a giant square it work good but a giant circle make them fail).
But I think moving Z motors to the ext board and x/y back to the main should solve that.

It's not normally necessary to add extra grounding to expansion boards if they are powered from the same PSU as the main board. However, we have found one situation in which CAN connection can be disrupted. In summary:

If you use a high powered bed heater powered directly from the Duer 3 MB6HC main board, then either:

Use very short and thick wires between then Duet 3 MB6HC and the PSU; or
Connect the PSU to the Duet 3 MB6HC, then daisy-chain the expansion board power wires from the MB6HC power connections, instead of running them direct from the PSU.