3.01RC11 on WiFi + Duex5 M950 Fan on Duex causes reboot loop.

Phaedrux

Moved to the Beta Firmware forum and updated the post title.

UticaTechClub

	M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.01-RC11 running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-917DA-G4MS8-6JTD4-3S86L-19S7AUsed output buffers: 3 of 24 (20 max)
=== RTOS ===
Static ram: 28060
Dynamic ram: 94724 of which 44 recycled
Exception stack ram used: 552
Never used ram: 7692
Tasks: NETWORK(blocked,172) HEAT(blocked,540) DUEX(blocked,160) MAIN(running,1816) IDLE(ready,76)
Owned mutexes: WiFi(NETWORK) I2C(DUEX)
=== Platform ===
Last reset 00:32:48 ago, cause: software
Last software reset time unknown, reason: User, spinning module GCodes, available RAM 7708 bytes (slot 2)
Software reset code 0x0003 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0441f000 BFAR 0xe000ed38 SP 0xffffffff Task 0x4e49414d
Error status: 8
MCU temperature: min 28.8, current 30.2, max 31.4
Supply voltage: min 24.0, current 24.2, max 24.4, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max 0/156
Driver 1: standstill, SG min/max 0/166
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max 0/152
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max not available
Driver 7: standstill, SG min/max not available
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Date/time: 2020-05-04 16:12:18
Cache data hit count 3253372840
Slowest loop: 38.63ms; fastest: 0.13ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 3.9ms, write time 1.2ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 125, MaxWait: 989754ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 13310, completed moves: 13310, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 15.71ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 2 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address 80:7d:3a:3f:f4:ce
WiFi Vcc 3.38, reset reason Unknown
WiFi flash size 4194304, free heap 22168
WiFi IP address 192.168.1.44
WiFi signal strength -43dBm, reconnections 0, sleep mode modem
Socket states: 0 4 0 0 0 0 0 0
=== DueX ===
Read count 325536, inf reads/min

T3P3Tony

@UticaTechClub thanks. just as a test can you try configure another fan on the Duex with M950 please?

UticaTechClub

@T3P3Tony
Testing...

UticaTechClub

@T3P3Tony
I tested them, - all end up rebooting the unit.

M950 F3 C"duex.fan3" Q500
M950 F4 C"duex.fan4" Q500
M950 F5 C"duex.fan5" Q500
M950 F6 C"duex.fan6" Q500
M950 F7 C"duex.fan7" Q500

However!
I ran the print job on two extruders, then paused and cancelled it, and then all of them worked.
It was one successful time. Subsequent reboot, caused the problem to return and I tried running and cancelling jobs again only to find the the same issue

With one time success, it presents itself as a possible firmware issue.
If some sort of low level logging was available, I could try to further track it.

I will try to derive as to what caused it to succeed one time and perhaps to find a pattern.

These boards were used in four extrude configuration with RRF2, so I am positive the hardware is OK.

Thank you!

dc42

Your M122 command contains little useful information because it shows that the last software reset done was commanded by you, for example because you used the Emergency Stop button.

To get further information, I suggest you get the machine in the reboot loop, then pop the SD card out so that it boots up. You can then either attach via USB/YAT to get the M122 report, or if you have a PanelDue then you can send M552 S1 from it to enable the network, then get the M122 report from DWC.

If the last software reset was caused by a firmware crash then the 'reason' field after 'Last software reset' will probably be "Hard fault".

UticaTechClub

@dc42
Thank you!

I followed the procedure and executed M122 over USB connection, right after the reboot:

SENDING:M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.01-RC11 running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-917DA-G4MS8-6JTD4-3S86L-19S7AUsed output buffers: 1 of 24 (2 max)
=== RTOS ===
Static ram: 28060
Dynamic ram: 92304 of which 20 recycled
Exception stack ram used: 312
Never used ram: 10376
Tasks: NETWORK(ready,1988) HEAT(blocked,1460) DUEX(blocked,160) MAIN(running,2508) IDLE(ready,76)
Owned mutexes: I2C(DUEX)
=== Platform ===
Last reset 00:00:08 ago, cause: software
Last software reset time unknown, reason: Stuck in spin loop, spinning module GCodes, available RAM 7944 bytes (slot 3)
Software reset code 0x0083 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80f BFAR 0xe000ed38 SP 0x20004334 Task 0x454c4449
Stack: 0045481d 0045524e 61000000 a5a5a5a5 0045481d a5a5a5a5 a5a5a5a5 2000435c 2000424c 00000002 20004d94
Error status: 0
[ERROR] Error status: 0

MCU temperature: min 27.8, current 28.3, max 28.4
Supply voltage: min 24.2, current 24.2, max 24.2, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max not available
Driver 7: standstill, SG min/max not available
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Date/time: 1970-01-01 00:00:00
Cache data hit count 8327457
Slowest loop: 13.57ms; fastest: 0.14ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 not detected, interface speed: 30.0MBytes/sec
SD card longest read time 0.0ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = -1 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is ready with "M122" in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 0.22ms; fastest: 0.00ms
Responder states: HTTP(0) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0)
HTTP sessions: 0 of 8
- WiFi -
Network state is disabled
WiFi module is disabled
Failed messages: pending 2779096485, notready 2779096485, noresp 2779096485
Socket states: 0 0 0 0 0 0 0 0
=== DueX ===
Read count 1306, inf reads/min

dc42

Thanks! How long does the reboot loop take, per iteration? The DIAG LED may give you a clue.

dc42

Please try 3.01-RC12 and see if the behaviour is any different.

UticaTechClub

@dc42
Thank you!
Every re-boot, red DIAG LED blinks.
It takes approximately 24 seconds between re-boots.

New discovery:
One time, it booted all the way in and discovered all four tools. It basically stopped re-booting.
I am trying this again, but I am not sure if I will succeed.

Looking forward:
I will try updating to RC12 after few more attempts at the above.

Greatly appreciated!

UticaTechClub

@dc42
The same exact pattern repeats with RC12.

Or, the reboot could be induced by simply issuing

M950 F3 C"duex.fan3" Q500

dc42

I've tested your config.g on my Duet WiFi + DueX setup, and it works here. I don't have a fan connected to Fan3 on the DueX yet. Does your machine boot up if no Fan3 is connected? Are you using VIN or 12V fan supply on the DueX?

I am beginning to suspect a hardware fault on your DueX5.

UticaTechClub

@dc42
I have tried different Duex board and it keeps rebooting. I doubt it is hardware related, it used to also work in this hardware configuration with version 2 firmware.

I also tried the very latest 3.1.1 firmware and it is still the same.

Perhaps something else in my configuration causes this problem indirectly.config-good.g config-bad.g

UticaTechClub

@dc42
I am using VIN, 24V.
No difference if Fan3 is connected or not.

dc42

Now that you are running firmware 3.1.1, please can you get and post another M122 report after the reboot happens.

UticaTechClub

@dc42
Thank you! Here it is:

M122
=== Diagnostics ===
RepRapFirmware for Duet 2 WiFi/Ethernet version 3.1.1 running on Duet WiFi 1.02 or later + DueX5
Board ID: 08DGM-917DA-G4MS8-6JTD4-3S86L-19S7A
Used output buffers: 3 of 24 (11 max)
=== RTOS ===
Static ram: 27980
Dynamic ram: 94788 of which 60 recycled
Exception stack ram used: 264
Never used ram: 7980
Tasks: NETWORK(ready,384) HEAT(blocked,1224) DUEX(blocked,160) MAIN(running,1812) IDLE(ready,80)
Owned mutexes: WiFi(NETWORK) I2C(DUEX)
=== Platform ===
Last reset 00:00:20 ago, cause: software
Last software reset at 2020-05-28 12:58, reason: Stuck in spin loop, spinning module GCodes, available RAM 7480 bytes (slot 3)
Software reset code 0x4083 HFSR 0x00000000 CFSR 0x00000000 ICSR 0x0041f80f BFAR 0xe000ed38 SP 0x200042e4 Task IDLE
Stack: 00454cdd 00455710 61000000 a5a5a5a5 00454cdd a5a5a5a5 20004308 200041f8 00000002 20004de4 00046153 
Error status: 0
MCU temperature: min 27.7, current 28.4, max 28.8
Supply voltage: min 24.1, current 24.2, max 24.3, under voltage events: 0, over voltage events: 0, power good: yes
Driver 0: standstill, SG min/max not available
Driver 1: standstill, SG min/max not available
Driver 2: standstill, SG min/max not available
Driver 3: standstill, SG min/max not available
Driver 4: standstill, SG min/max not available
Driver 5: standstill, SG min/max not available
Driver 6: standstill, SG min/max not available
Driver 7: standstill, SG min/max not available
Driver 8: standstill, SG min/max not available
Driver 9: standstill, SG min/max not available
Date/time: 2020-05-28 12:59:08
Cache data hit count 32201028
Slowest loop: 21.83ms; fastest: 0.13ms
I2C nak errors 0, send timeouts 0, receive timeouts 0, finishTimeouts 0, resets 0
=== Storage ===
Free file entries: 10
SD card 0 detected, interface speed: 20.0MBytes/sec
SD card longest read time 2.7ms, write time 0.0ms, max retries 0
=== Move ===
Hiccups: 0(0), FreeDm: 169, MinFreeDm: 169, MaxWait: 0ms
Bed compensation in use: none, comp offset 0.000
=== MainDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== AuxDDARing ===
Scheduled moves: 0, completed moves: 0, StepErrors: 0, LaErrors: 0, Underruns: 0, 0  CDDA state: -1
=== Heat ===
Bed heaters = 0 -1 -1 -1, chamberHeaters = -1 -1 -1 -1
=== GCodes ===
Segments left: 0
Movement lock held by null
HTTP is idle in state(s) 0
Telnet is idle in state(s) 0
File is idle in state(s) 0
USB is idle in state(s) 0
Aux is idle in state(s) 0
Trigger is idle in state(s) 0
Queue is idle in state(s) 0
Daemon is idle in state(s) 0
Autopause is idle in state(s) 0
Code queue is empty.
=== Network ===
Slowest loop: 15.74ms; fastest: 0.00ms
Responder states: HTTP(2) HTTP(0) HTTP(0) HTTP(0) FTP(0) Telnet(0), 0 sessions
HTTP sessions: 1 of 8
- WiFi -
Network state is active
WiFi module is connected to access point 
Failed messages: pending 0, notready 0, noresp 0
WiFi firmware version 1.23
WiFi MAC address 80:7d:3a:3f:f4:ce
WiFi Vcc 3.38, reset reason Unknown
WiFi flash size 4194304, free heap 25984
WiFi IP address 192.168.1.44
WiFi signal strength -47dBm, reconnections 0, sleep mode modem
Socket states: 0 0 0 0 0 0 0 0
=== DueX ===
Read count 3219, 9584.64 reads/min

UticaTechClub

@dc42
M122 was achieved by first booting with good configuration file and then issuing

M950 F3 C"duex.fan3" Q500

to cause the reboot.

UticaTechClub

@dc42
Here is something new... I don't know why I never tried it before.
I disconnected the the Duex bus cable and it no longer reboots. I feel bad that I never tried it before
Now, I could try few other things.

UticaTechClub

@dc42
@T3P3Tony
Duex board hosts five inputs (E2_STOP ... E6_STOP), one per extruder.
Our configuration hosts five tools in total, and used E0_STOP - E1_STOP on Duet WIfi and E2_STOP - E4_STOP on Duex5. Those ports have standard RepRep optical limit switches connected to them. Those limit switches monitor tools parking states - very useful application.

When tool is parked, it's 3D printed housing has a little tab which engages the optical limit switch. Depending on translucency of the material used for tool housing, the little tab may not be completely opaque, thus causing some light to travel through it, putting optical limit switch into the state other then ON or OFF. When you power up 3D printer, it will suffer continuous reboots. Please note (#1) that E0_STOP - E1_STOP ports are not susceptible to this, that is why, two tools configuration always worked. Note #2, is that continuous reboots will be only happening when FANs are configured using M950 F3 C"duex.fan3" Q500, M950 F4 C"duex.fan4" Q500, M950 F5 C"duex.fan5" Q500, M950 F6 C"duex.fan6" Q500, M950 F7 C"duex.fan7" Q500 gcodes.

I am planning to check the output voltage of such half-lit optical switch with the scope, and perhaps simulate the condition using the voltage source instead of the optical limit switch.

CONCLUSION:
There must be a slight difference in schematics of how E0_STOP - E1_STOP wired compared to E2_STOP ... E6_STOP on Duex5.
To prevent such condition, data feed must not be ambiguous.

A am very sorry, for taking your precious time with this, but exceedingly happy that the issue is now resolved.
Thank you guys!!!

dc42

Thanks for the update.

Do the outputs for fans 3 to 7 run close to the wires for the optical endstops connected to the DueX?

There is indeed a difference between endstops connected to the Duet and endstops connected to the DueX. Endstops connected to the Duet are polled during homing moves to see when they trigger. Endstops connected to the DueX generate an interrupt to the Duet, which then has to read them via the I2C connection. There is a mechanism to prevent the DueX interrupts causing excessive overload on the Duet main processor; but it sounds as though this mechanism isn't working as well as necessary.