Solved Duet3 + Pi DWC freezing

[chrishamm]

@jay_s_uk Yes, I think so. We'll provide a fix shortly.

[mwwhited]

The website is still responding and the SBC/RPi seems that's it's still running but I did get this message

[info] Starting macro file 0:/macros/Special Scripts/Check Relays on channel Daemon
[debug] Evaluating If block
[debug] Evaluating state.currentTool = -1 on channel Daemon
[debug] Evaluation result: state.currentTool = -1 = true
[debug] Evaluating If block
[debug] Evaluating 0 on channel Daemon
[debug] Evaluating 1 on channel Daemon
[debug] Evaluating 2 on channel Daemon
[debug] Evaluating state.gpOut[0].pwm > 0 || state.gpOut[1].pwm > 0 || state.gpOut[2].pwm > 0 on channel Daemon
[debug] Evaluation result: state.gpOut[0].pwm > 0 || state.gpOut[1].pwm > 0 || state.gpOut[2].pwm > 0 = false
[debug] End of If block
[debug] End of If block
[debug] Finished codes from macro file 0:/macros/Special Scripts/Check Relays
[info] Finished macro file 0:/macros/Special Scripts/Check Relays
[warn] Daemon: System macro daemon.g is requested but the stack is not empty. Discarding request.
[debug] Requesting update of key job, seq 1 -> 2
[debug] Daemon: ==> Finished starting code: M98 P"0:/macros/Special Scripts/Check Relays"
[debug] Completed M98 P"0:/macros/Special Scripts/Check Relays"
[debug] Daemon: Sent G4 S10 ; wait 10 seconds, remaining space 1504, needed 32
[debug] Daemon: -> Resumed suspended code
[debug] Updated key job
[warn] Daemon: System macro daemon.g is requested but the stack is not empty. Discarding request.

[mwwhited]

After the above message the daemon.g did not running. I was able to recover it by hitting emergency stop. I didn't get the total lockup like I did yesterday where the UI stopped responding as well as I was not able to SSH into the RPi... but this time is different as I cam logging the messages to the console as you suggested above. I'm going to allow it to continue running overnight to see if it freezes entirely.

[jay_s_uk]

@mwwhited

Thats the same message I was seeing. https://forum.duet3d.com/topic/16240/print-frozen-rrf3-01-rc12-and-dsf-2-2-0/12
When I got that, the system would lock up, the GUI would become unresponsive but I was still able to SSH into the pi.
Just to let you know that you're not alone.

[Carlo]

@jay_s_uk

Were you able to reload the screen to get control back ?

[jay_s_uk]

@Carlo

I could reload it but none of the buttons did anything.
Seems to be something locking the daemon file up and killing the pi side of things.

[Carlo]

@jay_s_uk
Yeah if I reloaded it worked as normal until cal or uploading gcode and the circle continued.

[Carlo]

Hi,
Connected to pi via desktop Chrome browser/and Pi screen
Still having this issue after update to 3.1.0 via apt-get
Also now get this error and constant network drops?
5/16/2020, 3:10:10 PM Failed to get file list (don't get this error on the Pi only over network)
Target drive is unmounted

No files show in system via web browser but are there on Pi screen
And still freezes after G32 or uploading gcode files may be more this is as far as I have gone on both Pi and web browser.

Cheers
C

[chrishamm]

@Carlo Can you confirm that you are on RRF 3.1.0? I haven't been able to reproduce it since that version. If it happens again, please run M122 from a serial monitor connected to the Duet 3, M122 "DSF" via the web interface, and post the outputs here.

The outputs of sudo ls -lR /opt/dsf/sd and sudo journalctl -u duetcontrolserver -e may be interesting, too (executed from a Linux terminal).

[Carlo]

@chrishamm

Hi Chris,

I have just installed a new image of duetpi and its the same issue file name 2020-05-15-DuetPi.img.

5/16/2020, 4:21:26 PM M122 "DSF"
=== Duet Control Server ===
Duet Control Server v3.1.0
Code buffer space: 4096
Configured SPI speed: 8000000 Hz
Full transfers per second: 31.0

duetcontrol to follow

cheers

[Carlo]

@Carlo

pi@duet3:~ $ sudo ls -lR /opt/dsf/sd
/opt/dsf/sd:
total 20
drwxr-xr-x 2 pi pi 4096 May 15 16:09 filaments
drwxr-xr-x 2 pi pi 4096 May 15 16:09 firmware
drwxr-xr-x 2 pi pi 4096 May 15 16:09 gcodes
drwxr-xr-x 2 pi pi 4096 May 15 16:09 macros
drwxr-xr-x 2 pi pi 4096 May 16 16:17 sys
lrwxrwxrwx 1 pi pi 7 May 15 16:25 www -> ../dwc2

/opt/dsf/sd/filaments:
total 0

/opt/dsf/sd/firmware:
total 0

/opt/dsf/sd/gcodes:
total 0

/opt/dsf/sd/macros:
total 0

/opt/dsf/sd/sys:
total 1004
-rw-r--r-- 1 root root 500 May 16 16:01 bed.g
-rw-r--r-- 1 root root 4758 May 16 16:17 config.g
-rw-r--r-- 1 root root 4758 May 16 16:01 config.g.bak
-rw-r--r-- 1 root root 4177 May 16 16:01 config.json
-rw-r--r-- 1 root root 595 May 16 16:16 config-override.g
-rwxr-xr-x 1 root root 10308 May 15 16:47 Duet3Bootloader_EXP.bin
-rwxr-xr-x 1 root root 9192 May 15 16:47 Duet3Bootloader_TOOL.bin
-rwxr-xr-x 1 root root 161956 May 15 16:47 Duet3Firmware_EXP3HC.bin
-rwxr-xr-x 1 root root 570184 May 15 16:47 Duet3Firmware_MB6HC.bin
-rwxr-xr-x 1 root root 163468 May 15 16:47 Duet3Firmware_TOOL1LC.bin
-rwxr-xr-x 1 root root 33804 May 15 16:47 Duet3iap_spi_MB6HC.bin
-rwxr-xr-x 1 root root 19908 May 15 16:47 Duet3_SBCiap_MB6HC.bin
-rw-r--r-- 1 root root 1185 May 16 16:21 dwc2settings.json
-rw-r--r-- 1 root root 2521 May 16 16:01 heightmap.csv
-rw-r--r-- 1 root root 630 May 16 16:01 homedelta.g
pi@duet3:~ $

[Carlo]

@Carlo

pi@duet3:~ $ sudo journalctl -u duetcontrolserver -e
May 16 16:17:15 duet3 systemd[1]: duetcontrolserver.service: Succeeded.
May 16 16:17:20 duet3 systemd[1]: duetcontrolserver.service: Service RestartSec=
May 16 16:17:20 duet3 systemd[1]: duetcontrolserver.service: Scheduled restart j
May 16 16:17:20 duet3 systemd[1]: Stopped Duet Control Server.
May 16 16:17:20 duet3 systemd[1]: Started Duet Control Server.
May 16 16:17:20 duet3 DuetControlServer[1282]: Duet Control Server v3.1.0
May 16 16:17:20 duet3 DuetControlServer[1282]: Written by Christian Hammacher fo
May 16 16:17:20 duet3 DuetControlServer[1282]: Licensed under the terms of the G
May 16 16:17:21 duet3 DuetControlServer[1282]: [info] Settings loaded
May 16 16:17:21 duet3 DuetControlServer[1282]: [info] Environment initialized
May 16 16:17:21 duet3 DuetControlServer[1282]: [info] Connection to Duet establi
May 16 16:17:21 duet3 DuetControlServer[1282]: [info] IPC socket created at /var
May 16 16:17:21 duet3 DuetControlServer[1282]: [info] Starting macro file config
May 16 16:17:22 duet3 DuetControlServer[1282]: [info] HTTP is enabled on port 80
May 16 16:17:22 duet3 DuetControlServer[1282]: [info] FTP is disabled
May 16 16:17:22 duet3 DuetControlServer[1282]: [info] TELNET is disabled
May 16 16:17:22 duet3 DuetControlServer[1282]: [warn] M307: Heater 1 appears to
May 16 16:17:22 duet3 DuetControlServer[1282]: [info] Starting macro file config
May 16 16:17:22 duet3 DuetControlServer[1282]: [info] Finished macro file config
May 16 16:17:23 duet3 DuetControlServer[1282]: [warn] M307: Heater 1 appears to
May 16 16:17:23 duet3 DuetControlServer[1282]: [info] Finished macro file config
May 16 16:17:27 duet3 DuetControlServer[1282]: [info] Starting macro file homede
May 16 16:17:28 duet3 DuetControlServer[1282]: [info] Finished macro file homede
lines 81-103/103 (END)

[chrishamm]

@Carlo Thanks, that's all looking good. I am a bit puzzled why DWC shows the first volume as not mounted. May you share the output of http://duet3/machine/status too? In most browsers, you can switch to raw view somewhere at the top, which makes copy&paste easier.

Please post those outputs only after a freeze has occurred and before a (firmware) reset.

[Carlo]

@chrishamm

Hope this right

{"result":{"boards":[{"bootloaderFileName":null,"canAddress":0,"firmwareDate":"2020-05-15b1","firmwareFileName":"Duet3Firmware_MB6HC.bin","firmwareName":"RepRapFirmware for Duet 3 MB6HC","firmwareVersion":"3.1.0","iapFileNameSBC":"Duet3_SBCiap_MB6HC.bin","iapFileNameSD":"Duet3_SDiap_MB6HC.bin","maxHeaters":32,"maxMotors":6,"mcuTemp":{"current":40.3,"min":39.7,"max":40.3},"name":"Duet 3 MB6HC","shortName":"MB6HC","supports12864":false,"uniqueId":"08DJM-956L2-G43S8-6J9DL-3SD6M-1S3UF","v12":{"current":12.1,"min":12.1,"max":12.2},"vIn":{"current":24,"min":24,"max":24}}],"directories":{"filaments":"0:/filaments/","firmware":"0:/sys/","gCodes":"0:/gcodes/","macros":"0:/macros/","menu":"0:/menu/","scans":"0:/scans/","system":"0:/sys/","web":"0:/www/"},"fans":[{"actualValue":0,"blip":0.1,"frequency":0,"max":1,"min":0.1,"name":"Berd Air","requestedValue":0,"rpm":-1,"thermostatic":{"heaters":[],"highTemperature":null,"lowTemperature":null}},{"actualValue":0,"blip":0.1,"frequency":0,"max":1,"min":0.1,"name":"","requestedValue":1,"rpm":-1,"thermostatic":{"heaters":[1],"highTemperature":45,"lowTemperature":45}}],"heat":{"bedHeaters":[0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],"chamberHeaters":[-1,-1,-1,-1],"coldExtrudeTemperature":160,"coldRetractTemperature":90,"heaters":[{"active":0,"current":26.8,"max":125,"min":-273.1,"model":{"deadTime":0.5,"enabled":true,"gain":122,"inverted":false,"maxPwm":1,"pid":{"overridden":false,"p":5.9,"i":1.8,"d":2.1,"used":true},"standardVoltage":24,"timeConstant":513.7},"monitors":[{"action":0,"condition":"tooHigh","limit":125},{"action":null,"condition":"disabled","limit":null},{"action":null,"condition":"disabled","limit":null}],"name":null,"sensor":0,"standby":0,"state":"off"},{"active":0,"current":40.4,"max":450,"min":-273.1,"model":{"deadTime":3.4,"enabled":true,"gain":718.2,"inverted":false,"maxPwm":1,"pid":{"overridden":false,"p":0.1,"i":0,"d":0.2,"used":true},"standardVoltage":24,"timeConstant":224.6},"monitors":[{"action":0,"condition":"tooHigh","limit":450},{"action":null,"condition":"disabled","limit":null},{"action":null,"condition":"disabled","limit":null}],"name":null,"sensor":1,"standby":0,"state":"off"}]},"httpEndpoints":[],"inputs":[{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"HTTP","stackDepth":0,"state":"idle","lineNumber":5,"volumetric":false},{"axesRelative":false,"compatibility":"Marlin","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"Telnet","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"File","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"Marlin","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"USB","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"Aux","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":true,"feedRate":50,"inMacro":true,"name":"Trigger","stackDepth":1,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"Queue","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"LCD","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"SBC","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"Daemon","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"Aux2","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false},{"axesRelative":false,"compatibility":"RepRapFirmware","distanceUnit":"mm","drivesRelative":false,"feedRate":50,"inMacro":false,"name":"Autopause","stackDepth":0,"state":"idle","lineNumber":0,"volumetric":false}],"job":{"build":null,"duration":null,"file":

[Carlo]

@Carlo
had to split at Filaments

{"filament":[],"fileName":null,"firstLayerHeight":0,"generatedBy":null,"height":0,"lastModified":null,"layerHeight":0,"numLayers":0,"printTime":null,"simulatedTime":null,"size":0},"filePosition":0,"firstLayerDuration":null,"lastDuration":0,"lastFileName":null,"lastFileAborted":false,"lastFileCancelled":false,"lastFileSimulated":false,"layer":null,"layerTime":null,"layers":[],"timesLeft":{"filament":null,"file":null,"layer":null},"warmUpDuration":null},"limits":{"axes":10,"axesPlusExtruders":20,"bedHeaters":12,"boards":21,"chamberHeaters":4,"drivers":26,"driversPerAxis":5,"extruders":16,"extrudersPerTool":8,"fans":16,"gpInPorts":16,"gpOutPorts":32,"heaters":32,"heatersPerTool":8,"monitorsPerHeater":3,"restorePoints":6,"sensors":56,"spindles":2,"tools":50,"trackedObjects":30,"triggers":32,"volumes":1,"workplaces":9,"zProbeProgramBytes":8,"zProbes":4},"messages":[],"move":{"axes":[{"acceleration":1000,"babystep":0,"current":1700,"drivers":["0.0"],"homed":true,"jerk":1200,"letter":"X","machinePosition":-0.012,"max":185,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":-185,"minProbed":false,"speed":18000,"stepsPerMm":160,"userPosition":-0.012,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]},{"acceleration":1000,"babystep":0,"current":1700,"drivers":["0.1"],"homed":true,"jerk":1200,"letter":"Y","machinePosition":0.013,"max":185,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":-185,"minProbed":false,"speed":18000,"stepsPerMm":160,"userPosition":0.013,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]},{"acceleration":1000,"babystep":0,"current":1700,"drivers":["0.2"],"homed":true,"jerk":1200,"letter":"Z","machinePosition":54.891,"max":420.6,"maxProbed":false,"microstepping":{"interpolated":true,"value":16},"min":-0.5,"minProbed":false,"speed":18000,"stepsPerMm":160,"userPosition":54.891,"visible":true,"workplaceOffsets":[0,0,0,0,0,0,0,0,0]}],"calibration":{"final":{"deviation":0.015,"mean":0},"initial":{"deviation":0.016,"mean":0.008},"numFactors":6},"compensation":{"fadeHeight":null,"file":null,"meshDeviation":null,"probeGrid":{"xMin":-160,"xMax":160.1,"xSpacing":20,"yMin":-160,"yMax":160.1,"ySpacing":20,"radius":165},"skew":{"tanXY":0,"tanXZ":0,"tanYZ":0},"type":"none"},"currentMove":{"acceleration":0,"deceleration":0,"laserPwm":null,"requestedSpeed":0,"topSpeed":0},"daa":{"enabled":false,"minimumAcceleration":10,"period":0},"extruders":[{"acceleration":120,"current":400,"driver":"0.3","filament":"","factor":1,"jerk":40,"microstepping":{"interpolated":false,"value":16},"nonlinear":{"a":0,"b":0,"upperLimit":0.2},"position":0,"pressureAdvance":0,"rawPosition":0,"speed":14000,"stepsPerMm":1837}],"idle":{"timeout":45,"factor":0.4},"kinematics":{"deltaRadius":231.747,"homedHeight":420.618,"printRadius":185,"towers":[{"angleCorrection":-0.126,"diagonal":440.47,"endstopAdjustment":0.212,"xPos":-200.954,"yPos":-115.431},{"angleCorrection":-0.369,"diagonal":440.47,"endstopAdjustment":-0.073,"xPos":199.949,"yPos":-117.163},{"angleCorrection":0,"diagonal":440.47,"endstopAdjustment":-0.139,"xPos":0,"yPos":231.747}],"xTilt":0,"yTilt":0,"name":"delta"},"printingAcceleration":10000,"speedFactor":1,"travelAcceleration":10000,"virtualEPos":0,"workspaceNumber":1},"network":{"hostname":"duet3","interfaces":[{"activeProtocols":[],"actualIP":"192.168.0.24","configuredIP":"192.168.0.24","firmwareVersion":null,"gateway":"192.168.0.1","mac":"DC:A6:32:A8:72:92","numReconnects":null,"signal":null,"speed":null,"subnet":"255.255.255.0","type":"wifi"}],"name":"Duet 3"},"scanner":{"progress":0,"status":"D"},"sensors":{"analog":[{"lastReading":26.8,"name":"","type":"thermistor"},{"lastReading":40.4,"name":"","type":"rtdmax31865"}],"endstops":[{"triggered":false,"type":"inputPin","probeNumber":null},{"triggered":false,"type":"inputPin","probeNumber":null},{"triggered":false,"type":"inputPin","probeNumber":null}],"filamentMonitors":[],"gpIn":[],"probes":[{"calibrationTemperature":25,"deployedByUser":false,"disablesHeaters":false,"diveHeight":5,"maxProbeCount":1,"offsets":[0,0],"recoveryTime":0.6,"speed":20,"temperatureCoefficient":0,"threshold":500,"tolerance":0.03,"travelSpeed":100,"triggerHeight":-0.1,"type":8,"value":[0]}]},"spindles":[{"active":0,"current":0,"frequency":0,"min":0,"max":10000,"tool":-1},{"active":0,"current":0,"frequency":0,"min":0,"max":10000,"tool":-1}],"state":{"atxPower":null,"beep":null,"currentTool":-1,"displayMessage":"","dsfVersion":"3.1.0","gpOut":[],"laserPwm":null,"logFile":"","messageBox":null,"machineMode":"FFF","nextTool":-1,"powerFailScript":"","previousTool":-1,"restorePoints":[{"coords":[0,0,0],"extruderPos":0,"feedRate":50,"ioBits":0,"laserPwm":null,"spindleSpeeds":[0,0],"toolNumber":-1},{"coords":[0,0,0],"extruderPos":0,"feedRate":50,"ioBits":0,"laserPwm":null,"spindleSpeeds":[0,0],"toolNumber":-1},{"coords":[0,0,0],"extruderPos":0,"feedRate":50,"ioBits":0,"laserPwm":null,"spindleSpeeds":[0,0],"toolNumber":-1},{"coords":[0,0,0],"extruderPos":0,"feedRate":50,"ioBits":0,"laserPwm":null,"spindleSpeeds":[0,0],"toolNumber":-1},{"coords":[0,0,0],"extruderPos":0,"feedRate":50,"ioBits":0,"laserPwm":null,"spindleSpeeds":[0,0],"toolNumber":-1},{"coords":[0,0,0],"extruderPos":0,"feedRate":50,"ioBits":0,"laserPwm":null,"spindleSpeeds":[0,0],"toolNumber":-1}],"status":"idle","upTime":590},"tools":[{"active":[0],"axes":[[0],[1]],"extruders":[0],"fans":[2],"filamentExtruder":0,"heaters":[1],"mix":[1],"name":"Nozzle X","number":0,"offsets":[0,0,0],"offsetsProbed":0,"retraction":{"extraRestart":0,"length":2,"speed":16.7,"unretractSpeed":16.7,"zHop":0},"standby":[0],"state":"off"}],"userSessions":[{"id":1,"accessLevel":"readWrite","sessionType":"http","origin":"::ffff:192.168.0.2","originId":49744}],"userVariables":[],"volumes":[{"capacity":13498773504,"freeSpace":10214948864,"mounted":true,"name":null,"openFiles":null,"path":"/","speed":null},{"capacity":2143281152,"freeSpace":2083966976,"mounted":true,"name":null,"openFiles":null,"path":"/boot","speed":null}]},"success":true}

[Carlo]

@chrishamm said in Duet3 + Pi DWC freezing:

Please post those outputs only after a freeze has occurred and before a (firmware) reset.

Sorry Chris I'm not sure I made this clear it freezes every time also sometimes when it echoes to the console DWC loses connection and needs me to reload the page not restart the system also I have reinstalled a number of times with the DuetPi image on brand new Sandisk Ultra SD cards always the same problem.
Just want to make sure I have given you as clear an understanding of the issue I'm having as my failing faculties possible can.

Cheers
C

[Carlo]

@Carlo

To rule the Pi out I swapped out the Pi4 for Pi 3B+ but has made no difference exactly the same ?
Thought it may help.

[chrishamm]

@Carlo I tried your config.g+bed.g on my test board and I haven't been able to provoke a freeze/crash using our latest builds. Can you please do the following:

1. Download and install this firmware build (upload on the System page, then confirm): https://chrishamm.io/Duet3Firmware_MB6HC.bin
2. Download and install this DuetWebControl version: https://chrishamm.io/DuetWebControl-SBC.zip (upload on the System page as well)
3. Stop DCS and DWS:

sudo systemctl stop duetcontrolserver
sudo systemctl stop duetwebserver

4. Download+extract this ZIP file: https://chrishamm.io/DSF.zip
5. Overwrite the files in /opt/dsf/bin
6. Restart DWS and start DCS in debug mode:

sudo systemctl start duetwebserver
sudo /opt/dsf/bin/DuetControlServer -l debug -r

And check if the problem persists.

[Carlo]

@chrishamm

Hi Chris,

All done as above however the last command returns:
pi@duet3:~ $ sudo /opt/dsf/bin/DuetControlServer -l debug -r
sudo: /opt/dsf/bin/DuetControlServer: command not found

And error in browser unable to connect to duet3 DCS is not started
I've redone it twice same result overwrote in Bin directory twice with the 10 file in the zip

[chrishamm]

@Carlo Maybe you need to run sudo chmod +x /opt/dsf/bin/DuetControlServer if you deleted it before.

Unfortunately I've found another issue which I am still debugging. I'll post an update here when I've fixed that.