DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1

gtj0

@Danal I wonder if DuetLapse is using just enough time to keep the DCS from hogging it all.

ChrisP

@gtj0 said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

@Garfield @ChrisP Have you tried my suggestion above to remove the two CPUScheduling lines from /lib/systemd/system/duetcontrolserver.service and rebooting?

I haven't yet, but only because I've not had chance (5 and a half hour Teams meeting today - these really are "special" times). I'll give it a go now.

Do you think the priority is like to have any effect on the cause of the crash though? Certainly if DCS does go nuts and it has a high priority, this will result in the total lock-ups that we're seeing, but is it going to be part of the in initial cause? I guess perhaps if DCS is waiting for something that can get CPU time because it's being blocked by DCS.... but if this were the case, I'd have thought it'd recover at some point?

@Danal said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

Data point: I have not changed the CPU scheduling and an approximate 18 hour print just finished.

To be honest, this doesn't entirely surprise me - I've managed to do a couple of decent length prints without issue... indeed, I've never had it crash mid-print. If my system is going to die, it usually does it pretty much immediately on startup and I think it's happened twice when I've opened a new DWC instance, but' I'm not confident to say that that is anything more than coincidence at the moment. My experience has mainly been that if it boots and doesn't crap out in the first few minutes then it'll just keep going.

The one thing that I have noticed that may or may not be related is that I've been noticing that on the DWC instance that running on the Pi, the left hand menu bar frequently becomes totally unresponsive, even after a Ctrl-F5 of the page. DWC otherwise seems to be working as I can see temperature fluctuations and print status progressing but the only way I've found to get the menu bar to respond is to reset DCS.

Anyway, I'll try removing the CPUScheduling since I have no better ideas currently.

gtj0

@ChrisP I don't think it's the cause of the crashes themselves but I do think that removing the real-time scheduling (fifo) will make it a LOT easier to troubleshoot further.

Danal

@ChrisP said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

To be honest, this doesn't entirely surprise me - I've managed to do a couple of decent length prints without issue... indeed, I've never had it crash mid-print. If my system is going to die, it usually does it pretty much immediately on startup and I think it's happened twice when I've opened a new DWC instance, but' I'm not confident to say that that is anything more than coincidence at the moment. My experience has mainly been that if it boots and doesn't crap out in the first few minutes then it'll just keep going.

I believe that fits the behaviors I saw on Sunday evening. I wasn't really "logging", not even mentally, so I am not absolutely certain.

Phaedrux

@deckingman said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

....................... but right now the learning curve is pretty steep and what I think should be simple tasks seem not to be - and I do this **** for a living !!

My God, what chance do us mere mortals have! As far as I'm concerned, you guys are all talking in some foreign language that I don't understand. I'll be sticking with stand alone operation for quite some time yet methinks.

Best not to tour the sausage factory before the wiener roast.

Garfield

Just did the scheduling thing - no change, I increased the timeout also but no difference.

I don't mind helping out here but I need some direction from the devs on what to look for - and where to look for it. So if it will be useful I will help work through this - otherwise I really need this printer back in service ....

Apr 27 20:17:09 duet3 systemd[1]: duetcontrolserver.service: Succeeded.
Apr 27 20:33:50 duet3 systemd[1]: duetcontrolserver.service: Service RestartSec=30s expired, scheduling restart.
Apr 27 20:33:50 duet3 systemd[1]: duetcontrolserver.service: Scheduled restart job, restart counter is at 1.
Apr 27 20:33:50 duet3 systemd[1]: Stopped Duet Control Server.
Apr 27 20:33:50 duet3 systemd[1]: Started Duet Control Server.
Apr 27 20:33:51 duet3 DuetControlServer[763]: Duet Control Server v2.1.1
Apr 27 20:33:51 duet3 DuetControlServer[763]: Written by Christian Hammacher for Duet3D
Apr 27 20:33:51 duet3 DuetControlServer[763]: Licensed under the terms of the GNU Public License Version 3
Apr 27 20:33:51 duet3 DuetControlServer[763]: [info] Settings loaded
Apr 27 20:33:51 duet3 DuetControlServer[763]: [info] Environment initialized
Apr 27 20:33:51 duet3 DuetControlServer[763]: [info] Connection to Duet established
Apr 27 20:33:51 duet3 DuetControlServer[763]: [info] IPC socket created at /var/run/dsf/dcs.sock
Apr 27 20:33:53 duet3 DuetControlServer[763]: [fatal] Abnormal program termination
Apr 27 20:33:53 duet3 DuetControlServer[763]: [fatal] Update task faulted
Apr 27 20:33:53 duet3 DuetControlServer[763]:    System.ArgumentException: Property set method not found.
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at System.Reflection.PropertyInfo.SetValue(Object obj, Object value)
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 237
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.FilamentMonitor.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Sensors/FilamentMonitors/FilamentMoni
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 209
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.FilamentMonitor.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Sensors/FilamentMonitors/FilamentMoni
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.ModelCollectionHelper.UpdateFromJson(IList list, Type itemType, JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 264
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.MachineModel.UpdateFromFirmwareModel(String key, JsonElement jsonElement) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/MachineModel.cs:line 149
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetControlServer.Model.Updater.Run() in /home/christian/duet/DuetSoftwareFramework/src/DuetControlServer/Model/Updater.cs:line 182
Apr 27 20:33:53 duet3 DuetControlServer[763]: [fatal] Update task faulted
Apr 27 20:33:53 duet3 DuetControlServer[763]:    System.ArgumentException: Property set method not found.
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at System.Reflection.PropertyInfo.SetValue(Object obj, Object value)
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 237
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.FilamentMonitor.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Sensors/FilamentMonitors/FilamentMoni
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 209
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.FilamentMonitor.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Sensors/FilamentMonitors/FilamentMoni
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.ModelCollectionHelper.UpdateFromJson(IList list, Type itemType, JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.ModelObject.UpdateFromJson(JsonElement jsonElement, Boolean ignoreSbcProperties) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/Base/ModelObject.cs:line 264
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetAPI.Machine.MachineModel.UpdateFromFirmwareModel(String key, JsonElement jsonElement) in /home/christian/duet/DuetSoftwareFramework/src/DuetAPI/Machine/MachineModel.cs:line 149
Apr 27 20:33:53 duet3 DuetControlServer[763]:    at DuetControlServer.Model.Updater.Run() in /home/christian/duet/DuetSoftwareFramework/src/DuetControlServer/Model/Updater.cs:line 182
Apr 27 20:33:53 duet3 DuetControlServer[763]: [info] Application has shut down
Apr 27 20:33:53 duet3 systemd[1]: duetcontrolserver.service: Succeeded.
Apr 27 20:34:23 duet3 systemd[1]: duetcontrolserver.service: Service RestartSec=30s expired, scheduling restart.
Apr 27 20:34:23 duet3 systemd[1]: duetcontrolserver.service: Scheduled restart job, restart counter is at 2.
Apr 27 20:34:23 duet3 systemd[1]: Stopped Duet Control Server.
Apr 27 20:34:23 duet3 systemd[1]: Started Duet Control Server.
Apr 27 20:34:23 duet3 DuetControlServer[879]: Duet Control Server v2.1.1
Apr 27 20:34:23 duet3 DuetControlServer[879]: Written by Christian Hammacher for Duet3D
Apr 27 20:34:23 duet3 DuetControlServer[879]: Licensed under the terms of the GNU Public License Version 3
Apr 27 20:34:24 duet3 DuetControlServer[879]: [info] Settings loaded
Apr 27 20:34:24 duet3 DuetControlServer[879]: [info] Environment initialized
Apr 27 20:34:24 duet3 DuetControlServer[879]: [info] Connection to Duet established
Apr 27 20:34:24 duet3 DuetControlServer[879]: [info] IPC socket created at /var/run/dsf/dcs.sock
Apr 27 20:34:24 duet3 DuetControlServer[879]: [warn] Daemon: Out-of-order reply: ''
Apr 27 20:34:25 duet3 DuetControlServer[879]: [fatal] Abnormal program termination

Garfield

I now have DCS up and running - this is something in config.g that it doesn't like and is bombing out. I looked at line 182 in updater.cs and it is immediately after it parses the fans. I have one fan defined as an output to enable a relay for lighting, this fan is not associated with anything - nor will it ever be - needed an on off output I could fire at will and this fitted the bill, I wonder if this is the trigger for the failure - it can't associate it with anything ???

Q : Is it possible to add an 'output' type that is neither a heater nor a fan to RRF3

I have almost my entire config.g commented out and will enable stuff line by line until it breaks again ..

My system.g is below

config.g

After uncommenting a line the reset takes an eternity, it works but it is a cause for concern - way too slow resetting

A Former User

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

Q : Is it possible to add an 'output' type that is neither a heater nor a fan to RRF3

look at M42

Garfield

So here is a 'partial' result.

DCS does not fail gracefully if it encounters something in .g files that it can't parse for whatever reason - it crashes and doesn't recover.

Sorry boys - needs to be a whole lot more friendly than this, errors should be reported and ignored, error trapping needs work.

Loss of DCS should not prevent the web server functioning - or how can you fix whatever the problem is. The web portal should report the loss of DCS, should report problems encountered parsing .g files - including the line concerned but should remain accessible to allow .g file edits / fixes. If you need DCS to make the edits then sort out the error trapping but might want to review the architecture in that case.

My config-override.g is currently empty .... this too blows DCS up ...

Apr 27 21:22:57 duet3 DuetControlServer[5864]: [info] Settings loaded
Apr 27 21:22:58 duet3 DuetControlServer[5864]: [info] Environment initialized
Apr 27 21:22:58 duet3 DuetControlServer[5864]: [info] Connection to Duet established
Apr 27 21:22:58 duet3 DuetControlServer[5864]: [info] IPC socket created at /var/run/dsf/dcs.sock
Apr 27 21:22:58 duet3 DuetControlServer[5864]: [info] Starting macro file config.g on channel Trigger
Apr 27 21:22:58 duet3 DuetControlServer[5864]: [warn] M307: Heater 0 appears to be over-powered. If left on at full power, its temperature is predicted to reach 875C
Apr 27 21:22:59 duet3 DuetControlServer[5864]: [info] Starting macro file config-override.g on channel Trigger
Apr 27 21:22:59 duet3 DuetControlServer[5864]: [info] Finished macro file config-override.g
Apr 27 21:23:00 duet3 DuetControlServer[5864]: [fatal] Abnormal program termination

gtj0

@Garfield Wow. There is somehting going on in your environment. I can't reproduce either the M950 or config-override.g issues.

Probably a good idea to just go back to a "stable" build and wait for things to shake out a bit more. Thanks for trying anyway.

Garfield

@gtj0 I wish I knew how to just get back to a base I can trust at this stage, was fine at RC6 but it seems that even that is busted now on my box.

I've firmware erased and tried all manner of 'stuff' now, if I comment out certain lines in config.g things keep running ...

Going to blow this back to the stable release but it has problems that I couldn't live with (some of which centred on the PINDA), I also had homing issues - hence I was into the RC's - seems like something is buried in there that erase and re-imaging isn't clearing ...

gtj0

@Garfield Yeah it's frustrating I know. You may want to create an issue in the DSF GitHub repo with that crash anyway. If not, would you mind if I opened an issue on your behalf?

Garfield

Absolutely no issue at all with you adding - I suppose I should create a GIT account .... even then though I need to be sure my own incompetence isn't the cause before logging anything - so I'll still run it by here.

gtj0

@Garfield https://github.com/chrishamm/DuetSoftwareFramework/issues/128

It looks like that crash is in filament monitor code. With your config.g, maybe I can reproduce it later today.

gtjoseph created this issue in chrishamm/DuetSoftwareFramework

closed DCS 2.1.1 Crashing with System.ArgumentException: Property set method not found #128

Garfield

Need some place I can send you a complete copy of my backup files (they're over 5 meg)

Here are the logs from tonights blood pressure testing ... if you need more please scream before I wipe this thing again ....

duetcontrolserver.txt

duetwebserver.txt

Tried to send only my .g files but system won't accept zip files ...

pause.g homez.g homey.g homex.g homeall.g tpre0.g tpost0.g tfree0.g stop.g sleep.g resume.g

A Former User

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

before I wipe this thing again ....

RRF3.01-RC10     duetsoftwareframework 2.1.1      reprapfirmware 2.1.1-1
RRF3.01-RC9      duetsoftwareframework 2.1.0      reprapfirmware 2.1.0-1
RRF3.01-RC8      duetsoftwareframework 2.0.0      reprapfirmware 2.0.0-1
RRF3.01-RC7      duetsoftwareframework 1.3.2      reprapfirmware 1.3.2-1
RRF3.01-RC6      duetsoftwareframework 1.3.1      reprapfirmware 1.3.1-1
RRF3.01-RC6      duetsoftwareframework 1.3.0      reprapfirmware 1.3.0-1
RRF3.01-RC4      duetsoftwareframework 1.2.5.0    reprapfirmware 1.2.5.0-1
RRF3.0           duetsoftwareframework 1.2.4.0    reprapfirmware 1.2.4.0-1
RRF3.0           duetsoftwareframework 1.2.3.1    reprapfirmware 1.2.3.1-1
RRF3.0           duetsoftwareframework 1.2.3.0    reprapfirmware 1.2.3.0-1
RRF3.0           duetsoftwareframework 1.2.2.1    reprapfirmware 1.2.2.1-1
RRF3.0RC2+1      duetsoftwareframework 1.2.2.0    reprapfirmware 1.2.2.0-1
RRF3.0RC1        duetsoftwareframework 1.2.1.0    reprapfirmware 1.2.1.0-1
RRF3.0RC1        duetsoftwareframework 1.2.0.0    reprapfirmware 1.2.0.0-1
RRF3.0beta11     duetsoftwareframework 1.1.0.5    reprapfirmware 1.1.0.5-1
RRF3.0beta11     duetsoftwareframework 1.1.0.4    reprapfirmware 1.1.0.4-1
RRF3.0beta10+2   duetsoftwareframework 1.1.0.3    reprapfirmware 1.1.0.3-1
RRF3.0beta10+2   duetsoftwareframework 1.1.0.2    reprapfirmware 1.1.0.2-1
RRF3.0beta10+2   duetsoftwareframework 1.1.0.1    reprapfirmware 1.1.0.1-1
RRF3.0beta10+2   duetsoftwareframework 1.1.0.0    reprapfirmware 1.1.0.0-1
RRF3.0beta10+1   duetsoftwareframework 1.0.4.1    reprapfirmware 1.0.4.1-1

https://forum.duet3d.com/post/149932 <-- quasi shortcut for installing a given DSF

edit: caveat emptor - full refund no questions asked if not correct...
(code is fugly as f, but shared )

Garfield

The gifts keep on coming ... 12 out of 10 - that's 2 cases of beer I owe you ...

A Former User

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

The gifts keep on coming ... 12 out of 10 - that's 2 cases of beer I owe you ...

haha, lets see if anything catches fire first - and i suspect you need that beer more than me atm.

Danal

@Garfield said in DCS Crash with 3.01-R10 / DWC 2.1.5 / DSF 2.1.1:

Could really do with that 'version compatibility' matrix ....

Your wish, and all that. See https://forum.duet3d.com/topic/15928/new-tool-duet-v3-xx-rcx-to-apt-dsf-2-x-x-version-mapping for more info.

Meanwhile a couple of examples:

./DuetVersionsAll.sh
Expect this to take several seconds per line of output.

duetsoftwareframework 2.1.1 contains reprapfirmware version 2.1.1-1 which is internal version 3.01-RC10
duetsoftwareframework 2.1.0 contains reprapfirmware version 2.1.0-1 which is internal version 3.01-RC9
duetsoftwareframework 2.0.0 contains reprapfirmware version 2.0.0-1 which is internal version 3.01-RC8
duetsoftwareframework 1.3.2 contains reprapfirmware version 1.3.2-1 which is internal version 3.01-RC7
duetsoftwareframework 1.3.1 contains reprapfirmware version 1.3.1-1 which is internal version 3.01-RC6
...snip...

 ./DuetVersion.sh
  Highest Available DuetSoftwareFramework = 2.1.1
Currently Installed DuetSoftwareFramework = 2.1.1

Dependencies for DSF version 2.1.1 are:
 duetcontrolserver (= 2.1.1)
 duetsd (= 1.0.6)
 duettools (= 2.1.1)
 duetwebserver (= 2.1.0)
 duetwebcontrol (= 2.1.5)
 reprapfirmware (>= 2.1.1-1)
 reprapfirmware (<= 2.1.1-999) )

reprapfirmware apt version 2.1.1-1 is internal version 3.01-RC10

./DuetVersion.sh 22
  Highest Available DuetSoftwareFramework = 2.1.1
Currently Installed DuetSoftwareFramework = 2.1.1

Command line arg specified to request Release -22 from highest available.
       Highest-22 is DuetSoftwareFramework = 1.0.4.1

Dependencies for DSF version 1.0.4.1 are:
 duetcontrolserver (= 1.0.4.1)
 duetsd (= 1.0.3)
 duettools (= 1.0.4.1)
 duetwebserver (= 1.1.1.0)
 duetwebcontrol (= 2.0.0-5)
 reprapfirmware (>= 1.0.4.1-1)
 reprapfirmware (<= 1.0.4.1-999) )

reprapfirmware apt version 1.0.4.1-1 is internal version 3.0beta10+1

The '22' argument is asking about the release 22 lines "back" from highest available.

gtj0

@Garfield I can reproduce your crash. It's the M591 command for the filament sensor. I'll add more info to the GitHub issue.