PS_ON safety question

Hi,

After having one of the heater mosfets fail open (hotend was at 400 degC before I noticed and turned everything off) , I've decided to improve the safety of my printer by controlling the 24V PSU with a relay on the PS_ON pin. I've managed to get it to work when there's a heater fault during printing. However, when I simulated a failed mosfet when the printer is idle, the relay didn't switch off.

Is there a way for the duet to monitor the heater temperatures when idle, so that when the temperature is above the set temperature and it is rising, it switches the PS_ON pin after a certain amount of time (for instance if it keeps rising for 10 seconds)?

Thanks!

DocTrucker

I've put in a request for idle ps_on thermal protection and saw it on dc42's todo list a few weeks back. I'll try to find a link for you.

DocTrucker

~~Looks like it is a long way down the list in 2.04+ release at the moment:~~ Edit: Long way down a non prioritised list of options that may make it into 2.03.

https://forum.duet3d.com/topic/8272/current-firmware-wishlist

@dc42 said in Current firmware wishlist:

As the firmware wishlist is so long now, I now keep it in the github RepRapFirmware project, at https://github.com/dc42/RepRapFirmware/blob/dev/src/BugList.txt. Here is the current one.

Todo before 2.02 release:
[clipped]

2.03:
[clipped]

Some features from the list below

Future:
[clipped]

Consider: Heater faults to run M81 even when not doing a build (https://forum.duet3d.com/topic/6225/configure-action-on-heater-fault/7)
[clipped]

I'ts good to know it's already on the wish list, thanks.
I found the thread with your request including the following quote from dc42:

Would anyone else consider this useful? I assumed that someone would always be attending the printer if it was commanded to heat but not printing from the SD card.

I'll just respond here, instead of bumping your old thread:
I'm often not in the room when the printer is idle, and even though it is idle, the heater can still be on due to a mosfet or 74HCT02 failure. In my case, I messed around with the connections on the board, and accidentaly shorted 5V to GND. The printer was still working fine, until somewhere mid-print, when the hotend suddenly heated up to above 400 degC, I didn't have a relay on the PSU at that time, so the duet couldn't stop it. I disconnected that hotend and started printing with my second hotend. All was fine, until after the print was finished and the printer was idle, when the second hotend heated up uncontrollably. I think that the 74HCT02 was damaged when I shorted 5V to GND, and it slowly died, latching the mosfets in their on state.

My point is that these issues can arise when you don't expect it (when the printer is idle and all heaters set to 0 degC). While it probably doesn't happen often, it is still a serious safety issue, especially because you expect an idle printer to be safe, so you pay less attention to it than to a printer that is currently printing. So I'd like to request that it is given a higher priority.

DocTrucker

No problem regarding leaving the old thread to collect dust. I hope this feature would be one of the things to make it onto one of the understandably occasional releases to the v0.6 compatible firmware.

General advice on this forum is tie psu ground to psu negative and ground chassis/motors. With bed slingers and many wires going to hot ends (particularly sensor, or direct extruder equipped) shorts to earth from any of the power rails is reasonably foreseeable with an ever increasing body of users either by user error or component failure.

dc42

I'd welcome more discussion on this. The main issues as I see it are:

If we turn off power to the printer due to a heater fault, the user won't know what happened unless logging was enabled and the heater faut was logged. However, I might be able to find some space in the software reset data to flag that a heater fault occurred, but probably without much detail.
When users are setting up printers, tuning heaters etc. it isn't appropriate to shut down the whole printer.

Should power off be triggered by some heater faults (e.g. temperature above limit) but not by others (e.g. temperature fluctuation outside allowed range)?

DocTrucker

I can't see that it will negatively affect many people? I'd guess if you are using ps_on you are either using a PC ATX PSU and use the 5V stand by for duet or have two power supplies and just cut power to heaters alone, or VIN?

I know some use one with a push button to start the duet until it can latch on a relay but is that the minority way of doing it?

Finally at the moment I think the ps_on trigger on fault is an opt in thing with gcode? If we're really concerned about users not liking the ps_on trigger while idle (or paused) then that could be a second gcode-config opt in?

Agree that some form of logging would be best, but would be interested to see if it is the minority or majority whose duet would shut down on ps_on trigger (and risk jammed hot end due to no fan)?

dc42 administrators 6 Jan 2019, 12:25

I'd welcome more discussion on this. The main issues as I see it are:

If we turn off power to the printer due to a heater fault, the user won't know what happened unless logging was enabled and the hearer faut was logged. However, I might be able to find some space in the software reset data to flag that a heater fault occurred, but probably without much detail.

When users are setting up printers, tuning heaters etc. it isn't appropriate to shut down the whole printer.

Should power off be triggered by some heater faults (e.g. temperature above limit) but not by others (e.g. temperature fluctuation outside allowed range)?

Issue 1: Even if you switch off the whole printer without logging, it's preferable to burning the house down. And you can always turn the printer back on, bypass the safety, and investigate what triggers the error.

Issue 2: If you can enable and disable it using gcode, it shouldn't be an issue. Or you could temporarily wire the ps_on wire to gnd to physically bypass the safety features.

I'd say that it should trigger at least when the temperature is above the limit.
What I'd also like to see, but is less important:

Trigger when the thermistor is loose, so when the temperature suddenly drops, or when the temperature doesn't rise as expected. This is only an issue when the printer is commanded to heat, but it is not running a print, so it's less of an issue.
Trigger when the set temperature is smaller than the actual temperature, but the temperature is rising. This can happen at any time the printer is on, if a mosfet fails. It's already covered by a trigger on the temp limit, but it would be nice if the fault can be identified before the heater reaches its limit. It can save you some burned fingers if, after switching on a cold printer, you touch a hot end that's supposed to be off, but is actually at 250 degC due to a mosfet fail.

I think it would be nice if the printer is always monitoring for heater faults (so also when it is idle or cooling down), and give the user the option to select ps_on behavior. So choose between: off, trigger only on max temp, and trigger on all heater faults.

Jacotheron

I agree with @Craven:

But maybe as an alternative option, make it opt-out: The protections are in place at the highest safety (triggering on any heater faults) by default, and using Gcode to reduce the safety. If someone is busy configuring heaters, they can run a single command to disable the protections (but still get the warnings), while they are busy, rather than having to remember to re-enable it once they are done. Also having the option to reduce the safety only for Max Temp excursions, can be a good idea. As an extra, perhaps a mode that in stead of switching the PS_ON off, only sends warnings to connected browsers and PanelDue (as a debugging tool).

Regarding the logging, as a must I would say that the printer should have a note that a Heater Fault Occurred (and a warning may be displayed on startup). If full logging is enabled, that should contain the additional information on what excursion happened (the symptom) that triggered the fault.

On my system i have the single power supply with bypass switch/relay mentioned above. My opinion is that if a heater fault occurs, the printer should get itself into a safe status - prevent a fire from starting. On my printer that would be completely shut down. If I then start it up, and see a warning that a heater fault occurred, I know why the printer was shut down, and can start investigating (enabling the logging and trying to replicate while I am at the printer, and have it to only give warnings).

I have just replaced the 74HCT02, and everything is working fine, so I can conclude the following:

I shorted 5V (I think, but it might have been 3.3V) to GND, damaging the 74HCT02, but it was still operating fine. After some time (a few hours), part of the 74HCT02 stopped working, latching one of the heater mosfets into an always-on state. Some time later the rest of the 74HCT02 broke down, latching all heater into their on state.

Reading this forum, it is clear that people often short one of the supply lines to GND. Since the 74HCT02 can fail some time after the short, a very dangerous situation can occur, because you expect everything to be fine (the printer is still working fine right after the short, so it must have survived the short), leave the printer to do its job, only to find out that it burned down when it was idle after it was finished printing.

Hopefully this will be solved in one of the next updates, but either way, I'd recommend anyone to invest in a dedicated 5V PSU (they aren't expensive anyway), and a relay module (I'm using a Grove Relay module, but you can use any relay module intended for Arduino or other development boards). 15 euro/dollar could save your printer, your house, or even you life. Additionally, put a (conventional) smoke detector above your printer, or even a smoke detector unit like the MQ-2, which you can connect to the relay, cutting power in case of a fire.

timcurtis67

@doctrucker said in PS_ON safety question:

I can't see that it will negatively affect many people? I'd guess if you are using ps_on you are either using a PC ATX PSU and use the 5V stand by for duet or have two power supplies and just cut power to heaters alone, or VIN?

I know some use one with a push button to start the duet until it can latch on a relay but is that the minority way of doing it?

Finally at the moment I think the ps_on trigger on fault is an opt in thing with gcode? If we're really concerned about users not liking the ps_on trigger while idle (or paused) then that could be a second gcode-config opt in?

Agree that some form of logging would be best, but would be interested to see if it is the minority or majority whose duet would shut down on ps_on trigger (and risk jammed hot end due to no fan)?

I would like the option of turning off the power supply for heater faults with the ATX pin.

I vote yes.

Martin1454

@timcurtis67 I suggested in another thread, that In case of a heater fault, the system should be able to execute an gcode script - And in that script we could put M81 in to dissable the power supply if we use that, and if somebody have some other way of dealing with heater faults, they could program it in the script.

Pato

I also vote on this feature, an script that runs when temp fail occurs. With m81 by default would be ok?

Maybe some conditions to choose whether it is over maxtemp, or any heater fault. Also a command to temporarily disable it in case someone is doing some tweaks and does not have an extra 5Vstandby source.

Hornetrider

I have followed this discussion with interest and I positively amazed how good this forum work!
From my side I vote for yes...if it helps...

But what about detection of a temperature sensor fault?
I don't mean a sensor break, this could easily be detected as the values will go to min or max, but what about a deviation of the signal. This signal is used to control the heating...
How could this be detected?
Could it help to measure the resistance of the heater while not on power to verify the reading of the temperature sensor?

Has somebody sketched the different failure causes for the heater complex and how to detect/analyze?

Jacotheron

@Hornetrider There are various failure modes that can be detected. The few I have come across:

Broken sensor or incorrectly configured sensor - as you mentioned in this case it will show at either the maximum or minimum.
Decoupled sensor - this is when the sensor reads a valid value, but is not behaving in an expected way. There are various stages, though I have seen 3. [1] heatup test -> when you command a heater to start heating, and the associated sensor is not showing an increase within a certain time period, it can be due to a decoupled sensor. [2] heating curve -> I did not know about this until it kicked in on my printer a while back (the sensor channel was failing^). In essence it measures the profile of the heater and if it heats too slowly, it triggers. [3] this last one monitors the temperature, and triggers if the sensor temperature rises or drops out of a specific range from the target.

^ The PT100 daughter board's first channel was failing and reading incorrect temperatures. Basically it would trigger while showing a temperature of only 170C, when I measured it with a k-type probe on my multimeter, I measured 250C+ -> I posted about it in this forum and the developers said that it is a confirmation to them that this protection actually worked.

There may be other failures as well (like detecting a significant rise in temperature while not actively heating), but the developers may have more details.

Knaudler

Hello

maybe it would be good to use a thermal fuse
280 grad / 10A
directly on the hotend, this will cut the power to the heater
(you have to crimp the cable)

and same on heatbed 4x thermo fuse (4x10A parallel) with 140 grad
so it will cut off power without elektronics

like this :
https://www.ebay.de/itm/15A-10A-250V-Thermosicherung-Temperatursicherung-Sicherung-66-bis-280-C-zur-Wahl-/122353697732

DocTrucker

The last time I had the chance to have a decent stab at an e-stop circuit that drew from industrial best practice I had two loops (grounded by PS_ON) in which items like this could be added. These would break the circuit that sunk current from the coils of a pair of guided contact relays, whose contacts broke the mains supply to a power supply that powered the heaters and steppers, but not the fans. Technically I had some fault shielding issues but overall it was robust, forced a local reset, and allowed basic visual fault checking.

The difficulty with thermal fuses in the hot end is two fold. Firstly manufacturers rarely allow for their mounting resulting in challenges securly mounting. Secondly it limits a machine to high or low melt point processing without changing the fuse, which itself introduces user error. Sure, for a ptfe lined hot end 280C is definitely error, but about ball park for processing of some hot materials. Raise it high enough for them and the margin between the upper process temperatures for the high temp material, and the auto-ignition temp of the lower is reduced.

Thermal fuse on the heatbed is more feasible, how are people generally mounting them?

DocTrucker

Any way, thermal fuses is drifting off topic here as what this is discussing is the system already capable (at least by physical specification) of detecting sensor issues, the reported temperature exceeding pre-defined min/max values, and an unexpected thermal rise of more than x degrees per unit of time while idle.

Currently the system ignores these errors unless the system is running a job. What this request is about is allowing the facility to turn off input connected to PS_ON in the event of an anomaly whatever the machine is doing.

My preferences would be:

Default to dropping PS_ON on heater faults whenever they occur.
Allow configuration file changes to disable (edit: or delay) this behaviour.
Allow an additional g code file to be fired on error if required by the user.

Reasoning (referring to same numbered items above):

Default setup should be as safe as possible. If a user attaches a power supply disable signal to PS_ON the natural assumption is this will disable on fault. Otherwise why not just loop PS_ON (from the power supply or relay) to ground? Likewise requiring users to research through the g code guide to find how to run the machine in the safe way seems back to front.
Naturally some people will want the option to disable it. At least from a liability point of view I'd expect the distributors of this product to be happier knowing that a user has to actively request the system is running in a less safe way rather than default to it. When the user does this they should be aware that the system needs more babying.
I have no problem with additional code running on fault, but am against only signalling PS_ON in here for three reasons. a) SD cards fail after a good amount of use. We see that on occasion on these forums and I've seen it with other devices too. I don't want the 'make it safe' bit of code in a corrupted bit of the SD card. b) If the system can't read the SD card on start up it would fall back to default (safest one would assume) levels of unexpected thermal rise or sensor fault and still have the capability of avoiding a fire. c) If you really want to specify when the M81 is fired in a sequence of g code you could disable it with the feature in 2 and then trigger it yourself. Better still would be having a time delay in 2 that would mean your M81 in the g code is nearly guaranteed to fire in the event of fault code being run. The system can then fire it off after a time delay to cover the case where the g code locks, or can't be read.

It might be nice to expand M570 with an extra option:
"Toggle ps_on in case of heater fault".

With the following options:
0: Off
1: Maxtemp only
2: All heater faults (default)

This would fulfil the first 2 preferences of DocTrucker (the delay could be implemented using the S parameter). I agree that a g code file would be nice, and that the ps_on should (by default) trigger regardsless of what's in that file.
I don't think there needs to be a distinction in ps_on behavior between idle and printing, if you can turn it off during set-up.

Fault detection is already enabled during idle, so I don't think it is that difficult to implement the above. The only thing still missing is the detection of a rising temperature when that shouldn't happen (due to mosfet or 74HCT02 fail), but maxtemp will eventually kick in to prevent serious damage, so this is less important.

BTW, slightly off-topic, but I'll be working on a dedicated safety board to cover any issues that can't be covered by the Duet. It will trip a relay and sound a buzzer on any of these issues:

Non-responsive Duet (implemented using a watchdog timer, similar to this)
Smoke detection (using MQ-2)
Failed hotend cooling fan (by measuring the current)
Thermistor senses too high temperature (will accomodate multiple thermistors that can be placed in strategic spots, such as near the Duet, the PSU, bed, etc)

I think I should be able to implement this without using a microcontroller, so it shouldn't fail easily. I'll start a new topic when I've made the first prototype (might take a while though).

DocTrucker

The too high fault current consideration could be considered for the Duet 3, don't some mosfets do current detection and if not could that be added?

I'll share the schematic for the safety circuit I built. Sometime tomorrow.