M911, Print not Resuming After Voltage Threshold is Reached

curieos

Running RRF 3.4.6 in SBC mode on a 6XD. The PSU is trimmed up to 27V nominal. A 24V supply buffer is being used to maintain power after the PSU shuts off. Once the buffer detects a voltage drop to 22V, it kicks in and maintains that level. This works, and can keep the DC system running for roughly 4 seconds.

Power loss detection works more or less perfectly, though the resume threshold does not seem to be working. If I simulate a momentary loss of power by flicking the power switch, the print pauses, but it does not resume. If I simulate a complete loss of power, the print pauses, but when power is restored it doesn't resume. I assume in both cases the print should be resumed automatically.

M911 S25.0 R26.0 P"G91 M83 G1 Z3 E-5 F450"		; set voltage thresholds and actions to run on power loss

Phaedrux

Are you assuming that the print should automatically resume when the power is brought back up? If so, this is not the case. The print must be manually resumed.

https://docs.duet3d.com/en/User_manual/Tuning/Resume#how-it-works

curieos

@Phaedrux That passage doesn't explicitly state that it won't autoresume on power up. I'd assume it should, otherwise what's the point?

Either way, it doesn't autoresume after a momentary blip in power, which I'm assuming it absolutely should, otherwise what's the point of the "R" Parameter?

Phaedrux

It has no way of knowing how long the power has been off, or if the print has detached from the build plate, or if it's safe to proceed. User intervention is the only way to proceed safely.

curieos

@Phaedrux I believe Prusa accomplishes this by checking the buildplate temp on resume and asking the user if the print is still attached if the buildplate temp dropped a certain amount. This logic can be implemented into the resurrect-prologue.g file very trivially.

infiniteloop

@curieos said in M911, Print not Resuming After Voltage Threshold is Reached:

That passage doesn't explicitly state that it won't autoresume on power up.

Sure it does:

…and the print is left in a paused state
After the power is restored, you can use command M916 to resume the print from where it stopped.

As @Phaedrux states, you need to check carefully whether it is safe to proceed. If you suffer from frequent power outages, there’s no other way than to install a UPS.

curieos

@infiniteloop said in M911, Print not Resuming After Voltage Threshold is Reached:

As @Phaedrux states, you need to check carefully whether it is safe to proceed. If you suffer from frequent power outages, there’s no other way than to install a UPS.

You're not understanding, this is supposed to be a large, industrial machine that offers power loss recovery as a selling point. At our shop we do occasionally suffer from momentary outages, but that's not the point. We can't account for a customer's situation.

For one, most UPSs can't handle the amount of AC power this draws (just shy of 1kW while printing). We got the DC supply buffer to ensure the DC system could remain powered long enough for the Duet to detect it.

Two, this machine is large enough that prints can go on for weeks. If a 2+ week long print using tens of kilos of plastic fails due to a power outage that lasts less than a minute, that could be hundreds or thousands of dollars in material that is now scrap.

@infiniteloop said in M911, Print not Resuming After Voltage Threshold is Reached:

…and the print is left in a paused state
After the power is restored, you can use command M916 to resume the print from where it stopped.

It does not say, explicitly, that it won't resume after a full power loss incident. It also doesn't clearly explain the "R" parameter and what the expected behavior is. This is all that I can find in the docs:

Raaa Resume threshold in volts. Must be greater than the auto save voltage. Set to a high value to disable auto resume.

Hence my confusion. I can make a macro to enable this behavior, so it's not a huge deal, but it's not clear from the docs.

Regardless, it doesn't do the 100% expected behavior of resuming after a blip, where AC power is restored before the DC system fully loses power.

Phaedrux

@curieos said in M911, Print not Resuming After Voltage Threshold is Reached:

@Phaedrux I believe Prusa accomplishes this by checking the buildplate temp on resume and asking the user if the print is still attached if the buildplate temp dropped a certain amount. This logic can be implemented into the resurrect-prologue.g file very trivially.

The same could likely be achieved with the object model and conditional gcode if you desired.

curieos

@Phaedrux said in M911, Print not Resuming After Voltage Threshold is Reached:

The same could likely be achieved with the object model and conditional gcode if you desired.

Yes, I know, that's what I meant by this:

@curieos said in M911, Print not Resuming After Voltage Threshold is Reached:

This logic can be implemented into the resurrect-prologue.g file very trivially.

In fact, I already did.

infiniteloop

@curieos said in M911, Print not Resuming After Voltage Threshold is Reached:

You're not understanding, this is supposed to be a large, industrial machine that offers power loss recovery as a selling point.

As long as you do not use automatic power loss recovery as a selling point …

OK, some clarifications. I understand why you want to buffer the electronics, it’s a good way to ensure completion of the M911 (state-saving) sequence.

But even a blip - which a buffered Duet might survive - can have effects on the printer’s mechanics, steppers may have missed steps or might have lost power. So you always have to re-home the printer after such an event - which might prove difficult, especially for Z, with a print on the bed. Depending on your hardware (servos, brakes, solenoids etc.) you might need to perform a reboot anyway just to return to a „known“ state.

Having some experience with the recovery from interrupted prints (with intentional or without power loss), a common reason for failure is the inaccuracy of homing (repowering the steppers can result in positions ±2 full steps off), leading to bad layer adhesion and thus failed prints. Sure, that depends on the specific printer - in my case, I can not probe the Z height with a print on the bulld plate. Most end stops are not accurate enough to level multiple steppers on an axis properly.

Depending on several aspects of your printer’s hardware, you may have a chance to implement automatic recovery under certain conditions - you mentioned checking the build plate temp, but you might want to treat blips (or brownouts) as a special case, too. For me, figuring out the situation after a reboot (like to check if the print file still exists) was the hardest part to script.

OwenD

@Phaedrux said in M911, Print not Resuming After Voltage Threshold is Reached:

Are you assuming that the print should automatically resume when the power is brought back up? If so, this is not the case. The print must be manually resumed.

https://docs.duet3d.com/en/User_manual/Tuning/Resume#how-it-works

The behaviour seems to have been changed around version 1.2 .
If you look at the docs for M911 for v1.19, it requires 3 values.
If the voltage drops below the middle value but doesn't go below the bottom value then the print will auto resume.

I've scanned the change logs since then and can't see anywhere it implicitly states that auto resume no longer happens under any circumstances.
There are still some references in the source code to it, but it may never get called.

Without a UPS and/capacitors, the drop is likely too fast for this to happen very often, but in the case shown by the OP it might as his voltages are well spread.

Clarification and updating the documentation might be required.

curieos

@infiniteloop I'm not expecting miracles, I know there are limits to recovering from this situation, I read the documentation. A decent attempt is better than none at all. I'm trying to make our machine give the best attempt it can. Part of that requires the bed to be turned back on as soon as possible, which can't happen if the resurrect process doesn't start as soon as possible. From there I can handle the logic of resurrect-prologue to take it the rest of the way.

dc42

@curieos auto resume has never been implemented in RRF. As has been mentioned in previous posts in this thread, unconditional auto resume is fraught with problems.

@curieos said in M911, Print not Resuming After Voltage Threshold is Reached:

Part of that requires the bed to be turned back on as soon as possible, which can't happen if the resurrect process doesn't start as soon as possible

If you always use the same bed temperature, or there is some minimum bed temperature that will be sufficient to prevent prints detaching even if they use a higher bed temperature, then in RRF 3.5 you can use conditional GCode to do this. Here's an outline of what you could do, to turn on the bed heater and offer the user the chance to resume the print immediately. Note, I have not tested this so may contain errors.

Create macro file /sys/offer-resume.g containing this:

M140 S60                       ; set bed temperature to 60C
M291 S4 P"There is an incomplete print on the bed. Please choose from the following:" K"Resume now":"Keep bed warm, resume later":"Don't resume, delete the resume file"
if input = 0
    M916                       ; resume the print
elif input = 2
    M140 S0                    ; turn off the bed heater
    M472 P"/sys/resurrect.g"   ; delete the resume file

At the end of config.g add this:

if fileexists("/sys/resurrect.g")   ; if there is a resurrect file
    M98 P"offer-resume.g"

Note, if your primary user interface relies on a network connection then you won't be able to use a dialog in config.g. In this case you can create a runonce,g file to do these actions instead, like this. In config.g add this near the end:

if fileexists("/sys/resurrect.g")
    echo >"/sys/runonce.g" "M98 P""offer-resume.g"""

RRF runs the file runonce.g if it exists after completing config.g and starting the network.

curieos

@dc42 Then what does the R parameter do in M911?

Thanks for the example macros, though we are still on 3.4.6 since it's stable. I'm sure I can figure something out in the meantime.

While we're on this train, I noticed some other odd behavior. I'm not sure if it's expected, but it appears that the resume.g file isn't deleted after it is executed, at least not consistently. I had a test I was running to verify the behavior, and I ran through two complete power loss incidents on the same print (I manually cut power by flipping the power switch). The first resurrect process completed mostly successfully, but the second one didn't overwrite the first resurrect.g file, so it ended up resuming from layer 2, where the first incident occurred.

infiniteloop

@curieos said in M911, Print not Resuming After Voltage Threshold is Reached:

The first resurrect process completed mostly successfully, …

Please explain "mostly" - what goes wrong?

… but the second one didn't overwrite the first resurrect.g file

In this case, I assume the Duet didn't have enough time to completely execute M911 before it ran out of power.

curieos

@infiniteloop said in M911, Print not Resuming After Voltage Threshold is Reached:

In this case, I assume the Duet didn't have enough time to completely execute M911 before it ran out of power.

That is highly unlikely, the buffer provides enough power for the M911 codes to be run to completion, plus an additional second or two. It's rated to supply 20A for 650ms iirc, and the bed is AC powered so it's not cutting into that budget. I estimate the buffer supplies a good 3-4 seconds of 22V. It can also be set to hold at Vin-1V, but I didn't do any testing in that configuration because that machine needed to run a big production job.

@infiniteloop said in M911, Print not Resuming After Voltage Threshold is Reached:

Please explain "mostly" - what goes wrong?

That's unrelated to this thread, it's related to fine-tuning the resurrect-prologue.g file behavior.

infiniteloop

@curieos said in M911, Print not Resuming After Voltage Threshold is Reached:

That is highly unlikely, the buffer provides enough power for the M911 codes to be run to completion

Sorry, I forgot:

this is supposed to be a large, industrial machine that offers power loss recovery as a selling point.

As a Pro, you sure have monitored the 24V and 3.3V rails’ behaviour on the event of sudden power loss - a cheapo two-channel scope is all you need. What I know for sure is that the resume.g file is generated by RRF on an undervoltage event as long as the Duet’s electronics have sufficient power to stay alive long enough. And I can assure you: 4 secs are long enough.

OTOH, your 3-4 seconds buffer supplies 22V, so it is meant to feed all steppers and heaters (excluding the bed) as well? These devices are operated by strong pulses - a voltage plot over these 3-4 secs might be interesting