Incident report: RRF 3 RC6 DWC 2.1.0 Lockup during print.

Danal

Just had the third failure in a row, about 30 minutes in. Absolutely no DWC after starting the job.

Now regressing. Deciding exactly how and to what...

Danal

My attempt at regressing to 1.2.5.0.

apt will not automatically deal with regressed dependencies, but it will tell you what would work. I've edited out 'false starts' and left only the commands that worked, in the order that they worked.

mkdir backup
 cp -r /opt/dsf/* backup
 ls -al backup/sd/sys
 sudo apt remove duetsoftwareframework
 sudo apt autopurge
 sudo apt autoclean
 sudo apt list -a duetsoftwareframework
 sudo apt install duetruntime=1.2.5.0
 sudo apt install duetcontrolserver=1.2.5.0
 sudo apt install duetsd=1.0.5
 sudo apt install duettools=1.2.5.0
 sudo apt install duetwebserver=1.2.3.1
 sudo apt install duetwebcontrol=2.0.7-1
 sudo apt install reprapfirmware=1.2.5.0-1
 sudo apt install duetsoftwareframework=1.2.5.0
 sudo reboot

The backup is not strictly required, apt uninstall won't delete non-empty directories.

Danal

At this point, duetcontrolserver is trying over and over to connect to the board. It can't, probably because of RRF mismatch. Time to 'bossac' the board. From the Pi:

~/BOSSA/bin/bossac -e -w -v -b /opt/dsf/sd/sys/Duet3Firmware_MB6HC.bin

This should be the correct release of firmware to match the DSF and friends, as installed by apt.

Worked... Had to press the reset button (or power cycle) after bossac. Also, good thing I made the backup, config.g was overlaid.

Now on:

Duet Web Control 2.0.7
Board: Duet 3 MB6HC v0.6 or 1.0 (MB6HC)
DSF Version: 1.2.5.0
Firmware: RepRapFirmware for Duet 3 MB6HC 3.01-RC4 (2020-03-16b1)

Danal

Next step:

M997 B1
M997 B2

M115 B0
M115 B1
M115 B2

4/12/2020, 1:24:31 AM	M115 B2
Board EXP3HC firmware 3.01-RC4 (2020-03-16b1)
4/12/2020, 1:24:25 AM	M115 B1
Board EXP3HC firmware 3.01-RC4 (2020-03-16b1)
4/12/2020, 1:24:16 AM	M115 B0
FIRMWARE_NAME: RepRapFirmware for Duet 3 MB6HC FIRMWARE_VERSION: 3.01-RC4 ELECTRONICS: Duet 3 MB6HC v0.6 or 1.0 FIRMWARE_DATE: 2020-03-16b1

Now to power cycle one more time, home, level, and run same print job.

tobias_munich

@Danal have exactly the same issue...

chas2706

@dc42 said:

There are no plans for a RRF 3.01-RC7 release, because there are no bugs in 3.01-RC6.

A Former User

@chas2706 said in Incident report: RRF 3 RC6 DWC 2.1.0 Lockup during print.:

because there are no bugs in 3.01-RC6.

I would think its obvious that we're talking about "known" bugs; and plans change.

Virtually no amount of testing can prove the absence of bugs.

Danal

Same print job finished just fine, about 3 hours, on RC4 and friends.

Duet Web Control 2.0.7
Board: Duet 3 MB6HC v0.6 or 1.0 (MB6HC)
DSF Version: 1.2.5.0
Firmware: RepRapFirmware for Duet 3 MB6HC 3.01-RC4 (2020-03-16b1)

gtj0

I would report all DSF issues to @chrishamm on github...
https://github.com/chrishamm/DuetSoftwareFramework/issues
or in this thread... https://forum.duet3d.com/topic/15343/dsf-1-3-1-unstable-released

Danal

@gtj0 How would I determine whether to post in RRF 3.0RC6 vs. DSF 1.3?

droftarts

@Danal run the job in Duet 3 standalone with the same settings and firmware version?

Ian

gtj0

@Danal said in Incident report: RRF 3 RC6 DWC 2.1.0 Lockup during print.:

@gtj0 How would I determine whether to post in RRF 3.0RC6 vs. DSF 1.3?

I have asked that question many times and I've never received a satisfactory response.

@droftarts said in Incident report: RRF 3 RC6 DWC 2.1.0 Lockup during print.:

@Danal run the job in Duet 3 standalone with the same settings and firmware version?

Ian

Don't get me started.

It's not always easy to just "run in standalone mode". In my case, I have to remove the covers from the printer to get to the sd card and remove the cable between the Duet and the SBC.

I also don't believe it's the user's responsibility to have to determine which component of the system is at fault. That should be Duet3D's responsibility. The fact that RRF and DSF seem to be owned and operated by 2 separate companies irks me to no end.

Danal

I agree. From a user's perspective, DSF and RRF are the same thing. Gcode goes in one end and motion comes out the other.

For example, it is incredibly weird that M999 restarts RRF but not DSF, even when it is absolutely reproducible that any number of hangs are cleared ONLY by restarting DSF. Which a user of this "gcode everywhere" system has no way to do (other than on the Pi, and with sudo no less!)

I love Duet and have been a happy advocate for at least a few years. I am aghast at the latest directions and actions. I sincerely hope they course correct. (To be clear, I'm all in favor of SBC/Pi integration. It is the way that's being accomplished that is going to send a great company downhill if they don't change something).

Danal

Also, as regards how easy to "reproduce in stand-alone", I don't have any ethernet that will reach. I am not the first owner of this house, and there is not an inch of Cat anything in it (except about 1 foot (1/2 meter) between the cable modem and the main wireless router). I'm also not real motivated to make a special SD card, run special ether, etc, etc, to run the printer in a mode which I will literally never run. It goes back to DSF and RRF being layers of the same thing, and they should be supported that way.

Danal

OK, rant over. Sort of.

bot

And here I am, just sad that RRF 2 isn't getting the attention it needs and deserves! We need an LTS team!

chas2706

@Danal said in Incident report: RRF 3 RC6 DWC 2.1.0 Lockup during print.:

I agree. From a user's perspective, DSF and RRF are the same thing.

I had my sarcastic rant earlier when I quoted DC42's statement that RRF 3.01-RC6 has no bugs.

I agree, I too am not interested if any particular bug is within the RRF, DCS, or DSF. They should all work together as one whole package.

The issue I have is that like you I love the idea of Duet 3 with SBC but there are many problems at the minute and all I seem to get is "It works ok in standalone mode".

If I wanted "stand alone" I would not have purchased RRF 3 and Raspberry Pi 4!

Rant over!

A Former User

@Danal said in Incident report: RRF 3 RC6 DWC 2.1.0 Lockup during print.:

and with sudo no less!

this was a topic way back when, sort of intermingled with permissions on the /opt/dsf/sd folder and /dev/spi nodes and the priority was to get it working first, then revisit.

as such i didn't poke in great detail, but as access to the spi node can be solved by group permissions, listening to port 80 (or any port below 1024) sounds like the last hurdle. the easy woraround would be nginx and a reverse proxy which would also ease setting up ssl with sometihng like letsencrypt (even if not exposed to the internet)

There are larger issues to deal with first i guess - but I will say the state of the supporting firmware and software has not been clearly communicated following the release of the hardware.

I believe I in August said I expected to run RRF2 as the stable version for 6-12 months, and the unfortunate truth of it is that with the limited team developing RRF3 + DSF they need the depend on the community for testing to stand a chance at getting ready for main stream use in such a short timeframe.

At the end of the day its up to the user to choose something tried and true, or accept that early adoption comes with a price tag in more than one sense.

droftarts

@gtj0 said

Don't get me started.

Okay, sorry I mentioned it.

@Danal said

OK, rant over. Sort of.

Okay, sorry, won't mention it again! We appreciate all your support!

@chas2706 said

Rant over!

No, really, I'm sorry for suggesting it, I'll never say it again!

@bearer said

At the end of the day its up to the user to choose something tried and true, or accept that early adoption comes with a price tag in more than one sense.

I agree. It's just taking time to get DSF (which is pretty much brand new) up to speed with the rest of the firmware (painstakingly developed over many years). But without community interest and expertise getting it working, reporting bugs and fixing, it will take much longer. So once again, thank you all for your continued support.

Ian

chas2706

@droftarts said in Incident report: RRF 3 RC6 DWC 2.1.0 Lockup during print.:

I agree. It's just taking time to get DSF (which is pretty much brand new) up to speed with the rest of the firmware

The activity shown on GitHub regards DSF says it all.