Duet 3/Rpi + toolboard on 3.2b3 - G29 fails

oozeBot

@oliof yes - and updated the bootloader.. will be testing with a usb cable attached at some point here soon.

oozeBot

I've had time to run mesh bed leveling again with a serial monitor attached. With M111 P3 S1 and M111 P16 S1 enabled, nothing worth reporting is captured. It simply just stops somewhere around 40~60 probe points in.

Below is what was captured. @dc42 - could you give me any guidance on what else I can try to capture the details of what is happening?

From the snippet below, it stopped on line 18, with the probe extended..

Requesting macro file 'deployprobe.g' (fromCode: true)
HTTP: M280 P0 S10
Requesting macro file 'daemon.g' (fromCode: false)
Macro completed on channel 0
Waiting macro completed on channel 9
Requesting macro file 'daemon.g' (fromCode: false)
Waiting macro completed on channel 9
Requesting macro file 'retractprobe0.g' (fromCode: true)
Waiting macro completed on channel 0
Requesting macro file 'retractprobe.g' (fromCode: true)
HTTP: M280 P0 S90
Macro completed on channel 0
Requesting macro file 'daemon.g' (fromCode: false)
Waiting macro completed on channel 9
Requesting macro file 'deployprobe0.g' (fromCode: true)
Waiting macro completed on channel 0
Requesting macro file 'deployprobe.g' (fromCode: true)
HTTP: M280 P0 S10
Aux: M409 F"d99f"
Macro completed on channel 0
Requesting macro file 'daemon.g' (fromCode: false)
Waiting macro completed on channel 9
Requesting macro file 'daemon.g' (fromCode: false)
Waiting macro completed on channel 9
Requesting macro file 'daemon.g' (fromCode: false)
Waiting macro completed on channel 9
Requesting macro file 'daemon.g' (fromCode: false)
Waiting macro completed on channel 9
Aux: M409 F"d99f"

oozeBot

I just tried to lower the routine to a 49 point mesh and it still failed multiple times at different locations.

Sigh.. I understand it's a Sunday, but if anyone sees this and has any ideas on how to capture detailed info on the issue before rolling back to 3.2b1 in a few hours, please let me know.

gloomyandy

Run the same test but also run DCS in debug mode and capture the output.

oozeBot

@gloomyandy DCS is not an option, unless there is another way?

Debugging disabled for modules: Platform(0) Network(1) Webserver(2) GCodes(3) Move(4) Heat(5) DDA(6) Roland(7) Scanner(8) PrintMonitor(9) Storage(10) PortControl(11) DuetExpansion(12) FilamentSensors(13) WiFi(14) Display(15) LinuxInterface(16) CAN(17)

gloomyandy

Sorry should have provided more details. You need to do this on the rPi. This is the official doc: https://duet3d.dozuki.com/Wiki/Getting_Started_With_Duet_3#Section_Monitoring_optional

I'm not sure if that is 100% correct for 3.2-beta3 though. In particular you may find that when you run DCS as root (which this will do), you can no longer connect via the web interface. But you should still be able to run the G29 via the USB console. You may also find that this produces a lot of output so you may find that changing the command to:
sudo /opt/dsf/bin/DuetControlServer -l debug | tee log
will help. This will send a copy of the output to a file called log in your current directory.

oozeBot

@gloomyandy here is the tail end of the output before it fails.. there is nothing valuable. It simply stops:

[info] Finished macro file retractprobe.g
[debug] HTTP: ==> Unfinished starting code: G29 S0
[debug] HTTP: Disposing macro file retractprobe.g
[debug] HTTP: Sent M566 Z600.00, remaining space 1472, needed 32
[debug] HTTP: -> Resumed suspended code
[debug] HTTP: Sent M201 Z600.00, remaining space 1440, needed 32
[debug] HTTP: -> Resumed suspended code
[debug] HTTP: Sent G1 X152.4 Y152.4 Z10, remaining space 1392, needed 48
[debug] HTTP: -> Resumed suspended code
[debug] HTTP: Suspending code M566 Z600.00
[debug] HTTP: Suspending code M201 Z600.00
[debug] HTTP: Suspending code G1 X152.4 Y152.4 Z10
[debug] Macro file deployprobe0.g not found
[debug] HTTP: ==> Starting code G29 S0
[debug] HTTP: Disposing macro file deployprobe0.g
[debug] HTTP: ==> Unfinished starting code: G29 S0
[debug] HTTP: Sent M566 Z600.00, remaining space 1472, needed 32
[debug] HTTP: -> Resumed suspended code
[debug] HTTP: Sent M201 Z600.00, remaining space 1440, needed 32
[debug] HTTP: -> Resumed suspended code
[debug] HTTP: Sent G1 X152.4 Y152.4 Z10, remaining space 1392, needed 48
[debug] HTTP: -> Resumed suspended code
[debug] HTTP: Suspending code M566 Z600.00
[debug] HTTP: Suspending code M201 Z600.00
[debug] HTTP: Suspending code G1 X152.4 Y152.4 Z10
[info] Starting macro file deployprobe.g on channel HTTP
[debug] HTTP: ==> Starting code G29 S0
[debug] Waiting for execution of M280 P0 S10 (macro code)
[debug] Processing M280 P0 S10
[debug] Waiting for finish of M280 P0 S10
[debug] HTTP: Sent M280 P0 S10, remaining space 1496, needed 40
[debug] Completed M280 P0 S10
[debug] Finished codes from macro file deployprobe.g
[info] Finished macro file deployprobe.g

I tried running M122. It's added to the queue but since G29 is still running, it's blocked from executing. The only way I know out of this is to reboot, which clears any errors.

gloomyandy

There may be something in there that is useful to Christian when he gets to look at it, so if you have the full log file please post it for him to be able to look at.

oozeBot

@gloomyandy @chrishamm Here is the log. It also contains a bonus crash at line 362-364 where it lost connection and soft-rebooted itself. I hadn't seen that before..

I'm going to be rolling back to 3.2b1 here soon. Please let me know if there is anything else I can assist with testing..

log.txt

jbarros

Just to add some data to this thread, I'm able to finish 210 point Mesh Probing on Duet3 with Toolboard with no SBC running 3.2 beta3:

09/11/2020, 12:27:29	210 points probed, min error -0.063, max error 0.349, mean 0.079, deviation 0.085
Height map saved to file 0:/sys/heightmap.csv

dc42

@jbarros said in Duet 3/Rpi + toolboard on 3.2b3 - G29 fails:

Just to add some data to this thread, I'm able to finish 210 point Mesh Probing on Duet3 with Toolboard with no SBC:
09/11/2020, 12:27:29	210 points probed, min error -0.063, max error 0.349, mean 0.079, deviation 0.085
Height map saved to file 0:/sys/heightmap.csv

Thanks, that's a very useful data point. Looks like it's something to do with running the deploy/retract files when they are held on the SBC, because the rest of the G29 probing function doesn't involve the SBC except at the start and end.

oozeBot

@dc42 I still have one machine running 3.2b3 if there is anything you or others want me to test on this issue.

I plan on rolling back to 3.2b1, creating a height map, and the moving then back to 3.2b3 for other testing by this afternoon though.

dc42

@oozeBot said in Duet 3/Rpi + toolboard on 3.2b3 - G29 fails:

@dc42 I still have one machine running 3.2b3 if there is anything you or others want me to test on this issue.

I plan on rolling back to 3.2b1, creating a height map, and the moving then back to 3.2b3 for other testing by this afternoon though.

@chrishamm, is there anything else you would like @oozeBot to test?

Touchthebitum

@dc42 said in Duet 3/Rpi + toolboard on 3.2b3 - G29 fails:

I plan on rolling back to 3.2b1, creating a height map, and the moving then back to 3.2b3 for other testing by this afternoon though.

I tried it yesterday and it seems to work but from 3.2b2 instead of 3.2b1 to 3.2b3

chrishamm

@oozeBot Thanks for reporting this issue, I could reproduce it and I'm testing a potential fix for this problem.

oozeBot

@chrishamm great! I didn't get around to rolling this machine back to 3.2b1, so please let me know if you want me to test your fix..

chrishamm

@oozeBot Yes, I am quite sure I could figure out what was going on. Please try out this firmware build with DSF 3.2-b3: https://pkg.duet3d.com/Duet3Firmware_MB6HC.bin You can upload it on the System page and confirm the update prompt.

oozeBot

@chrishamm This appears to be fixed. I just completed a 196 probe mesh without issue. Thanks!

oozeBot

@chrishamm scratch that - while it did complete, DWC is still in a busy state where it will not accept any new commands.

chrishamm

@oozeBot Please try to download and flash that file again - I just updated it once more with my other hotfix. If it remains in the "busy" state with that build, please share the debug log of DCS and I'll see what I can do.