G30 stop values increasing each time

Marcossf

We ( @CarlosR it's our developer) are using a self designed probe system comprised a optoendstop (Omron EE-SX670) and a solenoid to free a flag with a guided carbon fiber probe.
For the use of these printers we need to get the probe retracted from 20-30mm above the nozzles so it get enough clearance in the printable area to do complete objects one at time within this margin.

Deploy the probe, retract the probe and test a single point its done OK. It fails when sequential readings are done.

This design is rock solid in Marlin so far (same HW), and we are using it in our 3D custom printers from 2016.
Repetability or design is not the issue here, it's the way RRF does the mesh bed compensation, who it's unknown for us is the behaviour of G29 and the algorithm used in it.

T3P3Tony

@carlosr @Marcossf I have run a test on a machine with 3.4b5

27/10/2021, 13:43:51	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:43	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:38	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:35	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:31	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:28	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:25	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:23	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:20	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:18	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:13	G30 S-1 G1 Z0.5
Stopped at height 0.049 mm
27/10/2021, 13:43:03	Stopped at height 0.050 mm
27/10/2021, 13:42:58	G30 G30 S-1 G1 Z0.5

(read from the bottom up as a cut and paste from DWC) This is using a switch as the Z probe trigger with these settings:

M558
Z Probe 0: type 5, input pin (io2.in,i2c0.clk), output pin nil, dive height 2.0mm, probe speeds 20,20mm/min, travel speed 20000mm/min, recovery time 0.10 sec, heaters normal, max taps 1, max diff 0.01

which are quite a bit slower than yours.

Setting the probe speed to 800mm/min and rerunning the tests gives:

As you can see, the repeatability is as good, but because the switch i am using only has a very short travel I can afford to move only a little on Z between each probe

27/10/2021, 13:49:15	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:49:13	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:49:12	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:49:11	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:49:08	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:49:06	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:49:04	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:49:03	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:49:02	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:49:00	G30 S-1 G1 Z0.5
Stopped at height 0.056 mm
27/10/2021, 13:48:55	G30 G30 S-1 G1 Z0.5
Stopped at height 0.046 mm
27/10/2021, 13:48:07	M558
Z Probe 0: type 5, input pin (io2.in,i2c0.clk), output pin nil, dive height 2.0mm, probe speeds 800,800mm/min, travel speed 20000mm/min, recovery time 0.10 sec, heaters normal, max taps 1, max diff 0.01

Changing to 10mm in between each probe gives:

27/10/2021, 13:51:48	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:51:45	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:51:43	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:51:41	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:51:39	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:51:36	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:51:33	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:51:28	G30 G30 S-1 G1 Z10
Stopped at height 0.046 mm

So once again no difference.

Trying the other part of the issue - adding a large offset part way through (on the 5th measurement probe):

27/10/2021, 13:54:10	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:54:08	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:54:06	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:53:58	G30 S-1 G1 Z10
Stopped at height 1.436 mm
27/10/2021, 13:53:44	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:53:42	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:53:39	G30 S-1 G1 Z10
Stopped at height 0.049 mm
27/10/2021, 13:53:33	G30 G30 S-1 G1 Z10
Stopped at height 0.059 mm

gives the expected result of a large offset, followed by a reset.

when you are running these tests are you deploying and retracting the probe in between each test or leaving it deployed?

Can you send the sequence of commands you are using to run this test where you see an error accumulating.

T3P3Tony

@fcwilt said in G30 stop values increasing each time:

What type of probe do you have that has no X or Y offset and a trigger height of 0?

For multi tool printers I have used the z probe as my 0,0,0 point and then every tool has an offset from that. - so that is not an invalid approach.

gloomyandy

@marcossf said in G30 stop values increasing each time:

Repetability or design is not the issue here, it's the way RRF does the mesh bed compensation, who it's unknown for us is the behaviour of G29 and the algorithm used in it.

I'm a little confused as to what G29 has to do with this? I thought your test was probing the same point over and over? Have you tried the same test with bed compensation turned on and off? Do you see any difference?

I'm not sure that any comparison with Marlin is valid unless it is made using the same printer hardware (your colleague stated above that this was not the case). Given what has been described so far I'd be checking to make sure there is not either some loss of steps in the Z axis or other mechanical issue.

dc42

@carlosr said in G30 stop values increasing each time:

If we run a G30 to stablish the Z datum, and then multiple G30 S-1, each time the reported stop height value increases (0.010 -> 0.030 -> 0.040 -> 0.050-> 0.080 -> ... -> 0.120 ...). Sometimes the value decreases a bit (0.040 -> 0.030), but the tendency is to increase. If we run again G30, the values seem to be reset and they start again increasing from 0.010 when multiple G30 S-1 sent.

There are two know reason why this can happen:

If your Z axis motor is driven by a CAN-connected expansion board. See https://duet3d.dozuki.com/Wiki/Duet_3_firmware_configuration_limitations?revisionid=HEAD#Section_Temporary_limitations.
If you use a high probing speed and have a high Z steps/mm.

In the first case, the additional latency of the CAN bus can delay stopping the Z motor by a few milliseconds after the probe triggers, so that it overshoots. The increase in reported stop height in this case would be a multiple of 1 microstep. In the second case, when the probe triggers and the Z motor is commanded to stop, it could skip steps because of inertia. In this case the increase in reported stop height would be a multiple of 4 full steps.

Marcossf

@dc42 said in G30 stop values increasing each time:

There are two know reason why this can happen:

If your Z axis motor is driven by a CAN-connected expansion board. See https://duet3d.dozuki.com/Wiki/Duet_3_firmware_configuration_limitations?revisionid=HEAD#Section_Temporary_limitations.

If you use a high probing speed and have a high Z steps/mm.

In the first case, the additional latency of the CAN bus can delay stopping the Z motor by a few milliseconds after the probe triggers, so that it overshoots. The increase in reported stop height in this case would be a multiple of 1 microstep. In the second case, when the probe triggers and the Z motor is commanded to stop, it could skip steps because of inertia. In this case the increase in reported stop height would be a multiple of 4 full steps.

We use 1XD boards to drive an absolute encoder closed loop stepper motor (Oriental Motor AZM48AK with a pulse driver AZD-K connected to step/dir in the 1XD board). They are ultra precise.

There isn't any inertia in Z motor motion as it is controlled in the encoder.
We can calibrate the external driver to get exact movement for each step (up to 10000 step/rev resolution available) and we did same steps on the motor driver than RRF configuration.

So I think is the limitation you said:

Due to CAN latency the motors connected to expansion boards may slightly overshoot the position at which the endstop or Z probe was triggered. This would not usually matter for an endstop switch, but it does mean that if the Z motor(s) is/are connected to an expansion board then repeated probing with a Z probe (e.g. for mesh bed compensation) is not advisable. This will be fixed in a future release

Did you know if that problem which it could be fixed quite soon?

We will do some tests modifying the step/microstep logic into the external driver software and look for any hint to minimize the problem.

With normal axis movements we obtain a ±0.001mm repetability and positioning, so we thought the Z probe could achieve same precision given from the axis movement, limited by the optoendstop resolution ±0.0025mm, enough to get a very precise mesh.

We did not take into account the possible delay of the CANbus<

EDIT: In fact, the opto endstop is connected to the 6HC board, and the Z motor is in a 1XD board, so the endstop isn't connected to the same board it drives the steps for that axis.

Marcossf

@gloomyandy said in G30 stop values increasing each time:

I'm not sure that any comparison with Marlin is valid unless it is made using the same printer hardware (your colleague stated above that this was not the case). Given what has been described so far I'd be checking to make sure there is not either some loss of steps in the Z axis or other mechanical issue.

Same autolevel hardware used in a printer driven by Marlin (SKR Pro + TMC5160) to check if there's any HW deviation.
It's not driven by the same Z stepper motor/driver obviously.

Marcossf

@t3p3tony said in G30 stop values increasing each time:

when you are running these tests are you deploying and retracting the probe in between each test or leaving it deployed?

Retracting and deploying it, except when G29 S0 who it does all the mesh probing deployed. We tried several speeds and distances but the patterns was, as @CarlosR said, incremental values.

We do some other test after David's prompt about CAN delays in the encoder driver side. We are seriously concerned if this problem couldn't be addresable.

dc42

@marcossf we had scheduled to fix this limitation in release 3.5, but I'll se if it can be brought forward to 3.4.

Marcossf

@dc42 It would be awesome.

This prototype printer is our biggest project so far, in we invest tons of money and time. We would like have an stable version soon comprised all the features needed, since we are near to end right now.

Marcossf

@dc42 Hi David.

Any news about correct the CAN delays limitation?
We can't use the mesh bed leveling in our printer because this issue.

We drive all our axis with 1XD boards connected by CANbus, and driven by absolute encoder closed loop motors (Oriental Motor AZD-K drivers and motors). So we think all the problems regarding probing (located in Z axis) are caused by this timing problem.

dc42

Hi @marcossf, I have put preliminary firmware that resolves this issue at https://www.dropbox.com/sh/cx760ysonlzzkjd/AACfsVfX4olHipuqmbWut5EKa?dl=0. Please test this firmware with care because I have only tested it using Duet 3 MB6HC + EXP3HC (the Z motor was connected to the EXP3HC).

If you are running your Duet with an attached Raspberry Pi then you must upgrade to RRF 3.4.0beta7 from the package server before you use these files.

You may also be interested in our forthcoming 6XD board, which is similar in many ways to the 6HC board but has ports for connecting external drivers instead of having internal drivers.

Marcossf

@dc42 OK, we will test it very soon. Thanks!

The 6XD looks like it will be ideal, for us at least, as the internal motor drivers have never been used in the 6HC. Hopefully I'll be able to buy and test one soon!

dc42

@marcossf have you had a chance to test this yet?

Marcossf

@dc42 Still not, i'm sorry.
We are waiting for some complementary HW to arrive.

MikeDC

@dc42 said in G30 stop values increasing each time:

Hi @marcossf, I have put preliminary firmware that resolves this issue at https://www.dropbox.com/sh/cx760ysonlzzkjd/AACfsVfX4olHipuqmbWut5EKa?dl=0. Please test this firmware with care because I have only tested it using Duet 3 MB6HC + EXP3HC (the Z motor was connected to the EXP3HC).

If you are running your Duet with an attached Raspberry Pi then you must upgrade to RRF 3.4.0beta7 from the package server before you use these files.

You may also be interested in our forthcoming 6XD board, which is similar in many ways to the 6HC board but has ports for connecting external drivers instead of having internal drivers.

@dc42
I think I have the same issue with my 3 Z motors on the 3HC
I just moved them back to the mainboard.
please see my thread called levelling and mesh.
I can test these files if you think it may help resolve my issue too and will report back.
If so, do i need to copy all the files to my firmware folder or just the mainboards bin file and the 3hc bin file ?

dc42

@mikedc yes they should resolve your issue. You need just the main board file and the files for the 3HC board and any other expansion boards (e.g. tool boards) that you are using.

MikeDC

@dc42 are the map files required and do i just upload them via dwc to the firmware folder ?

dc42

@mikedc you do not need the map files.

MikeDC

@dc42 Thankyou

I will upload as soon as my current print is finished and put Z back on the 3hc.
Ill let you know here how i get on