Unsolved Bad command & Missing characters in Gcode since firmware upgrade

[Vinculum]

@dragonn
I thought of the exact same thing but i tried to do an M38 but with these large files it took forever. After 20 minutes I had to reset the duet because I couldn't wait for that anymore. So unless it is a few seconds thing I would not want this unless it is something that can be turned off.

[norbs12]

Having this exact issue on multiple boards now.

Currently the problems is happening on:
Firmware Name: RepRapFirmware for Duet 2 WiFi/Ethernet
Firmware Electronics: Duet Ethernet 1.02 or later + DueX5
Firmware Version: 2.03 (2019-06-13b2)
Web Interface Version: 1.22.6

I swapped out sd cards. Seems to only work when machine has been off for long period of time. I've been uploading my gcode then downloading it to verify before printing, otherwise I'll get "bad command" due to dropped characters or just sloppy prints.

Has there been any update to how to get this fixed?

[Vinculum]

@norbs12 Was this also when uploading new files while printing?

[norbs12]

@HBM-3D Yep.

I also did some minor additional testing.

I got a sha1 sum before upload, checked it again after it got uploaded to the printer and it changed. Afterward, I downloading back to my computer and the sha1 sum remained what it showed on the printer (so no longer the matching the original file).

So this means the failure is one-way, going to the SD card (or writing to it), but not reading down from it. @dc42 does this give you any ideas on possible causes? I now have two boards with this exact behavior. Both were normal for a good year or so before they started doing this.

[T3P3Tony]

@norbs12 can you confirm this only occurs if you are uploading while printing, this is what @HBM-3D found.

[dc42]

I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

[norbs12]

@dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

diff Orig.gcode Downloaded.gcode 
863362c863362
< G1 X69.968 Y5.986 E0.00274
---
> G1 X69.968 Y59.986 E0.00274
863364c863364
< G1 X70.502 Y59.729 E.00274
---
> G1 X70.502 Y59.729 E0.00274
863391c863391
< G1 X57.055 Y6 X4.310 E0.00357
---
> G1 X57.055 Y64.310 E0.00357
908950c908950
< 1 X62.824 Y98.654
---
> G1 X62.824 Y98.654
908990c908990
< G1 X1G02.446 Y66.725 E0.00361
---
> G1 X102.446 Y66.725 E0.00361
909168c909168
< G1 X103.024Y66.246 E0.00529
---
> G1 X103.024 Y66.246 E0.00529
909193c909193
< G1 X109.288 YY66.944 E0.00303
---
> G1 X109.288 Y66.944 E0.00303
909325c909325
< G1 105.718 Y70.234 E0.00378
---
> G1 X105.718 Y70.234 E0.00378
909346c909346
< G1 X103.8273 Y65.606 E0.00274
---
> G1 X103.873 Y65.606 E0.00274
909374c909374
< G1X108.523 Y68.658 E0.00274
---
> G1 X108.523 Y68.658 E0.00274
909399c909399
< G1 F21800
---
> G1 F1800
909426c909426
< G1 X05.883 Y46.145 E0.00355
---
> G1 X105.883 Y46.145 E0.00355
909435c909435
< G1 X08.937 Y47.507 E0.00355
---
> G1 X108.937 Y47.507 E0.00355
909450c909450
< G1 X1F207.974 Y52.969 E0.00373
---
> G1 X107.974 Y52.969 E0.00373
909488c909488
< G1 X06.991 Y46.409 E0.00341
---
> G1 X106.991 Y46.409 E0.00341
909501c909501
< G1 X2109.593 Y50.121 E0.00329
---
> G1 X109.593 Y50.121 E0.00329
909799c909799
< G1 X108.652 Y50891 E0.00274
---
> G1 X108.652 Y50.891 E0.00274
909806c909806
< G1 X106.9987 Y52.455 E0.00269
---
> G1 X106.987 Y52.455 E0.00269
933045c933045
< G1 X129.607 Y68.005 F1000.000
---
> G1 X129.607 Y68.005 F18000.000
933048c933048
< G1 X133.198 Y70.51 F18000.000
---
> G1 X133.198 Y70.751 F18000.000
933062c933062
< G1 X122.179 Y59.2 319 E0.00576
---
> G1 X122.179 Y59.319 E0.00576
933509c933509
< G1 X68.243 Y2.863 E0.00312
---
> G1 X68.243 Y62.863 E0.00312
933546c933546
< G1 X7Y3.197 Y64.893 E0.00307
---
> G1 X73.197 Y64.893 E0.00307
933736c933736
< G1 X53.455 Y66.03 E0.00355
---
> G1 X53.455 Y66.003 E0.00355
933738c933738
< G1 X52.239 Y65.621 E0.0009
---
> G1 X52.239 Y65.621 E0.00609
933760c933760
< G1 X53.400 Y.058.721 E0.00358
---
> G1 X53.400 Y58.721 E0.00358
1181427c1181427
< G1 X27.108 Y2.058 E0.00194
---
> G1 X27.108 Y42.058 E0.00194
1181429c1181429
< G1 X31.272 Y46.431 0.00343
---
> G1 X31.272 Y46.431 E0.00343
1181461c1181461
< G.01 X27.108 Y45.205 E0.12357
---
> G1 X27.108 Y45.205 E0.12357
1181488c1181488
< G1 X34.926 Y55.959 E000139
---
> G1 X34.926 Y55.959 E0.00139
1181513c1181513
< G1 X207.108 Y50.658 E0.09256
---
> G1 X27.108 Y50.658 E0.09256

[Vinculum]

Thanx. My files were all too big to compare. I tried 4 different program but they all crashed or simply refused to do it.

[jay_s_uk]

Try notepad++ It has a compare plugin

[dc42]

@HBM-3D said in Bad command & Missing characters in Gcode since firmware upgrade:

Thanx. My files were all too big to compare. I tried 4 different program but they all crashed or simply refused to do it.

Did you try Windows fc /b ? AFAIK it can handle very large files. The output when using /b isn't line oriented, but that might actually make things easier for me.

Cam you confirm that the original and corrupt files have exactly the same length? They should have, because the upload mechanism does a check on the file size at the end.

[dc42]

@norbs12 said in Bad command & Missing characters in Gcode since firmware upgrade:

@dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

diff Orig.gcode Downloaded.gcode
...

Thanks. There is something very odd about those diffs. For example:

> 863364c863364
> < G1 X70.502 Y59.729 E.00274
> ---
> > G1 X70.502 Y59.729 E0.00274

Clearly, a 0 has been dropped here (after the E). In most other instances, one character has again been dropped.

> 863391c863391
> < G1 X57.055 Y6 X4.310 E0.00357
> ---
> > G1 X57.055 Y64.310 E0.00357

But here, " X" appears to have been inserted.

> 908990c908990
> < G1 X1G02.446 Y66.725 E0.00361
> ---
> > G1 X102.446 Y66.725 E0.00361

This time, "G" appears to have been inserted.

> 909193c909193
> < G1 X109.288 YY66.944 E0.00303
> ---
> > G1 X109.288 Y66.944 E0.00303

And here, 'Y' has been inserted.

The fact that both these reports are using Duet Ethernet and not Duet WiFi leads me to believe that this is an issue with the transfer of data by DMA from the Ethernet chip, or just possibly an interaction between DMA to the SD card and DMA from the Ethernet chip..

[norbs12]

@dc42 I do have another ethernet chip, from a different board, I can swap it out today and try again. It might take me some time to get back to you since usually this only happens when the printer has been active and warmed up.

[dc42]

PS - please can you try again using Windows fc /b to show the differences. Given that RRF checks the received file size, I have a theory that the DMA from the Ethernet chip is dropping bytes, then inserting extra ones at the end of the block in order to finish the block. A diff using fc /b will make it clear whether or not this is the case.

[bot]

@HBM-3D said in Bad command & Missing characters in Gcode since firmware upgrade:

Thanx. My files were all too big to compare. I tried 4 different program but they all crashed or simply refused to do it.

Visual Studio Code is my new go-to GCode editor. It has comparing built in and can open massssssssive files.

[tekkydave]

On Windows I always use Textpad as a great editor. Also has a good visual built-in file compare facility.

[norbs12]

@dc42 quick question, I noticed that M38 doesn't seem to return the exact sha1 even when files are the same. Is this just a bug or am I misunderstanding something.

(using "shasum <filename>" on a Mac)
a03e3b0b03697fccb2997d3e33140801713830c0
-vs-
(M38 on Duet)
a03e3b0b3697fccb2997d3e33140801713830c0

There is an extra '0' using 'shasum' near the beginning of the checksum. I ran the m38 command twice just to make sure it was not a fluke.

Just to make sure I ran a diff and it found no differences.

[dc42]

Thanks for pointing this out. It's a bug, zeros are sometimes omitted. The SHA1 function was contributed code which I didn't check thoroughly. Will be fixed in next release.

[norbs12]

@dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:
OK I got to testing a different ethernet board. Same issues unfortunately. I was thinking of trying out the secondardy SD card slot on my paneldue. I would have to wire it up, do you think it's worth the effort?

Also is there a way to slow down the communication between the network card and the processor in maybe some test firmware? I'd try this myself if I knew how... Was thinking just cut the speed in half just to see if it fixes anything, then if it works make this somehow accessible through a custom gcode command and allow tuning of it (if that's even possible).

Thanks for pointing this out. It's a bug, zeros are sometimes omitted. The SHA1 function was contributed code which I didn't check thoroughly. Will be fixed in next release.

No worries, it's close enough to get a good idea if it's the same file. Was just using 'grep' to speed up verifying if it matched and it didn't show up but looked like it should have, which is the only reason I noticed it.

[T3P3Tony]

@T3P3Tony said in Bad command & Missing characters in Gcode since firmware upgrade:

@norbs12 can you confirm this only occurs if you are uploading while printing, this is what @HBM-3D found.

@norbs12 is this only an issue if you upload a file while it is printing? That appeared tto be what @HBM-3D reported

[norbs12]

@T3P3Tony It does seem to be only while printing though I have not had a chance to confirm that it's not related to temperature of the board. Meaning I have not tried uploading immediately after a long print.