Unsolved Bad command & Missing characters in Gcode since firmware upgrade

[dc42]

@HBM-3D , have the files you upload been getting bigger?

[Vinculum]

@dc42 Yes I do run bigger prints with the new machines. Our gcodes are generally between 150 & 550Mb.
In this example the file was "just" 230mb which is for us a medium size run.

[Vinculum]

I now have it on another machine as well. So there is something common that drives this issue.
This was with a 190Mb file. If I hash (CRC32) them I get this result:
Original file: 5A759A1C
Downloaded file: 654B92EC

I did a reupload of the original file and did a download immediately after uploading it. All via the DWC.
The has now is has the same code as the original file: 5A759A1C

So whatever happens doesn't appear to be consistent or it happens during/after printing.

[dc42]

I guess possible solutions are to slow down the SPI clock to see if we can find a speed that is always reliable, or to send the CRC with the uploaded data and check it at the end of the upload.

Btw there is a gcode command to compute and return the SHA1 hash of a GCode file on the SD card. You may find it useful for checking the integrity of uploaded files.

Can you do a compare of original vs corrupted files? I think the ancient DOS/Windows command fc /b works on large files.

[Vinculum]

I figured something out this weekend and tested it. If I upload the files while the printer is idle then the files are OK. If I do so when the printer is running it drops the ball and alters the file.
This weekend I had another strange situation with this. Some line of code was wrong but ended up being a valid gcode. The printer went all the way to x, y & z-max and tried to resume printing from there. Naturally it was spaghetti and a busted hotend but nothing serious. Weird stuff and dangerous though...

So uploading while printing is a no go, at least for bigger files i guess.

[dc42]

@HBM-3D said in Bad command & Missing characters in Gcode since firmware upgrade:

I figured something out this weekend and tested it. If I upload the files while the printer is idle then the files are OK. If I do so when the printer is running it drops the ball and alters the file.

Thanks for the update, that gives me something to test.

[dragonn]

@dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

Btw there is a gcode command to compute and return the SHA1 hash of a GCode file on the SD card. You may find it useful for checking the integrity of uploaded files.

Maybe could this be implemented in DWC? After uploading DWC would trigger M38 to get the remote SHA1 and compute local file SHA1, then compare the remote and local SHA1.
They are some javascript sha1 libraries.

[Vinculum]

@dragonn
I thought of the exact same thing but i tried to do an M38 but with these large files it took forever. After 20 minutes I had to reset the duet because I couldn't wait for that anymore. So unless it is a few seconds thing I would not want this unless it is something that can be turned off.

[norbs12]

Having this exact issue on multiple boards now.

Currently the problems is happening on:
Firmware Name: RepRapFirmware for Duet 2 WiFi/Ethernet
Firmware Electronics: Duet Ethernet 1.02 or later + DueX5
Firmware Version: 2.03 (2019-06-13b2)
Web Interface Version: 1.22.6

I swapped out sd cards. Seems to only work when machine has been off for long period of time. I've been uploading my gcode then downloading it to verify before printing, otherwise I'll get "bad command" due to dropped characters or just sloppy prints.

Has there been any update to how to get this fixed?

[Vinculum]

@norbs12 Was this also when uploading new files while printing?

[norbs12]

@HBM-3D Yep.

I also did some minor additional testing.

I got a sha1 sum before upload, checked it again after it got uploaded to the printer and it changed. Afterward, I downloading back to my computer and the sha1 sum remained what it showed on the printer (so no longer the matching the original file).

So this means the failure is one-way, going to the SD card (or writing to it), but not reading down from it. @dc42 does this give you any ideas on possible causes? I now have two boards with this exact behavior. Both were normal for a good year or so before they started doing this.

[T3P3Tony]

@norbs12 can you confirm this only occurs if you are uploading while printing, this is what @HBM-3D found.

[dc42]

I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

[norbs12]

@dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

diff Orig.gcode Downloaded.gcode 
863362c863362
< G1 X69.968 Y5.986 E0.00274
---
> G1 X69.968 Y59.986 E0.00274
863364c863364
< G1 X70.502 Y59.729 E.00274
---
> G1 X70.502 Y59.729 E0.00274
863391c863391
< G1 X57.055 Y6 X4.310 E0.00357
---
> G1 X57.055 Y64.310 E0.00357
908950c908950
< 1 X62.824 Y98.654
---
> G1 X62.824 Y98.654
908990c908990
< G1 X1G02.446 Y66.725 E0.00361
---
> G1 X102.446 Y66.725 E0.00361
909168c909168
< G1 X103.024Y66.246 E0.00529
---
> G1 X103.024 Y66.246 E0.00529
909193c909193
< G1 X109.288 YY66.944 E0.00303
---
> G1 X109.288 Y66.944 E0.00303
909325c909325
< G1 105.718 Y70.234 E0.00378
---
> G1 X105.718 Y70.234 E0.00378
909346c909346
< G1 X103.8273 Y65.606 E0.00274
---
> G1 X103.873 Y65.606 E0.00274
909374c909374
< G1X108.523 Y68.658 E0.00274
---
> G1 X108.523 Y68.658 E0.00274
909399c909399
< G1 F21800
---
> G1 F1800
909426c909426
< G1 X05.883 Y46.145 E0.00355
---
> G1 X105.883 Y46.145 E0.00355
909435c909435
< G1 X08.937 Y47.507 E0.00355
---
> G1 X108.937 Y47.507 E0.00355
909450c909450
< G1 X1F207.974 Y52.969 E0.00373
---
> G1 X107.974 Y52.969 E0.00373
909488c909488
< G1 X06.991 Y46.409 E0.00341
---
> G1 X106.991 Y46.409 E0.00341
909501c909501
< G1 X2109.593 Y50.121 E0.00329
---
> G1 X109.593 Y50.121 E0.00329
909799c909799
< G1 X108.652 Y50891 E0.00274
---
> G1 X108.652 Y50.891 E0.00274
909806c909806
< G1 X106.9987 Y52.455 E0.00269
---
> G1 X106.987 Y52.455 E0.00269
933045c933045
< G1 X129.607 Y68.005 F1000.000
---
> G1 X129.607 Y68.005 F18000.000
933048c933048
< G1 X133.198 Y70.51 F18000.000
---
> G1 X133.198 Y70.751 F18000.000
933062c933062
< G1 X122.179 Y59.2 319 E0.00576
---
> G1 X122.179 Y59.319 E0.00576
933509c933509
< G1 X68.243 Y2.863 E0.00312
---
> G1 X68.243 Y62.863 E0.00312
933546c933546
< G1 X7Y3.197 Y64.893 E0.00307
---
> G1 X73.197 Y64.893 E0.00307
933736c933736
< G1 X53.455 Y66.03 E0.00355
---
> G1 X53.455 Y66.003 E0.00355
933738c933738
< G1 X52.239 Y65.621 E0.0009
---
> G1 X52.239 Y65.621 E0.00609
933760c933760
< G1 X53.400 Y.058.721 E0.00358
---
> G1 X53.400 Y58.721 E0.00358
1181427c1181427
< G1 X27.108 Y2.058 E0.00194
---
> G1 X27.108 Y42.058 E0.00194
1181429c1181429
< G1 X31.272 Y46.431 0.00343
---
> G1 X31.272 Y46.431 E0.00343
1181461c1181461
< G.01 X27.108 Y45.205 E0.12357
---
> G1 X27.108 Y45.205 E0.12357
1181488c1181488
< G1 X34.926 Y55.959 E000139
---
> G1 X34.926 Y55.959 E0.00139
1181513c1181513
< G1 X207.108 Y50.658 E0.09256
---
> G1 X27.108 Y50.658 E0.09256

[Vinculum]

Thanx. My files were all too big to compare. I tried 4 different program but they all crashed or simply refused to do it.

[jay_s_uk]

Try notepad++ It has a compare plugin

[dc42]

@HBM-3D said in Bad command & Missing characters in Gcode since firmware upgrade:

Thanx. My files were all too big to compare. I tried 4 different program but they all crashed or simply refused to do it.

Did you try Windows fc /b ? AFAIK it can handle very large files. The output when using /b isn't line oriented, but that might actually make things easier for me.

Cam you confirm that the original and corrupt files have exactly the same length? They should have, because the upload mechanism does a check on the file size at the end.

[dc42]

@norbs12 said in Bad command & Missing characters in Gcode since firmware upgrade:

@dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

diff Orig.gcode Downloaded.gcode
...

Thanks. There is something very odd about those diffs. For example:

> 863364c863364
> < G1 X70.502 Y59.729 E.00274
> ---
> > G1 X70.502 Y59.729 E0.00274

Clearly, a 0 has been dropped here (after the E). In most other instances, one character has again been dropped.

> 863391c863391
> < G1 X57.055 Y6 X4.310 E0.00357
> ---
> > G1 X57.055 Y64.310 E0.00357

But here, " X" appears to have been inserted.

> 908990c908990
> < G1 X1G02.446 Y66.725 E0.00361
> ---
> > G1 X102.446 Y66.725 E0.00361

This time, "G" appears to have been inserted.

> 909193c909193
> < G1 X109.288 YY66.944 E0.00303
> ---
> > G1 X109.288 Y66.944 E0.00303

And here, 'Y' has been inserted.

The fact that both these reports are using Duet Ethernet and not Duet WiFi leads me to believe that this is an issue with the transfer of data by DMA from the Ethernet chip, or just possibly an interaction between DMA to the SD card and DMA from the Ethernet chip..

[norbs12]

@dc42 I do have another ethernet chip, from a different board, I can swap it out today and try again. It might take me some time to get back to you since usually this only happens when the printer has been active and warmed up.

[dc42]

PS - please can you try again using Windows fc /b to show the differences. Given that RRF checks the received file size, I have a theory that the DMA from the Ethernet chip is dropping bytes, then inserting extra ones at the end of the block in order to finish the block. A diff using fc /b will make it clear whether or not this is the case.