Unsolved Bad command & Missing characters in Gcode since firmware upgrade

[Vinculum]

I tried the new SD card and that seems for now to do the trick. Thank you for the help. If the issue happens again I will let you know.

[Vinculum]

Well sadly changing the SD card did not solve it. Same issues.

And results in huge blobs:

[dc42]

It could be that occasional SPI overrun errors are occurring between the Duet and the Ethernet module. We saw something similar very occasionally during initial development of the Duet Ethernet when we were using a higher SPI clock speed. So we reduced the SPI clock speed, and we never saw the problem again.

If you like I can do a build with the SPI clock speed reduced further, for you to try out. This will have the unfortunate side effect of reducing the upload speed a little.

[mwolter]

How are you uploading the gcode to the Duet? I have seen the gcode change if you are uploading directly from the slicer. Possibly try uploading directly from DWC and download it again to see if it has changed.

[Vinculum]

@mwolter I always upload through the DWC, never through a slicer.

@dc42 I am happy to try but this was never an issue before. The hardware did not change. And it is on 4 out of currently 6 printers (3 more under construction). If this is an very unlikely scenario as you describe than I must buy a lottery ticket if 4 out of my 6 have the issue.

[dc42]

@HBM-3D , have the files you upload been getting bigger?

[Vinculum]

@dc42 Yes I do run bigger prints with the new machines. Our gcodes are generally between 150 & 550Mb.
In this example the file was "just" 230mb which is for us a medium size run.

[Vinculum]

I now have it on another machine as well. So there is something common that drives this issue.
This was with a 190Mb file. If I hash (CRC32) them I get this result:
Original file: 5A759A1C
Downloaded file: 654B92EC

I did a reupload of the original file and did a download immediately after uploading it. All via the DWC.
The has now is has the same code as the original file: 5A759A1C

So whatever happens doesn't appear to be consistent or it happens during/after printing.

[dc42]

I guess possible solutions are to slow down the SPI clock to see if we can find a speed that is always reliable, or to send the CRC with the uploaded data and check it at the end of the upload.

Btw there is a gcode command to compute and return the SHA1 hash of a GCode file on the SD card. You may find it useful for checking the integrity of uploaded files.

Can you do a compare of original vs corrupted files? I think the ancient DOS/Windows command fc /b works on large files.

[Vinculum]

I figured something out this weekend and tested it. If I upload the files while the printer is idle then the files are OK. If I do so when the printer is running it drops the ball and alters the file.
This weekend I had another strange situation with this. Some line of code was wrong but ended up being a valid gcode. The printer went all the way to x, y & z-max and tried to resume printing from there. Naturally it was spaghetti and a busted hotend but nothing serious. Weird stuff and dangerous though...

So uploading while printing is a no go, at least for bigger files i guess.

[dc42]

@HBM-3D said in Bad command & Missing characters in Gcode since firmware upgrade:

I figured something out this weekend and tested it. If I upload the files while the printer is idle then the files are OK. If I do so when the printer is running it drops the ball and alters the file.

Thanks for the update, that gives me something to test.

[dragonn]

@dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

Btw there is a gcode command to compute and return the SHA1 hash of a GCode file on the SD card. You may find it useful for checking the integrity of uploaded files.

Maybe could this be implemented in DWC? After uploading DWC would trigger M38 to get the remote SHA1 and compute local file SHA1, then compare the remote and local SHA1.
They are some javascript sha1 libraries.

[Vinculum]

@dragonn
I thought of the exact same thing but i tried to do an M38 but with these large files it took forever. After 20 minutes I had to reset the duet because I couldn't wait for that anymore. So unless it is a few seconds thing I would not want this unless it is something that can be turned off.

[norbs12]

Having this exact issue on multiple boards now.

Currently the problems is happening on:
Firmware Name: RepRapFirmware for Duet 2 WiFi/Ethernet
Firmware Electronics: Duet Ethernet 1.02 or later + DueX5
Firmware Version: 2.03 (2019-06-13b2)
Web Interface Version: 1.22.6

I swapped out sd cards. Seems to only work when machine has been off for long period of time. I've been uploading my gcode then downloading it to verify before printing, otherwise I'll get "bad command" due to dropped characters or just sloppy prints.

Has there been any update to how to get this fixed?

[Vinculum]

@norbs12 Was this also when uploading new files while printing?

[norbs12]

@HBM-3D Yep.

I also did some minor additional testing.

I got a sha1 sum before upload, checked it again after it got uploaded to the printer and it changed. Afterward, I downloading back to my computer and the sha1 sum remained what it showed on the printer (so no longer the matching the original file).

So this means the failure is one-way, going to the SD card (or writing to it), but not reading down from it. @dc42 does this give you any ideas on possible causes? I now have two boards with this exact behavior. Both were normal for a good year or so before they started doing this.

[T3P3Tony]

@norbs12 can you confirm this only occurs if you are uploading while printing, this is what @HBM-3D found.

[dc42]

I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

[norbs12]

@dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.

diff Orig.gcode Downloaded.gcode 
863362c863362
< G1 X69.968 Y5.986 E0.00274
---
> G1 X69.968 Y59.986 E0.00274
863364c863364
< G1 X70.502 Y59.729 E.00274
---
> G1 X70.502 Y59.729 E0.00274
863391c863391
< G1 X57.055 Y6 X4.310 E0.00357
---
> G1 X57.055 Y64.310 E0.00357
908950c908950
< 1 X62.824 Y98.654
---
> G1 X62.824 Y98.654
908990c908990
< G1 X1G02.446 Y66.725 E0.00361
---
> G1 X102.446 Y66.725 E0.00361
909168c909168
< G1 X103.024Y66.246 E0.00529
---
> G1 X103.024 Y66.246 E0.00529
909193c909193
< G1 X109.288 YY66.944 E0.00303
---
> G1 X109.288 Y66.944 E0.00303
909325c909325
< G1 105.718 Y70.234 E0.00378
---
> G1 X105.718 Y70.234 E0.00378
909346c909346
< G1 X103.8273 Y65.606 E0.00274
---
> G1 X103.873 Y65.606 E0.00274
909374c909374
< G1X108.523 Y68.658 E0.00274
---
> G1 X108.523 Y68.658 E0.00274
909399c909399
< G1 F21800
---
> G1 F1800
909426c909426
< G1 X05.883 Y46.145 E0.00355
---
> G1 X105.883 Y46.145 E0.00355
909435c909435
< G1 X08.937 Y47.507 E0.00355
---
> G1 X108.937 Y47.507 E0.00355
909450c909450
< G1 X1F207.974 Y52.969 E0.00373
---
> G1 X107.974 Y52.969 E0.00373
909488c909488
< G1 X06.991 Y46.409 E0.00341
---
> G1 X106.991 Y46.409 E0.00341
909501c909501
< G1 X2109.593 Y50.121 E0.00329
---
> G1 X109.593 Y50.121 E0.00329
909799c909799
< G1 X108.652 Y50891 E0.00274
---
> G1 X108.652 Y50.891 E0.00274
909806c909806
< G1 X106.9987 Y52.455 E0.00269
---
> G1 X106.987 Y52.455 E0.00269
933045c933045
< G1 X129.607 Y68.005 F1000.000
---
> G1 X129.607 Y68.005 F18000.000
933048c933048
< G1 X133.198 Y70.51 F18000.000
---
> G1 X133.198 Y70.751 F18000.000
933062c933062
< G1 X122.179 Y59.2 319 E0.00576
---
> G1 X122.179 Y59.319 E0.00576
933509c933509
< G1 X68.243 Y2.863 E0.00312
---
> G1 X68.243 Y62.863 E0.00312
933546c933546
< G1 X7Y3.197 Y64.893 E0.00307
---
> G1 X73.197 Y64.893 E0.00307
933736c933736
< G1 X53.455 Y66.03 E0.00355
---
> G1 X53.455 Y66.003 E0.00355
933738c933738
< G1 X52.239 Y65.621 E0.0009
---
> G1 X52.239 Y65.621 E0.00609
933760c933760
< G1 X53.400 Y.058.721 E0.00358
---
> G1 X53.400 Y58.721 E0.00358
1181427c1181427
< G1 X27.108 Y2.058 E0.00194
---
> G1 X27.108 Y42.058 E0.00194
1181429c1181429
< G1 X31.272 Y46.431 0.00343
---
> G1 X31.272 Y46.431 E0.00343
1181461c1181461
< G.01 X27.108 Y45.205 E0.12357
---
> G1 X27.108 Y45.205 E0.12357
1181488c1181488
< G1 X34.926 Y55.959 E000139
---
> G1 X34.926 Y55.959 E0.00139
1181513c1181513
< G1 X207.108 Y50.658 E0.09256
---
> G1 X27.108 Y50.658 E0.09256

[Vinculum]

Thanx. My files were all too big to compare. I tried 4 different program but they all crashed or simply refused to do it.