Unsolved Bad command & Missing characters in Gcode since firmware upgrade

[Vinculum]

Just a little history first. We've build 4 large scale CoreXY machines (buildvolume: 900x1200x1500mm) running on the Duet Ethernet & Duex 5 with 4-point bed leveling. They run solidly @ 150mm/s actual printing speed but transport speeds up to 500 mm/s.
They run 24/7 and have actually very few issues and do runs lasting from 24h to almost a week for a single print.

However, recently I upgraded them to2.04RC1 (2019-07-14b1) and since then i noticed these kinds of error messages (copied from console):
Error: Bad command: G X455.496 Y726.708 E0.0500
Error: Bad command: EG1 X530.431 Y651.772 E0.0471
Error: Bad command: 11 X326.547 Y526.795 E0.0545
Error: Bad command: .703 Y237.446 E0.0093
Error: Bad command: 1G1 X110.272 Y541.859 E0.2472
Error: Bad command: E0.4945

Just a few of them. Now normally i would expect that to not be an issue. It would just reject that line and move on to the next. But what I now see is large blobs of extruded plastic on my model. These are so large that sometimes the printheads slams into them at high speed and breaks at the heatbreak. Mayhem follows...

The blob in this picture is about 4cm big...

I've checked the gcode generated from Simplify. I do find the lines that are mentioned but they are correct. For instance this error.
This is what is in the console:

When I open that line in the original gcode file i see:

While I was writing this I wanted to do a double check. So I downloaded the gcode file from the printer back to my computer and checked that same line. This is what I saw:

So it seems that something goes wrong during the transfer of files. Can anyone shed some light on the matter?

Update: I have it different machines and not only on the 2.04 firmware. 1 machine is running 2.02(RTOS) (2018-12-24b1) and has the same issue. I dit a checksum compare between the original file and the downloaded file and they do not match. I wanted to do a 1:1 comparison but the files are too big (300+Mb) and my software crashes. Tried that on Notepad++, Examdiff and some other tools.

[T3P3Tony]

@hbm-3d if the checksums don't match then the upload is not working right. Can you try a new SD card, maybe the card you have is failing after a long usage? Also i assume you are uploading over DWC (as opposed to another method , say a slicer plugin?)

[Vinculum]

I tried the new SD card and that seems for now to do the trick. Thank you for the help. If the issue happens again I will let you know.

[Vinculum]

Well sadly changing the SD card did not solve it. Same issues.

And results in huge blobs:

[dc42]

It could be that occasional SPI overrun errors are occurring between the Duet and the Ethernet module. We saw something similar very occasionally during initial development of the Duet Ethernet when we were using a higher SPI clock speed. So we reduced the SPI clock speed, and we never saw the problem again.

If you like I can do a build with the SPI clock speed reduced further, for you to try out. This will have the unfortunate side effect of reducing the upload speed a little.

[mwolter]

How are you uploading the gcode to the Duet? I have seen the gcode change if you are uploading directly from the slicer. Possibly try uploading directly from DWC and download it again to see if it has changed.

[Vinculum]

@mwolter I always upload through the DWC, never through a slicer.

@dc42 I am happy to try but this was never an issue before. The hardware did not change. And it is on 4 out of currently 6 printers (3 more under construction). If this is an very unlikely scenario as you describe than I must buy a lottery ticket if 4 out of my 6 have the issue.

[dc42]

@HBM-3D , have the files you upload been getting bigger?

[Vinculum]

@dc42 Yes I do run bigger prints with the new machines. Our gcodes are generally between 150 & 550Mb.
In this example the file was "just" 230mb which is for us a medium size run.

[Vinculum]

I now have it on another machine as well. So there is something common that drives this issue.
This was with a 190Mb file. If I hash (CRC32) them I get this result:
Original file: 5A759A1C
Downloaded file: 654B92EC

I did a reupload of the original file and did a download immediately after uploading it. All via the DWC.
The has now is has the same code as the original file: 5A759A1C

So whatever happens doesn't appear to be consistent or it happens during/after printing.

[dc42]

I guess possible solutions are to slow down the SPI clock to see if we can find a speed that is always reliable, or to send the CRC with the uploaded data and check it at the end of the upload.

Btw there is a gcode command to compute and return the SHA1 hash of a GCode file on the SD card. You may find it useful for checking the integrity of uploaded files.

Can you do a compare of original vs corrupted files? I think the ancient DOS/Windows command fc /b works on large files.

[Vinculum]

I figured something out this weekend and tested it. If I upload the files while the printer is idle then the files are OK. If I do so when the printer is running it drops the ball and alters the file.
This weekend I had another strange situation with this. Some line of code was wrong but ended up being a valid gcode. The printer went all the way to x, y & z-max and tried to resume printing from there. Naturally it was spaghetti and a busted hotend but nothing serious. Weird stuff and dangerous though...

So uploading while printing is a no go, at least for bigger files i guess.

[dc42]

@HBM-3D said in Bad command & Missing characters in Gcode since firmware upgrade:

I figured something out this weekend and tested it. If I upload the files while the printer is idle then the files are OK. If I do so when the printer is running it drops the ball and alters the file.

Thanks for the update, that gives me something to test.

[dragonn]

@dc42 said in Bad command & Missing characters in Gcode since firmware upgrade:

Btw there is a gcode command to compute and return the SHA1 hash of a GCode file on the SD card. You may find it useful for checking the integrity of uploaded files.

Maybe could this be implemented in DWC? After uploading DWC would trigger M38 to get the remote SHA1 and compute local file SHA1, then compare the remote and local SHA1.
They are some javascript sha1 libraries.

[Vinculum]

@dragonn
I thought of the exact same thing but i tried to do an M38 but with these large files it took forever. After 20 minutes I had to reset the duet because I couldn't wait for that anymore. So unless it is a few seconds thing I would not want this unless it is something that can be turned off.

[norbs12]

Having this exact issue on multiple boards now.

Currently the problems is happening on:
Firmware Name: RepRapFirmware for Duet 2 WiFi/Ethernet
Firmware Electronics: Duet Ethernet 1.02 or later + DueX5
Firmware Version: 2.03 (2019-06-13b2)
Web Interface Version: 1.22.6

I swapped out sd cards. Seems to only work when machine has been off for long period of time. I've been uploading my gcode then downloading it to verify before printing, otherwise I'll get "bad command" due to dropped characters or just sloppy prints.

Has there been any update to how to get this fixed?

[Vinculum]

@norbs12 Was this also when uploading new files while printing?

[norbs12]

@HBM-3D Yep.

I also did some minor additional testing.

I got a sha1 sum before upload, checked it again after it got uploaded to the printer and it changed. Afterward, I downloading back to my computer and the sha1 sum remained what it showed on the printer (so no longer the matching the original file).

So this means the failure is one-way, going to the SD card (or writing to it), but not reading down from it. @dc42 does this give you any ideas on possible causes? I now have two boards with this exact behavior. Both were normal for a good year or so before they started doing this.

[T3P3Tony]

@norbs12 can you confirm this only occurs if you are uploading while printing, this is what @HBM-3D found.

[dc42]

I am still hoping that one of you can send me a comparison of the original vs. corrupted files, using Windows fc /b or some similar command. See my reply of 20 September.