Completely replace SD Card? (duet ethernet)
-
Wiki page https://duet3d.com/wiki/SD_card_folder_structure includes a link to the files that we put on the SD card during manufacturing
-
Is FAT16 still the recommended file system? I just noticed the card that came with my Duet is formatted as FAT32.
I've been running some benchmarks on some SD cards I have laying around as well as a new 32gb Class 10 A1 Sandisk Ultra card. The newer card has random uncached writes of 14MB/s with 256k blocks. 10 times the speed of the stock card. The A1 rating denotes a card optimized for smartphone applications. Meaning optimized for random writes. See here for more explanation.
I'm going to copy the stock duet image over to that card and test the upload speed. On the stock card it was reporting peaks of about 300k but based on the length of time and the size of the file 100k average is probably more accurate.
Update: Well it's slightly faster. I'm going to do some more tests though.
Update 2: Well I take that back. The stock card seems to be doing better than I thought it was initially.
I uploaded a 10Mb gcode file a number of times to each card. The Stock card took 15 seconds most of the time. The larger A1 card took 25 seconds most of the time. Not a huge difference though. If 4gb isn't large enough for you, I guess you've got options that perform alright at least. I tried with an 8gb Class 10 card from Kingston that came with a Raspberry Pi Zero W and it performed abysmally. Benchmarks showed a 0.2MB random write speed, and uploading the same file took over a minute. So it would appear the A1 cards do help.
The cards I used were formatted with the SD Formatter tool, which is supposed to ensure properly formatted cards, whereas the OS tools may not. https://www.sdcard.org/downloads/formatter_4/ I tried a few other formatting attempts but couldn't improve performance. I'm not sure how the stock card is formatted, and I didn't mess with it.
Are the stock cards formatted any particular way, or is the performance simply from their smaller size?
-
For best performance, the card should be formatted with the cluster size as large as possible. On smaller capacity cards this is achieved using FAT16 format, but that is not an option on larger capacity cards.
-
Thanks David, I think you missed my last edit. I don't suppose you could share some more details on how exactly the cards are formatted to ensure best performance. When I tried to manually format for largest cluster size I got worse performance than when I used the SD Formatter.
-
Thanks David, I think you missed my last edit. I don't suppose you could share some more details on how exactly the cards are formatted to ensure best performance. When I tried to manually format for largest cluster size I got worse performance than when I used the SD Formatter.
We don't format the cards specially, we take them as they come.
I am very surprised that using a larger cluster size made the performance worse, and I suggest you re-test this in case a variation in WiFi performance affected the speed.
It makes sense to me that cards optimised for random access would perform better than cards optimised for digital cameras.
The next RC firmware will include the cluster size in the M39 response.
-
I re-ran the tests a little more methodically. Here are the results and how I got them.
I used the SD associations SD Formatter tool to format the cards for defaults. It uses 32k cluster size for cards 32gb and under.
To get 64k cluster sizes I used the OSX terminal to run sudo newfs_msdos -F 32 -c 128 -v DUETWIFI disk1
It is interesting to note that the stock SD card that comes with the Duet is actually formatted with 32k cluster sizes. I was actually unable to format it using that command, as it gave me an error. 61416 clusters too few clusters for FAT32, need 65525. Perhaps 4gb is too small to use 64k clusters?
Cluster sizes were verified on a windows system using chkdsk.To test the transfer speeds I used an 18Mb gcode file uploaded from my laptop over wifi to the duet. The router is about 20 feet away. I probably could have gone wired for the laptop to take that out of the equation, but I wouldn't be doing that on a regular basis anyway, so this test was more realistic. 18Mb is a very large gcode file for me. I originally was going to test with a 200Mb file, but it was taking too long. I uploaded the file 3 times in a row and timed it using my smartphone timer and averaging the times.
Here are the results.
[c]
Transcend 4gb Class 4 FAT32 with 32kb clusters 1:00 (stock)
Transcend 4gb Class 4 FAT32 with 64kb clusters Could not format
Sandisk Ultra 32gb Class 10 A1 FAT32 with 32kb clusters 1:10
Sandisk Ultra 32gb Class 10 A1 FAT32 with 64kb clusters 1:00
Kingston 8gb Class 10 FAT32 with 32kb clusters 0:55
Kingston 8gb Class 10 FAT32 with 64kb clusters 0:56
[/c]As you can see, not very exciting, and probably more limited by wifi than the SD card. Very little difference between cluster sizes, except in the case of the larger 32Gb card. Most surprisingly for me, the 8gb class 10 card performed terribly in synthetic benchmarks, but once formatted properly (I think it was a bit messed up at first) it turned in the best real world results.
I should also note that cards over 32Gb will default to being formatted in exFAT, and the SD Formatter tool will format it as such. You'd have to use the command line or another tool to format those any other way.
Also, if you are going to use a very large card, look for one with the A1 classification for improved random write performance. That may become more important over time as the large card gets filled up with more files and things get fragmented.
-
For 4Gb and under cards, use FAT16 format. FAT16 provides an additional performance boost over FAT32.
-
Good to know. I'll try retesting with the 4GB class 4 card that came with the Duet with FAT16 and 64k clusters. Is that how they are supposed to come formatted? Mine was FAT32 with 32k clusters.
-
We have been using FAT32 for the 4GB cards. I was not actually aware of the FAT16 performance increase that David mentioned.
Something to change on the next batch.
-
One final data point then.
[c]
Transcend 4gb Class 4 FAT16 with 64kb clusters 00:50
[/c]10 seconds off the FAT32 format.
So there you go. For best speeds the stock card formatted at FAT16 with 64k clusters is the way to go. This might require you to reformat the card yourself though, as out of the box mine was FAT32 and 32k clusters.
If you need a larger card, FAT32 with 64kb clusters and A1 classification.
I may try again with an ideal wireless situation with the router directly line of site to the printer and my laptop on a wired connection, just to isolate the SD card performance. In actual practice there isn't much difference.
-
Sorry to be pedantic, would using fAT16 provide a significant upload speed boost?
If I am driving prints remotely (http xfer gcode to Duet, start print remotely, from a script on a server that it will automatically remove completed print files too); Would it provide a useful/noteworthy speed boost to the upload If I reformatted my card to FAT16.
(I'm suggesting not using the whole drive.. eg just force a FAT16 filesystem onto the first four gigs and ignore the rest.. I've done something similar to this before.)
Edit: Looking at the above.. I guess not.
-
One final data point then.
[c]
Transcend 4gb Class 4 FAT16 with 64kb clusters 00:50
[/c]10 seconds off the FAT32 format.
So there you go. For best speeds the stock card formatted at FAT16 with 64k clusters is the way to go. This might require you to reformat the card yourself though, as out of the box mine was FAT32 and 32k clusters.
If you need a larger card, FAT32 with 64kb clusters and A1 classification.
I may try again with an ideal wireless situation with the router directly line of site to the printer and my laptop on a wired connection, just to isolate the SD card performance. In actual practice there isn't much difference.
Thanks for running the tests!
-
Sorry to be pedantic, would using fAT16 provide a significant upload speed boost?
If I am driving prints remotely (http xfer gcode to Duet, start print remotely, from a script on a server that it will automatically remove completed print files too); Would it provide a useful/noteworthy speed boost to the upload If I reformatted my card to FAT16.
(I'm suggesting not using the whole drive.. eg just force a FAT16 filesystem onto the first four gigs and ignore the rest.. I've done something similar to this before.)
Edit: Looking at the above.. I guess not.
Short stroking a drive like that used to be popular with spinning platters back when they topped out at 120ish gigs, but for an SD card it might actually degrade performance further due to the wear leveling algorithm having less space to work with. If there is some performance to be gained from FAT16 it would probably be lost from that.
As it is, I think unless you have a really poor SD card, you're probably more limited by wireless transfer than write speed.
-
The wear levelling algorithm works at a level underneath the FAT format, so it shouldn't be affected.
-
I have tried to reproduce this issue, but so far without success. I copied a 384Mbyte file over wifi to the Duet. The speed started at over 1Mbyte/sec but had dropped to 657kbytes/sec by the time it finished. DWC shows file information for it. Then I tacked half a megabyte of useless comments on the end so that DWC would have to search a long way back from the end of the file without finding what it wanted, and uploaded that. Again, no problems but file info is not displayed because the Duet does not search back as far as half a megabyte from the end. When I refresh the file list, it pauses on that file for 4 seconds, but carries on after that.
So I conclude that you can use files up to at least 385Mbytes long if your choice of SD card and format is suitable. This card is a Sandisk 16Gb card with a class 4 speed rating, formatted to FAT32 with 64Kb clusters. In DWC I have the status update interval set to 250ms which I think is the default, and the maximum number of AJAX retries set to 1 which is wrong - I recommend at least 3 when using WiFi.
-
Thanks for the feedback, and the testing! super helpful
I'm certainly not going this route for a modest gain, I guess I was optimistically hoping for a dramatic (2x or whatever) speedup
Once this weekend's re-organising and wiring is complete I should have my Pi acting as the AP for my print cupboard, I just, errm, 'acquired' a 10m CAT5 cable for this purpose from the nest at work. This will put the AP ~10cm from the Duet in line-of-sight; so I'm hoping to push the speed if I can, and see if this solves the disconnects that plague me from webcontrol. (yes, I know that may be too close; I'll play with separating them if I see weirdness before complaining here)
-
If you are getting disconnects, check that you have the retry count set to at least 3 in DWC.
-
20 enough?I've seen all the advice, but since I have a signal strength that flaps between -50 and -70 I'll wait till that is stable before passing judgement.. at the moment I'm happy to continue blaming my end of the network.
-
With the signal strength changing that much, I suspect you are getting interference from another device in the same band. Have you tried changing the WiFi channel?
-
Just found something odd. I have near identical printers setup, one which has a Duet Wifi, and one which is a Duet Ethernet. Using DWC to upload a 30MB file shows the performance is severely different between the two, but not in a way that I would expect. The Wifi connected Duet can be uploaded to significantly quicker. The Duet Wifi seems to top out at around 580k/sec, whereas the Ethernet only gets to about 340k/s. Could this be a feature of the firmware on the Ethernet version? Does anyone else have similar results?
The SD card in both is a brand new 16GB SanDisk Ultra SDHC Class 10.
