duetbackup - CLI tool to backup your Duet SD Card [deprecated]

mudcruzr

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

Release 1.0.2

I just released version v1.0.2.

This release should fix the time zone issue @mudcruzr described.

Also starting from this release the (callable) filename for all target platforms now is duetbackup - for this the files are package as either tar.gz or zip depending on target

Works for me, many thanks! (Windows 10)

I have a suggestion (only semi-seriously), if it's not too cheeky: an exclude switch, e.g.

./duetbackup -dirToBackup "0:/" -excludeDir "0:/gcodes" -domain "192.168.0.53" -outDir "./bkup"

(I didn't realise how many files I had accumulated in my gcodes directory, lol)

wilriker

@mudcruzr said in duetbackup - CLI tool to backup your Duet SD Card:

I have a suggestion (only semi-seriously), if it's not too cheeky: an exclude switch, e.g.

./duetbackup -dirToBackup "0:/" -excludeDir "0:/gcodes" -domain "192.168.0.53" -outDir "./bkup"

(I didn't realise how many files I had accumulated in my gcodes directory, lol)

I already have thought of that. I will see if I incorporate that - but on the other hand on successive calls it will only download new or changed files. GCodes files change rather seldomly so most of them won't get downloaded again.

wilriker

Release 1.1.0

I just released v1.1.0 and I think this will make @mudcruzr quite happy - as well as I hope that other users will also be happy with the new feature. Release notes below.

Exclude Feature

A new flag -exclude has been introduced to exclude remote paths starting with the given value. This flag can be used multiple times.

Simple Example

./duetbackup -domain duet.localdomain -outDir "sys" -exclude "0:/sys/c"

will skip all files and folders in /sys that start with "c" like config.g or cancel.g.

Multi Example

./duetbackup -domain duet.localdomain -outDir "sys" -exclude "0:/sys/c" -exclude "0:/sys/home

will skip all files and folders in /sys that start with "c" like config.g or cancel.g as well as all files starting with "home" like homex.g, homey.g, homez.g and homeall.g.

Example Full Backup excluding GCode Files

./duetbackup -domain duet.localdomain -outDir "backup" -dirToBackup "0:/" -exclude "0:/gcodes"

Other Changes

Some internal code simplifications
Logging has been reformatted and reduced - the former amount of logging can be restored with the new flag -verbose

wilriker

Release 1.1.1

I had to release v.1.1.1 to fix a bug where the local directory has to exist unless the corresponding remote directory has a sub-directory.

Now all missing local directories are being created independently of the contents of the remote directory.

Also the order of file handling has been adjusted to first handle all files in the remote directory before traversing into sub-directories. This way the order of log statements makes much more sense.

mudcruzr

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

Release 1.1.0

I just released v1.1.0 and I think this will make @mudcruzr quite happy - as well as I hope that other users will also be happy with the new feature. Release notes below.

LOL

Happy? Happy? I'm absolutely ecstatic mate! Seriously, many thanks for a very useful tool.

Dougal1957

Finally got it working (Due to me not knowing what I am doing with Mac command line I might add) but does it normally take for ever to complete i normally achieve 600kib/s or more on uploads?

Phaedrux

@dougal1957 said in duetbackup - CLI tool to backup your Duet SD Card:

Finally got it working (Due to me not knowing what I am doing with Mac command line I might add) but does it normally take for ever to complete i normally achieve 600kib/s or more on uploads?

I went and cleaned out my Gcode directory first so that it didn't have to download all of them again.

mudcruzr

@mudcruzr said in duetbackup - CLI tool to backup your Duet SD Card:

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

Release 1.1.0

I just released v1.1.0 and I think this will make @mudcruzr quite happy - as well as I hope that other users will also be happy with the new feature. Release notes below.

LOL

Happy? Happy? I'm absolutely ecstatic mate! Seriously, many thanks for a very useful tool.

Ah! not so happy now even with the exclude flag it's still including all the gcode files.

Command line:

./duetbackup -domain "192.168.0.52" -outDir "Feynman" -dirToBackup "0:/" -exclude "0:/gcodes"

(Sorry!)

gtj0

@mudcruzr said in duetbackup - CLI tool to backup your Duet SD Card:

@mudcruzr said in duetbackup - CLI tool to backup your Duet SD Card:

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

Release 1.1.0

I just released v1.1.0 and I think this will make @mudcruzr quite happy - as well as I hope that other users will also be happy with the new feature. Release notes below.

LOL

Happy? Happy? I'm absolutely ecstatic mate! Seriously, many thanks for a very useful tool.

Ah! not so happy now even with the exclude flag it's still including all the gcode files.

Command line:

./duetbackup -domain "192.168.0.52" -outDir "Feynman" -dirToBackup "0:/" -exclude "0:/gcodes"

(Sorry!)

confirmed....

$ ./duetbackup -domain duet0 -outDir "backup" -dirToBackup "0:/" -exclude "0:/gcodes"
2019/06/04 10:42:56 Fetching filelist for 0:/
2019/06/04 10:42:57 Downloading new/changed files from 0:/ to /usr/src/3d/duet0/backup
2019/06/04 10:42:57 Fetching filelist for 0://Probe
2019/06/04 10:42:57 Downloading new/changed files from 0://Probe to /usr/src/3d/duet0/backup/Probe
2019/06/04 10:42:57 Fetching filelist for 0://System Volume Information
2019/06/04 10:42:57 Downloading new/changed files from 0://System Volume Information to /usr/src/3d/duet0/backup/System Volume Information
2019/06/04 10:42:57 Fetching filelist for 0://filaments
2019/06/04 10:42:57 Downloading new/changed files from 0://filaments to /usr/src/3d/duet0/backup/filaments
2019/06/04 10:42:57 Fetching filelist for 0://filaments/PLA
2019/06/04 10:42:57 Downloading new/changed files from 0://filaments/PLA to /usr/src/3d/duet0/backup/filaments/PLA
2019/06/04 10:42:57 Fetching filelist for 0://gcodes
2019/06/04 10:42:57 Downloading new/changed files from 0://gcodes to /usr/src/3d/duet0/backup/gcodes

Note the double //. If you specify -exclude "0://gcodes" it works.

wilriker

@gtj0 @mudcruzr I can see where the bug is. Will fix it tomorrow.

P.S.: Another workaround in addition to what @gtj0 already said would be to specify -dirToBackup 0: without the trailing slash.

wilriker

Release 1.1.2

Here we go with v1.1.2.

This release fixes the issue where trailing (and/or possibly multiple successive) slashes can break the exclude feature.

Also in this release:

Excluding is now more efficient by checking if a directory can be excluded as a whole instead of only excluding everything inside it
Approximate download speeds have been added to the output
Remove local directories recursively if they no longer exist remote and -removeLocal has been given

mloidl

@wilriker
Hi,
Many thanks for the update, i've not tried it yet, but as i saw in the code you're now using os.RemoveAll instead of os.Remove.
This would mean that non-empty directories will be deleted too. Including e.g. .git or similar folders.

I already thought about an additional option to black list some folders/files from cleanup, but had no idea where to put that.
In my local branch (not pushed) i've created a hard-coded list but this might not be very useful for others.
Another option would be to store the filenames in a kind of config file or (.blacklist, whatever) which can be changed by users without modifying the code.

Do you have some suggestions? If you wan't i can then try to implement it.

wilriker

@mloidl You are right I did not think of VCS. I'll think about a better solution.

wilriker

Release 1.2.0

And another one named v1.2.0 has been released to address the issue brought up by @mloidl.

To prevent removing directories not managed by duetbackup the tool will now create files named .duetbackup in each directory it creates.

If for example the user will then initialize a git repository inside the local backup directory the .git directory will be skipped when removing files that do not exist remote.

wilriker

7 Releases in 5 days - you guys are awesome in spotting issues! I really appreciate the feedback from all of you.

mloidl

That was really fast.
Will test it when i'm at home on the machine.

wilriker

@mloidl Thanks. I borrowed the idea from tools like Dropbox or Syncthing that also use marker files (for different purposes but that doesn't matter) so it was only implementation and local testing effort.

Fun fact: while implementing everything worked immediately protecting unmanaged directories but strangely there was not a single marker file inside the directories after the tool had finished. It took me quite a while to figure out that I forgot to also exclude the marker file itself from being removed. It would even have worked this way since I create the file everytime I visit a managed directory but that would have been too much voodoo-magic-implementation to keep.

dc42

@wilriker said in duetbackup - CLI tool to backup your Duet SD Card:

@dc42 Still if the file is there Content-Encoding: gzip is wrong.

This should be fixed in 2.03RC5 which I have just released.

Phaedrux

Just tested 1.2.0 on Mac using a command file to automate the process and it works great.

For those interested in making a shortcut file to run instead of using the command line each time. It will create a time stamped directory to download the backup to. You can create separate shortcut files to run different backups, different printers, etc.

This is Mac specific, but similar could be achieved on other OS as well.

Save the following into a text file and save it with .command extension. You will need to modify the following script to suit your setup

#!/bin/bash
_timestamp=$(date +"%Y-%m-%d-%H%M")
_filename="Backup-$_timestamp"
cd "/PATH/TO/DUETBACKUP/duetbackup-darwin_amd64"
./duetbackup -domain duet.local -outDir "$_filename" -dirToBackup "0:/" -exclude "0:/gcodes"

You may need to make the resulting .command file executable for it to work from the Finder. In Terminal, navigate to the directory the .command file is in, and use chmod +x FILENAME.command to make it executable so you can double click on it from the Finder to run it. Change FILENAME to be whatever the filename you saved it as.

Thanks @wilriker for the awesome backup tool!

mloidl

Also tested v1.2.0 on linux. Works like a charm.

Thanks @wilriker for this awesome tool.