Linux : Sync/Backup USB Drives

I have a lot of content on my home network but backing it up has always been a pain.

When I say content I mean everything in the computing world that is important to me like all of my images, some video files,  CAD files, schematics, reference documents, install files, machine backups, automation scripts, ham radio logs, lots of engineering notes, etc.

  • When my network was W2K3 and XP I used Robocopy to backup the W2K3 network shares to an external USB drive every two weeks. (Sometimes I was better about the two week rotation than other times.)

  • As I started to transition from Windows (servers and workstations) to Linux and my network became more of a hybrid I bought a 1TB external USB drive. I moved most of the Windows and Linux content to this drive as I decommissioned machines. 

    I had numbered several other external USB drives and would rotate which one I used every 2 weeks for the backup of the main USB drive. Once again I used Robocopy to do the heavy lifting. The initial mirror took 6-8 hours to run (a good overnight job) and then additional syncs generally ran in less than 1-2 hours.

  • Now that my network is about 95% Linux I need to rotate the backup USB drive(s) again as it has been about 30 days. I am going to use RSYNC to mirror the primary 1TB USB drive to the numbered USB drives every two weeks or so.
  • I live in the middle of rural Minnesota so using a service like Amazon S3 or one of the other Internet backup services is unrealistic given my bandwidth combined with the amount of content that I have today and produce on-going. (RAW photos from my digital camera + any video work that I am producing is too large for the humble amount of Internet connectivity that I have.)

    At some point I should just take the USB drive some place with ubber Internet connectivity and upload it to S3 (or whoever) and then try to let the nightly jobs keep it in sync... but that will take some research to figure out.


    My answer to this issue for the time being is that I will need to get a safety deposit box at the bank and then rotate several numbered USB drives between the house and safety deposit box at the bank for my off-site rotation.

  • Today I don't backup any of the PCs on our LAN. I probably should look at backing up my PC and the Amateur Radio PC and/or at least the contents of my home directories.  That might be a phase two thing as I refresh my backup process as part of the Linux migration.

Here is how the USB mirror process works:
rsync -vrlptg --delete "/media/SOURCEUSB/" "/media/TARGETUSB/"
replace SOURCEUSB and TARGETUSB with the name of the drive

This should provide a "mirror" between the two locations.

rsync flags:
# -v   verbose output
# -r   recurse into directories
# -l   copy symlinks as symlinks
# -p   preserve permissions
# -t   preserve times
# -g   preserve group
# --delete   delete extraneous files from dest dirs
# --delete-before   receiver deletes before transfer (default)
# --delete-during   receiver deletes during xfer, not before
# --delete-delay   find deletions during, delete after
# --delete-after   receiver deletes after transfer, not before

You can confirm that the drive/content sizes match with:
df -h

For other Linux newbies (like me) here is a link to a webpage that provides some simple background on the rsync backup process I am using.
http://www.basicallytech.com/blog/index.php?/archives/73-Using-a-USB-external-hard-disk-for-backups-with-Linux.html

Scott Hanselman has a great post on the Family Backup Strategy that I like a lot:
http://www.hanselman.com/blog/OnLosingDataAndAFamilyBackupStrategy.aspx