Subscribe to
NSLog(); Header Image

rsync Out, SuperDuper In

For weeks now my nightly schedule rsync copy would fail mysteriously, quietly. The command was simply two lines:

/usr/sbin/diskutil repairPermissions /;
/usr/local/bin/rsync -axq --delete / /Volumes/Roger

In other words, I repaired permissions and then archived (which enables a few options), quietly, and deleted files/folders which no longer existed.

But rsync would launch three processes, and two would sit rather idly and the third would get hung up on a file for hours and hours and hours and hours and hours, using 100% (or close to it) CPU time.

I ran the line with "-v" and without "-q" and saw that, for example, the "store.db" file would halt rsync's progress. I'd let it go for hours and it simply wouldn't budge past store.db. Or some other random file really, REALLY early on in the process.

Not worth the hassle. I'd sudo kill -9 17236 or whatever the PIDs were and they'd eventually (after five minutes or more) bail out.

So now SuperDuper - yeah, a GUI app - is part of my nightly backup. I'm not entirely comfortable with it, but it seems to work okay, and nightly backups take only 20 minutes or so (the first took, predictably, eight or nine hours). I already owned a license, so that was all good.

I don't know why rsync stopped working though. Bummer.

3 Responses to "rsync Out, SuperDuper In"

  1. I finally figured out what's going on with my rsync: it's going crazy on dot files. In an directory, GoneDark.AdiumMessageStyle had 120,000 files in it, all named ..typeattributes.xxxxxx, where xxxxxx is a random 6-character string.

    I'm just erasing all of my synced directories and am starting over.

  2. Well, I finally figured out exactly what was going on with rsync for me. There is a bug in it (including the latest source you can download) that doesn't properly handle dot files on OS X.

    It's due to the way rsync works. As it is copying file foo.txt, for example, it is named as .foo.txt.xxxxxx, where xxxxx is a 6-character string.

    In my case, I was excluding .DS_Store files, but there were other dot files (e.g., .localized) in the directories I was syncing. So, rsync would create ..localized.xxxxxx (yes, dot dot localized dot xxxxxx). As it was doing this, it would fail and start over, to the point that I had 120,000 ..localized.xxxxxx files in one of my directories.

    The solution I found is to sync via ssh instead of a local filesystem. This works for me, because I was copying between computers, but that would not work for you, if indeed this is the problem you were having.

    You could use something like this to see if dot files might've been the source of the problem you were having:

    find SOURCE_DIR -name ".??*" -print

    That's how I found things like:
    ./DOS Games.localized/.background
    ./DOS Games.localized/.localized
    ./Movies/iMovie Events.localized/.localized
    ./Movies/iMovie Projects.localized/.localized

    Those are the files that were causing my rsync to fail.

  3. I also use SD and I'm quite fond of it. I use it on some servers for backups as well.

    Great software. Makes bootable backups as well.