rsync Out, SuperDuper In
Posted July 17th, 2010 @ 09:51pm by Erik J. Barzeski
For weeks now my nightly schedule rsync copy would fail mysteriously, quietly. The command was simply two lines:
/usr/sbin/diskutil repairPermissions /;
/usr/local/bin/rsync -axq --delete / /Volumes/Roger
In other words, I repaired permissions and then archived (which enables a few options), quietly, and deleted files/folders which no longer existed.
But rsync would launch three processes, and two would sit rather idly and the third would get hung up on a file for hours and hours and hours and hours and hours, using 100% (or close to it) CPU time.
I ran the line with "-v" and without "-q" and saw that, for example, the "store.db" file would halt rsync's progress. I'd let it go for hours and it simply wouldn't budge past store.db. Or some other random file really, REALLY early on in the process.
Not worth the hassle. I'd sudo kill -9 17236 or whatever the PIDs were and they'd eventually (after five minutes or more) bail out.
So now SuperDuper - yeah, a GUI app - is part of my nightly backup. I'm not entirely comfortable with it, but it seems to work okay, and nightly backups take only 20 minutes or so (the first took, predictably, eight or nine hours). I already owned a license, so that was all good.
I don't know why rsync stopped working though. Bummer.
Donate Life
Posted 20 Aug 2010 at 4:15pm #
I finally figured out what's going on with my rsync: it's going crazy on dot files. In an Adium.app directory, GoneDark.AdiumMessageStyle had 120,000 files in it, all named ..typeattributes.xxxxxx, where xxxxxx is a random 6-character string.
I'm just erasing all of my synced directories and am starting over.
Posted 23 Aug 2010 at 8:42am #
Well, I finally figured out exactly what was going on with rsync for me. There is a bug in it (including the latest source you can download) that doesn't properly handle dot files on OS X.
It's due to the way rsync works. As it is copying file foo.txt, for example, it is named as .foo.txt.xxxxxx, where xxxxx is a 6-character string.
In my case, I was excluding .DS_Store files, but there were other dot files (e.g., .localized) in the directories I was syncing. So, rsync would create ..localized.xxxxxx (yes, dot dot localized dot xxxxxx). As it was doing this, it would fail and start over, to the point that I had 120,000 ..localized.xxxxxx files in one of my directories.
The solution I found is to sync via ssh instead of a local filesystem. This works for me, because I was copying between computers, but that would not work for you, if indeed this is the problem you were having.
You could use something like this to see if dot files might've been the source of the problem you were having:
find SOURCE_DIR -name ".??*" -print
That's how I found things like:
./Desktop/.localized
./Documents/.localized
./DOS Games.localized/.background
./DOS Games.localized/.localized
./Movies/iMovie Events.localized/.localized
./Movies/iMovie Projects.localized/.localized
Those are the files that were causing my rsync to fail.
Posted 01 Sep 2010 at 6:10am #
I also use SD and I'm quite fond of it. I use it on some servers for backups as well.
Great software. Makes bootable backups as well.
Heiko