I bought Apple’s new operating system, 10.5 Leopard, on Saturday. One of the main selling points for me was its new automated backup software called Time Machine. I have gigs and gigs of personal data that screams at me every night to take proper care of it. Movies, Photos, Music. I had one near catastrophe a couple months ago and took measures then to avoid a total loss.
At the time I had one external 250 gig drive attached to my Mac Mini. This stored all of my data that I catagorize as ‘media’ files. Music and Movies. I don’t take enough photos, yet, to feel like I need to lump those in as well so they remain on my internal drive in the default location.
One night a couple months ago my external drive just flat out locked up. I rebooted my computer because everything else I tried failed to free up the drive. Upon reboot, the drive was unreadable. Fudge! Fortunately I had the large majority of my media on my previous external hard drive, which I replaced because it was too small for my media (125 gigs), and on my iPod.
I went out that weekend and bought two new 1.5 TB hard drives for $129 each. Buy them from Amazon here for $129 each! This gives me a larger external media drive and a matching external drive for backup. I also picked up a small USB 2.0 hub for about $20 to connect all of my USB devices to.
I installed both external drives on my Mac Mini. I restored most of my lost data to the first external drive. I then manually copied everything from this drive and my user folder to the second drive. Now I had a proper backup! The only problem with this plan is that it is not automated. I have to copy the data myself. Apple’s new operating system, 10.5, Leopard was coming out in about a month so I continued to do my backups by hand every week until Leopard was released.
This past weekend I installed Leopard on my Mac Mini. After completion, the first thing I wanted to configure was Time Machine to automate my data backup. I selected the backup drive and waited for it to start by itself.
Time Machine started processing a couple minutes later. Eventually it gave an error that there wasn’t enough room on the backup drive to complete the backup. I looked at the contents of the backup drive and sure enough, I forgot to erase all of the data that I manually copied. I reformatted my designated backup drive, using Disk Utility and formatted the external drive with Mac OS Extended (Journaled), and waited for Time Machine to start up again. In retrospect, this was probably stupid because now I’m running a new OS and just killed my backup. This isn’t the smartest way to go about this but I was willing to take the chance. I did copy the data from my new 500 gig drive back to my old 250 gig drive so in reality it wasn’t all that stupid but it wasn’t my most recent backup.
A couple minutes later, Time Machine started up again. Another failure. The error message was completely useless and gave no hints as to why it failed. I found two system logs that gave a little more information but they were too vague to really be of help. I opened up the Console.app application from the Utilities folder and found Time Machine (backupd) references in these two logs:
I resorted to my old Mac skills from the old OS 9 days and did some “Extension Conflict”-like troubleshooting. Within the Time Machine preferences, in System Preferences, I told Time Machine to ignore everything on my internal drive, selecting the root of the internal drive. (/Macintosh HD) I told it to ignore everything on my external drive 1 except for one folder. I waited for Time Machine to start.
Success! It successfully backed up the one folder on my external media drive. So I removed all of the exclusions that referred to my external drive. Again it worked! So something is causing this on my internal Macintosh HD drive.
I removed the exclusion of my internal drive (Macintosh HD) and then added exclusions for all of the next level folders except for my personal User folder.
Again it worked! So something on my internal hard drive is causing Time Machine to fail that does not reside in my User folder.
While waiting for various backup test to finish or fail, I researched Time Machine on discussions.apple.com. It is recommended that during Time Machine’s initial backup that you do not let your computer go to sleep or let the screensaver come on. The initial backup is probably the most intensive because Time Machine needs to construct the required databases and other initialization files. Some people are also recommending that you exclude large files like those created by Entourage and Parallels.
/Users/[user folder]/Documents/Microsoft User Data/
The Entourage database file is an absolute beast! It encapsulates all data into one file which can realistically be over 2 gigabytes! If you make one small change to your email data within Entourage then this database changes modification dates and Time Machine will have to backup the entire database again. So your options are to use another mail application instead of Entourage or backup this database file hourly, which is Time Machine’s default setting. So if you do use Entourage for email, you’ll want to back this data up but I’m just mentioning it here as another possible cause for Time Machine failing. You can also create a smart folder with the Finder to show files of a certain size to help you find files that may be larger than your preferred tolerance for what you want to backup.
Time Machine tip:
To change Time Machine’s default backup interval you’ll need to edit /System/Library/LaunchDaemons/com.apple.backupd-auto.plist. Open it in your favorite text editor and look for this key:
Change the number 3600 to some other time interval in seconds. You’ll have to do this using ‘sudo’ or root. I won’t go into that here because that’s beyond the scope of this article.
I have a hunch that different files will cause success or failure for different users at different times but I haven’t proven this yet.
I will continue my testing with Time Machine exclusions allowing different folders within my internal hard drive to be backed up by Time Machine until I narrow it down to the folder(s) and file(s) that are causing it to fail. I will report back and update this post with my findings. I just thought that it would be better to give some people some direction to get themselves started with at least some sort of successful backup.
other Time Machine tips:
update: 10/31/07 1:30 PM PST
Further testing has revealed that something within my iPhoto Library package is causing fits with both the Finder AND Time Machine.
If I tell Time Machine to only backup my User folder but not backup my /Users/[shortusername]/Pictures directory, Time Machine succeeds.
If I tell Time Machine to back up my entire User folder including the Pictures directory, it fails to backup about 3 gigs of data.
Using the Finder, if I try to manually copy the ‘iPhoto Library’ package from my Pictures directory to an external drive, it hangs about about the same point and eventually fails with error -36.
ioErr (-36): I/O error (bummers)
If I open this package and manually copy each folder it works. (huh?)
I discovered that some files within my iPhoto Library are JPGs but showing up in the Terminal as files with ‘extended attributes’, note the @ after the read/write/execute permissions:
-rwxrwxr-x@ 1 usershortname usershortname 3869764 May 20 19:59 CIMG0886.JPG
xattr -l CIMG0886.JPG
0000 4A 50 45 47 00 00 00 00 00 00 00 00 00 00 00 00 JPEG
0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
I think this is the cause of my Time Machine troubles with this particular directory. I am using the latest version of iPhoto, version 7.1.
I don’t think the ‘iPhoto Library’ is the cause of everyone’s problems but probably whatever traits my troubled files have may be the same traits others have with other files.
I tried removing the one picture folder from within ‘iPhoto Library’ that contained 4 jpgs and both Time Machine and the Finder both still failed to copy the data. iPhoto does something internally to the way it stores data to cause this.
An interesting note is that after successfully backing up everything except this Pictures folder with Time Machine, removing the exclusion for this folder and trying again produces odd results. Time Machine shows correctly that about 3 gigs of data needs to be backed up. Time Machine hangs a bit early on in the process and quickly ends with no errors/failures. It also doesn’t jump past the hang point nor does it change the “Latest Backup” date. It just flat out ends like everything was fine when obviously it isn’t.
Notice that in my original post, I reported that I had success backing up my User folder the first time I tried without backing up anything else on my internal drive. Either I made a mistake in remembering what my test was or Time Machine is spotty and succeeds or fails with different backups. I wouldn’t be surprised if it were both. :)
update: 11/16/07 9:30 AM PST
I installed the 10.5.1 OS update last night. I removed the Time Machine exclusion rule for a couple of items that were failing before. “iPhoto Library” and /Library/Application Support/. I ran Time Machine and it worked successfully on all items. I wonder if this bug fix is what resolved my problem even though my external drive is less than 512 GB:
“* Addresses formatting issues with certain drives used with Time Machine (specifically, single-partition MBR drives greater than 512 GB in size as well as NTFS drives of any size and partition scheme).”