At WWDC, TMO’s Dave Hamilton met with Mike Bombich, author of Carbon Copy Cloner for OS X. Mr. Bombich left Apple two years ago to pursue his passion for being a creator of something great. A short respite breathed new life into his work with his quintessential OS X copy and backup tool, CCC. This interview includes some intriguing, little known facts about that app.
___________________
Dave Hamilton: I’m here with Mike Bombich of Bombich Software, Inc. You have an interesting story. You have been developing essentially one app in your app developer role for many, many years, and that’s Carbon Copy Cloner. But you’ve had an interesting history with Apple that goes back further than that.
Mike Bombich: Yes, I’ve been working on CCC for ten years. I’ve certainly developed a lot of other applications in the interim. Way back in the day, I did things like Share My Desktop, Stream Catcher, DeLocalizer. These are just some of the popular ones.
And then there are others that I don’t even remember. It’s been at least a couple of dozen over the years, and most of them are just small utilities. Honestly, for most of the stuff that I’ve developed, it’s just been because I wanted it or my wife wanted it, and I just sat down and started writing it.
Carbon Copy Cloner was a little bit different because I was actually working at a university at the time that I developed it. And Mac OS X had just come out, and I wanted a way to foster adoption of Mac OS X. The way we did that in education, we couldn’t adopt it unless there was a way to mass deploy it.
When the Unix underpinnings of Mac OS X came out, there was some mythology about whether or not you could even copy Mac OS X. There was such a paradigm about how Mac OS 9 works and this mythology about how the UNIX underside of Mac OS X worked. People just didn’t believe you could do it. And I thought that’s insanity. Obviously Apple does it.
TMO: That’s a good point.
MB: I’m sure we can figure this out. So I sat down and I figured out what the method had to be to copy the right stuff to make the right tweaks and things like that. And then I just wrapped an application around that. That has certainly evolved over the years.
Lately, especially Mac OS X, it’s has gotten a lot more complicated. I think Apple is hiring an army of engineers with the specific purpose of making it more complicated. That’s certainly what it feels like. And if you look at Xcode [development system for OS X] lately, it’s the same kind of thing. There’s certainly a lot of nice things that are coming along, some conveniences, but there’s a lot more complexity. It’s my job to cover up as much of that as possible and make the task that we’re trying to achieve, you know, cloning one disk to another that happens to contain Mac OS X and make it work as seamlessly as possible.
TMO: And clearly Lion added a whole other layer of complexity to that by, in some ways hiding the recovery partition out there. Of course when you clone a drive, if you don’t clone that with it, you actually lose some functionality.
MB: Potentially. There’s a lot of misunderstanding about this. And the reason for the misunderstanding is because it is hidden. And I don’t just mean that you can’t see the partition, I mean that it’s not documented whatsoever. It’s not like a developer like me can go to apple.com/developer and hit the Recovery HD link and see exactly how the Recovery HD works, how big it is, what’s on it, how you bless it. It’s 100 percent sitting down and tearing it apart, figuring out how it works, and then making a guess about why it’s there, what it’s supposed to do, what Apple’s vision for it is, where it’s going to go.
Indeed, Mountain Lion came along and it was like taking a square and turning it into a cube. I just got a completely different perspective on it and realized that … well I don’t want to get into the weeds, but I thought it was kind of one Recovery HD per disk with Lion, but I think you actually need a Recovery HD attached to each installation of OS X. So if you have a volume with Mac OS X, you need a Recovery HD attached to each one. Now I say “need,” but you don’t actually need it. As far as I’m concerned, the only reason you need a Recovery HD is if you’re going to use the volume in production.
TMO: And also for Find My Mac. Find My Mac won’t function without it?
MB: If you’re going to use the volume in production.
TMO: Ah, okay. Fair enough.
MB: So if you’re just need a backup and you just need to restore from your bootable backup, you actually don’t need it. I always archive it so that we can recreate it. And that’s one of the reasons that I don’t automatically create it. I certainly could. But I think that there’s probably too many people [for whom] it would cause some hassle for them if I did automatically create it.
I just happened to be creating the Disk Center when I learned about Lion’s new functionality, and I thought “Hey, this is great. I’ll just pop a Recovery HD in there” and now you have an opportunity within Disk Center to go ahead and attach a Recovery HD to each of your Mac OS X installations, as you need it.
TMO: As needed. Okay. So you’re archiving that in Disk Center ahead of time.
MB: I actually archive it as part of the backup process. So right now, in 3.4, I’ll only archive the Recovery HD if you’re backing up the startup disk. In 3.5, I’m actually changing that functionality so that I look to see if there’s a Recovery HD associated with the source volume, whether it’s your startup disk or not, I’m going to go ahead and archive it. A lot of that is for the guys around here. The end user, most of them, you’ve got one volume and it’s your startup disk and that’s the typical case, so it’s always getting backed up.
The very first thing I do with a standard backup is: I just create an archive of that Recovery HD and in the future we can go and create another one.
TMO: As a user of Carbon Copy Cloner, I come at it from a completely different standpoint than you do, obviously, as the developer. You started it as a tool to aid in mass deployment. My guess is that most of your users are using it as a tool to backup in some capacity or another.
MB: And it’s certainly changed a lot over the years. I actually kind of held off development for a while and went even more towards the mass deployment perspective. For a while, there was CCC 2.3 and not much development there. That was the time that I was doing lots of development on other products like Net Restore and Boot Picker and a few other things that catered specifically to the education and the mass deployment market.
I’m actually glad I took that kind of sabbatical. Because it gave me an opportunity to get very, very technical and really understand the underpinnings of Mac OS X and HFS from the cloning perspective. Once I got tired of that, I retired that stuff and put my focus back to CCC and back to backup.
CCC has always been a cloning utility, but more recently I’ve spent a lot more effort implementing the backup functionality. Archiving, incremental backups, things like that.
TMO: You’re doing some stuff that certainly falls outside the bounds of the Mac App Store obviously.
MB: Absolutely.
TMO: It seems like Apple is, in its own way, embracing that with developer ID and that sort of thing to allow you to bless your app.
MB: I think embracing is the wrong word.
TMO: Yeah, maybe. Perhaps.
MB: It’s been frustrating for me. I would love to add that as a distribution mechanism. There are so many conveniences, not to mention the fact that you’re put in front of so many more users. I actually had a chance to talk to some of the developer technical support folks who are responsible for supporting the Mac App Store, right around the time they were announcing it, and rejecting me from it. The analogy the guy made that I talked to said I wouldn’t want my grandmother using an application like that.
I found that I had the complete opposite perspective. I would rather that my grandmother not be forced to go in and use the Terminal to do something that required administrative privileges. I honestly prefer that she had an application that had a nice interface and nice bounds and things like that. It’s a policy decision that Apple has made, they’re not going to allow applications that require administrative privileges. So as long as I need to backup your entire operating system, I really need to be able to read all of those system files. I don’t anticipate Apple ever changing that policy.
TMO: I think you’re in a situation though, because the policy is pretty clear and pretty broad, people that want to do really any kind of backup of the whole system, or any maintenance on the system, any disk utilities, anything like that, all have to live outside of the Mac App Store. In that sense, safety in numbers. Because people will at least think to look outside the Mac App Store for something that does this.
MB: It’s interesting that it really does force people. Because that’s not a small niche of software. Hard drives fail. And I see it every day on my help desk. People come to me, the biggest driver of support requests, probably 90 percent, is a problem with the file system, failing media, a bad cable, a bad enclosure, hardware incompatibility.
For a lot of these kinds of problems, you can’t just chuck the disk in the trash and go buy a new one. You want to try and fix the problem, you want to try and assess the problem. And Disk Utility really doesn’t do it. Disk Utility is nice for doing a very basic analysis of the file system structures. But you look to something else that goes into a lot more detail looking at specific file system objects, really exercising the file system to find problems that Disk Utility doesn’t find. Lots of people are going to find themselves looking for those solutions. And they can’t find it in the Mac App Store. It’s an interesting decision that Apple made there.
TMO: And it’s one we all have to live with. I don’t see them reversing it. Or expanding it. Or anything. They’re not clear about much, but they’re pretty clear about that. Can we talk about the fact that you worked for Apple?
MB: Sure.
TMO: For years, I’ve had a policy of preferring to use software where I can support the developer. Because, frankly, it’s a selfish policy. It’s because I want to use software where I can know that in a year or two years there’s a good chance that this person is still going to be developing the software. And we’ve all seen it, where somebody just has to abandon the project. I had always stayed away from Carbon Copy Cloner for that reason alone, not because the functionality was bad or anything. There was no way for me to pay you. That changed recently. Perhaps you can explain why.
MB: I solved a lot of those problems. First of all, I enjoyed working for Apple. I was there for eight years. I was the technical side of the sales team for higher education accounts. It was a good stint. I loved my sales team. We were a great team. We had lots of fun together. I sold hundreds and hundreds of servers to education in a big cluster. I helped with that mass deployment as well, it was part of my net restore stint.
It wasn’t enough for me. One of the things that I discovered about myself is that I need to be creating something. I actually have a masters in aquatic ecology. I studied biology for many years. I love that too. I loved doing field research. But I couldn’t take putting months and months into a project and have it fail completely. Some people can do that. They do it over and over and eventually you discover something amazing. But I’m not a discoverer. If I can sit down and in five minutes create something, I get so much gratification from that.
So I decided that I need to go and do something where I can be creating. And this is it for me. I absolutely love developing in Mac OS X. It’s so much fun. That’s why I left Apple. I left so that I could go and do great things and build great things. That was one of the things I had to solve. So now it’s my full time job — and a half.
TMO: And just to be clear, when you worked for Apple, you chose, for a good reason I’m guessing, not to take money in for Carbon Copy Cloner.
MB: It was entirely donationware at the time. As of recently, I’ve started to make a transition away from the donation ware model. The reason for that … there are some things that have weighed upon me that have been very frustrating and that’s exactly what you’re saying. When you look at a product, you compare a product that costs $30 and you compare it to something that has an apparent cost of nothing, some people think that there’s less value to a product where they don’t ask for anything.
Fair enough. I don’t necessarily agree with that, but I certainly understand that perspective. A lot of people also assess the value of a product based upon the price that the developer or the vendor puts upon that. Again, I totally understand that. But again I have a philosophical problem with that, having been a college student, having been not a wealthy person.
And there are people who use the product for different things. I’ve got lots of people that will download it once, clone a volume, and then they’re done. They’re using Time Machine, or no backup at all (to each their own). There’s that kind of person, then there’s the person that’s got 29 scheduled tasks, who I just worked with yesterday (it was a record), they backup a terabyte of data a day and they’re using it on a couple of servers.
So there’s really a wide gamut of uses. It’s hard to place a single monetary value on the one product that they download and use on one machine. It’s hard to say. I’ve got people who love it so much that they’ll send me $100. That’s great, and that’s the value of the product to them. And then there are other people who, you know, think it’s worth $5 or $20 or something like that. And that has worked really, really well for me. And I’ve really enjoyed it.
At least initially, what I’m going to try is moving to the commercial [market], where you purchase CCC, there’s a 60-day trial. The trial is supported by the banners that people have seen in the software. That primarily covers the people who download it and use it once just to clone a volume. That’s kind of what that covers, I certainly don’t get massive amount of money from that. But then, there’s the purchase button. So when the trial expires, you’re expected to purchase. But when you purchase, you actually get to choose the amount that you want to pay. Which is strange.
TMO: No, that’s good.
MB: This may cause some other logistical problems. And maybe I’m just being naive about this, I don’t know. I’m hopeful that I can make this work and it will work everywhere that I need it to work. We’ll see. If not, then I’ll go to a flat rate kind of deal. I’m always evolving my business plan and trying to make it work. As of the last version I released in April, I moved away from PayPal to a new payment processor. I had a lot of trouble with PayPal. They’re not a great vendor to work with.
TMO: No, they’re not.
MB: Now, I’m with FastSpring and their customer support has been outstanding. They really have a great product. So I’m working with them now, and as soon as Mountain Lion ships, or hopefully a little bit before that, I’ve got a new release coming out 3.5. It’s currently out in beta right now. It’s primary purpose is Mountain Lion compatibility. It’s at least my goal to not have any new features in it. It’s my goal to focus entirely on Mountain Lion compatibility.
TMO: And as a creator, I’m sure that’s not an easy thing to do — not add new features.
MB: I actually had one guy, he said “You know, I contacted you last July. I’ve really been wanting to do x, y, and z” and I’d put him off and finally I was like “oh, yeah, you know and there have been a couple of other cases where people have really needed this.” Oh all right, I’ll stick it in the experimental settings box. Now there are some things I exclude by default, globally. There are so many idiosyncrasies to backing up Mac OS X. You can’t back up the Spotlight database, it just doesn’t make sense. You can’t backup the mobile backups database, it also doesn’t make sense.
[There are] lots and lots of little things that nobody knows or cares about; they just want it to work on the backend. Well, the list is huge, it’s like 29 items. This guy wanted to not hide a specific folder on the destination. So that’s one tweak. Some people want to exclude the .ds_store file. They don’t want it copied to their SMB volumes, their network volumes. So I just added a little window that allows you to customize that functionality. It’s technically a new feature, but most people aren’t going to see it because it’s kind of hidden away in the kitchen sink area.
The goal is to really focus on the Mountain Lion compatibility. And I’m pretty much ready. I’m not going to give it the official seal of approval until I see the GM because as with every single release of Mac OS X, there are huge differences.
That’s actually one of the really frustrating things about Mac OS X development. Apple’s pace is breakneck and they’ve got an army of engineers that are going as fast as they can. They crank out lots of new features, file system changes. There’s always some flaming hoop that I have to jump through. Especially with the Mac App Store, it has never been easier for users to get it faster. I was actually scared to death last year when it GM’d, I thought as soon as it GMs the users are going to have it. But there was actually a two-week grace period. And I was thankful for every second I was given. But this year I’m pretty much ready. All of the major functionality I’ve already got plugged in and we’re just making tweaks and catching bugs right now.
TMO: Anything else before we wrap up?
MB: You know, I could talk about lots of things. I guess there’s some features in CCC that a lot of people aren’t aware of. I see a lot of comparisons of CCC to some of my competition, and a lot of people don’t even realize that CCC is a full-fledged backup product. They don’t realize that it can do things like an incremental backup, copy only the things that have changed. They don’t realize that you can copy to a network volume.
Actually, I was reading on an IRC channel, somebody said “I thought CCC only backed up to disk images.” It’s funny. People have lots of interesting conceptions about what it does or what it doesn’t do. It runs the full gamut. I’ll do basic backups of Mac OS X from one disk to another, backing up to a disk image, if you wanted to archive to a disk image, I can backup to a network volume. That was a huge one that people wanted that I delivered.
TMO: Tell us the top misconceptions of what people think.
MB: A lot of people think that it’s going to copy everything every time. And that’s certainly not the case. CCC will always evaluate the files on the source and destination and copy only the things that have changed since the last backup. A lot of people are like “Where’s the option to do this?” There is no option. It just does that. That’s the way that it works. And the trial, as far as that’s concerned, is completely open. There’s no hidden functionality, there’s nothing that’s locked until you pay. Someone actually thought that the performance was degraded until you pay. That’s also not the case. There’s lots of crazy ideas out there.
TMO: That would be a lot of work to write.
MB: It really would. I have absolutely no desire or time to play those kinds of shenanigans. And that’s certainly not how you build the users trust. I think I have a state of the art support mechanism built into CCC. The documentation window has three tabs. It has the documentation built in.
There’s also a support request tab. I’ve improved it in 3.5. You basically plug in your name, you can give me any name you want, it’s just so I can say “Hi, so-and-so” instead of just laying into it. Then your email address so we can send you an email confirmation. And then your text. When you hit the submit button, it submits the request to my help desk, and it gives you the opportunity to submit your logs to my server separately. So to improve privacy, I like to keep all of the logs in a secure location, not on my public help desk. I can see the logs, but everyone and his brother isn’t going to see all the stuff in your logs. And that also enables me to give support very quickly.
I get a lot of kudos on my support. I literally get people who will click the submit button and all they’ve said is “It’s not working.” And because they submitted the logs, I can browse through it and in just a matter of minutes I can say “Oh, here’s a problem with your configuration” or “Oh, that disk is definitely hosed. We’re having a problem with the disk. Here’s what we can do to isolate that problem.” So a lot of people are just amazed that they can give me almost no information — there’s never a back and forth anymore. There’s never “What OS are you running?” “What are your settings?” Things like that. You click the button.
TMO: Gosh, that’s really smart. I mean, it’s clear you spent time building your support mechanism and that’s probably an afterthought for most people.
MB: For me, support’s an interesting thing. A lot of people see it as a burden. It certainly can be a burden on your time. But I see it as an opportunity to improve the product, an opportunity to see how my users are using the product, and where they’re struggling. Obviously I get a time savings if I can fix the problem, if I can shorten that turnaround time, not have that back-and-forth with the user, if I can immediately dive into the details and solve the problem the first time I reply, that’s going to save me time, it’s going to save the customer time, it’s going to make everyone happier. So that’s always been my goal ever since I left Apple two years ago. That was kind of the first thing that I did was I wanted to completely turn around the support experience. And I’m happy to say it’s been working really well.
So there’s the support aspect, backing up to network volumes, I can backup to a Time Capsule for example, which I don’t particularly care for, it’s not a very strong device. If you backup to a Mac Mini, you get much better performance and I think a better experience. Backing up to disk images. I can do a block-level clone for a very fast clone. There’s the Recovery HD functionality for cloning the Recovery HD volume.
A hidden feature, I think, is my Disk Center. Within the Disk Center you can actually see performance statistics. As your backup task is running you can see what the data rate is going into the destination and out of the source. This is also really useful for troubleshooting. So if a backup task isn’t running, you can open that up and see, for example, if something is cranking away on your disk. A lot of people are like “It’s running really, really slowly. I don’t get it.” You open up the Disk Center and realize that something’s putting 8 MB per second of bandwidth to my destination disk.
TMO: So is it just a completely separate troubleshooting utility?
MB: Exactly. A lot of the stuff that I built into CCC really has nothing to do with backup but it’s more about the atmosphere around your backup. It’s about figuring out the problems that are affecting your backup.
The number one driver of support requests is problems with your disks and your file system, things like that. So another kind of hidden feature is the Cloning Coach and the Error Console. So if you do run into errors, I tally them all up into an array and at the end of the backup task, I present them to you in a table and you can click on each one and I give context-specific advice for each error. So if I see an error that looks likes a file system problem, I can say “Hey, this looks like a file system problem, why don’t you try this. And if you’re still having trouble with it, click the ‘Help Me’ button in the bottom and submit a request directly to my help desk.”
TMO: Very cool.
MB: I think that that right there, the context-specific help, the looking at individual error messages, and I looked at over a thousand error reports that were submitted to me, and I just poured through them finding the patterns of various errors that are reported back within my software, and looked at the resolution on the help desk that was associated with those error reports, and finally figured out “Why, I can look at these errors and I can make some pretty good guesses about what’s going on.” I have a pretty good resolution rate right there from within the Error Center.
The most common cause of problems is media failure. You get a bad sector on your disk, what do you do with it? I used to get that support request like six times a day. “What’s this error mean? What do I need to do with it?” And every once in a while I still get it even though it says right there you need to delete this corrupted file from your source volume. But I think it’s really helped a lot of users. And a lot of users appreciate the fact that as I’m copying, if I run into a corrupted file, I don’t just throw up my arms and bail.
Media failure is not a good thing, especially if your data’s on it. But it doesn’t mean your disk is failing necessarily. It usually just means there’s a bad sector on the disk, which is bad luck for that file. But delete the file and move on with life and we’re getting the rest of your data backed up. That’s especially important, because if your disk is failing, you don’t want to have to go through and exclude that bad file and then make another try just to find there’s another file. I go through in one shot and get everything I can read and I give you a list of the files that I couldn’t get. Right there, people love it.
TMO: Well have to wrap it up there. Thanks for taking the time to chat. This is great stuff. Thanks, Mike.
________________
Interview by Dave Hamilton with his iPhone. Transcription by Julie Kuehl with the Scrivener Transcription Tool. Editing by John Martellaro