Data Safety - Trust but Verify

After a recent internal event, I asked my IT department to write up a commentary of what happened which I could publish in order to warn my fellow agents. The cautionary tale that follows is one I urge everyone to read and heed when considering the safety of the data that is used every day to run your business.

Carbonite is a commercial off-site data backup and recovery system.  The principle is simple, elegant, and attractive. You sign up for a subscription, a small piece of software is installed on your PC, and data from your PC is backed up continuously - as it is changed - with nearly no disruption to your day-to-day activities.  The backup occurs to an off site location, providing safety against physical disasters at your location. The data is backed up on redundant systems to provide further safety against hardware failures. They even promise some level of version control which could allow you to recover from a human error by recovering a file prior to the human involvement that generated the error. Carbonite just sits there in the background taking care of your data so that you need not worry about it. That's the problem...

All of our office systems were "protected" by Carbonite.  We installed the software on every PC and laptop nearly 4 years ago. Approximately 2 years ago, one of the computers experienced a hardware failure corrupting a percentage of the files. We were able to recover all of this data using Carbonite. Around six months before that, a laptop hard disk failed. We recovered the entire backup to the new disk. For a lengthy period of time, a member of the team was away from the office for weeks at a time. Every week, he received a notice that the PC had not been backed up (because it was turned off). This is a good safety measure.

We, the IT department, were feeling pretty good about our selected data safety strategy.  For a small investment, our office had a multiply redundant off-site backup and disaster recovery strategy. Those happy feelings ended last week.

One laptop experienced physical damage to its disk drive (the read head scratched the disk surfaces). No problem, the PC was operating slowly and was aging, it was time to be replaced anyway. So the IT Department goes to work locating backups and evaluating replacement laptops. Then came the ugly surprise. The last backup of this laptop occurred two years earlier. For two years, no backups have been occurring, and no notification had been sent. The nightmare scenario was playing out. It was lucky that this PC was not the sole repository of absolutely business-critical information. It did have, however, a significant amount of unique data that existed nowhere else.

We attempted a forensic recovery of the disk platters, but the damage was too severe. The technician quoted us $1300 for a 50% chance that any data could be recovered.  Two years of important data has simply vanished.

Carbonite's technical support team was astonishingly unhelpful and uncaring. They simply confirmed the date of the last backup, with no comment on how the backups might have stopped or why there was no email notification.  We found it interesting that the date of the last backup within one week of the subscription renewal date. We also confirmed that we had been paying for the backup subscription for this laptop for two years while no backups were occurring.  This is an unacceptable failure on the part of Carbonite. 

We are requesting a refund of the two years subscriptions, and we are in the process of moving our backup procedures to an alternate service. Rest assured that verification of successful backups will be a part of that plan. We strongly encourage members of both the real estate and IT industries to learn from this cautionary tale and ensure the safety of their own backup systems. Not just when something fails, not just once in a while, but on a regular periodic basis. Also, before selecting a Internet backup service, check out the online reputation of those services.  When we signed up for Carbonite, the company had not been around long enough to have established a reputation - we took them on faith. Now there are multiple companies, and each has been around long enough to have built up an independently reviewed reputation. For example: we just googled Carbonite backup failure during this process and found some interesting results. 

Our entire team hopes that you learn from our problems, so that you never have to experience it yourself. Imagine how bad it could have been if the loss included are only copies of critical customer information, contracts, financial documents, and company books. To this day, we are in awe of how bad it could have been. Don't let it be that bad for you.

As long as you are here reading this article, help me impress upon all the other readers how important data safety is.  Share your stories with the audience by posting a comment. Feel free to re-blog this article in any venue that might help others to avoid this problem (with proper attribution).