The folly of "Do It Yourself"

About a year ago, a friend of mine called me in a panic about the storage devices at his company. A power brownout at the office hung both a recycled Compaq server running a open source storage platform called OpenFiler and Maxtor MaxAttach NAS appliance . It seems that their revenue-generating advertising graphics were on the Maxstor NAS appliance, while the rest of the workflow data was on the OpenFiler box.
No problem, I said, I can help put a new disk in the NAS and your IT guys run a restore of both devices. "Backups?!? We didn't back it up because it was a NAS. I thought that was what RAID and everything else was for!"
Ugh. After meeting with the IT staff, I helped them replace the disk in the Adaptec NAS, which promptly rebuilt and recovered. The OpenFiler server, however, suffered corrupt LVM information and we couldn't recover the root volume group in any way, costing them 2TB of unrecoverable data.

What went wrong? How did they get themselves into this predicament? There's nothing wrong with OpenFiler - it's a solid, stable platform. Nor was the Maxtor NAS the culprit - it was actually an award-winning product when it was introduced. The problem lie in the "Do It Yourself" approach the IT team had. They focused on creating point solutions to solve problems at hand, without looking at longterm impacts or objectives. I found out that the "backup" service at the company was a series of NTbackup and rsync scripts that copied data from desktop PCs to the OpenFiler box while similar scripts on the Macs copied advertising graphics onto the Maxtor NAS. No thought was given to real business continuity or data availability - these were both solutions that implemented because someone asked them. Open source software and integrated appliances make it easy for us to buy our way into and out of trouble, unless we take the time to chart a course and pay attention to turns we make.

With lessons learned, the company implemented a brand new server to run OpenFiler, configuring one share for important, revenue-generating work flow and another share for desktop file backups. All files on the revenue share were replicated to a managed hosting provider's content management system, with indexing and versioning, giving them an off-site copy of data with a clear way to recover from a local disaster. They IT manager crafted a storage strategy to reduce the amount of unprotected data on desktops and laptops, with important business data residing on the (replicated) NAS.

The "Do It Yourself" days of rolling together disparate products, with no thought to overall strategy, have come to an end - our 24/7, non-stop Internet economy won't support it any more.