04 January 2009

journalspace - Gone

"The list of potential causes for this disaster is a short one. It includes a catastrophic failure by the operating system (OS X Server, in case you're interested), or a deliberate effort. A disgruntled member of the Lagomorphics team sabotaged some key servers several months ago after he was caught stealing from the company; as awful as the thought is, we can't rule out the possibility of additional sabotage. But, clearly, we failed to take the steps to prevent this from happening. And for that we are very sorry"
After nearly six years, journalspace is no more. The drives that housed their entire database were overwritten.

The problem was that their backups weren't actually backups at all. journalspace's servers were set up with a mirrored RAID system so that if the primary drive should fail the secondary drive would be used to recover the primary. This alone is risky business, as it only protects you from the failure of one drive. In the case of journalspace, the drive didn't fail but instead the data was overwritten/erased on one drive, leading the other drive to follow suit and clear itself. A data recovery team from DriveSavers was unable to retrieve the database.

5 comments:

Paul Mohr said...

I knew a fellow who got his backup and restores in the wrong order once. He happened to be managing the data base for all the American Airlines flights at the time. He managed to restore before backing up and lost an entire week of airline reservations. He developed a twitch after that. It is a dangerous and expensive game dealing with data sometimes. You need the best people in charge of it, but companies often like the cheaper alternatives. When I do any sort of cloud computing, I keep my own backups. I have a mirror of my blog even. Just in case Google shells out ;)

Wei-Yee Chan said...

>I knew a fellow who got his backup
>and restores in the wrong order once.

*Rotfl*

>It is a dangerous and expensive game
>dealing with data sometimes. You need
>the best people in charge of it, but
>companies often like the cheaper
>alternatives.

Pay peanuts and you'll get monkeys.

>When I do any sort of cloud computing,
>I keep my own backups. I have a mirror
>of my blog even. Just in case Google
>shells out ;)

Well, good for you.

Wei-Yee Chan said...

-------- Original Message --------
Subject: Re: journalspace - Gone
Date: Sun, 04 Jan 2009 15:17:05 +0800
From: Wei-Yee Chan
Organization: Google Inc.
To: fmiller@lightlink.com

On Sunday, January 04, 2009 02:36 PM, Fred A. Miller wrote:
> Wei-Yee Chan wrote:
>
>> http://chanweiyee.blogspot.com/2009/01/journalspace-gone.html
>>
>
> 'Good example of when a plan really doesn't "come together", eh?
>
> Fred
>
Yep. For instance, the $in-gaporeans swear by RAID but fail to realise what it does and what it doesn't do. I always keep a 'real' backup. I recall that the last time I had a conversation with a $in-gaporean over RAID, it ended with him shouting and swearing at me, cos he got everything wrong.

unwesen said...

RAID as a backup, huh? Stupid, stupid.

Wei-Yee Chan said...

>RAID as a backup, huh? Stupid, stupid.

Yep, that's what you get when you have a monkey majority in an organisation.