Happy Penguin - Not so happy
They've had a hardware crash and are struggling to recover from it since 9th of October (more than a month as of this blog entry).
They've managed to bring up a simple HTML page to tell us about the progress they're making and as it's likely to be removed once the site is back up I thought I'd save it for the world.
I hope it helps teaching a lesson or two about data recovery tests...
Due to a hard drive failure which occurred at the same time as our backup system failed, the site is currently down. We expect to have full data recovery within the next few days.
As of 9th Oct 2010 the data recovery is still ongoing. We have been told to expect a result early next week, so crossed fingers, we are hoping to have service resumed on or around the 13th Oct.
As of 15th Oct 2010 the data recovery is still ongoing. Apparently the Western Digital HDD that died has corrupted itsown firmware, making it hard to extract the data from the disc.
As such, we are beginning the process of backup recovery from our most recent tier 2 backups which are a few weeks out of date, but waiting on the tier 1 recovery is getting ridiculous. We plan on working over the weekend on this recovery, and hope to have more news for you shortly.
As of 22nd Oct 2010 data recovery is STILL ongoing. The Western Digital Caviar Green 2TB drive that failed has chemical degredation on the surface making the data recovery much slower and harder.
All going well, we will have the older backups making sense soon, and will be back up to speed.
As of Nov 4th, we have been unable to bring the site back up yet, due to the discovery that the older backups bring brought online would end up with a number of users being unable to access their games due to copy protection issues. As such, rather than give ANY user a problem with their legally purchased game, we will be keeping the site down for the next few days while the final stages of data recovery are performed. All going well, we hope to be back up towards the start of next week.
As of Nov 8th, we have a report from the data recovery company of a complete recovery. We should be receiving the recovered data on Wednesday 10th and we can then begin the process of finally getting things back to normal around here.
As of Nov 12th, we have now received the data recovery files, but there has been a fair amount of filesystem damage, and it will take us some time to locate the files, rebuild the filesystem, and get everything back on its feet. We are still hoping for being back online next week, all going well.