dsleports.com offline: "Lacking a good explanation of what happened to "redundant" disk storage..."

[bananas]

http://www.dslreports.com/front/shutdown.html

dslreports.com is offline
-------------------------

Update Wed Apr 18 09:54:54 EDT 2012

Well, the good news is we have a plan & I hope things will remain offline for no
longer than the balance of this week. Unfortunately, equipment must be sourced
installed, configured, and so on. Lacking a good explanation of what happened to
"redundant" disk storage, I am reluctantly faced with just replacing it with more
expensive gear. No certainty on times, yet. I will post a link to the gory details soon.

Update Tue Apr 17 04:14:29 EDT 2012

Things have turned out to be more complicated. Although no hardware was fried,
our large storage array has decided, probably as a result of going dark in mid write,
that our two sql servers can no longer have access to one of the small,
but important, partitions that makes up the larger SQL data area. All data
is present, but it refuses to offer it.
At this point all courses of action involve long delays. When I know more I'll post
-Justin

Update 15:35 EDT
Our data center experienced a power outage which affected our entire system. We are in the process of recovering everything
and turning a few cranky servers into happy ones.

Apologies for the inconvenience, we expect to be up in a few hours

Update 16:18 EDT
-------------------------
The power outage affected the entire data center and we are still awaiting our turn to power up some machines.
We will keep this page updated.

Update 19:20 EDT
-------------------------
Unfortunately, a database server doesn't take kindly to being taken down without proper shutdown procedures and the
restoration efforts are still under way. No ETA at this time.

[SlipperySlope]

1. "Redundant" isn't an absolute safety net

Redundant is generally intended to prevent catastrophic hardware failure. Redundant doesn't do anything to prevent data corruption. If bad data is written to your redundant storage array, you now have redundant bad data.

Only backups can recover from data corruption.

[RC]

2. It sounds to me like someone needs to make a call to a UPS sales person.

Good insurance. And they can be setup to orderly shut down the servers they are powering.

[discntnt_irny_srcsm]

3. My thoughts exactly n/t