HPE SANs Not Designed for Reliability in the Australian Tax Office
-
Another HPE 3PAR horror story (the other big one recently was the debacle with King's College London.) HPE 3PAR SANs in the Australian Tax Office crashed twice because they were built for speed, not for reliability, and had error reporting disabled, according to The Register.
Some of the issues reported by the Senate Committee overseeing the situation include:
- The fibre optic cables feeding the SAN were not optimally fitted
- Disk drives on the SAN had software bugs that made the stored data on the drives inaccessible or unable to be read
- Some monitoring features were not activated, including a "back-to-base" tool to report operating errors
The Senate also said: "The SAN design and configuration meant we had an overemphasis on performance features rather than stability or resilience – a relatively small disk drive failure had a large impact – only 12 of some 800 disk drives failed, but they impacted most ATO systems."
After the SAN failed, it didn't help that "some of the recovery tools required were stored on the same SAN that failed". <- Sound family to King's College... the SAN was used as its own backup!
Remember, even the best SAN in the world is only as good as the implementation engineers that install it and the support people who stand behind it.
-
So is this another IPOD, but at a government level?
Cause that is what it sounds like. . .
-
This, of course, is where making a system more complicated than necessary can be a huge risk. The tax office is understandably huge and needs a lot of complication, but clearly the SANs in use were a bit too much unknown for the people involved. Whether this was that there wasn't a competent SAN storage engineer team with the office, whether they relied on outside contractors that were not properly vetted or engaged, that needs were not communicated, that auditing and governance were missed or whatever - the bottom line is that this was a complex system that wasn't setup properly, even after there had been an outage and a chance to investigate!
-
@DustinB3403 said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
So is this another IPOD, but at a government level?
Cause that is what it sounds like. . .
It IS an IPOD, but we assume at a massive scale where an IPOD would make sense. In theory, these are insanely high availability SANs (except that was apparently not how they were used here) and in theory there would always be replication between SANs at this level. However, that's not what King's College London did, and that was HPE consulting there as well.
-
Sounds like they were victims of Shiny New Device syndrome.
-
@nadnerB said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
Sounds like they were victims of Shiny New Device syndrome.
Likely victims of "I'll just let a salesman do my job and get paid to not know what I'm doing" syndrome.
-
@scottalanmiller said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
@nadnerB said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
Sounds like they were victims of Shiny New Device syndrome.
Likely victims of "I'll just let a salesman do my job and get paid to not know what I'm doing" syndrome.
At my work right now... we just implemented a SAN device which supposed to store our Files servers.... Even if there are multiple drives, and Raid 10, there is only one SAN. I was aggressively oppose this idea, but it was approved by my manger anyhow. Can't wait for it to fail. * grab popcorn *
-
The moral of this story is that listening to HPE consulting will totally bone your systems at maximum profit to HPE.
-
@RojoLoco said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
The moral of this story is that listening to HPE consulting will totally bone your systems at maximum profit to HPE.
Yup, it's like RAID 5 - people forget that it makes vendors money to recommend reckless systems. The risk goes to you, the profits go to them.
It's far from specific to HPE, but HPE seems to have a track record of it recently.
-
@scottalanmiller said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
@RojoLoco said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
The moral of this story is that listening to HPE consulting will totally bone your systems at maximum profit to HPE.
Yup, it's like RAID 5 - people forget that it makes vendors money to recommend reckless systems. The risk goes to you, the profits go to them.
It's far from specific to HPE, but HPE seems to have a track record of it recently.
Maybe the "E" stands for "evil".
-
@RojoLoco said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
@scottalanmiller said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
@RojoLoco said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
The moral of this story is that listening to HPE consulting will totally bone your systems at maximum profit to HPE.
Yup, it's like RAID 5 - people forget that it makes vendors money to recommend reckless systems. The risk goes to you, the profits go to them.
It's far from specific to HPE, but HPE seems to have a track record of it recently.
Maybe the "E" stands for "evil".
That's still Lenovo in my book, but HPE is working on it.
-
-
Listening to any vendor consulting will lead to a p*** poor system but for the vendors profit.
-
@Breffni-Potter said in HPE SANs Not Designed for Reliability in the Australian Tax Office:
Listening to any vendor consulting will lead to a p*** poor system but for the vendors profit.
Yeah, they brought in the wrong people here. Fundamental business flaws. Listening to the sales people instead of hiring someone whose job it is to know what is needed.
