Visit the full NBR website

IBM's state-of-the-art, $80m Auckland data centre hit by new problems

IBM's $80 million, state-of-the-art data centre in Highbrook, Auckland is suffering a second day of outages - and a "new technical issue."

The company has played its cards close to its chest, refusing to comment in detail on the cause of the service problems, or even when they first hit.

This afternoon, an IBM spokeswoman sent NBR ONLINE the following statement:

"While working to resolve the outage of New Zealand Virtual Server Services [a cloud computing service], a new technical issue was experienced which continues to affect the delivery of this particular service from the data centre. Our team of
global experts continue to work on this as a high priority and remain committed to restoring service for our clients."

The fault is thought to have first struck around 3am yesterday. IBM has not commented on timing.

When the Highbrook centre opened in May 2011, IBM touted its layers of phyical protection. These included a second roof under the first, and a 30,000 litres of petrol of hoarded petrol - enough to run the centre for around three days after a power cut. 

However, it seems all that was no protection against a technical glitch.

Comments and questions

Any chance of a comment being sought from Rob Fyfe.... ;-)

It was AirNZ's DataCentre (managed by IBM under contract) that had those power outages when Rob Fyfe lashed IBM in the media. I don't think AirNZ are in Highbrook

I think the point was more they talk a big game and charge a big price but deliver - not so much.

The data centre was owned by Air NZ but leased back to IBM, which was responsible for running it. The outage was caused by a UPS failure (also the responsibility of IBM) taking down the Air NZ owned/IBM supplied mainframe which ran the airline's reservation and flight control systems. Air NZ had a contract with IBM to run and manage its infrastructure, including the mission critical mainframe. Like this latest outage, it would seem the problem is in IBM short cutting or failing to manage its own processes as it strips cost to the bone to try and hit its touted EPS target of $20 by 2015. It has sent jobs to India, slashed its retirement plans and gutted its employee contracts. Those decisions are now coming home to roost. Go and Google 'IBM sued over failed contract' and see how many hits you get.

Is this tier 4 five nines reliability?

Their marketing for virtual shared services (main, but not only platform impacted) seems to sugget they're aiming for 3 nines "only" but even at that point, they can allow 8.7 hours per year to be within that SLA.

I understand they were offline for something in the ballpark of 40 hours, so they'll want to have about 5 perfect years to even get back to that.

Wonder what the contingency is for crappy software?

This long of an outage is absolutely TERRIBLE for a data center like this...

IBM nz is a backwater operation of cloud Wannabees. Google they are not!

Did anyone think about how often the petrol needs replacing? 30,000 litres @ $2.209? That's $66,270.00 worth! Oh well, the end user probably will have to pick up the tab.

Guys, the article title is misleading. This is not a data centre outage but rather Virtual Server Services, one service offering out of this data centre.

As far as I know, the data centre itself is still, and always has been, 100% functional. Power, cooling, etc, has not been compromised. The fault lies with IBM-managed infrastructure (servers and storage) within the data centre that runs their virtual server platform which clients subscribe to.

Well that's completely different. It changes it from being an IBM screw up to .... an IBM screw up.

(Feels smug that I decided to go with self hosting rather than this a year ago)...

Reliability relates to everything required to provide the contracted seamless service to the client.

Virtual servers don't run so well when your SAN goes offline and you are struggling to get your non existent backups back. Seems cloud has many definitions and IBM's version of cloud computing is much like others version of single site virtual server hosting. Maybe the big VPS offerings are not all equal and buyers should be more aware of what they are getting. The SAN outage in this case is a clear single point of failure and therefore this is NOT a resilient cloud offering....hmmm

IBM NZ lack quality engineers to maintain the VSS services infrastructure. Somewhere at some point some one or more than one person has commited an act of engineering technical negligence on the VSS front end and back end components, which torpedoed into various directions bringing down the whole stack of cards down and there may have been no real way of undoing the negligent engineering act. Or they may have tried to undo it, which then probably made it more worse. An $80 million data centre running into issues suddenly when it is supposedly world class standard. Don't they have monitoring, proactive checks and detection, prevention mechanisms in place, any regular health checks? Go figure.

« Back to home page