IBM's state-of-the-art, $80m Auckland data centre hit by new problems

The Highbrook data centre in Auckland's East Tamaki

IBM's $80 million, state-of-the-art data centre in Highbrook, Auckland is suffering a second day of outages - and a "new technical issue."

The company has played its cards close to its chest, refusing to comment in detail on the cause of the service problems, or even when they first hit.

This afternoon, an IBM spokeswoman sent NBR ONLINE the following statement:

"While working to resolve the outage of New Zealand Virtual Server Services [a cloud computing service], a new technical issue was experienced which continues to affect the delivery of this particular service from the data centre. Our team of
global experts continue to work on this as a high priority and remain committed to restoring service for our clients."

The fault is thought to have first struck around 3am yesterday. IBM has not commented on timing.

When the Highbrook centre opened in May 2011, IBM touted its layers of phyical protection. These included a second roof under the first, and a 30,000 litres of petrol of hoarded petrol - enough to run the centre for around three days after a power cut. 

However, it seems all that was no protection against a technical glitch.

This article is tagged with the following keywords. Find out more about My Tags

Post Comment

16 Comments & Questions

Commenter icon key: Subscriber Verified

Any chance of a comment being sought from Rob Fyfe.... ;-)

Reply
Share

It was AirNZ's DataCentre (managed by IBM under contract) that had those power outages when Rob Fyfe lashed IBM in the media. I don't think AirNZ are in Highbrook

Reply
Share

I think the point was more they talk a big game and charge a big price but deliver - not so much.

Reply
Share

The data centre was owned by Air NZ but leased back to IBM, which was responsible for running it. The outage was caused by a UPS failure (also the responsibility of IBM) taking down the Air NZ owned/IBM supplied mainframe which ran the airline's reservation and flight control systems. Air NZ had a contract with IBM to run and manage its infrastructure, including the mission critical mainframe. Like this latest outage, it would seem the problem is in IBM short cutting or failing to manage its own processes as it strips cost to the bone to try and hit its touted EPS target of $20 by 2015. It has sent jobs to India, slashed its retirement plans and gutted its employee contracts. Those decisions are now coming home to roost. Go and Google 'IBM sued over failed contract' and see how many hits you get.

Reply
Share

Is this tier 4 five nines reliability?

Reply
Share

Their marketing for virtual shared services (main, but not only platform impacted) seems to sugget they're aiming for 3 nines "only" but even at that point, they can allow 8.7 hours per year to be within that SLA.

I understand they were offline for something in the ballpark of 40 hours, so they'll want to have about 5 perfect years to even get back to that.

Reply
Share

Wonder what the contingency is for crappy software?

Reply
Share

This long of an outage is absolutely TERRIBLE for a data center like this...

Reply
Share

IBM nz is a backwater operation of cloud Wannabees. Google they are not!

Reply
Share

Did anyone think about how often the petrol needs replacing? 30,000 litres @ $2.209? That's $66,270.00 worth! Oh well, the end user probably will have to pick up the tab.

Reply
Share

Guys, the article title is misleading. This is not a data centre outage but rather Virtual Server Services, one service offering out of this data centre.

Reply
Share

As far as I know, the data centre itself is still, and always has been, 100% functional. Power, cooling, etc, has not been compromised. The fault lies with IBM-managed infrastructure (servers and storage) within the data centre that runs their virtual server platform which clients subscribe to.

Reply
Share

Well that's completely different. It changes it from being an IBM screw up to .... an IBM screw up.

(Feels smug that I decided to go with self hosting rather than this a year ago)...

Reply
Share

Reliability relates to everything required to provide the contracted seamless service to the client.

Reply
Share

Virtual servers don't run so well when your SAN goes offline and you are struggling to get your non existent backups back. Seems cloud has many definitions and IBM's version of cloud computing is much like others version of single site virtual server hosting. Maybe the big VPS offerings are not all equal and buyers should be more aware of what they are getting. The SAN outage in this case is a clear single point of failure and therefore this is NOT a resilient cloud offering....hmmm

Reply
Share

IBM NZ lack quality engineers to maintain the VSS services infrastructure. Somewhere at some point some one or more than one person has commited an act of engineering technical negligence on the VSS front end and back end components, which torpedoed into various directions bringing down the whole stack of cards down and there may have been no real way of undoing the negligent engineering act. Or they may have tried to undo it, which then probably made it more worse. An $80 million data centre running into issues suddenly when it is supposedly world class standard. Don't they have monitoring, proactive checks and detection, prevention mechanisms in place, any regular health checks? Go figure.

Reply
Share

Post New comment or question

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

NZ Market Snapshot

Forex

Sym Price Change
USD 0.7850 -0.0001 -0.01%
AUD 0.8957 -0.0002 -0.02%
EUR 0.6214 -0.0002 -0.03%
GBP 0.4894 0.0000 0.00%
HKD 6.0919 0.0019 0.03%
JPY 84.1530 0.0310 0.04%

Commods

Commodity Price Change Time
Gold Index 1241.1 -7.450 2014-10-22T00:
Oil Brent 84.7 -1.510 2014-10-22T00:
Oil Nymex 80.5 -1.970 2014-10-22T00:
Silver Index 17.2 -0.310 2014-10-22T00:

Indices

Symbol Open High Last %
NZX 50 5279.7 5298.2 5279.7 0.32%
NASDAQ 4429.2 4435.9 4419.5 -0.83%
DAX 8934.5 8957.2 8887.0 0.60%
DJI 16615.3 16653.9 16614.8 -0.92%
FTSE 6372.3 6401.5 6372.3 0.43%
HKSE 23296.0 23397.2 23404.0 -0.40%
NI225 15093.8 15137.4 15195.8 -0.52%