Optus customers cut-off by the Queensland telecommunications black-out yesterday may be able to seek compensation from the company.
The meltdown occurred when the main east coast line was severed by a Gold Coast excavator, nearby a separate work site operated by the Southern Regional Pipeline Alliance which was connecting a desalination plant to the main pipeline.
A secondary incident, reported yesterday by Computerworld, blocked traffic on the remaining Optus inland backbone when the Stanthorpe Point of Presence (POP) failed.
Optus customers were stranded at airports and electronic financial systems failed as all telecommunications links outside the state were dead while the telco rushed new hardware in by helicopter to the Stanthorpe exchange.
An Optus spokesperson issued an unreserved apology and invited affected customers to lodge compensation claims with the telco.
"The events that occurred on the Optus network in Queensland yesterday were exceptional and highly unusual," he said.
"We plan to meet obligations to customers on a one on one basis. If customers wish to discuss yesterday's events, we ask them to contact their account manager or Optus customer service."
The construction company which severed the link, an alliance of Gold Coast Water, the Gold Coast City Council and Abigroup, denied responsibility for the incident.
While Optus has kept mum on the failure, industry experts suggested the failure occurred due to the telco's lax redundancy capabilities because it did not have adequate planning agreements with network peers, nor sufficient hardware backup on site.
Market Clarity director Shara Evans said the company should have had backup hardware on site in the Stanthorpe premises to avoid delays in the event of network failure.
"It is surprising that hardware located at a major POP facility was not made fault tolerant, and that there was not a backup switch," Evans said.
"General practise says if you have a mission critical component that you are unable to backup with redundant features, then you [should] buy another piece of identical hardware or buy spare parts to get it back up online."
"The cut was restricted to south-east Queensland, so mobile phones should have still worked, [albeit] congested."
Evans said the mobile phone failure could have been caused by the interstate location of Optus' Queensland mobile subscriber registration database, which must be contacted by mobile phones before they can place a call over the network.
"If the link to the database is broken, phone and Internet services could be [taken] offline," she said.
Isolated traffic, normally re-routed over neighbouring lines such as PIPE Networks, could not be redirected, Evans said, because the required configuration and switching arrangements were not put in place earlier.
"That needs to be done early; it's not the sort of thing you want to do live in a crisis situation," she said.
Evans stopped short of saying the lack of redundancy could shift blame away from the construction company which severed the cable, because too much is left to speculation until Optus releases further detail.
The QLD Department of Natural Resources and Water was blacked-out from the network failure.
Staff mobile phones subscribing to the Optus network and resellers, including 3 Mobile, were dead along with department sites that use the telco's land line network.
A spokesperson for the department said the downed services were restored and verified by internal IT staff about four hours after the failure.
"Departmental staff were unable to access external Internet sites from our network [but] access to other government Web sites was not affected," the spokesperson said.
"Our public-facing services, [including] Websites and eBusiness servers, remained available during the outage."