Amazon cloud outage was triggered by configuration error

Company's postmortem and apology wins praise for transparency

Amazon has released a detailed postmortem and mea culpa about the partial outage of its cloud services platform last week and identified the culprit: A configuration error made during a network upgrade.

During this configuration change, a traffic shift "was executed incorrectly," Amazon said, noting that traffic that should have gone to a primary network was routed to a lower capacity one instead. The error occurred at 12:47 p.m. on April 21 and led to a partial outage that lingered through last weekend.

The outage sent a number of prominent Web sites offline, including Quora, Foursquare and Reddit, and renewed an industry-wide debate over the maturity of cloud services.

Amazon posted updates, short and bulletin-like, throughout the outage, but what it offered in its postmortem is entirely different. This nearly 5,700-word document includes a detailed look at what happened, an apology, a credit to affected customers, as well a commitment to improve its customer communications.

Amazon didn't say explicitly whether it was human error that touched off the event, but hints at that possibility when it wrote that "we will audit our change process and increase the automation to prevent this mistake from happening in the future."

The initial mistake, followed by the subsequent increase in network load, exposed a cascading series of issues, including a "re-mirroring storm" with systems continuously searching for a storage space.

Amazon also said in its explanation of the outage that it will work to ensure that it builds software and services that can survive failures.

Matt Stevens, the CTO of AppNeta, a cloud performance network performance management company and an Amazon cloud user, praised Amazon's postmortem for its transparency. "As a technical architect, I thought it was actually amazing how deep they went into it," said Stevens, adding that he wished the company had offered more detail about the initial network change that started the problem.

In terms of the overall issue, Stevens said: "How does anybody who runs their own private data center know how it's going to hold up until you have a massive issue?"

Jim Damoulakis, CTO of GlassHouse Technologies, an enterprise storage services provider, called it "a pretty through postmortem and I think for the most part they are being transparent about it."

Damoulakis said that while Amazon will take steps to keep the problem from happening again -- and to make their availability zones more robust -- customers will ultimately be responsible for having a good disaster recovery plan.

"I think there is blame on both sides," said Justin Alexander, who heads strategic research and development at Hyland Software, an enterprise content management software firm, referring to both Amazon and its customers.

"Clearly, Amazon needs to take accountability for their services. But at the same time there were a variety of customers who were using the EC2 platform that did not suffer any period of unavailability," said Alexander, citing their disaster recovery plans.

Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov or subscribe to Patrick's RSS feed. His e-mail address is pthibodeau@computerworld.com.

Read more about cloud computing in Computerworld's Cloud Computing Topic Center.

Join the newsletter!

Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection

Tags Configuration / maintenancedisaster recoveryapplicationsNetworkinghardware systemssoftwareBusiness ContinuityData Centercloud computinginternet

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Patrick Thibodeau

Computerworld (US)
Show Comments

Cool Tech

Breitling Superocean Heritage Chronographe 44

Learn more >

SanDisk MicroSDXC™ for Nintendo® Switch™

Learn more >

Toys for Boys

Family Friendly

Panasonic 4K UHD Blu-Ray Player and Full HD Recorder with Netflix - UBT1GL-K

Learn more >

Stocking Stuffer

Razer DeathAdder Expert Ergonomic Gaming Mouse

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Walid Mikhael

Brother QL-820NWB Professional Label Printer

It’s easy to set up, it’s compact and quiet when printing and to top if off, the print quality is excellent. This is hands down the best printer I’ve used for printing labels.

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Ratchada Dunn

Sharp PN-40TC1 Huddle Board

The Huddle Board’s built in program; Sharp Touch Viewing software allows us to easily manipulate and edit our documents (jpegs and PDFs) all at the same time on the dashboard.

George Khoury

Sharp PN-40TC1 Huddle Board

The biggest perks for me would be that it comes with easy to use and comprehensive programs that make the collaboration process a whole lot more intuitive and organic

David Coyle

Brother PocketJet PJ-773 A4 Portable Thermal Printer

I rate the printer as a 5 out of 5 stars as it has been able to fit seamlessly into my busy and mobile lifestyle.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?