How Netflix survived the Amazon EC2 reboot

The video streaming service was able to stay online even as its cloud hosting provider rebooted its servers

Sometimes the best path to success is to learn how to avoid failure.

Netflix was able to keep serving its customers while its cloud hosting provider, Amazon Web Services (AWS), rebooted servers, because it had prepared for that happening.

"When we got the news about the emergency EC2 [Elastic Cloud Compute] reboots, our jaws dropped. When we got the list of how many Cassandra nodes would be affected, I felt ill," said Christos Kalantzis, Netflix engineering manager of cloud database engineering, in a Netflix blog post discussing the outage.

Amazon announced to EC2 customers on Sept. 25 that it would be updating its servers and that a small percentage would require a reboot, which could potentially disrupt customer services. AWS did not specify which of their virtual hosts would be rebooted or when. It was revealed later that AWS was fixing a vulnerability in the Xen hypervisor, which underpins EC2.

Netflix is one of Amazon's largest customers. And its 50 million customers expect to be able to stream TV shows, movies and other content at any time. If Netflix wasn't prepared to mitigate potential outages, the company -- and not Amazon -- would have a lot of angry customers.

But Netflix had architected its service to be resilient, so that if one Amazon data center went down, operations could be switched over to another with barely a noticeable bump to customers. It also looked for ways to minimize downtime that occurred when its services did need to be rebooted.

The company even went the extra mile and aggressively looked for ways to try to disrupt its own services through a set of tools called the Simian Army that are designed to periodically and randomly kill Netflix services. The thinking goes that any Netflix service should be resilient enough to keep running through an attack from one such tool. If it isn't, then the Netflix engineers redesign the service to make it more reliable.

Even with its systems hardened by abuse from Chaos Monkey and other Simian Army tools, the engineers were still worried about the AWS reboot.

In particular, concern centered around the 2,700 Cassandra databases that the company runs on AWS.

Databases, as the blog post pointed out, are "the pampered and spoiled princes of the application world." They are run on the best hardware, get lots of attention from database engineers and still can be fussy creatures.

Netflix deliberately chose to use the Cassandra database over more traditional choices such as Oracle's databases because, as a NoSQL database, Cassandra could be spread across multiple servers in such a way that if one of the nodes failed, the database could keep running. Over the past year, the company had been subjecting Cassandra to Chaos Monkey testing, with promising results.

The AWS reboot would be the first true test of Cassandra's reliability, however. The entire cloud database engineering team was on alert.

In the end, and thanks to Chaos Monkey testing, most all of the Cassandra nodes remained online. Of the 218 Cassandra nodes that were rebooted, only 22 did not return to a full operational state, and those were successfully restarted with minimal human intervention.

"Repeatedly and regularly exercising failure, even in the persistence layer, should be part of every company's resilience planning," the blog concluded. "If it wasn't for Cassandra's participation in Chaos Monkey, this story would have ended much differently."

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Tags amazoncloud computingnetflixinternetInfrastructure services

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Essentials

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Kathy Cassidy

STYLISTIC Q702

First impression on unpacking the Q702 test unit was the solid feel and clean, minimalist styling.

Anthony Grifoni

STYLISTIC Q572

For work use, Microsoft Word and Excel programs pre-installed on the device are adequate for preparing short documents.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?