Facebook prepares to use 'cold storage' to deal with vast amounts of data

Facebook rethinks the data center to reduce costs and still be able to store billions of photos effectively

Facebook is rethinking the way it stores data to cope with the 7 petabytes of new photos the social network's users upload every month. As the number of photos grows, Facebook needs find cheaper, less power-hungry ways to store them all, according to the company's vice-president of infrastructure engineering.

Users upload about 300 million photos a day, more on special occasions, Facebook's Jay Parikh [CQ] told the Structure Europe conference in Amsterdam on Wednesday. "Halloween is one of our biggest photo upload days of the year. We will get somewhere between probably 1 and 2 billion photos uploaded just in a single day," he said.

Photos like the ones taken at Halloween soon lose their interest, with no one looking at them after a few days or weeks, but "Our contract with our users is that we can't delete the data when it is not accessed, we have to keep it," he said. That led to the idea of putting the photos into a sort of "cold storage," Parikh said. To do that, Facebook plans to build a new data center with different types of storage, server hardware and network equipment that consumes less power and costs less than existing data centers -- all without changing servers response times, said Parikh.

But how efficient can Facebook make its cold storage? When costs and power consumption in data centers are lowered, this usually happens at the expense of access speeds.

Storing data on tapes, for instance, lowers power consumption but severely slows down data access.

Amazon Web Services is following a middle path with its Glacier cloud storage service, which it pitches as an alternative to tape. The service is optimized for data that is infrequently accessed and for which retrieval times of several hours are acceptable.

That's much too slow for Facebook, according to Parikh. "I can't have a photo that you go access from five or ten years ago, and for me to show up a banner to the user that says: 'Hey, why don't you try again in 24 hours?' It's still got to be relatively real time," he said.

Most data centers that are used today are optimized to use a lot of power to deal with tasks that need big computing power. The "cold storage" technology Facebook is thinking of is at the other extreme, said Parikh. "You need lots and lots of space but you don't need as much power," he said, adding that everything about the data center needs to be rethought to handle the problem at the scale Facebook faces.

At a high level, Facebook is working on software that will figure out how and where to store a piece of content in the infrastructure when it ages, said Parikh. "That will mean that the copies of the data will move around over time and utilize the different pieces of infrastructure that we will have optimized for the age of content." Some of the inventions in the software layer will allow Facebook to still respond quickly but to store data more cost effectively, he said.

Cold storage will be part of Facebook's infrastructure in the next year or two, he said. Facebook plans to disclose and share the parts that it thinks are relevant through the Open Compute Project, an initiative started by Facebook to apply the open-source software collaboration model to the world of data center hardware.

Loek is Amsterdam Correspondent and covers online privacy, intellectual property, open-source and online payment issues for the IDG News Service. Follow him on Twitter at @loekessers or email tips and comments to loek_essers@idg.com

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Loek Essers

IDG News Service
Show Comments

Essentials

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?