Twitter solves its data formatting challenge

At HadoopWorld, Twitter's analytics chief discussed the new technologies it uses to handle its daily data deluge

Eschewing popular choices such as XML, CSV and JSON, Twitter has opted to format the back-end storage of its user and systems data with a relatively unknown format pioneered by Google, called Protocol Buffers.

With the company storing 12TB of this data each day for later use, the decision of which format to use was a crucial one.

"Getting your data formats right is everything," said Twitter analytics lead Kevin Weil, during a talk at the HadoopWorld conference in New York on Tuesday.

The company is planning for the time when it will have to house "a trillion Tweets," Weil said, and it wants tools in place to analyze this information. The combination of Protocol Buffers, along with Hadoop and other associated technologies, should streamline this job, Weil said.

When stored, each short message, or "tweet," consists of 17 fields, six of which have at least one subfield, he explained. And the company will probably add more fields to these schema in the years to come.

In addition to the tweets the company's users supply, Twitter keeps internal log data on more than 80 different types of operations that occur within its systems, Weil said. Much of this log data is aggregated by Facebook's open-source technology Scribe.

The choice of a format to store all this data was a difficult one. One obvious choice is XML (Extensible Markup Language), but that protocol is "very wordy," Weil said, referring to how the name of the tag accompanies each data element.

Under XML, "one petabyte for a trillion Tweets might become 10 petabytes for a trillion Tweets," he said.

JSON (JavaScript Object Notation), though it was designed to simplify XML, is also wordy, in that it also stores the name of the key with every entry.

At the other end of the spectrum is CSV (Comma Separated Values). As the name suggests, CSV separates each data element only with a comma. While simple, it is not good for nesting data elements in subfields, Weil explained. Also, if the schema is changed, the resulting programming it would take to accommodate data in the old schema would be considerable.

A downside to all of these protocols is that, in order to get the data in and out of applications, developers have to repeatedly create data structures to encode and parse the data, work Weil considers "rote."

Protocol Buffers, used widely within Google, is an extensible protocol for serializing data, one Google claims is simpler than XML. And it can automate the process of recreating the data structures within applications.

"You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages," a Google tutorial on Protocol Buffers states. "You can even update your data structure without breaking deployed programs that are compiled against the 'old' format."

For Twitter, this automation would allow the company to spin up new features more quickly.

"Protocol Buffers will generate code in a number of different languages, so you don't have to write code beyond IDL," or Interface Description Language, Weil said. It also ensures that should the schema be changed, the older information will remain accessible.

While primary copies of user Tweets are kept in MySQL and Cassandra databases, the company is also building a second data repository, running on Hadoop, that can be used for analytics and applications.

The information in this system can be queried using Java MapReduce or Pig, which is Hadoop's own SQL-like query language. Already one feature, Twitter's name search, runs on this system, and more are expected to be built.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Or
Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection

Tags social mediainternetGoogletwittersocial networkingstoragesoftwaremiddlewareapplication developmentdata integrationWeb services developmentInternet-based applications and servicesDevelopment toolsLanguages and standards

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Essentials

Mobile

Sansai 6-Outlet Power Board + 4-Port USB Charging Station

Learn more >

Exec

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Louise Coady

Brother MFC-L9570CDW Multifunction Printer

The printer was convenient, produced clear and vibrant images and was very easy to use

Edwina Hargreaves

WD My Cloud Home

I would recommend this device for families and small businesses who want one safe place to store all their important digital content and a way to easily share it with friends, family, business partners, or customers.

Walid Mikhael

Brother QL-820NWB Professional Label Printer

It’s easy to set up, it’s compact and quiet when printing and to top if off, the print quality is excellent. This is hands down the best printer I’ve used for printing labels.

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Ratchada Dunn

Sharp PN-40TC1 Huddle Board

The Huddle Board’s built in program; Sharp Touch Viewing software allows us to easily manipulate and edit our documents (jpegs and PDFs) all at the same time on the dashboard.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?