Microsoft beats data-sorting record with new approach

A new approach, called Flat Datacenter Storage, pushes computational sorting to each data server

Besting a record set by Yahoo in 2009, the research arm of Microsoft have deployed a new technique for quickly sorting large amounts of data, called Flat Datacenter Storage (FDS).

The researchers will discuss their work at an Association for Computing Machinery conference dedicated to databases this week in Scottsdale, Arizona. They are also implanting their data-sorting techniques in Microsoft's Bing search engine, where it could boost response times to user queries.

"Improving big-data performance has a wide range of implications across a huge number of businesses," said Microsoft Research project leader Jeremy Elson, in an online entry describing the work. "Almost any big-data problem now becomes more efficient, which, in many cases, will be the difference between the work being economically feasible or not."

In tests conducted under the MinuteSort benchmark, the system set up by Elson and his colleagues was able to sort 1,401Gb of data in a minute, which beat Yahoo's previous record of 500GB in the same time. Microsoft also boasted of sorting the data using fewer resources: The system used 1,033 disks in 250 machines while Yahoo required 5,624 disks across 1,406 machines to complete their operation.

FDS starts with a similar approach as Google's MapReduce -- as it is implemented in Apache Hadoop -- by moving the computational sorting to each individual data server. Unlike Hadoop, however, every server trades information with all the other server in the sorting cluster. The researchers used an additional Microsoft networking technology, called full bisection bandwidth networks, to boost the bandwidth, allowing each computer to both send a receive send up to 2GB per second.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Cate Bacon

Aruba Instant On AP11D

The strength of the Aruba Instant On AP11D is that the design and feature set support the modern, flexible, and mobile way of working.

Dr Prabigya Shiwakoti

Aruba Instant On AP11D

Aruba backs the AP11D up with a two-year warranty and 24/7 phone support.

Tom Pope

Dynabook Portégé X30L-G

Ultimately this laptop has achieved everything I would hope for in a laptop for work, while fitting that into a form factor and weight that is remarkable.

Tom Sellers

MSI P65

This smart laptop was enjoyable to use and great to work on – creating content was super simple.

Lolita Wang

MSI GT76

It really doesn’t get more “gaming laptop” than this.

Featured Content

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?