Cray brings Hadoop to supercomputing

Cray has released a package designed to allow XC30 users to easily deploy Hadoop

Helping scientific supercomputing take advantage of emerging big-data technologies, high-performance computing manufacturer Cray is releasing a set of packages promising to optimize the process of running Hadoop on the company's XC30 machines.

The Cray Framework for Hadoop, along with the Cray Performance Pack for Hadoop, provides a set of tools and best practices for configuring and optimizing an XC30 to run Hadoop for scientific big-data-style projects, according to the company.

Hadoop's Java-based MapReduce model of data analysis could bring a number of benefits to supercomputing, though it has not found widespread acceptance in that community yet, even though both deploy parallel processing and extremely large data sets.

Cray has seen some interest in Hadoop from its users, though the open-source data processing platform was not set up to meet most scientific supercomputing use cases, said Bill Blake, chief technical architect of Cray, in a statement.

Hadoop's approach of bringing the computation to the data differs from the traditional supercomputing approach of moving the data to the processors.

Traditional supercomputing scientific number-crunching tends to rely on large hierarchical file formats and libraries for boosting rates of I/O (input/output), neither of which Hadoop was geared well for handling. Scientific computing relies on parallel file systems and fast interconnects typically not found in Hadoop deployments.

Scientific workloads also tend to have more complex workflows, incorporating both scientific compute and analytics workloads. Data models are also co-mingled with math models in scientific computing, also not the norm for Hadoop.

The Cray Framework for Hadoop and the Cray Performance Pack for Hadoop will address these issues, allowing users to get the most computational power out of the XC30s for Hadoop jobs, according to the company.

An update to the performance pack, to be made available in early 2014, will also include additional system code to optimize the XC30's use of the Lustre file system library and the Aries system interconnect used on Cray machines.

The XC30 is Cray's premier supercomputer, featuring integrated servers and switches, the Lustre parallel file system, Aries high-speed interconnects, an innovative cooling system, and the Dragonfly network topology for minimizing locality constraints.

Cray announced the packages at the SC2013 supercomputing conference, being held this week in Denver.

Cray also announced that it is upgrading the University of Stuttgart's XC30, nicknamed "Hornet," so it will offer more than seven petaflops (quadrillion mathematical calculations per second) of processing power.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Tags open sourceClusterssupercomputersHigh performanceapplicationshardware systemsCraydata miningsoftwaredata warehousing

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Cool Tech

D-Link TAIPAN AC3200 Ultra Wi-Fi Modem Router (DSL-4320L)

Learn more >

Crucial® BX200 SATA 2.5” 7mm (with 9.5mm adapter) Internal Solid State Drive

Learn more >

D-Link PowerLine AV2 2000 Gigabit Network Kit

Learn more >

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

ASUS ROG Swift PG279Q – Reign beyond virtual world

Learn more >

Gadgets & Things

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >


Learn more >

Lexar Professional 2000x SDHC™/SDXC™ UHS-II cards

Learn more >

Family Friendly

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

ASUS VivoPC VM62 - Incredibly Powerful, Unbelievably Small

Learn more >

Lexar Professional 2000x SDHC™/SDXC™ UHS-II cards

Learn more >

Stocking Stuffer

Lexar Professional 2000x SDHC™/SDXC™ UHS-II cards

Learn more >

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Best Deals on Good Gear Guide

Latest News Articles


GGG Evaluation Team

Kathy Cassidy


First impression on unpacking the Q702 test unit was the solid feel and clean, minimalist styling.

Anthony Grifoni


For work use, Microsoft Word and Excel programs pre-installed on the device are adequate for preparing short documents.

Steph Mundell


The Fujitsu LifeBook UH574 allowed for great mobility without being obnoxiously heavy or clunky. Its twelve hours of battery life did not disappoint.

Andrew Mitsi


The screen was particularly good. It is bright and visible from most angles, however heat is an issue, particularly around the Windows button on the front, and on the back where the battery housing is located.

Simon Harriott


My first impression after unboxing the Q702 is that it is a nice looking unit. Styling is somewhat minimalist but very effective. The tablet part, once detached, has a nice weight, and no buttons or switches are located in awkward or intrusive positions.


Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?