IBM speeds storage retrieval with flash drives

IBM has demonstrated a high-speed storage system that uses solid-state memory and a parallel file system

With an eye toward helping tomorrow's data-deluged organizations, IBM researchers have created a super-fast storage system capable of scanning in 10 billion files in 43 minutes.

This system handily bested their previous system, demonstrated at Supercomputing 2007, which scanned 1 billion files in three hours.

Key to the increased performance was the use of speedy flash memory to store the metadata that the storage system uses to locate requested information. Traditionally, metadata repositories reside on disk, access to which slows operations.

"If we have that data on very fast storage, then we can do those operations much more quickly," said Bruce Hillsberg, director of storage systems at IBM Research Almaden, where the cluster was built. "Being able to use solid-state storage for metadata operations really allows us to do some of these management tasks more quickly than we could ever do if it was all on disk."

IBM foresees that its customers will be grappling with a lot more information in the years to come.

"As customers have to store and process large amounts of data for large periods of time, they will need efficient ways of managing that data," Hillsberg said.

For the new demonstration, IBM built a cluster of 10 eight-core servers equipped with a total of 6.8 terabytes of solid-state memory. IBM used four 3205 solid-state Storage Systems from Violin Memory. The resulting system was able to read files at a rate of almost 5 GB/s (gigabytes per second).

The system used a tuned version of IBM's General Parallel File System (GPFS), version 3.4. Originally developed for high-performance computing systems, GPFS is becoming increasingly relevant for other data-heavy enterprise workloads as well, Hillsberg said. GPFS allows all the processor cores to write to and from disks in parallel, which can significantly improve storage system responsiveness.

Today's file systems are not well-suited for managing data across multiple storage systems as a single namespace, Hillsberg explained. The 2007 demonstration showed how a parallel file system such as GPFS could be used as the basis for highly scalable storage systems. The new work demonstrates how such a system could be improved even more with the addition of solid-state disks.

The researchers posted a white paper that describes the architecture in a level of detail that could help other parties recreate similar systems.

IBM may also fold the ideas into its own products, Hillsberg said. Earlier IBM research work in building experimental solid-state systems has led to the creation of new software, such as IBM Easy Tier, which helps systems automatically balance data between solid-state disks and regular disks.

"I think you will see some really interesting things come out of this research," Hillsberg said of the demonstration.

IBM is not alone in its enthusiasm for using solid-state storage as a way to speed operations. In this month's Association for Computing Machinery "Communications" publication, a group of researchers from Carnegie Mellon University and Intel Labs described a server architecture that combines low-power processors and flash memory, a design that could significantly speed operations for transaction-heavy large websites.

Like IBM's setup, the researchers' Fast Array of Wimpy Nodes (FAWN) architecture only requires a relatively small amount of flash memory, on which the most frequently consulted data can be stored. They noted that while solid-state storage can cost 10 times as much as traditional disks, they can offer 100 percent performance boost.

The idea of building flash-memory-assisted servers "is not that far out. The technology already exists," said Luiz André Barroso, a Google distinguished engineer who was not involved in FAWN.

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Tags Storage serversIBMstorageStorage Management

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Essentials

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?