AOL installs 50TB of SSD; boosts DB performance by 4X

SSD SAN installed to eliminate I/O bottlenecks caused by backend storage

GRAPEVINE, Texas -- When it came to managing most of AOL's 6 petabytes of data, a Fibre Channel SAN sufficed. But for its most critical relational database, AOL found that the SAN was too constrained and caused its IT shop to miss business unit service levels more than 50% of the time.

After investigating what may have been causing I/O bottlenecks, AOL found the problem was backend storage. To fix the problem, AOL decided to build a 50TB storage area network (SAN) from solid state technology.

The upgrade worked, increasing throughput to the SQL database by four times over the Fibre Channel SAN, while still providing storage admins the flexibility to migrate data between storage systems because the NAND flash memory sat behind an existing virtualization appliance, which aggregates all of the backend storage and serves it up as from a single pool.

AOL engineers found that the throughput problems didn't appear to be based on hardware. The company has five large high-end arrays with 15,000 or 10,000 rpm drives for primary storage and two lower-end arrays with serial ATA (SATA) drives for nearline backup. But the storage arrays could only feed data to internal drives as fast as the SAS backbone allowed, which was about 6Gbit/sec.

AOL's storage infrastructure supports about 4,000 servers that feed information to both online users as well as to the company's own backend applications.

Dan Pollack, senior operations architect for AOL, said he considered adding solid state drives to the company's existing legacy storage arrays, but decided against it because they would be held back by the array's controller bandwidth.

"So you find you can take a high-performance SSD device and half the performance is lost at the head of the array," Pollack told an audience at the Storage Networking World conference here this week.

Pollack also considered using solid state drives in servers, but didn't because he would not have been able to cluster them into his SAN, they don't offer the required capacity, such drives could not be non-disruptively swapped out and data couldn't be migrated between systems.

Pollack settled on a solid state array from Violin Memory . It could plug directly into his Fibre Channel network, and could sit behind his storage virtualization appliance from YadaYada, which was purchased last year by storage giant EMC.

Pollack said the rollout, including planning and testing, took only eight weeks. The system has been live since July 1. Since then, the SSD array has had zero downtime and it has allowed his shop to meet all of the business SLAs without exception. I/O response times are typically less than 1 millisecond and there was no impact in his IT team's ability to manage the backend storage.

"It's very easy to fall in love with this stuff once you're on it," he said.

But love can be expensive. Without offering an exact price tag, Pollack said the solid state array cost AOL about $20 per gigabyte, which with 50TB of capacity, adds up to about $1 million.

Pollack said that money was less of an obstacle than that alternative -- throwing more manpower at the database problems.

The flash array has RAID 5 internal protection, so the drives are hot swappable. For added protection, Pollack configured the "preliminary" installation so that it mirrored data across two clusters of Violin appliances -- each cluster being made up of six Violin appliances.

"Because this is all new and unproven, and this is a tier one heavily visible application, we felt it was prudent to spend the extra money and time and provide that additional protection. In the future we won't do that," he said.

And, while the array didn't offer the 1 million-plus I/Os per second (IOPS) that vendor marketing material boasts, it does come in at around 250,000 IOPS, which Pollack said is more than enough for his purposes.

There were also significant cost savings associated with the flash memory, he added.

For one, Pollack said, Fibre Channel arrays often only use about 10% of the capacity in its top tier drives because storage admins often short-stroke the drives, using only the outer sectors of the drive to increase I/O response times. That can eat as much as 20 kilowatts of power per array, he said.

By comparison, the flash array only uses 2 kilowatts and 90% of its capacity is utilized. On top of that, the Violin storage array takes up less floor space and produces less heat, which also helps to reduce power required from HVAC systems.

Violin doesn't use NAND flash in hard drive form factors, as many solid state companies do today. Instead it uses flash chips directly on cards called Violin Intelligent Memory Modules (VIMMs). VIMMs are similar to DIMMs, only built out of flash instead of DRAM.

"So you're getting the 4GB/sec of PCIe bandwidth not the 5Gbit/sec or 6 Gbit/sec SAS bandwidth. You're getting almost an order of magnitude of bandwidth to the storage internally just because you're using an interface that's capable of it," Pollack said

Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian , or subscribe to Lucas's RSS feed . His e-mail address is .

Read more about storage in Computerworld's Storage Topic Center.

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Tags network-attached storageNetworked Storagestoragestorage hardware

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Lucas Mearian

Computerworld (US)
Show Comments

Most Popular Reviews

Latest News Articles


PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?