IBM speeds up data analysis with new algorithm

The algorithm could help to more quickly predict weather and financial risks

IBM researchers have developed a new algorithm that could in minutes analyze terabytes' worth of raw data to more quickly predict weather and electricity usage, the company said Thursday.

The mathematical algorithm, developed by IBM's laboratories in Zurich, can sort, correlate and analyze millions of random data sets, a task that could otherwise take days for supercomputers to process, said Costas Bekas, a researcher at IBM.

The algorithm is just under a thousand lines of code and will be instrumental in establishing usage patterns or trends based on data gathered from sources such as sensors or smart meters, he said. The algorithm could be used to analyze a growing mass of data measuring electricity usage trends as well as air or water pollution levels. The algorithm could also break down data from global financial markets and assess individual and collective exposure to risk, Bekas said.

"We are interested in measuring the quality of data," Bekas said. Efficient analysis of large data sets requires new mathematical techniques that reduce computational complexity, Bekas said.

The algorithm combines models of data calibration and statistical analysis that can assess measurement models and hidden relationships between data sets. IBM has been working on the research for two years, Bekas said.

The algorithm can also reduce the cost burden on companies by analyzing data in a more energy-efficient way, Bekas said. The lab used a Blue Gene/P Solution system at the Forschungszentrum Julich research center in Germany to validate 9TBs of data in less than 20 minutes. Analyzing the same amount of data without the algorithm would have taken a day with the supercomputer operating at peak speeds, which would have added up to higher electricity bills, Bekas said.

According to Top500.org, the Blue Gene/P is the fourth-fastest supercomputer in the world as of last November, with 294,912 IBM Power processing cores that can provide peak performance of up to 1 petaflop.

The traditional approach to data analysis is to take multiple data sets and look at them individually, said Eleni Pratsini, manager of mathematical and computational sciences at the IBM research labs. However, the algorithm compares data sets against each other, which could help enterprises point toward larger trends in particular areas, such as risk reduction in financial portfolios.

Enterprises will want faster ways of generating business intelligence as masses of data flood servers with the expansion of computing to new devices, he said.

Now that the algorithm has been proven to work scientifically, the research lab is collaborating with IBM's Global Services unit to use it for specific services, Pratsini said. Ultimately, the algorithm could make its way to IBM applications such as the SPSS statistical analysis software, but the company didn't provide a specific time frame for that.

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Tags analyticsIBMalgorithmsdata analysis

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Agam Shah

IDG News Service
Show Comments

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?