Aiming for SEC's big data project, Sungard and Google bet on the cloud

Sungard is using Google cloud services to prototype a system that stores and formats all U.S. financial trading data

At the Google Next roadshow in New York, held on June 12, the company extolled the virtues of cloud computing.

At the Google Next roadshow in New York, held on June 12, the company extolled the virtues of cloud computing.

To bid for a massive U.S. government contract, Sungard has teamed up with Google to build a prototype cloud system that could store six years' worth of U.S. stock and financial trading data and let regulators and stock traders scrutinize this mountain of information.

The work is being done to compete for a U.S. Securities and Exchange Commission contract, called the Consolidate Auditing Trail (CAT). The SEC's goal is to build a system that provides more transparency into financial markets, a response, in part, to the computer-driven 2010 "flash crash" that briefly cratered U.S. stock prices.

"The CAT is a huge undertaking," said Neil Palmer, Sungard's chief technology officer for its consulting services practice. "It is the biggest big data problem in the financial industry today."

Palmer described the prototype Friday at the Google Next user conference in New York. Sungard, a provider of financial software and services, is one of six finalists for the work, and has partnered with Google for technology infrastructure.

The flexibility of cloud computing provides the ability for Sungard to pursue such an ambitious job, Palmer told a group of reporters after the keynote.

With building a system in-house, "there are just too many unknowns," he said, referring to the intense hardware and operational demands that would come with building an on-premises system to execute this work.

The system will cost anywhere from $350 million to $1 billion to build, the SEC has estimated.

Once operational, CAT will generate a tremendous amount of data, Palmer said. The system must record every quote and every trade from every financial company participating in the public U.S. markets. The companies must submit their data on a daily basis, and the system must keep this data for six years.

Each day, the system will ingest about 50 terabytes of data, comprised of about 100 billion events. The six year window of when records will be actively kept will amount to about 30 petabytes of data, Sungard estimated.

All this data must be validated, indexed, and posted within four hours.

Tools must also be available to query all this data. "There is no point storing that much data and not being able to generate any actionable information from it," Palmer said.

In addition to the SEC, cloud accessible financial data could also be of immense value to the financial firms themselves, Palmer noted. A comprehensive centralized copy of all financial trading information would reduce the needs for such firms to store that data in house. They could test algorithms on the history market data, to see how will they can predict upcoming changes.

Sungard assembled the infrastructure for the prototype using a variety of Google Cloud Platform components.

The prototype uses Google Cloud Storage to hold the data, and Google BigTable to structure the information. A Google Dataflow service can validate the data. Google's Big Query can be used to publish the data and provide a way for users to analyze it, either directly or through third-part software such as Microsoft Excel or various business intelligence (BI) packages.

In the first round of tests, the prototype was able to process 10 billion events an hour, or about three gigabytes of data process a second.

Best of all, the prototype was built in six weeks.

"There is no way we could have done that, even with similar technologies, if we had to stand up our own infrastructure," Palmer said.

The team still has work to do, Palmer said. The system must be able to ingest data at four times the speed of the current prototype.

If Sungard wins the contract, or if another team with a cloud-based approach wins, it will represent a significant step towards the acceptance of the cloud computing model in the U.S. financial industry, which to date has been reluctant to embrace the approach.

Sungard's testimony was thematically in tune with the overall theme of the keynote, which was how cloud computing can give companies resources to forge into new markets that they otherwise wouldn't have had access to.

Carl Schachter, Google vice president of the cloud platform, said that companies like Uber and AirBnB have used cloud computing to disrupt traditional markets. "Markets that have been previously thought established are now re-inventable," he said.

The other finalists for the CAT contract are Epam systems; Thesys; the Financial Industry Regulatory Authority (FINRA); a team comprised of AxiomSL and Computer Sciences Corporation; and a consortium of companies including Hewlett Packard and Booz Allen.

The SEC has not offered a date when it expects to award the contract.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Tags Googlesungardcloud computinginternetInfrastructure services

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Joab Jackson

IDG News Service
Show Comments

Most Popular Reviews

Latest News Articles


PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?