Hackathon geared toward the 'liberation' of data from public PDF documents

The Sunlight Foundation and others will sponsor a three-day hackathon starting Friday

Massive amounts of unstructured data are held in the form of PDF documents, but extracting key figures and words out of PDFs in a programmatic manner can be difficult and costly. This poses a challenge to public-interest groups, journalists and others who are interested in running large-scale analyses on PDF documents in order to uncover valuable insights.

In a hackathon set for this week, participants will work on ways to improve the open-source software tools available for PDF data extraction.

"Say, for example, you want to model student loan securitizations," wrote Marc Joffe, principal consultant at Public Sector Credit Solutions and an organizer along with the Sunlight Foundation and others of the PDF Liberation Hackathon, in a guest post on the Mathbabe blog. "A corporation or well funded research institution can purchase an expensive, enterprise-level ETL (Extract-Transform-Load) tool to migrate data from the PDFs into a database. But this is not much help to insurgent modelers who want to produce open source work."

"Data journalists face a similar challenge," he added. "They often need to extract bulk data from PDFs to support their reporting. Examples include IRS Form 990s filed by non-profits and budgets issued by governments at all levels."

Data journalists have developed open-source PDF harvesting tools such as Tabula, Joffe added.

"Unfortunately, the free and low cost tools available to modelers, data journalists and transparency advocates have limitations that hinder their ability to handle large scale tasks," he wrote. "If, like me, you want to submit hundreds of PDFs to a software tool, press 'Go' and see large volumes of cleanly formatted data, you are out of luck."

The hackathon runs from Friday through Sunday and will be held at six sites, including the Sunlight Foundation's headquarters in Washington, D.C., according to the event's website. Remote participation is also possible.

Contestants will be able to work on "a PDF extraction challenge provided by one of our sponsoring organizations, can work on their own challenges or develop enhancements to an open source PDF extraction tool," according to the site.

While the use of open-source tools is encouraged, commercial tools are allowed as long as licensing costs less than US$1,000 and an unlimited trial is available.

It's true that some of the best tools for PDF extraction are proprietary and expensive, said analyst Curt Monash of Monash Research, who closely tracks the database and data-analysis market as well as public policy on technology.

"One of the leading filter/extraction libraries was bought by Verity, which was bought by Autonomy, which was bought by HP," he said via email on Thursday. "Another one, with a somewhat different orientation, was developed by Xerox, which spun it out as Inxight, which was bought by Business Objects, which was bought by SAP."

"It's worth remembering that there's a multi-stage process here," Monash added. "For example, a PDF can be converted to text (and image) data, (Name, value) pairs can be extracted. Those can have their spelling corrected. Then the company names can be regularized. In real life, there can be tens of steps."

As for the hackathon's potential value, "a large fraction of the world's interesting information is on paper, or in paper-like formats such as PDF," he added. "Of course it's worthwhile to make all that more accessible."

Chris Kanaracus covers enterprise software and general technology breaking news for The IDG News Service. Chris' email address is Chris_Kanaracus@idg.com

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Tags application developmentSunlight Foundationapplicationssecuritylegalsoftwareinternetprivacy

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Chris Kanaracus

IDG News Service
Show Comments

Essentials

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?