Twitter revamps search engine

It's now much faster and scalable

Twitter has overhauled the back-end infrastructure of its search engine, boosting its speed and capacity to index posts, process queries and deliver results, while making the system more stable and better suited for the addition of new features.

Twitter transferred its search engine to the new platform in recent weeks, after working on the new back-end system for about six months, according to the company.

Twitter's search engine ran on a system based on MySQL from Summize, a company Twitter acquired in mid-2008, but scaling up the system had become difficult.

The engineering team in charge of the project decided to do an extreme makeover of the search engine using a different technology: the open-source, text search engine Lucene, which is written in Java.

Twitter modified some aspects of Lucene, including its garbage collection, query termination, posting lists, and data structures and algorithms, and was left with an inverted, index-based search engine that scales much more and has better performance.

Twitter fields 12,000 search queries per second -- or more than 1 billion per day -- and "tweets" become part of its search index less than 10 seconds after they are posted.

"We estimate that we're only using about 5 percent of the available backend resources, which means we have a lot of headroom. Our new indexer could also index roughly 50 times more Tweets per second than we currently get," Twitter official Michael Busch wrote in a blog post.

Twitter will contribute back to the Lucene project the modifications and improvements it made to the code.

Although Twitter makes available its index of "tweets" to external search engines like Google and Microsoft's Bing, its internal search engine is a key component of its microblogging service.

In addition to being the preferred vehicle among private citizens, public figures and companies for broadcasting short status updates, Twitter has become an increasingly valued repository of real-time data, tapped for following news, trends and collective musings.

To maximize the value of this "tweet" repository, the company must have a search engine that is fast, comprehensive and scalable, and its massive revamping of its search technology shows the company recognizes the importance of its internal search capabilities.

Join the newsletter!

Error: Please check your email address.
Rocket to Success - Your 10 Tips for Smarter ERP System Selection

Tags Internet-based applications and servicestwittersocial mediainternetsearch engines

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Juan Carlos Perez

IDG News Service
Show Comments

Most Popular Reviews

Latest Articles

Resources

PCW Evaluation Team

Ben Ramsden

Sharp PN-40TC1 Huddle Board

Brainstorming, innovation, problem solving, and negotiation have all become much more productive and valuable if people can easily collaborate in real time with minimal friction.

Sarah Ieroianni

Brother QL-820NWB Professional Label Printer

The print quality also does not disappoint, it’s clear, bold, doesn’t smudge and the text is perfectly sized.

Ratchada Dunn

Sharp PN-40TC1 Huddle Board

The Huddle Board’s built in program; Sharp Touch Viewing software allows us to easily manipulate and edit our documents (jpegs and PDFs) all at the same time on the dashboard.

George Khoury

Sharp PN-40TC1 Huddle Board

The biggest perks for me would be that it comes with easy to use and comprehensive programs that make the collaboration process a whole lot more intuitive and organic

David Coyle

Brother PocketJet PJ-773 A4 Portable Thermal Printer

I rate the printer as a 5 out of 5 stars as it has been able to fit seamlessly into my busy and mobile lifestyle.

Kurt Hegetschweiler

Brother PocketJet PJ-773 A4 Portable Thermal Printer

It’s perfect for mobile workers. Just take it out — it’s small enough to sit anywhere — turn it on, load a sheet of paper, and start printing.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?