Group releases guidelines for future AV software tests

The guidelines seek to account for other security technology that can detect malware for fair tests

A coalition of security companies and researchers has agreed on guidelines for how security software products should be tested, which may help put an end to long-running disputes about different testing methodologies.

Two sets of guidelines covering principles for testing security software for performance and testing entire security suites were adopted by the Anti-Malware Testing Standards Organization (AMTSO) at its latest meeting in Helsinki.

The guidelines are part of a series of documents on AMTSO's Web site aimed at introducing a set of standards broadly agreed through the industry as appropriate for ranking the effectiveness of security software.

Security companies have bickered over tests run by organizations such as and Virus Bulletin and others, which regularly test and rank security software.

Among the contentious issues are the age of the malicious software samples used to test antivirus programs. Companies that failed to detect certain kinds of malware have argued in the past that the samples weren't threats any more and that they had removed the signatures from their databases.

Other companies have countered that their failed test wasn't accurate since other technologies in their software would have stopped the particular threat. Some antivirus testing programs perform static testing, where an antivirus engine is run against a set of samples.

But many security companies have incorporated other complex ways to detect malware. One such method, known as behavioral detection, checks to see how malware behaves on a computer.

The new guidelines are voluntary, but many testing organizations have agreed to go along with them, such as Virus Bulletin, AV-Comparatives, West Coast Labs and ICSA Labs, said David Harley, an AMTSO board member and director of malware intelligence for ESET. will also use the guidelines, according to Andreas Marx, who founded the company.

"We're just trying to get people to think harder their methodologies so that they actually make sense," Harley said. "It doesn't mean you can't do things different ways, it just means you have to try and conform to a rationality."

Virus Bulletin has contributed to the guidelines covering performance testing, said John Hawes, technical consultant and test team director.

"We've already started implementing some of the ideas developed while discussing and designing this document, with some major expansions to the performance data we report in our comparatives in recent months and more improvements on the way," Hawes said. "We're also hard at work developing a new style of test which will allow us to measure the full range of features in many of today's security solutions."

The guidelines may not entirely end the feuding between security companies and testing organizations but will better inform those who write reviews of security software, said Stuart Taylor, manager of Sophos' threat lab and chairman of the board for AMTSO.

"What we are trying to do is get everyone involved to think more widely about decisions made in designing tests," Taylor said. "In this case we are talking about testing a whole product and not just part of a product, which can be very misleading, and also about measuring performance where it is so easy to create a performance test that is really just meaningless."

AMTSO has reached agreement with many security companies, but the nonprofit organization is still viewed with some suspicion by other consumer organizations outside of the security industry that do security software testing, Harley said.

AMTSO's membership comprises mostly vendors, but the organization will work to communicate that the standards that have been developed are sound and can be trusted to provide more accurate test results, Harley said. "That's something we are going to have to work on," Harley said.

Send news tips and comments to

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags AV-Test.orgAnti-Malware Testing Standards Organization (AMTSO)

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Jeremy Kirk

IDG News Service
Show Comments

Cool Tech

Toys for Boys

Family Friendly

Stocking Stuffer

SmartLens - Clip on Phone Camera Lens Set of 3

Learn more >

Christmas Gift Guide

Click for more ›

Brand Post

Most Popular Reviews

Latest Articles


PCW Evaluation Team

Aysha Strobbe

Microsoft Office 365/HP Spectre x360

Microsoft Office continues to make a student’s life that little bit easier by offering reliable, easy to use, time-saving functionality, while continuing to develop new features that further enhance what is already a formidable collection of applications

Michael Hargreaves

Microsoft Office 365/Dell XPS 15 2-in-1

I’d recommend a Dell XPS 15 2-in-1 and the new Windows 10 to anyone who needs to get serious work done (before you kick back on your couch with your favourite Netflix show.)

Maryellen Rose George

Brother PT-P750W

It’s useful for office tasks as well as pragmatic labelling of equipment and storage – just don’t get too excited and label everything in sight!

Cathy Giles

Brother MFC-L8900CDW

The Brother MFC-L8900CDW is an absolute stand out. I struggle to fault it.

Luke Hill


I need power and lots of it. As a Front End Web developer anything less just won’t cut it which is why the MSI GT75 is an outstanding laptop for me. It’s a sleek and futuristic looking, high quality, beast that has a touch of sci-fi flare about it.

Emily Tyson

MSI GE63 Raider

If you’re looking to invest in your next work horse laptop for work or home use, you can’t go wrong with the MSI GE63.

Featured Content

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?