Lies, damned lies and benchmarks

Volkswagen cheated on the emissions tests for 11 million vehicles. But it’s not the only company that has built lying into its testing and benchmarking.

It always struck me, as I recoiled from the stink of diesel fumes on the interstate, that the notion of “clean” diesel had to be wrong. It turns out I was right — at least in the case of Volkswagen, which cheated on emissions tests for about 11 million vehicles. Now countries are banning sales of the makes that had been programmed to lie about pollution results.

Shocking? Only if you think that VW is the only company that’s ever shipped products that faked test results.

If you haven’t heard about it, several models of diesel VW vehicles built between 2009 and 2015 and diesel Audi A3 models from 2010 to 2015 had software installed in their electronic control modules that sensed when an emissions test was being conducted.

When the car figured out that it was being tested, which it seemed able to do almost perfectly, it would switch to “clean mode.” According to the Environmental Protection Agency, “This results in cars that meet emissions standards in the laboratory or testing station, but during normal operation, emit nitrogen oxides, or NOx, at up to 40 times the standard.”

Able to tout great numbers on the test stand with no performance hit on the road, VW was able to post excellent financial numbers — until it was caught. Now VW CEO Martin Winterkorn has resigned because of the cheating scandal.

Oh, well, so much for VW’s next quarterly results.

Why did the third-biggest carmaker in the world do it? Because cheating was easier than building competitive “clean” diesel vehicles.

When the U.S. adopted stricter pollution rules in 2008 (with the EU following in 2015), VW was caught between a rock and a hard place. It found it impossible to build clean diesels that had good fuel mileage. So it programmed its cars to fib.

I am not among those who find this sordid tale surprising.

I’ve been running and building benchmarks for close to three decades now, and I’ve been an expert witness in cases involving them. And I can tell you that tech companies have been double-dealing on tests since day one.

Back in the 1980s, I and other technology journalists discovered a brand of Ethernet card that knocked the others out of the race. How could it be so much faster than the others? It cheated.

When this card “saw” a then common traffic pattern involving Novell network interface cards (NIC), as my colleague Wayne Rash observed, “it simply passed packets through without processing.

There were others. Many others.

Just a few years after the NIC cheat, for example, Rash encountered a brand of Gigabit Ethernet switches that had results that were too good to be believed. In this case, when tested with real-world data, the switch not only had poor performance, it “failed spectacularly,” as Rash put it.

Another writer and programming buddy of mine, Stephen Satchell, reminded me of another case. This time it was a software trick very reminiscent of what VW did. As he recounted this incident in a recent email, a “C compiler (names removed to protect the guilty) would detect that the Sieve of Eratosthenes [a simple algorithm for finding prime numbers often used to test the speed of compilers] was being compiled, and would inject handcrafted intermediate code that had been highly optimized.” This made the compiler look faster, although the trick could sometimes cause programs to break.

In the ’90s, I also encountered a brand of graphic cards that always won our benchmarks. They were good, but that good? We didn’t think so.

So we looked closer and found that the cards were set to detect our benchmarks. When they did, they either speeded up the GPU or output the correct graphics from embedded memory, depending on the particular cheat.

Like VW, all these companies took shortcuts to make more money on the cheap. People willing to break rules to look better than the competition in “objective” benchmarks and tests will always be with us.

So whenever you see benchmark results that appear too good to be true, go with your gut reaction. It’s all too likely the vendor has been playing games with numbers just so it can swindle you out of some money.

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Steven J. Vaughan-Nichols

Computerworld (US)
Show Comments

Cool Tech

Crucial Ballistix Elite 32GB Kit (4 x 8GB) DDR4-3000 UDIMM

Learn more >

Gadgets & Things

Lexar® Professional 1000x microSDHC™/microSDXC™ UHS-II cards

Learn more >

Family Friendly

Lexar® JumpDrive® S57 USB 3.0 flash drive 

Learn more >

Stocking Stuffer

Plox Star Wars Death Star Levitating Bluetooth Speaker

Learn more >

Christmas Gift Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

GGG Evaluation Team

Kathy Cassidy

STYLISTIC Q702

First impression on unpacking the Q702 test unit was the solid feel and clean, minimalist styling.

Anthony Grifoni

STYLISTIC Q572

For work use, Microsoft Word and Excel programs pre-installed on the device are adequate for preparing short documents.

Steph Mundell

LIFEBOOK UH574

The Fujitsu LifeBook UH574 allowed for great mobility without being obnoxiously heavy or clunky. Its twelve hours of battery life did not disappoint.

Andrew Mitsi

STYLISTIC Q702

The screen was particularly good. It is bright and visible from most angles, however heat is an issue, particularly around the Windows button on the front, and on the back where the battery housing is located.

Simon Harriott

STYLISTIC Q702

My first impression after unboxing the Q702 is that it is a nice looking unit. Styling is somewhat minimalist but very effective. The tablet part, once detached, has a nice weight, and no buttons or switches are located in awkward or intrusive positions.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?