Lies, damned lies and benchmarks

Volkswagen cheated on the emissions tests for 11 million vehicles. But it’s not the only company that has built lying into its testing and benchmarking.

It always struck me, as I recoiled from the stink of diesel fumes on the interstate, that the notion of “clean” diesel had to be wrong. It turns out I was right — at least in the case of Volkswagen, which cheated on emissions tests for about 11 million vehicles. Now countries are banning sales of the makes that had been programmed to lie about pollution results.

Shocking? Only if you think that VW is the only company that’s ever shipped products that faked test results.

If you haven’t heard about it, several models of diesel VW vehicles built between 2009 and 2015 and diesel Audi A3 models from 2010 to 2015 had software installed in their electronic control modules that sensed when an emissions test was being conducted.

When the car figured out that it was being tested, which it seemed able to do almost perfectly, it would switch to “clean mode.” According to the Environmental Protection Agency, “This results in cars that meet emissions standards in the laboratory or testing station, but during normal operation, emit nitrogen oxides, or NOx, at up to 40 times the standard.”

Able to tout great numbers on the test stand with no performance hit on the road, VW was able to post excellent financial numbers — until it was caught. Now VW CEO Martin Winterkorn has resigned because of the cheating scandal.

Oh, well, so much for VW’s next quarterly results.

Why did the third-biggest carmaker in the world do it? Because cheating was easier than building competitive “clean” diesel vehicles.

When the U.S. adopted stricter pollution rules in 2008 (with the EU following in 2015), VW was caught between a rock and a hard place. It found it impossible to build clean diesels that had good fuel mileage. So it programmed its cars to fib.

I am not among those who find this sordid tale surprising.

I’ve been running and building benchmarks for close to three decades now, and I’ve been an expert witness in cases involving them. And I can tell you that tech companies have been double-dealing on tests since day one.

Back in the 1980s, I and other technology journalists discovered a brand of Ethernet card that knocked the others out of the race. How could it be so much faster than the others? It cheated.

When this card “saw” a then common traffic pattern involving Novell network interface cards (NIC), as my colleague Wayne Rash observed, “it simply passed packets through without processing.

There were others. Many others.

Just a few years after the NIC cheat, for example, Rash encountered a brand of Gigabit Ethernet switches that had results that were too good to be believed. In this case, when tested with real-world data, the switch not only had poor performance, it “failed spectacularly,” as Rash put it.

Another writer and programming buddy of mine, Stephen Satchell, reminded me of another case. This time it was a software trick very reminiscent of what VW did. As he recounted this incident in a recent email, a “C compiler (names removed to protect the guilty) would detect that the Sieve of Eratosthenes [a simple algorithm for finding prime numbers often used to test the speed of compilers] was being compiled, and would inject handcrafted intermediate code that had been highly optimized.” This made the compiler look faster, although the trick could sometimes cause programs to break.

In the ’90s, I also encountered a brand of graphic cards that always won our benchmarks. They were good, but that good? We didn’t think so.

So we looked closer and found that the cards were set to detect our benchmarks. When they did, they either speeded up the GPU or output the correct graphics from embedded memory, depending on the particular cheat.

Like VW, all these companies took shortcuts to make more money on the cheap. People willing to break rules to look better than the competition in “objective” benchmarks and tests will always be with us.

So whenever you see benchmark results that appear too good to be true, go with your gut reaction. It’s all too likely the vendor has been playing games with numbers just so it can swindle you out of some money.

Join the Good Gear Guide newsletter!

Error: Please check your email address.

Our Back to Business guide highlights the best products for you to boost your productivity at home, on the road, at the office, or in the classroom.

Keep up with the latest tech news, reviews and previews by subscribing to the Good Gear Guide newsletter.

Steven J. Vaughan-Nichols

Computerworld (US)
Show Comments

Essentials

Microsoft L5V-00027 Sculpt Ergonomic Keyboard Desktop

Learn more >

Lexar® JumpDrive® S57 USB 3.0 flash drive

Learn more >

Mobile

Lexar® JumpDrive® S45 USB 3.0 flash drive 

Learn more >

Exec

Audio-Technica ATH-ANC70 Noise Cancelling Headphones

Learn more >

Lexar® Professional 1800x microSDHC™/microSDXC™ UHS-II cards 

Learn more >

HD Pan/Tilt Wi-Fi Camera with Night Vision NC450

Learn more >

Lexar® JumpDrive® C20c USB Type-C flash drive 

Learn more >

Budget

Back To Business Guide

Click for more ›

Most Popular Reviews

Latest News Articles

Resources

PCW Evaluation Team

Azadeh Williams

HP OfficeJet Pro 8730

A smarter way to print for busy small business owners, combining speedy printing with scanning and copying, making it easier to produce high quality documents and images at a touch of a button.

Andrew Grant

HP OfficeJet Pro 8730

I've had a multifunction printer in the office going on 10 years now. It was a neat bit of kit back in the day -- print, copy, scan, fax -- when printing over WiFi felt a bit like magic. It’s seen better days though and an upgrade’s well overdue. This HP OfficeJet Pro 8730 looks like it ticks all the same boxes: print, copy, scan, and fax. (Really? Does anyone fax anything any more? I guess it's good to know the facility’s there, just in case.) Printing over WiFi is more-or- less standard these days.

Ed Dawson

HP OfficeJet Pro 8730

As a freelance writer who is always on the go, I like my technology to be both efficient and effective so I can do my job well. The HP OfficeJet Pro 8730 Inkjet Printer ticks all the boxes in terms of form factor, performance and user interface.

Michael Hargreaves

Windows 10 for Business / Dell XPS 13

I’d happily recommend this touchscreen laptop and Windows 10 as a great way to get serious work done at a desk or on the road.

Aysha Strobbe

Windows 10 / HP Spectre x360

Ultimately, I think the Windows 10 environment is excellent for me as it caters for so many different uses. The inclusion of the Xbox app is also great for when you need some downtime too!

Mark Escubio

Windows 10 / Lenovo Yoga 910

For me, the Xbox Play Anywhere is a great new feature as it allows you to play your current Xbox games with higher resolutions and better graphics without forking out extra cash for another copy. Although available titles are still scarce, but I’m sure it will grow in time.

Featured Content

Latest Jobs

Don’t have an account? Sign up here

Don't have an account? Sign up now

Forgot password?