Limited time offer









Why a Security Product Fails a Test. And Why It Is Redeemed.

You might think that testing security software is an exercise in scientific, unbiased objectivity - a study of mathematical precision. This might lead you to think that such test results are The Truth, and that there is no room for discussion. If that’s what you think, you’re wrong. Partly wrong, at least.

The test results drew headlines in tech media around the world, with Neil Rubenking of PC Mag taking the lead in reporting that, basically, most security products for Android phones failed the test miserably. As a result, the smartphone owners using the products would be at risk whenever they went online from their phone, even though they pay good money to stay safe from malware.

Among those who scored really poorly in the test was … yes, you guessed it: BullGuard! Very upsetting, we thought, as we’re generally not used to scoring low on malware detection.

Other companies, whose products did poorly, were also quite taken aback by the results. One of those companies, MYMobileSecurity, very quickly got in touch with AV Test Labs, and voiced their frustration with the results, causing AV-Test Labs to revisit their test methods.

It turned out that an incompatibility between many of the products and the testing environment made the product appear ineffective. Nice to get that confirmed. Even if it wasn’t entirely surprising to the industry because, as the wide range in detection results shows, the testers are mapping completely unknown territories.

The new tests proved worthwhile, as many of the products that at first had been dismissed as poor performers, now came into their own and proved their effectiveness.

For BullGuard, this meant that our Mobile Security product ended up with a catch rate of 65%-90%, and as AV-Test Labs point out in their summary:

“Products with a detection rate between 90% and 65% are still very good and could move to the top range depending on changes to the tested malware set. Some of these products just miss one or two malware families, which might be not prevalent in certain environments.”

All in all, the score is well above average and showed BullGuard to miss the elite +90% group by a hairsbreadth since we provided excellent detection rates for more than half of the tested malware families, and well above average detection for other 5 types. The other 3 families where our detection was either average or the family was missed them, tell us where we should direct our efforts. Improving those areas will certainly put us in the +90% detection group.

But how does it happen in the first place?

There is no question that AV-Test Labs do a very good job when it comes to testing security products – they’ve tested PC security products for  years, and I think it’s safe to say that they have the respect of the industry, as well as that of the media covering it.

Here’s the deal, though:

  • This is the first sizeable malware test for the Android platform performed by AV-test Labs and mobile security is still unchartered waters. Android malware testing is a brand new discipline, and therefore there is no well-established methodology for these kinds of tests.
  • The very way the Android platform works is a challenge to testing which directly influences how testing is performed.
  • Testing rounds are currently mostly hands-on approaches by manually copying malware packages and installing them one by one. That’s tedious work and making sure all possible test scenarios are covered takes quite some time, thus limiting the amount of malware samples used during the tests.

So bottom line: Testing a new technology is not as precise and sophisticated a discipline as you might think. But naturally, as the market matures, test methods will improve. And this first large scale Android malware testing round will prove to be quite useful by showing areas that security software developers will want to focus on.

Until then, it’s certainly worthwhile to keep this note in mind which AV-Test Labs included in their original comments to the test results:

“Please bear in mind, that malware may not be the only or the most important threat to your Anti-Malware solutions for Android 4 device. Even if a product scores poorly in malware detection it may have other convenient features, such as remote lock and wipe, backup and phone locating, that make it useful for your purposes. It is also possible to run two or more security apps on your device at the same time, using only the best features of the single apps.” 

What is the most important feature for you, when it comes to mobile security? Antivirus protection or features like Antitheft and  Parental Control. Or maybe online backup?


PS from BullGuard:


We’d like to thank AV-Test Labs for the way they’ve handled this. The entire test was a very good initial evaluation of the AV engines for the Android platform and we are really grateful for AV-Test’s cooperation and prompt reaction in updating the test results.

Written by Katrine Palsby

More articles by Katrine Palsby

Leave a Reply




Please enter the code

Please enter the captcha code!

Security code

Ranked #1 by industry experts

BullGuard Internet Security Cup

Internet Security

Free download
We use cookies to ensure that we give you the best experience on our website. By continuing to browse, we are assuming that you have no objection in accepting cookies. You can change your cookie settings at any time.