In Search of
Evidence-Based IT-Security

Hanno Böck


Hanno Böck, freelance journalist and hacker.

Writing for and others.

Fuzzing Project, funded by Linux Foundation's Core Infrastructure Initiative.

Author of monthly Bulletproof TLS Newsletter.

IT Security Products

Picture: Hanno Böck, Black Hat USA 2016

Picture: Hanno Böck, Black Hat USA 2016

Picture: Hanno Böck, Black Hat USA 2016

Picture: Hanno Böck, Black Hat USA 2016

I'm a bit skeptical

Source: Twitter / Tavis Ormandy ( [1], [2], [3], [4])

Sources: PCWorld, The Register, April King/Twitter, Justin Schuh/Twitter.

New research: Comparing how security experts and non-experts stay safe online (Google Security)

There is considerable disagreement whether many IT security and especially Antivirus products are a good idea.

How do we know if these things work?

Let's look at something completely different

Another Anti-Virus

Pictures: CDC/Wikimedia Commons, Hanno Böck

Vitamin C against the Common Cold

It's probably not very useful.

Why do we know that?


Amitchell125, Wikimedia Commons, CC by-sa 3.0

Vitamin C

Regular ingestion of vitamin C had no effect on common cold incidence in the ordinary population [...]. However, regular supplementation had a modest but consistent effect in reducing the duration of common cold symptoms [...].

Trials of high doses of vitamin C administered therapeutically, starting after the onset of symptoms, showed no consistent effect on the duration or severity of common cold symptoms.

Vitamin C for preventing and treating the common cold, H. Hemilä, E. Chalker, Cochrane Library (2013)

Randomized Controlled Trial (RCT)

Randomly split patients in groups.

Simple: Group A gets medication, Group B gets placebo

More complex: Group A gets new medication, Group B gets best old medication, Group C does alternative to medication, e.g. exercise.

See who gets better.

Meta analyis

We don't care about single studies. We care about all the evidence we have.

Meta analysis: Pool results from all available studies.

Evidence-based Medicine

Ideally all decisions in medicine should be based on high quality scientific evidence.

Science has problems

Sources: PLOS One, Nature News, STAT

Spotting bad science (1)

Small number of research subject (underpowered studies).

Making causal claims although the data only supports a correlation (too many results purely rely on observational data).

Single or few studies. Good science needs to be replicated.

Spotting bad science (2)

Publication bias - only studies with "postive" results get published.

Fishing for results and outcome switching. (If we don't find X in our data, maybe we find something else.)

Ideally all empirical studies should be preregistered (but we're very far from that).

Now let's get back to IT security

This was an empty slide.

It was also the complete list of all randomized controlled trials ever done on the effectiveness of Antivirus applications or other IT security products.

Questionable Antivirus tests

There are some tests that compare Antivirus prudcts against each other (AV-Test, AV comparatives), but the methodology is extremely flawed.

Problems with naive Antivirus tests

If a software detects a malware it does not mean it would've caused harm if undetected.

Alternatives to Antivirus software are not considered.

Antivirus software as a security risk is not considered.

None of these tests are with real users.

Questionable statistics

Bad statistics about IT security are common.

The most notorious example is probably CVE counting.

(see also Black Hat USA 2013 - Buying into the Bias: Why Vulnerability Statistics Suck)

IT security is largely not based on Scientific Evidence

Wait, aren't there plenty of scientific papers and conferences on IT security?

Most Cited Academic Security & Crypto Papers

  1. Candidate Indistinguishability Obfuscation and Functional Encryption for all Circuits
    S Garg, C Gentry, S Halevi, M Raykova, A Sahai, B Waters, FOCS, 2013
  2. Candidate Multilinear Maps from Ideal Lattices.
    S Garg, C Gentry, S Halevi, Eurocrypt, 2013
  3. FRESCO: Modular Composable Security Services for Software-Defined Networks.
    S Shin, PA Porras, V Yegneswaran, MW Fong, G Gu, M Tyson, NDSS, 2013
  4. Homomorphic Encryption from Learning with Errors: Conceptually-Simpler, Asymptotically-Faster, Attribute-Based.
    C Gentry, A Sahai, B Waters, Crypto, 2013

Google Scholar Metrics 2013/2014

Papers that affect real software

  1. DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket.
    D Arp, M Spreitzenbarth, M Hubner, H Gascon, K Rieck, NDSS, 2014
  2. Lucky Thirteen: Breaking the TLS and DTLS Record Protocols
    NJ Al Fardan, KG Paterson, IEEE S&P, 2013

Not a single paper that tests with real users.

IT security and science

Most academic research in IT security is comparable to basic research.

Practical research tends to investigate interesting, but probably not very relevant parts of the problem.

Proposal for a randomized controlled trial on security software

Get a large group of users, randomly split them in groups:

  • n groups using Security products.
  • 1 group using "alternative treatment", e. g. automatic regular updates / application whitelisting.
  • 1 group gets a user training ("Don't click on that attachment").
  • Placebo (let them do whatever they did before).

Randomized controlled trial for security software

  • Measure security incidents.
  • Measure "side effects" (performance slowdowns, costs, downtimes, ...).
  • Compare result after some amount of time.

"But this is really hard!"

Yes, it is. Sorry, science is hard.


  • Ethics (situation comparable to medicine).
  • How to measure incidents?
  • Standard attacks versus targetted attacks?

Other things that could be tested

Safety of programming languages (e. g. Rust versus C++).

Application security (e. g. different browser brands).

An example that is both good and bad

FTC on Twitter

FTC and passwords

The Federal Trade Commission (FTC) found out they had no scientific evidence for their recommendation to change passwords often.

FTC recommends the opposite

Regular mandatory password changes are probably not a good idea.

We have studies that say so.


All of the studies are based on observational data, no intervention studies (Correlation != Causation). Nothing that comes close to a randomized controlled trial.

The studies measure things like password entropy, not real incidents (in medicine you would call that a surrogate endpoint).

FTC and passwords

Good: The FTC looked at the scientific evidence.

Not so good: The quality of the evidence was relatively low.

Caveats of Evidence-based IT-security

Future or very rare threats

Post-Quantum Cryptography: We want to protect against future attacks on cryptography.

Reproducible Builds: Protect against rare, but powerful attack scenario.

There are things you shouldn't study

Some IT security products make impossible claims.

"Full protection from malware" - that violates the halting problem.

Related debate in medicine: Should you study claims that violate the laws of physics? (Homeopathy)

Evidence-based IT-security

Today IT security is largely not based on scientific evidence - instead we rely on experience, expert advice or - even worse - marketing.

We should use Evidence-based IT-Security based on high-quality science. However the science largely doesn't exist.