Fix all the bugs

American Fuzzy Lop and Address Sanitizer

Hanno Böck


Hanno Böck, freelance journalist and hacker.

Writing for and others.

Author of monthly Bulletproof TLS Newsletter.

Fuzzing Project, funded by Linux Foundation's Core Infrastructure Initiative.

Bug example

(QT file src/plugins/platforminputcontexts/compose/qcomposeplatforminputcontext.cpp)

KDE / QT bugs

Use after free in qmake

Underflow / out of bounds read in QT / QCompose

Out of bounds read in QtGui

Out of bounds read in Kwin

(last 2 not sure if I should blame Xorg/xcb API)

For fairness: GNOME

#762417: Out of bounds read in glib / token parser

#762483: Out of bounds read in glib / unicode parser

#768441: Heap overflow in gnome-session parameter parsing

#770027: Out of bounds read in pango / test suite

Bug example

/* +2 for our new arguments,
	+1 for NULL */
new_argv = g_malloc (argc + 3 *
	sizeof (*argv));

(gnome-session 3.20.1, bug #768441)

Address Sanitizer (ASAN)

All this bugs can be trivially found with Address Sanitizer.

Just add -fsanitize=address to the compiler flags in GCC/CLANG.

Find bugs with ASAN

./configure CFLAGS="-fsanitize=address -g" \
	CXXFLAGS="-fsanitize=address -g" \
make check

What is ASAN doing?

Shadow memory tracking which memory areas are valid.

Finds out of bounds access (read/write) and use after free bugs (and other less common issues).

Out of bounds read

#include <stdio.h>
int main() {
	int a[2] = {3,1};
	int i = 2;
	printf("%i\n", a[i]);

Use after free

#include <stdio.h>
#include <stdlib.h>
int main() {
	char *c = calloc(10,1);
	printf("%i\n", c[0]);
	printf("%i\n", c[1]);

Mission: Test everyting with ASAN

Every project using C/C++ code should test with ASAN.

Gentoo with ASAN

Why not build everything in a Linux system with ASAN?

Gentoo + ASAN: It runs!

Found bugs in Bash, Coreutils/Shred, man-db, Pidgin-OTR, Courier, Syslog-NG, Screen, Claws-Mail, ProFTPD ICU, TCL, Dovecot, Glib, GNOME, Qt, KDE, Libarchive, Squid, CMake, Gettext, SpamAssassin, ...


Throw garbage at software.


Example: Image parser

Take valid image, add random errors to it, see if parser crashes.

Darpa Cyber Grand challenge

Rarely told story

Most teams and all three winners of the Darpa Cyber Grand Challenge used American Fuzzy Lop (AFL) with some addons as a bug finding tool.

American Fuzzy Lop

Fuzzing strategies

Dumb fuzzing: Easy, but not very effective.

Template-based fuzzing: More effective, lots of work, doesn't scale.

Coverage-based fuzzing: Easy and effective.

American Fuzzy Lop

American Fuzzy Lop (AFL) made the idea of coverage-based fuzzing popular.

Step 1: Compile with afl-wrapper (afl-gcc or afl-clang-fast)

Step 2: Fuzz

AFL is easy

./configure CC=afl-clang-fast CXX=afl-clang-fast++ --disable-shared; make

[put sample file into directory in/]

afl-fuzz -i in -o out [path_to_parser_executable] @@

American Fuzzy Lop

AFL found bugs in ...

OpenSSL, OpenSSH, libjpeg-turbo, libpng, sqlite, GnuPG, Bash, Stagefright, BIND, NTPD, ...

There isn't any major piece of C parser code where AFL hasn't found bugs.

Fuzzing with superpowers

AFL finds bugs, ASAN finds more bugs.

Best to combine AFL and ASAN.

Set AFL_USE_ASAN=1 and add "-m none".

AFL/ASAN meet Heartbleed

Could Fuzzing find the Heartbleed bug?

Experiment: Implement wrapper that accepts handshake messages as file input.

Success after ~ 6 hours.

(Kostya Serebryany showed that LibFuzzer finds it in 5 minutes)


Also coverage based fuzzing.


AFL fuzzes executables, LibFuzzer fuzzes functions.

Faster, but more initial work (write code).

LibFuzzer example

#include <stdint.h>
#include <stddef.h>
#include <openssl/asn1.h>

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
	ASN1_STRING *out = 0;
	ASN1_mbstring_copy(&out, Data, Size, MBSTRING_BMP, 0);
	if (out!=0) ASN1_STRING_free(out);
	return 0;

Differential testing

Typical fuzzing: look for crashes / memory corruption.

Differential testing: Feed two different implementations doing the same thing the same input, compare the output.

Differential testing on math

Crypto is important.

Crypto uses mathematics - but is the math correct?

BN_sqr() bug (CVE-2014-3570)

OpenSSL had a bug in the squaring function.

On very rare inputs (1 in 2^128) it produced wrong results.

Surprising: AFL found this bug (first tested by Ralph-Philipp Weinmann).

AFL is good at this

OpenSSL / BN_mod_exp (CVE-2015-3193)

Nettle / ECC (CVE-2015-8803, CVE-2015-8804)

NSS / mp_div() / exptmod() (CVE-2016-1938)

OpenSSL / Poly1305

MatrixSSL / pstm_exptmod (CVE-2016-6885, CVE-2016-6886, CVE-2016-6887)

What is a vulnerabilit?

The vast majority of bugs found with AFL+ASAN are heap out of bounds reads.

Are these vulnerabilities? Sometimes (Heartbleed!)

Be prepared for pointless discussions whether these should be called vulnerabilities.

IMHO: Just fix them and skip that discussion.

More Tools

Other Sanitizers

Undefined Behavior Sanitizer (UBSAN) - easy to use, but finds many bugs, mostly not very interesting.

Memory Sanitizer (MSAN) - finds uninitialized memory, tricky to use.

Thread Sanitizer (TSAN) - mostly interesting for larger C++ projects.

Undefined behavior sanitizer (UBSAN)

#include <limits.h>
int main() {
	int i = 10;
	int j = -1;
	i <<= j;
	i = INT_MAX;

Memory Sanitizer (MSAN)

int main(int argc, char **argv) {
	int x[10];
	x[1] = 1;
	if (x[argc]) return 1;


Sanitizers and coverage-based fuzzing have been adapted for the Linux Kernel.


Tricky - no really good solution yet.

Preeny - uses LD_PRELOADing.

Patch from Doug Birdwell for AFL, fragile.

Wrappers to parser functions.

AFL + symbolic execution

Some work on this (e. g. in Darpa Challenge), but nothing easily usable yet.

Will have to proove it's useful.

It's free

All presented tools (AFL, LibFuzzer, ASAN, other Sanitizers, Preeny, KASAN, Syzcaller) are published as Free and Open Source Software.

The C/C++ problem

Most fuzzing/ASAN-related bugs are typical C/C++ problems.

Maybe we should just rewrite everything in Rust.

Comparing vendor reactions



2015-11-18: Reported 2 bugs in .deb parsing

2015-11-26: Debian and Ubuntu publish updates and security advisories (USN-2820-1, DSA-3407-1)



2015-11-20: Reported 3 bugs in .rpm parsing to Red Hat

Answer: We already got 30 crash reports, may take some time.


RPM is an independent project since 2007, used by Red Hat, Suse and others.

Or not? belongs to Red Hat Inc.

Red Hat Security: "However, we don't own domain, it's upstream project, so there's not much we can do about it."

Trac installation.

Trying to register account: Certificate error.

To create a bug you should ask for permission in IRC or on the mailing list.

rpm repository

RPM development happens on Github these days.

The webpage does not mention that.

What's the latest version of RPM?

According to

According to Github repository: 4.12.0

According to Fedora: 4.13.0

Status RPM

One Stack Buffer Overflow still unfixed in the latest Git Code.

No release (that can be found) since 2014.

There are more bugs, including ones that happen pre-signature-check.

Advertisement block: BerlinSec Meetup

Tomorrow (5th Sept) at Mozilla Berlin Community space

Thanks for listening

Test with Address Sanitizer

Fuzz your software