Fix all the bugs
American Fuzzy Lop and Address Sanitizer
Hanno Böck
Introduction
Hanno Böck, freelance journalist and hacker.
Writing for Golem.de and others.
Author of monthly Bulletproof TLS Newsletter.
Trying to fix TLS (more tomorrow).
Fuzzing Project, funded by Linux Foundation's Core Infrastructure Initiative.
Bug example
/* +2 for our new arguments, +1 for NULL */ new_argv = g_malloc (argc + 3 * sizeof (*argv));
(gnome-session 3.20.1, bug #768441)
GNOME bug examples
#762417: Out of bounds read in glib / token parser
#762483: Out of bounds read in glib / unicode parser
#770027: Out of bounds read in pango / test suite
KDE (we have to be fair)
Use after free in qmake
Out of bounds read in QtGui
Out of bounds read in Kwin
(not sure if I should blame Xorg/xcb API)
Address Sanitizer (ASAN)
All this bugs can be trivially found with Address Sanitizer.
Just add -fsanitize=address to the compiler flags in GCC/CLANG.
Find bugs with ASAN
./configure CFLAGS="-fsanitize=address -g" \ CXXFLAGS="-fsanitize=address -g" \ LDFLAGS="-fsanitize=address" make make check
What is ASAN doing?
Shadow memory tracking which memory areas are valid.
Finds out of bounds access (read/write) and use after free bugs (and other less common issues).
Out of bounds read
#include <stdio.h>
int main() {
int a[2] = {3,1};
int i = 2;
printf("%i\n", a[i]);
}
Use after free
#include <stdio.h>
#include <stdlib.h>
int main() {
char *c = calloc(10,1);
printf("%i\n", c[0]);
free(c);
printf("%i\n", c[1]);
}
Mission: Test everyting with ASAN
Every project using C/C++ code should test with ASAN.
Gentoo with ASAN
Why not build everything in a Linux system with ASAN?
Gentoo + ASAN: It runs!
Found bugs in Bash, Coreutils/Shred, man-db, Pidgin-OTR, Courier, Syslog-NG, Screen, Claws-Mail, ProFTPD ICU, TCL, Dovecot, Glib, GNOME, Qt, KDE, Libarchive, Squid, CMake, Gettext, SpamAssassin, ...
Fuzzing
Throw garbage at software.
Fuzzing
Example: Image parser
Take valid image, add random errors to it, see if parser crashes.
Darpa Cyber Grand challenge
Rarely told story
Most teams and all three winners of the Darpa Cyber Grand Challenge used American Fuzzy Lop (AFL) with some addons as a bug finding tool.
American Fuzzy Lop
Fuzzing strategies
Dumb fuzzing: Easy, but not very effective.
Template-based fuzzing: More effective, lots of work, doesn't scale.
Coverage-based fuzzing: Easy and effective.
American Fuzzy Lop
American Fuzzy Lop (AFL) made the idea of coverage-based fuzzing popular.
Step 1: Compile with afl-wrapper (afl-gcc or afl-clang-fast)
Step 2: Fuzz
AFL is easy
./configure CC=afl-clang-fast CXX=afl-clang-fast++ --disable-shared; make
[put sample file into directory in/]
afl-fuzz -i in -o out [path_to_parser_executable] @@
American Fuzzy Lop
AFL found bugs in ...
OpenSSL, OpenSSH, libjpeg-turbo, libpng, sqlite, GnuPG, Bash, Stagefright, BIND, NTPD, ...
There isn't any major piece of C parser code where AFL hasn't found bugs.
Fuzzing with superpowers
AFL finds bugs, ASAN finds more bugs.
Best to combine AFL and ASAN.
Set AFL_USE_ASAN=1 and add "-m none".
AFL/ASAN meet Heartbleed
Could Fuzzing find the Heartbleed bug?
Experiment: Implement wrapper that accepts handshake messages as file input.
Success after ~ 6 hours.
(Kostya Serebryany showed that LibFuzzer finds it in 5 minutes)
LibFuzzer
Also coverage based fuzzing.
Part of LLVM/CLANG.
AFL fuzzes executables, LibFuzzer fuzzes functions.
Faster, but more initial work (write code).
LibFuzzer example
#include <stdint.h>
#include <stddef.h>
#include <openssl/asn1.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
ASN1_STRING *out = 0;
ASN1_mbstring_copy(&out, Data, Size, MBSTRING_BMP, 0);
if (out!=0) ASN1_STRING_free(out);
return 0;
}
Differential testing
Typical fuzzing: look for crashes / memory corruption.
Differential testing: Feed two different implementations doing the same thing the same input, compare the output.
Differential testing on math
Crypto is important.
Crypto uses mathematics - but is the math correct?
BN_sqr() bug (CVE-2014-3570)
OpenSSL had a bug in the squaring function.
On very rare inputs (1 in 2^128) it produced wrong results.
Surprising: AFL found this bug (first tested by Ralph-Philipp Weinmann).
AFL is good at this
OpenSSL / BN_mod_exp (CVE-2015-3193)
Nettle / ECC (CVE-2015-8803, CVE-2015-8804)
NSS / mp_div() / exptmod() (CVE-2016-1938)
OpenSSL / Poly1305
MatrixSSL / pstm_exptmod (CVE-2016-6885, CVE-2016-6886, CVE-2016-6887)
What is a vulnerabilit?
The vast majority of bugs found with AFL+ASAN are heap out of bounds reads.
Are these vulnerabilities? Sometimes (Heartbleed!)
Be prepared for pointless discussions whether these should be called vulnerabilities.
IMHO: Just fix them and skip that discussion.
More Tools
Other Sanitizers
Undefined Behavior Sanitizer (UBSAN) - easy to use, but finds many bugs, mostly not very interesting.
Memory Sanitizer (MSAN) - finds uninitialized memory, tricky to use.
Thread Sanitizer (TSAN) - mostly interesting for larger C++ projects.
Undefined behavior sanitizer (UBSAN)
#include <limits.h>
int main() {
int i = 10;
int j = -1;
i <<= j;
i = INT_MAX;
i++;
}
Memory Sanitizer (MSAN)
int main(int argc, char **argv) {
int x[10];
x[1] = 1;
if (x[argc]) return 1;
}
KASAN, KUBSAN, KTSAN, Syzcaller
Sanitizers and coverage-based fuzzing have been adapted for the Linux Kernel.
Network-fuzzing
Tricky - no really good solution yet.
Preeny - uses LD_PRELOADing.
Patch from Doug Birdwell for AFL, fragile.
Wrappers to parser functions.
AFL + symbolic execution
Some work on this (e. g. in Darpa Challenge), but nothing easily usable yet.
Will have to proove it's useful.
It's free
All presented tools (AFL, LibFuzzer, ASAN, other Sanitizers, Preeny, KASAN, Syzcaller) are published as Free and Open Source Software.
The C/C++ problem
Most fuzzing/ASAN-related bugs are typical C/C++ problems.
Maybe we should just rewrite everything in [Rust/Go/OCAML/...].
Comparing vendor reactions
DPKG
DPKG
2015-11-18: Reported 2 bugs in .deb parsing
2015-11-26: Debian and Ubuntu publish updates and security advisories (USN-2820-1, DSA-3407-1)
RPM
RPM
2015-11-20: Reported 3 bugs in .rpm parsing to Red Hat
Answer: We already got 30 crash reports, may take some time.
RPM...
RPM is an independent project since 2007, used by Red Hat, Suse and others.
Or not? rpm.org belongs to Red Hat Inc.
Red Hat Security: "However, we don't own rpm.org domain, it's upstream project, so there's not much we can do about it."
rpm.org
Trac installation.
Trying to register account: Certificate error.
To create a bug you should ask for permission in IRC or on the mailing list.
rpm repository
RPM development happens on Github these days.
The rpm.org webpage does not mention that.
What's the latest version of RPM?
According to rpm.org/releases: 4.12.0.1
According to Github repository: 4.12.0
According to Fedora: 4.13.0
Status RPM
One Stack Buffer Overflow still unfixed in the latest Git Code.
No release (that can be found) since 2014.
There are more bugs, including ones that happen pre-signature-check.
Thanks for listening
Test with Address Sanitizer
Fuzz your software
Questions?
