Syzkaller安装 Fuzz Qemu amd64 Kernel

syzkaller官网上有介绍如何在Ubuntu宿主机上用qemu方法fuzz x86_64的Linux内核，但是步骤很分散，在好几个页面上，而且还可能有一些坑，后面会讲到。
首先介绍一下我的环境：

Ubuntu 16.04 x86_64
gcc 8.2.0
linux-5.1
go1.12.5

安装新版gcc

Syzkaller是一个coverage-guided fuzzer，因此需要编译内核有coverage support，gcc 6.1.0以后加入了coverage support。这里我是源码编译安装了gcc 8.2.0，大体步骤如下：

$ wget http://ftp.tsukuba.wide.ad.jp/software/gcc/releases/gcc-8.2.0/gcc-8.2.0.tar.gz
$ tar xzvf gcc-8.2.0.tar.gz
$ cd gcc-8.2.0/
$ ./contrib/download_prerequisites
$ sudo apt install texinfo bison flex
$ mkdir build
$ cd build
$ ../configure --prefix=/usr/local/gcc --enable-bootstrap --enable-checking=release --enable-languages=c,c++ --disable-multilib
$ make -j8
$ sudo make install
$ vim ~/.bashrc
export PATH=/usr/local/gcc/bin:$PATH
$ source ~/.bashrc
$ gcc -v
...
gcc version 8.2.0 (GCC)

编译新版linux内核

linux kernel也需要coverage support，KCOV在linux kernel 4.6以后加入，可以用CONFIG_KCOV=y配置。
编译内核的大体步骤如下：

$ wget http://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.1.tar.gz
$ tar zxvf linux-5.1.tar.gz
$ cd linux-5.1
$ make defconfig
$ make kvmconfig
$ vim .config
CONFIG_KCOV=y
CONFIG_DEBUG_INFO=y
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_CONFIGFS_FS=y
CONFIG_SECURITYFS=y
$ make oldconfig # 使能这些选项使得一些子选项可用，一路回车即可
$ make -j8

创建linux镜像

接着需要创建一个debian-strech的Linux镜像：

$ sudo apt install debootstrap
$ mkdir IMAGE
$ cd IMAGE
$ wget https://raw.githubusercontent.com/google/syzkaller/master/tools/create-image.sh -O create-image.sh
$ chmod +x create-image.sh
$ ./create-image.sh
$ ls
chroot  create-image.sh  stretch.id_rsa  stretch.id_rsa.pub  stretch.img

安装qemu

1	sudo apt install qemu-system-x86

下面需要确保kernel能正常启动，sshd能正常运行，这里是一个坑点：

$ qemu-system-x86_64 \
  -kernel linux-5.1/arch/x86/boot/bzImage \
  -append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
  -hda IMAGE/stretch.img \
  -net user,hostfwd=tcp::10021-:22 -net nic \
  -enable-kvm \
  -nographic \
  -m 2G \
  -smp 2 \
  -pidfile vm.pid \
  2>&1 | tee vm.log

这样启动的qemu可能存在[FAILED] Failed to start Raise network interfaces.的错误，原因是虚拟机启动后网卡名称为enp0s3（ip a s命令可以查看），而/etc/network/interfaces里的默认配置为eth0，网卡名称配置错误，启动不了网络接口，需要在虚拟机里修改interfaces文件。

之后可以在另一个终端ssh连接到虚拟机：

1	ssh -i IMAGE/stretch.id_rsa -p 10021 -o "StrictHostKeyChecking no" root@localhost

安装golang

syzkaller是go语言实现的，要编译安装需要Go 1.11+工具链。我安装的是go1.12.5。将go解压到/usr/local/目录，在~/创建gopath文件夹，在环境变量中添加GOROOT变量和GOPATH等：

$ vim ~/.bashrc
export GOROOT=/usr/local/go
export GOPATH=/home/fanrong/gopath
export PATH=$GOROOT/bin:$PATH

编译syzkaller

下载编译syzkaller源码：

$ go get -u -d github.com/google/syzkaller/...
$ cd $GOPATH/src/github.com/google/syzkaller
$ make
$ ls bin
linux_amd64  syz-db  syz-manager  syz-mutate  syz-prog2c  syz-repro  syz-runtest  syz-upgrade

直接运行make是在amd64平台上编译amd64版本的syzkaller，如果要交叉编译需要设置TARGETOS、TARGETARCH等参数，详见Makefile。

启动syzkaller开始fuzz

在syzkaller目录创建一个配置文件my.cfg来指定fuzz中相关的信息：

{
	"target": "linux/amd64",
	"http": "127.0.0.1:56741",
	"workdir": "workdir",
	"kernel_obj": "/home/fanrong/Computer/kernel/linux-5.1",
	"image": "/home/fanrong/Computer/kernel/IMAGE/stretch.img",
	"sshkey": "/home/fanrong/Computer/kernel/IMAGE/stretch.id_rsa",
	"syzkaller": ".",
	"procs": 8,
	"type": "qemu",
	"vm": {
		"count": 4,
		"kernel": "$KERNEL/arch/x86/boot/bzImage",
		"cpu": 2,
		"mem": 2048
	}
}

然后就可以在syzkaller目录中运行fuzz了，这里会遇到刚才启动qemu那个坑，前面把网卡改成了enp0s3才能正常运行sshd，而这里又出现了网卡不能启动的错误。这里要找出错误原因需要知道syzkaller启动虚拟机时的参数，一开始我打算看源码，在源码中log启动参数，后来发现syz-manager本身提供了调试信息的打印，只要在启动syz-manager的时候加上-debug选项，运行发现启动qemu的命令如下：

$ qemu-system-x86_64 -m 1024 -smp 1 -net nic,model=e1000 -net user,host=10.0.2.10,hostfwd=tcp::1569-:22 -display none -serial stdio -no-reboot -enable-kvm -cpu host,migratable=off -hda /home/fanrong/Computer/IoT/IMAGE/stretch.img -snapshot -kernel /home/fanrong/Computer/IoT/linux-5.1/arch/x86/boot/bzImage -append "earlyprintk=serial oops=panic nmi_watchdog=panic panic_on_warn=1 panic=1 ftrace_dump_on_oops=orig_cpu rodata=n vsyscall=native net.ifnames=0 biosdevname=0 root=/dev/sda console=ttyS0 kvm-intel.nested=1 kvm-intel.unrestricted_guest=1 kvm-intel.vmm_exclusive=1 kvm-intel.fasteoi=1 kvm-intel.ept=1 kvm-intel.flexpriority=1 kvm-intel.vpid=1 kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 kvm-intel.enable_apicv=1"

单独运行这个命令，发现网卡确实不能启动，在qemu里运行ip a s命令，网卡名变成了eth0，所以又要把/etc/network/interfaces里的网卡名改回eth0，然后就可以正常启动syzkaller了：

$ mkdir workdir
$ ./bin/syz-manager -config=my.cfg
2019/05/30 18:29:01 loading corpus...
2019/05/30 18:29:01 serving http on http://127.0.0.1:56741
2019/05/30 18:29:01 serving rpc on tcp://[::]:42897
2019/05/30 18:29:01 booting test machines...
2019/05/30 18:29:01 wait for the connection from test machine...
2019/05/30 18:29:25 machine check:
2019/05/30 18:29:25 syscalls                : 1387/2695
2019/05/30 18:29:25 code coverage           : enabled
2019/05/30 18:29:25 comparison tracing      : enabled
2019/05/30 18:29:25 extra coverage          : extra coverage is not supported by the kernel
2019/05/30 18:29:25 setuid sandbox          : enabled
2019/05/30 18:29:25 namespace sandbox       : /proc/self/ns/user does not exist
2019/05/30 18:29:25 Android sandbox         : enabled
2019/05/30 18:29:25 fault injection         : CONFIG_FAULT_INJECTION is not enabled
2019/05/30 18:29:25 leak checking           : CONFIG_DEBUG_KMEMLEAK is not enabled
2019/05/30 18:29:25 net packet injection    : /dev/net/tun does not exist
2019/05/30 18:29:25 net device setup        : enabled
2019/05/30 18:29:25 corpus                  : 0 (0 deleted)
2019/05/30 18:29:31 VMs 1, executed 0, cover 1349, crashes 0, repro 0
2019/05/30 18:29:41 VMs 1, executed 595, cover 2456, crashes 0, repro 0
2019/05/30 18:29:51 VMs 1, executed 1156, cover 3520, crashes 0, repro 0
2019/05/30 18:30:01 VMs 1, executed 1156, cover 7565, crashes 0, repro 0
2019/05/30 18:30:11 VMs 1, executed 1538, cover 8745, crashes 0, repro 0
...

可以在浏览器中查看fuzz进度：

reference
How to set up syzkaller
Setup: Ubuntu host, QEMU vm, x86-64 kernel
ubuntu16.04 编译安装gcc8.2.0