BruceFan's Blog

Stay hungry, stay foolish

0%

Syzkaller安装 Fuzz Qemu amd64 Kernel

syzkaller官网上有介绍如何在Ubuntu宿主机上用qemu方法fuzz x86_64的Linux内核,但是步骤很分散,在好几个页面上,而且还可能有一些坑,后面会讲到。
首先介绍一下我的环境:

  • Ubuntu 16.04 x86_64
  • gcc 8.2.0
  • linux-5.1
  • go1.12.5

安装新版gcc

Syzkaller是一个coverage-guided fuzzer,因此需要编译内核有coverage support,gcc 6.1.0以后加入了coverage support。这里我是源码编译安装了gcc 8.2.0,大体步骤如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ wget http://ftp.tsukuba.wide.ad.jp/software/gcc/releases/gcc-8.2.0/gcc-8.2.0.tar.gz
$ tar xzvf gcc-8.2.0.tar.gz
$ cd gcc-8.2.0/
$ ./contrib/download_prerequisites
$ sudo apt install texinfo bison flex
$ mkdir build
$ cd build
$ ../configure --prefix=/usr/local/gcc --enable-bootstrap --enable-checking=release --enable-languages=c,c++ --disable-multilib
$ make -j8
$ sudo make install
$ vim ~/.bashrc
export PATH=/usr/local/gcc/bin:$PATH
$ source ~/.bashrc
$ gcc -v
...
gcc version 8.2.0 (GCC)

编译新版linux内核

linux kernel也需要coverage support,KCOV在linux kernel 4.6以后加入,可以用CONFIG_KCOV=y配置。
编译内核的大体步骤如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ wget http://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.1.tar.gz
$ tar zxvf linux-5.1.tar.gz
$ cd linux-5.1
$ make defconfig
$ make kvmconfig
$ vim .config
CONFIG_KCOV=y
CONFIG_DEBUG_INFO=y
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_CONFIGFS_FS=y
CONFIG_SECURITYFS=y
$ make oldconfig # 使能这些选项使得一些子选项可用,一路回车即可
$ make -j8

创建linux镜像

接着需要创建一个debian-strech的Linux镜像:

1
2
3
4
5
6
7
8
$ sudo apt install debootstrap
$ mkdir IMAGE
$ cd IMAGE
$ wget https://raw.githubusercontent.com/google/syzkaller/master/tools/create-image.sh -O create-image.sh
$ chmod +x create-image.sh
$ ./create-image.sh
$ ls
chroot create-image.sh stretch.id_rsa stretch.id_rsa.pub stretch.img

安装qemu

1
sudo apt install qemu-system-x86

下面需要确保kernel能正常启动,sshd能正常运行,这里是一个坑点:

1
2
3
4
5
6
7
8
9
10
11
$ qemu-system-x86_64 \
-kernel linux-5.1/arch/x86/boot/bzImage \
-append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
-hda IMAGE/stretch.img \
-net user,hostfwd=tcp::10021-:22 -net nic \
-enable-kvm \
-nographic \
-m 2G \
-smp 2 \
-pidfile vm.pid \
2>&1 | tee vm.log

这样启动的qemu可能存在[FAILED] Failed to start Raise network interfaces.的错误,原因是虚拟机启动后网卡名称为enp0s3(ip a s命令可以查看),而/etc/network/interfaces里的默认配置为eth0,网卡名称配置错误,启动不了网络接口,需要在虚拟机里修改interfaces文件。

之后可以在另一个终端ssh连接到虚拟机:

1
ssh -i IMAGE/stretch.id_rsa -p 10021 -o "StrictHostKeyChecking no" root@localhost

安装golang

syzkaller是go语言实现的,要编译安装需要Go 1.11+工具链。我安装的是go1.12.5。将go解压到/usr/local/目录,在~/创建gopath文件夹,在环境变量中添加GOROOT变量和GOPATH等:

1
2
3
4
$ vim ~/.bashrc
export GOROOT=/usr/local/go
export GOPATH=/home/fanrong/gopath
export PATH=$GOROOT/bin:$PATH

编译syzkaller

下载编译syzkaller源码:

1
2
3
4
5
$ go get -u -d github.com/google/syzkaller/...
$ cd $GOPATH/src/github.com/google/syzkaller
$ make
$ ls bin
linux_amd64 syz-db syz-manager syz-mutate syz-prog2c syz-repro syz-runtest syz-upgrade

直接运行make是在amd64平台上编译amd64版本的syzkaller,如果要交叉编译需要设置TARGETOSTARGETARCH等参数,详见Makefile。

启动syzkaller开始fuzz

在syzkaller目录创建一个配置文件my.cfg来指定fuzz中相关的信息:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"target": "linux/amd64",
"http": "127.0.0.1:56741",
"workdir": "workdir",
"kernel_obj": "/home/fanrong/Computer/kernel/linux-5.1",
"image": "/home/fanrong/Computer/kernel/IMAGE/stretch.img",
"sshkey": "/home/fanrong/Computer/kernel/IMAGE/stretch.id_rsa",
"syzkaller": ".",
"procs": 8,
"type": "qemu",
"vm": {
"count": 4,
"kernel": "$KERNEL/arch/x86/boot/bzImage",
"cpu": 2,
"mem": 2048
}
}

然后就可以在syzkaller目录中运行fuzz了,这里会遇到刚才启动qemu那个坑,前面把网卡改成了enp0s3才能正常运行sshd,而这里又出现了网卡不能启动的错误。这里要找出错误原因需要知道syzkaller启动虚拟机时的参数,一开始我打算看源码,在源码中log启动参数,后来发现syz-manager本身提供了调试信息的打印,只要在启动syz-manager的时候加上-debug选项,运行发现启动qemu的命令如下:

1
$ qemu-system-x86_64 -m 1024 -smp 1 -net nic,model=e1000 -net user,host=10.0.2.10,hostfwd=tcp::1569-:22 -display none -serial stdio -no-reboot -enable-kvm -cpu host,migratable=off -hda /home/fanrong/Computer/IoT/IMAGE/stretch.img -snapshot -kernel /home/fanrong/Computer/IoT/linux-5.1/arch/x86/boot/bzImage -append "earlyprintk=serial oops=panic nmi_watchdog=panic panic_on_warn=1 panic=1 ftrace_dump_on_oops=orig_cpu rodata=n vsyscall=native net.ifnames=0 biosdevname=0 root=/dev/sda console=ttyS0 kvm-intel.nested=1 kvm-intel.unrestricted_guest=1 kvm-intel.vmm_exclusive=1 kvm-intel.fasteoi=1 kvm-intel.ept=1 kvm-intel.flexpriority=1 kvm-intel.vpid=1 kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 kvm-intel.enable_apicv=1"

单独运行这个命令,发现网卡确实不能启动,在qemu里运行ip a s命令,网卡名变成了eth0,所以又要把/etc/network/interfaces里的网卡名改回eth0,然后就可以正常启动syzkaller了:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ mkdir workdir
$ ./bin/syz-manager -config=my.cfg
2019/05/30 18:29:01 loading corpus...
2019/05/30 18:29:01 serving http on http://127.0.0.1:56741
2019/05/30 18:29:01 serving rpc on tcp://[::]:42897
2019/05/30 18:29:01 booting test machines...
2019/05/30 18:29:01 wait for the connection from test machine...
2019/05/30 18:29:25 machine check:
2019/05/30 18:29:25 syscalls : 1387/2695
2019/05/30 18:29:25 code coverage : enabled
2019/05/30 18:29:25 comparison tracing : enabled
2019/05/30 18:29:25 extra coverage : extra coverage is not supported by the kernel
2019/05/30 18:29:25 setuid sandbox : enabled
2019/05/30 18:29:25 namespace sandbox : /proc/self/ns/user does not exist
2019/05/30 18:29:25 Android sandbox : enabled
2019/05/30 18:29:25 fault injection : CONFIG_FAULT_INJECTION is not enabled
2019/05/30 18:29:25 leak checking : CONFIG_DEBUG_KMEMLEAK is not enabled
2019/05/30 18:29:25 net packet injection : /dev/net/tun does not exist
2019/05/30 18:29:25 net device setup : enabled
2019/05/30 18:29:25 corpus : 0 (0 deleted)
2019/05/30 18:29:31 VMs 1, executed 0, cover 1349, crashes 0, repro 0
2019/05/30 18:29:41 VMs 1, executed 595, cover 2456, crashes 0, repro 0
2019/05/30 18:29:51 VMs 1, executed 1156, cover 3520, crashes 0, repro 0
2019/05/30 18:30:01 VMs 1, executed 1156, cover 7565, crashes 0, repro 0
2019/05/30 18:30:11 VMs 1, executed 1538, cover 8745, crashes 0, repro 0
...

可以在浏览器中查看fuzz进度:

reference
How to set up syzkaller
Setup: Ubuntu host, QEMU vm, x86-64 kernel
ubuntu16.04 编译安装gcc8.2.0