A common complaint from people who would like to learn more about TEE is that it is difficult to get started.
In this sequence, we are going to build a TEE running on QEMU. This means that we don’t need any special hardware, and that the same set of instructions should work for everyone. There won’t be any real security, but this is just an educational environment so it doesn’t matter much. Any Trusted Application code you write should simply need to be recompiled for any other target TEE.
What we are going to do
I haven’t found anywhere which covers, in one place, how to put everything together (almost) from scratch. You will need:
- A computer capable of running Docker or a Linux Virtual Machine. Nowadays that should mean almost anything, but lots of RAM and a fast CPU will obviously help.
- A barebones Debian image (smaller is generally faster when running in a VM).
- Linux kernel source code
- Cross compilation tools
- A Raspbian Linux distribution
- QEMU compiled with ARM v8 support
- OP-TEE source code
You can short-cut building your own kernel by downloading one but I am a bit fussy about running a kernel compiled by random people on the Internet. I’ll trust the Debian and Raspbian distos as building everything from scratch would probably take weeks.
Dhruv Vyas on GitHub has instructions on building a a suitable Kernel including some script automation, and some of what follows is heavily inspired by his (and others’) work. He provides a set of pretty up to date pre-built kernels if you prefer.
QEMU is capable of almost emulating a Raspberry Pi 3, but it is not quite a sufficiently faithful emulation to run the stock Raspbian kernel. This means we have to build our own and use some QEMU magic to graft it into the Raspbian image. That done, we can build OP-TEE and finally actually write some code for a TEE.
Step 1: I’d like a Debian, please
I’m using Docker to host my Debian install, but most of these instructions will work just as well on a Debian install, Debian running in a VM and Linux distros derived from Debian (Ubuntu, Mint etc). If you’re using something else you’re on your own but it probably isn’t too hard.
docker pull debian docker run -it debian bash
You should now see a root prompt on your docker container. If you are not using Docker, almost everything that follows is just stock Debian. Let’s install the tools you need:
apt-get update apt-get install git libncurses5-dev gcc-arm-linux-gnueabihf make gcc bc binutils build-essential bzip2 file gettext kmod lzma xmlto xz-utils cpio bison flex
Answer ‘Y’ and wait a while – there are a good number of packages to be installed. Note that you might also need an editor to be installed.
Let’s also not build as root, just because it doesn’t feel right. You will most likely want a different username if you don’t have one already (on a VM or real install you can most likely skip this bit).
adduser jodonoghue su jodonoghue cd /home/jodonoghue
Step 2: Getting the right Linux Kernel sources
The kernel sources you choose must match the Raspbian distro you plan to run. These things change often enough that you should check the following details yourself before blindly building.
Since the image we will run is stock Raspbian, you can go to https://www.raspberrypi.org/downloads/raspbian/ and download a distibution. I chose the “Raspbian Stretch Lite” image, which is current at the time of writing.
There are both desktop and “lite” variants – the “lite” variant is likely to run a bit faster under QEMU, but either should work.
The Raspbian website tells you which kernel version each download image was built with. At the time of writing that is Kernel version 4.14.
mkdir kernel cd kernel git clone https://github.com/raspberrypi/linux.git
Depending on what you are planning to use, you may need a specific branch, but if you are pulling the latest these instructions should be OK. There are plenty of good git tutorials out there if you need one. This is a good time to make a refreshing hot beverage – the Linux kernel tree is quite large.
Dhruv Vyas’ GitHub scripts include a patch, which modifies the ARM Versatile build variant (which is fully supported on QEMU) so that it provides the hooks needed to run Raspbian.
In my case, and with a newer kernel, there are some minor differences to the necessary patch, shown below after I made the required edits.
cd linux git diff
Step 3: Building the Kernel
We are going to build for the ARM Versatile board, as described previously. First build a baseline configuration.
make ARCH=arm versatile_defconfig
Next set some necessary environment variables
export KERNEL_VERSION=$(make kernel version) export KERNEL_TARGET_FILE_NAME=../qemu-kernel-$KERNEL_VERSION export MODULES_INSTALL_PATH=../qemu-kernel-$KERNEL_VERSION-modules export TOOLCHAIN=arm-linux-gnueabihf
You will need to append the following to your .config file:
make -j 4 -k ARCH=arm CROSS_COMPILE=${TOOLCHAIN}- menuconfig
make -j 4 -k ARCH=arm CROSS_COMPILE=${TOOLCHAIN}- bzImage dtbs
cp arch/arm/boot/zImage $KERNEL_TARGET_FILE_NAME
cp arch/arm/boot/dts/versatile-pb.dtb ..
make -j 4 -k ARCH=arm CROSS_COMPILE=${TOOLCHAIN}- modules
make -j 4 -k ARCH=arm CROSS_COMPILE=${TOOLCHAIN} INSTALL_MOD_PATH=$MODULES_INSTALL_PATH modules_install
We should now have a built kernel in the kernel directory.
Step 4: Booting the Raspbian image
At this point we are more or less ready to go. It’s a good idea to resize your image so that there’s some space to work in:
qemu-img resize 2018-06-27-raspbian-stretch-lite.img +1G
Now we can launch our system. The things you may need to change are the kernel name, the DTS filename and the Raspbian image name.
qemu-system-arm -kernel qemu-kernel-4.14.69 -cpu arm1176 -m 256 -M versatilepb -dtb versatile-pb.dtb -no-reboot -append “root=/dev/sda2 panic=1 rootfstype=ext4 rw” -net nic -net user,hostfwd=tcp::5022-:22 -hda 2018-06-27-raspbian-stretch-lite.img
With luck and a following wind, after a minute or so you will see something like. The default login is “pi” with password “raspberry”, to save you looking it up.
While this post hasn’t really had very much to do with TEE, we are well on our way to having a usable TEE development environment. In the next part we will add OP-TEE to our emulated Raspberry Pi.
Credits
John Lane originally developed build-kernel-emu. Even though I am not using it directly, I have taken liberally from it.
Dhruv Vyas authored much of the RPi specific material I have adapted.