A TEE to play with (Part 1)

A common complaint from people who would like to learn more about TEE is that it is difficult to get started.

In this sequence, we are going to build a TEE running on QEMU. This means that we don’t need any special hardware, and that the same set of instructions should work for everyone. There won’t be any real security, but this is just an educational environment so it doesn’t matter much. Any Trusted Application code you write should simply need to be recompiled for any other target TEE.

What we are going to do

I haven’t found anywhere which covers, in one place, how to put everything together (almost) from scratch. You will need:

  • A computer capable of running Docker or a Linux Virtual Machine. Nowadays that should mean almost anything, but lots of RAM and a fast CPU will obviously help.
  • A barebones Debian image (smaller is generally faster when running in a VM).
  • Linux kernel source code
  • Cross compilation tools
  • A Raspbian Linux distribution
  • QEMU compiled with ARM v8 support
  • OP-TEE source code

You can short-cut building your own kernel by downloading one but I am a bit fussy about running a kernel compiled by random people on the Internet. I’ll trust the Debian and Raspbian distos as building everything from scratch would probably take weeks.

Dhruv Vyas on GitHub has instructions on building a a suitable Kernel including some script automation, and some of what follows is heavily inspired by his (and others’) work. He provides a set of pretty up to date pre-built kernels if you prefer.

QEMU is capable of almost emulating a Raspberry Pi 3, but it is not quite a sufficiently faithful emulation to run the stock Raspbian kernel. This means we have to build our own and use some QEMU magic to graft it into the Raspbian image. That done, we can build OP-TEE and finally actually write some code for a TEE.

Step 1: I’d like a Debian, please

I’m using Docker to host my Debian install, but most of these instructions will work just as well on a Debian install, Debian running in a VM and Linux distros derived from Debian (Ubuntu, Mint etc). If you’re using something else you’re on your own but it probably isn’t too hard.

docker pull debian
docker run -it debian bash

You should now see a root prompt on your docker container. If you are not using Docker, almost everything that follows is just stock Debian. Let’s install the tools you need:

apt-get update
apt-get install git libncurses5-dev gcc-arm-linux-gnueabihf make gcc bc binutils build-essential bzip2 file gettext kmod lzma xmlto xz-utils cpio

Answer ‘Y’ and wait a while – there are a good number of packages to be installed.

Let’s also not build as root, just because it doesn’t feel right. You will most likely want a different username if you don’t have one already (on a VM or real install you can most likely skip this bit).

adduser jodonoghue
su jodonoghue
cd /home/jodonoghue

Step 2: Getting the right Linux Kernel sources

The kernel sources you choose must match the Raspbian distro you plan to run. These things change often enough that you should check the following details yourself before blindly building.

Since the image we will run is stock Raspbian, you can go to https://www.raspberrypi.org/downloads/raspbian/ and download a distibution. I chose the “Raspbian Stretch Lite” image, which is current at the time of writing.

There are both desktop and “lite” variants – the “lite” variant is likely to run a bit faster under QEMU, but either should work.

The Raspbian website tells you which kernel version each download image was built with. At the time of writing that is Kernel version 4.14.

mkdir kernel
cd kernel
git clone https://github.com/raspberrypi/linux.git

Depending on what you are planning to use, you may need a specific branch, but if you are pulling the latest these instructions should be OK. There are plenty of good git tutorials out there if you need one. This is a good time to make a refreshing hot beverage – the Linux kernel tree is quite large.

Dhruv Vyas’ GitHub scripts include a patch, which modifies the ARM Versatile build variant (which is fully supported on QEMU) so that it provides the hooks needed to run Raspbian.

In my case, and with a newer kernel, there are some minor differences to the necessary patch, shown below after I made the required edits.

cd linux
git diff

Step 3: Building the Kernel

We are going to build for the ARM Versatile board, as described previously. First build a baseline configuration.

make ARCH=arm versatile_defconfig

Next set some necessary environment variables

export KERNEL_VERSION=$(make kernel version)
export MODULES_INSTALL_PATH=../qemu-kernel-$KERNEL_VERSION-modules
export TOOLCHAIN=arm-linux-gnueabihf

You will need to append the following to your .config file:

make -j 4 -k ARCH=arm CROSS_COMPILE=${TOOLCHAIN}- menuconfig
make -j 4 -k ARCH=arm CROSS_COMPILE=${TOOLCHAIN}- bzImage dtbs
cp arch/arm/boot/zImage $KERNEL_TARGET_FILE_NAME
cp arch/arm/boot/dts/versatile-pb.dtb ..
make -j 4 -k ARCH=arm CROSS_COMPILE=${TOOLCHAIN}- modules

We should now have a built kernel in the kernel directory.

Step 4: Booting the Raspbian image

At this point we are more or less ready to go. It’s a good idea to resize your image so that there’s some space to work in:

qemu-img resize 2018-06-27-raspbian-stretch-lite.img +1G

Now we can launch our system. The things you may need to change are the kernel name, the DTS filename and the Raspbian image name.

qemu-system-arm -kernel qemu-kernel-4.14.69 -cpu arm1176 -m 256 -M versatilepb -dtb versatile-pb.dtb -no-reboot -append “root=/dev/sda2 panic=1 rootfstype=ext4 rw” -net nic -net user,hostfwd=tcp::5022-:22 -hda 2018-06-27-raspbian-stretch-lite.img

With luck and a following wind, after a minute or so you will see something like. The default login is “pi” with password “raspberry”, to save you looking it up.

2018 09 13 00 20 08

While this post hasn’t really had very much to do with TEE, we are well on our way to having a usable TEE development environment. In the next part we will add OP-TEE to our emulated Raspberry Pi.


John Lane originally developed build-kernel-emu. Even though I am not using it directly, I have taken liberally from it.

Dhruv Vyas authored much of the RPi specific material I have adapted.

What does a hiring manager expect?

It is very difficult for a new graduate to know what a hiring manager is expecting of them. The lucky minority may have had the benefit of an internship doing something similar, but for those who have not, the gap between University and work is unknown, and it would not be unreasonable to feel like Christopher Columbus: confident that the Earth was round and that he would eventually hit land if he continued to sail west, but at the same time knowing nothing about where he was sailing.

If you are a manger hiring graduates for the first time, you might want to reflect on this and set your expectations appropriately.

Things University generally does pretty well:

  • Ensuring that you know how to learn. This is, by far, the most important skill you can possibly have as it is the foundation of developing new skills.
  • Ensuring that you have a good basic knowledge of everything covered in the scope of your degree. Good graduates generally know a bit about quite a few things.
  • Ensuring that you know how to solve solved problems. Many graduates know how to code a balanced red-black tree from memory as well as knowing when (at least in theory) this would be a good data structure to choose. I have no clue about either of these things, but I have a good algorithms book and I know what I do not know (i.e. when I need to consult a suitable resource). Thus far, in over 25 years working as a software engineer, I have never had to code a balanced red-black tree, so this seems like a reasonable trade-off to me.
  • Teaching you basic (as opposed to BASIC) programming skills.
  • Teaching you algorithms, and good ways to handle data in general.

Things University generally doesn’t do so well:

  • Teaching you about programming in the large – version control, code reviews, continuous integration, regression testing and the like. You will most likely be initially overwhelmed by the sheer size of some codebases, and I will not be surprised by this.
  • Teaching you about control and management code. Most students are introduced to state machines and finite automata, so you have a good theoretical grounding, but control and management can be complex and messy – not something that is always amenable to academic study.
  • Teaching real-world concurrency and parallel programming. There are exceptions (functional programming courses often teach this as it is an area where FP has significant advantages over imperative programming), but I have interviewed relatively few students who have much understanding of the basic synchronisation primitives.
  • Debugging. Most of the projects students are expected to undertake are not sufficiently complex to allow you to develop good debugging techniques.

Most of my expectations should now be pretty clear: I expect you to have a good understanding in the areas Universities generally do quite well, and maybe a very basic understanding in some of the areas that are done somewhat less well.

There are a few other basics that you should have picked up long before you started your tertiary studies.

  • You need to be able to communicate in clear and reasonably concise English – both written and spoken.
  • You need to be able to get on with a variety of people of different backgrounds, ages and opinions in a professional environment. You don’t need to be super-outgoing, or be everyone’s best mate, but you are going to be spending over 1600 hours with your close team mates every year, so getting on at a basic level is important.

I’m a graduate. Get me in here.

I’ve been hiring engineers for over 15 years now, and I find that many of the same questions come up time after time when talking to graduates. They generally fall into categories like “what is work like?” and “how do I choose the right job?” Something I am not asked as often as I should be is “how do large companies behave?”.

Before you read on, it’s important to bear a few things in mind when weighing my advice.

  • I work for a relatively large company. Smaller companies are often more flexible which has both benefits and downsides.
  • I work in a company department which is focussed on embedded and security-related software. I kind of know – in the abstract – about enterprise software and web development, but just as you wouldn’t buy a car from a second hand horse salesman, please don’t assume I know much of what I am talking about outside my field of expertise.
  • I am located in the UK. Cultural and employment practices vary considerably between countries. In general, the further you are from the General Western Hegemony, the less what I have to say in some areas will apply.

This is mostly aimed at software engineers who are just starting on their careers, but new managers may find the odd useful idea somewhere if they look hard enough.

I’ve split this into a series, which will appear in due course.

  • The interview process.
  • What does a hiring manager expect?
  • How do I decide if this is the right job for me?
  • What should I expect from my first few months at work?
  • I just love bureaucracy. Tell me about it.
  • How do I manage my career?
  • What do large companies do that I might not expect?

Why TEE matters (Part 1)

In this article I am going to talk about the Trusted Execution Environment, or TEE. This is a relatively little understood subsystem on many different types of device which can help to protect security assets rather better than they can be protected by the main device operating system.

It is my intention to include some much more developer-focussed material in future articles with guidance on how to develop Trusted Applications to run on a TEE.

There are many different types of TEE, and quite a number of different hardware architectures that can support them. While some of what I will discuss is applicable to most systems, some is quite specific to the most commonly deployed form of TEE: the GlobalPlatform TEE running on ARM TrustZone.

In the interest of fair disclosure, I should mention that I am active in GlobalPlatform as well as being employed by a company which sells devices that support ARM TrustZone. It is perhaps unsurprising that my expertise is in this architecture.

Security Levels

It is worth starting any discussion by thinking about how the security provided by an environment can be described. The Common Criteria define a complete methodology for evaluating the security of information systems which looks at almost every aspect of the development of a product in terms of its impact on security. An in-depth introduction to the methodology is in the Common Criteria Evaluation Methodology document (CEM).

The usual approach to evaluating a product under the Common Criteria to to develop a Protection Profile – a generic document describing how a particular class of products provides security (and what is secured – which is equally important). If a particular product is to be evaluated, a Security Target is developed which shows how the product meets the requirements for the chosen Protection Profile. For a formal certification, an independent security lab will perform an evaluation of the product based on its Security Target and will write a report. If all is well, this report is submitted to a Certifying Body (which is usually a Government security agency such as ANSSI in France) which will issue a certificate.

The traditional way to  “measure” security under this methodology is the EAL (Evaluation Assurance Level). Typical Secure Elements (SIM Cards, chip cards used in credit cards and the like) are evaluated at a level of EAL4+ or better (the “+” indicates that they go beyond the EAL4 requirements in some respects). For products evaluated according to the same Protection Profile, this is a useful comparison – a Secure Element evaluated according to the usual Protection Profile for such devices (the so-called PP084 developed by the German BSI) at EAL6+ should be more secure than a product evaluated at EAL4+.

However, it is not meaningful – indeed it can be extremely misleading – to compare EAL levels for products evaluated according to different Protection Profiles. As a result, many in the industry have tried to find a simpler way of describing the type of security offered by an environment. One approach which is becoming increasingly accepted is to define essentially three types of environment (there is not yet broad agreement on there exact terms to use):

  • “Unrestricted” – these are environments which are designed to support a wide range of applications and have a broad functionality. While such environments usually provide security functionality, it is not the main purpose of the environment. Linux, Windows, OSX, iOS, Android and the like are typical examples.
  • “Restricted” – these are environments which are designed primarily to provide security functionality and generally little else. Such environments are normally intended to protect against attacks by remote attackers. A TEE is an example of such an environment.
  • “Tamper Resistant” – these are environments which are designed primarily to provide security functionality, with protection from sophisticated local attackers as well as remote attackers. Secure Elements and TPMs are examples of such environments.

When I talk about TEE, I hope it is therefore clear that I am describing an environment that is designed primarily to protect certain assets against disclosure to attackers who are using software techniques (e.g. malware) as their attack vector.

What is a TEE

At the simplest level, a TEE is a code execution environment in which:

  • All code executing is trusted for authenticity and integrity.
  • Other assets are protected in confidentiality.

Broken down further, this means that any executing code comes from a known source, and has not been tampered with, and that the TEE takes measures to ensure that protected information (“assets” in the security sense – usually secret or private keys, although you could protect almost any information using the mechanisms available) cannot be easily extracted from the TEE by an attacker.

In the TEE security model, the main attacker is assumed to be the REE (this is GlobalPlatform-speak for “Rich Execution Environment”) which is typically a Linux or Windows OS that is running a range of complex applications. The TEE therefore does not trust the REE, and assumes that it is potentially malicious.

This is not to say that Windows or Linux are insecure – far from it – it is simply a model in which we choose to prevent disclosure of TEE assets to the REE. It is much better to think of the security model as resembling a castle.

Alnwick Castle from the Gary Danvers Collection of Flickr

The image of Alnwick castle shows how castles were designed with multiple layers of defence. There might be a moat (although not here), some outer walls and a central keep which was usually built on a hill. The castle designers assumed that each layer of castle defences was breached when designing the next layer.

The TEE security model is similar: the REE provides an outer layer of security – process separation, well designed general IPC mechanisms, permissions and the like. The TEE is then isolated and provides its own additional strong defences in an environment designed to provide security at the expense of comfort. The TEE does not trust the REE simply because it assumes that an attacker has penetrated the REE, because this is the worst-case scenario.

What Mechanisms does a TEE use to protect itself

There are many potential ways to implement a TEE – the GlobalPlatform TEE System Architecture describes a number of options. Generally the architectural starting point is either a separation mechanism allowing the TEE to be isolated from the REE while running on the same CPU (architectures such as ARM Trustzone do this), or the provisioning of a separate CPU dedicated to running the TEE. There are several characteristics of the security mechanisms used by a GlobalPlatform TEE which are common:

  • The TEE is protected from the REE by hardware mechanisms that the REE cannot control. This means that an attacker compromising the REE cannot reconfigure hardware so as to make the TEE more vulnerable to attack. The statement “cannot” is an extremely challenging requirement in practice, and many of the successful attack vectors on TEEs find unexpected (by the system designer) routes to change hardware behaviour.
  • The TEE is booted by secure mechanisms which strongly bind the TEE to the hardware it runs on. In practice this means that the core of the TEE code is digitally signed by the device vendor or an entity trusted by the device vendor (e.g. SoC vendor, TEE vendor). This guarantee of code integrity and authenticity extends over the lifetime of the TEE including any upgrades. The stronger TEE implementations provide protection against rollback of TEE code to older versions which might have security vulnerabilities using hardware mechanisms.
  • The TEE provides mechanisms for securely storing data and keys which are at least as strongly protected as the TEE environment itself. These prevent an attacker from accessing or modifying the stored assets. In most cases this storage is actually provided by the REE (e.g. on the main REE storage), but it is encrypted using keys known only to the TEE and thus an attacker can only access the encrypted and integrity protected data.
  • The TEE provides mechanisms which limit the ability of debuggers and loggers to observe sensitive parts of the system. This means, for example, that if an attacker is using a debugger to trace execution of REE code, once control transfers to the TEE the debugger will be unable to observe system operation until control is transferred back to the REE.
  • The interface between the REE and the TEE needs to be designed in such a way that the TEE has the opportunity to verify (and potentially reject) requests to perform operations from the REE.

There is one aspect of the TEE that sometimes surprises those unfamiliar with the security architecture and it is this: the TEE usually does not offer very much protection against denial of service attacks. It is, for example, fairly straightforward for an attacker with control of the REE to delete any files on the REE storage that are used for Trusted Storage. This sounds much worse than it is in practice – since the purpose of the TEE is to perform security-sensitive operations on behalf of the REE, it follows that if the REE is compromised, we may not care very much that the TEE operation has been impacted provided that the protected assets are not revealed.

Anatomy of an application using a TEE

The TEE allows application developers to separate the most security sensitive parts of their applications and to execute these as Trusted Applications (TAs) on the TEE while retaining the bulk of the application functionality in the REE.

2018 08 08 09 30 33

The diagram above, taken from the GlobalPlatform TEE System Architecture, shows how this works in practice. An application running in the REE, which GlobalPlatform calls a Client Application (or CA – a term which is unfortunately rather overloaded in information security) communicates with its associated Trusted Application (TA) using the TEE Client API. The TA owns security assets – keys and other secrets. The TA performs operations which use these on behalf of the CA. The TEE provides a library of functions – the TEE Internal API (now renamed the TEE Internal Core API) which support many of the operations a TA might wish to perform.

To make this more concrete, let’s imagine that I am developing a mobile payment application. I want to ensure a few things in my security model.

  • Only the authorised registered user of a payment account can authorise a payment. This is intended to make it difficult for an attacker who has stolen a device from the authorised user to make fraudulent payments.
  • Only devices that have been enrolled in the service by the authorised user can be used to authorise payments. This is intended to ensure that an attacker who somehow obtains credentials (e.g. PIN, password) of the authorised user – e.g. using malware – cannot make fraudulent payments using a different device.

In Part 2 of this article we will take a high-level view of how the TEE can help to achieve these goals.

Credit for the image of Alnwick Castle to Gary Danvers Collection on Flickr.

Credit for images of the TEE System Architecture to GlobalPlatform (they are drawn from the TEE System Architecture document).

The Interview Process

Interview processes tend to be fairly company-specific, so I should be clear that this is how I do things rather than a general guide. There are a few questions you can ask to help you to prepare for different approaches, and I’ll mention some of these as I go.


The first stage of the interview process is the application itself. Probably 80% of applications are rejected without so much as an interview, so it’s important to get this right. The following are basics – if you fail on these, you should expect to be rejected, quite honestly.

  • Actually read the job description. Make sure that you fit the criteria – or at least most of them, possibly with a mild stretch for one or two. If you do not, you are probably wasting your time. In particular, if the job description says you must have the right to live and work in the UK (other countries are available…) and you don’t have that right, you are wasting your time and mine. It is generally somewhere between difficult and impossible to justify getting a work permit for a graduate.
  • Your CV must contain contact information. You might be surprised how often people forget this. E-mail address and cellphone are essential.
  • Be aware of norms for the country you are applying to. Photos and date of birth are almost obligatory in some countries. In others, such as the UK, they are seen as a waste of perfectly good space on your CV.
  • Your CV needs to have proper spelling and grammar. The odd minor slip might be forgiven, but generally a sloppy CV is a sign of a sloppy mind.
  • Never, never, never, ever lie. No matter how much you want the job. You will almost certainly be caught and the interview will end the moment this happens. Mild exaggeration is somewhat acceptable (the British “false modesty” thing doesn’t really work anymore – not even in the UK), but no more than that.

Assuming you have done everything here, we come to the content of the CV itself. Most graduate CVs inevitably read similarly as few people have much relevant experience to fall back on. There are a few things that can make your CV stand out.

People with GitHub accounts or other evidence of contribution to Open Source projects always stand out. Of all the things I look for, a passion for software engineering is probably at the top of the list. I will almost certainly take a quick look through any Open Source code you have authored and it’s likely to be enough on its own to get you a telephone interview.

Not everyone is a prolific GitHub contributor, and not doing so doesn’t count you out. Evidence of commercial software development (some students work for small companies during their free time) is nice. Evidence of other work is helpful. You might think that waitressing in a local restaurant is not very relevant, but it shows that you can be committed and professional. If you ran the student yoga with chainsaws club, that can help as it shows evidence of leadership and commitment.

You need to give an indication of why you want the job – usually via a cover letter. Modifying a CV for each company you apply to is tedious and unnecessary at this level.

You do not have to give details of your social media, although if you do I may take a look. I was, believe it or not, young once and may have occasionally imbibed alcoholic beverages myself. That means I won’t care about party photos where you dressed up as a toadstool while drinking Guinness from a Viking helmet. If it turns out that you are an international gold bullion thief or prolific generator of hate postings things may be different.

Once submitted, your application will go through a screening process. These vary, largely depending on the size of the company you are applying to and the number of vacancies. At some point your application will hopefully be selected for a phone screen.

Initial contact

You will be contacted by e-mail or phone (remember those contact details!), usually by someone from a staffing department rather than by the hiring manager him/herself, who will offer you a time for a phone interview or whatever else passes for the first stage of the process.

It is OK to ask for the interview time to be changed if the first time offered does not work for you, but if you do you should be proactive and offer several times that do work. In doing so, it is good to remember that there person calling you probably has a fairly full calendar themselves, so offering only a single time is not the best idea. Few managers want to interview outside of normal office hours at this stage.

This is also an opportunity to ask about the interview process: how many stages, how long does it take and what should you expect. Remember that the person you are talking to is not actually the hiring manager, and may only be able to answer in general terms. Some companies will send you an e-mail that details their process.

When the time comes for the call itself, make sure that you are:

  • In a quiet location where you will not be disturbed. If you live in a shared house it may be an idea to ask your housemates to refrain from practicing “Stairway” on their electric guitars.
  • Somewhere where the cellular signal is fairly strong (if you are taking the call on a mobile phone).
  • Somewhere where you have access to a computer, if you have been told to do this (some companies use “live coding” platforms in the early interview stages.

Please make sure that you are as comfortable and relaxed as possible just before the call comes in. Do not be late.

The phone screen

As far as I am concerned, the phone screening interview has three main purposes.

  • Firstly, I want to establish that what you have on your CV is essentially truthful, and that you haven’t missed out anything I might consider important.
  • Secondly, I want to understand why you are interested in the role on offer.
  • Lastly, I want to tell you about my expectations, some more about the job and generally to help you to go away knowing about the job you have applied for.

I usually spend about half of a phone interview talking about the role and answering candidate questions and half asking the candidate about themselves. I do not ask detailed technical questions during the phone screen.

Opinions differ, but I believe that asking someone to do detailed problem solving over the phone is too difficult on both sides. I haven’t tried any of the online live coding platforms which might make this a more realistic prospect, but in practice I have not found that I have selected too may candidates for face to face interview who were not up to scratch without it.

My main reason for setting up the interview in this way is to make sure that candidates really understand the job they are applying for. It is somewhat of a sales pitch, but I also point out what I am not offering, and why a candidate might care. In short, I want people to come to an interview well equipped.

The interview

I am not going to give all of my secrets away here, but as a candidate you should expect to do most of the following, possibly more than once:

  • Write some code on a whiteboard and step through it. You need to show that you have a good mental model of program execution.
  • Answer some questions on things you should know about. I will often probe to see how much more than “theoretical” knowledge you have.
  • Solve a problem that you are unlikely to have seen before. This is a test of your analytical and design skills, and you will most probably be guided in the right direction from time to time if you need it.
  • Talk about yourself, your career aspirations and your interests.
  • Talk about projects you have undertaken – usually at University.

You do not need to come to interview with me in a suit if you do not possess one, but even if a company has a fairly relaxed dress code, you should probably be at least “business casual”. The phone interview is a good time to ask whether there is a dress code (the finance sector often has one, for example).

Most larger companies will give you a separate “HR” interview, which may occur on the same day as the technical interview or may be separate. This is your opportunity to ask about salary, company benefits and the like. I try to avoid talking too much about these in the technical interviews – for all that I know they are important to candidates – because there are sufficient legal and ethical areas around discussing some aspects that I prefer to leave it entirely to the experts to discuss them with you.

My approach to interviews is to try to put candidates at their ease as much as possible. I understand that many people are nervous and that they may well feel that the stakes are high. I am trying to give people the space to show what they know rather than to highlight what they don’t. Some of the questions you are asked may be tough, but you will be guided and assisted to do your best.

A major reason for my approach is simply this: I want every candidate to show me the best of themselves.

The aftermath, hopefully an offer.

Relatively few companies will give you an offer on the spot. This will usually come later, and how much later is fairly dependent on company policy. In many cases, a manager will have to obtain authorisation to make an offer from a very high level in the company, and this can take a while. You should ask the hiring manager – if s/he does not volunteer this information – how long you can expect to wait before receiving an offer and how the rest of the process runs.