Chapter 137: Jailhouse in QEMU ARM64¶

What: boot a QEMU ARM64 Linux system, enable Jailhouse, and start a small inmate cell.

Why: Jailhouse is useful only when you understand what it removes from Linux and gives to another cell.

Focus: no magic VM language. We will track CPUs, RAM, MMIO, interrupts, and console ownership.

137.1 Why this chapter is not on i.MX6ULL¶

The i.MX6ULL has one Cortex-A7 core.

That matters.

Jailhouse is a partitioning hypervisor. Its cleanest teaching example is:

CPU 0..N: Linux root cell
CPU M: inmate cell
some RAM: Linux root cell
some RAM: inmate cell
some devices: Linux root cell
some devices: inmate cell

On a one-core chip, there is no spare CPU to hand to the inmate while Linux retains a core. The ownership model remains useful to study, but this multicore example is not a deployment recipe for the i.MX6ULL.

QEMU ARM64 can give us a fake machine with many CPUs. The official Jailhouse QEMU ARM64 configuration uses a virtual ARM machine with 1 GiB of RAM and multiple virtual CPUs. That is a much better initial example.

This is the honest path:

first: QEMU ARM64, so the model is visible
later: real board only when the hardware shape fits

The goal is not to pretend QEMU is your product. The goal is to learn the Jailhouse workflow without fighting board bring-up at the same time.

137.2 Jailhouse in one picture¶

Xen boots before Linux:

firmware -> Xen -> Dom0 Linux -> DomU guests

Jailhouse starts after Linux:

firmware -> Linux -> jailhouse.ko -> Jailhouse active -> inmate cells

Before Jailhouse is enabled, Linux owns the whole machine.

After Jailhouse is enabled, Linux becomes the root cell. Jailhouse then blocks Linux from touching resources that belong to other cells.

That is the core idea:

Linux does not become stronger.
Linux gives resources away.
Jailhouse enforces the new ownership table.

Jailhouse is not trying to emulate a PC for each guest. It is trying to partition real hardware.

137.3 Vocabulary we need¶

Root cell

The Linux system that booted first. It loads jailhouse.ko, enables Jailhouse, creates inmate cells, and remains responsible for management.

Inmate cell

The isolated workload. It can be bare-metal code, an RTOS, or sometimes a small Linux guest.

Cell config

A compiled hardware ownership table. It describes CPUs, memory regions, interrupt controllers, PCI or MMIO devices, and console settings.

Root-cell config

The system-wide config used when enabling Jailhouse. On QEMU ARM64 this is commonly built from configs/arm64/qemu-arm64.c.

Inmate config

The config for one non-root cell. On the QEMU ARM64 demo path this is commonly built from configs/arm64/qemu-arm64-inmate-demo.c.

Inmate binary

The code loaded into the inmate cell. For the first demo this is usually a small test binary such as gic-demo.bin.

137.4 What the QEMU ARM64 demo proves¶

The first Jailhouse demo should prove only five things:

Linux can boot on the QEMU ARM64 virtual machine.
Linux can load the Jailhouse kernel module.
Jailhouse can be enabled with the QEMU ARM64 root-cell config.
A non-root inmate cell can be created, loaded, and started.
Linux root cell stays alive while the inmate runs.

Do not add Zephyr yet.

Do not add a second Linux yet.

Do not edit the cell config yet.

First make the known-good demo work. Then change one thing at a time.

137.5 Workspace¶

Use a separate workspace because this example will create images, logs, configs, and notes:

$ mkdir -p ~/imx6ull/jailhouse-lab/{src,build,logs,notes}
$ cd ~/imx6ull/jailhouse-lab

Record host versions:

$ uname -a | tee logs/host-uname.txt
$ lsb_release -a 2>/dev/null | tee logs/host-release.txt

If your distribution does not have lsb_release, use:

$ cat /etc/os-release | tee logs/host-release.txt

137.6 Install host tools¶

Install the tools used by the demo and by later inspection:

$ sudo apt update
$ sudo apt install qemu-system-aarch64 git make gcc-aarch64-linux-gnu \
    device-tree-compiler flex bison libssl-dev bc cpio rsync file \
    docker.io

Check the important tools:

$ qemu-system-aarch64 --version | tee logs/qemu-version.txt
$ aarch64-linux-gnu-gcc --version | head -n 1 | tee logs/aarch64-gcc-version.txt
$ dtc --version | tee logs/dtc-version.txt

The upstream Jailhouse README says the ARM64 QEMU demo needs QEMU 3.0 or newer. Modern Ubuntu and Debian releases are normally far beyond that.

The jailhouse-images reference flow uses kas-container, so Docker must also work for your normal user or through sudo. Check:

$ docker --version
$ docker run --rm hello-world

If Docker permission fails, fix that before starting the image build. Do not debug Jailhouse while Docker itself is broken.

137.7 Get the source¶

Clone Jailhouse:

$ cd ~/imx6ull/jailhouse-lab/src
$ git clone https://github.com/siemens/jailhouse.git
$ cd jailhouse
$ git rev-parse HEAD | tee ../../logs/jailhouse-commit.txt

Now find the QEMU ARM64 configs:

$ ls configs/arm64 | tee ../../logs/jailhouse-arm64-configs.txt
$ ls inmates/demos/arm64 | tee ../../logs/jailhouse-arm64-inmates.txt

Expected names may include:

qemu-arm64.c
qemu-arm64-inmate-demo.c
gic-demo.c

Do not worry if the exact list changes across releases. The concepts do not change:

root-cell config
inmate-cell config
inmate binary

137.8 Use the reference image path first¶

Jailhouse has a lot of moving parts:

QEMU command line
Linux kernel image
root filesystem image
Jailhouse kernel module
Jailhouse user tool
root-cell config
inmate config
inmate binary

For the first successful run, use the Jailhouse reference image project:

$ cd ~/imx6ull/jailhouse-lab/src
$ git clone https://github.com/siemens/jailhouse-images.git
$ cd jailhouse-images
$ git rev-parse HEAD | tee ../../logs/jailhouse-images-commit.txt

The reference image project is built around kas-container. It provides a menu for virtual targets:

$ ./kas-container menu

Select the QEMU ARM64 Jailhouse target from the menu. After the image is generated, start it with:

$ ./start-qemu.sh arm64

This is intentionally the shortest path. The first time, your job is to get a working root cell and inmate cell, not to debug every package in the image build.

Expect this build to download and compile a full reference image stack. It is much heavier than building the tiny BusyBox initramfs in Chapter 129.

If the menu names have changed in your release, look at the repository README and conf/multiconfig:

$ ls conf/multiconfig
$ grep -R "arm64" -n conf scripts start-qemu.sh

Write down the exact target name you used:

Jailhouse image target:

That line belongs in your configuration notes.

137.9 The manual QEMU shape¶

The reference image scripts hide details. Before running Jailhouse commands, understand the QEMU machine they are creating.

The upstream ARM64 demo has this shape:

$ qemu-system-aarch64 \
    -cpu cortex-a57 \
    -smp 16 \
    -m 1G \
    -machine virt,gic-version=3,virtualization=on,its=off \
    -nographic \
    -netdev user,id=net \
    -device virtio-net-device,netdev=net \
    -drive file=LinuxInstallation.img,format=raw,id=disk,if=none \
    -device virtio-blk-device,drive=disk \
    -kernel Image \
    -append "root=/dev/vda1 mem=768M"

Do not copy this blindly yet. Use it as a map.

Important pieces:

Option	Meaning
`-cpu cortex-a57`	Run an ARMv8-A CPU model.
`-smp 16`	Create sixteen virtual CPUs.
`-m 1G`	Give the machine 1 GiB RAM.
`virtualization=on`	Expose virtualization support to the guest.
`gic-version=3`	Use a GICv3 interrupt controller.
`its=off`	Match the demo platform expectation.
`mem=768M`	Leave the top RAM area unused by Linux.

That last line is critical.

Linux sees only 768 MiB:

0x40000000 .. lower RAM used by Linux

The rest can be used by Jailhouse and inmate cells:

top RAM reserved for Jailhouse and inmates

If Linux uses the same memory as an inmate, isolation is already broken. The system may crash before you learn anything useful.

137.10 Inspect the root-cell config¶

Open the QEMU ARM64 root-cell config:

$ cd ~/imx6ull/jailhouse-lab/src/jailhouse
$ less configs/arm64/qemu-arm64.c

Find the comment near the top:

NOTE: Add "mem=768M" to the kernel command line.

That comment is not decoration. It is the contract between Linux boot arguments and the Jailhouse memory map.

Now search for the hypervisor memory region:

$ grep -n "hypervisor_memory" -A5 configs/arm64/qemu-arm64.c

You should see a physical start address and a size.

Write it in your notes:

Jailhouse hypervisor memory:

Now search for CPUs:

$ grep -n "cpus" -A4 configs/arm64/qemu-arm64.c

On this demo config, the CPU bitmap represents the CPUs available to the root cell config. A bitmap of 0xffff means sixteen CPUs are described.

This is why the demo QEMU command uses:

-smp 16

The QEMU command and the cell config must describe the same machine.

137.11 Inspect the inmate config¶

Now open the inmate config:

$ less configs/arm64/qemu-arm64-inmate-demo.c

Look for:

.name
.cpus
.mem_regions
.irqchips
.console

The inmate config answers four practical questions:

Which CPU can the inmate run on?
Where is the inmate RAM?
Which interrupts can reach the inmate?
How does the inmate print?

This is the file you will annotate later.

137.12 Boot the reference image¶

Start the reference image:

$ cd ~/imx6ull/jailhouse-lab/src/jailhouse-images
$ ./start-qemu.sh arm64 | tee ../../logs/qemu-arm64-boot.txt

Inside the QEMU guest, collect basic evidence:

# uname -a
# cat /proc/cmdline
# nproc
# free -m

Expected ideas:

many CPUs visible before Jailhouse is enabled
kernel command line contains mem=768M
memory is less than the full 1 GiB QEMU machine

The exact numbers depend on the image, but the shape should match.

137.13 Find Jailhouse inside the guest¶

Inside the QEMU guest:

# which jailhouse
# find /lib/modules -name "jailhouse.ko*"
# find /usr -name "*.cell" | grep jailhouse
# find /usr -name "gic-demo.bin" -o -name "*demo*.bin"

Common locations include:

/usr/share/jailhouse/cells/
/usr/libexec/jailhouse/
/lib/modules/.../extra/

Do not memorize paths. Learn to find the artifacts.

Record them:

jailhouse tool:
jailhouse module:
root-cell config:
inmate-cell config:
inmate binary:

137.14 Enable Jailhouse¶

Load the module:

# modprobe jailhouse

If that fails, check:

# dmesg | tail -n 80
# uname -r
# find /lib/modules/$(uname -r) -name "jailhouse.ko*"

The module must match the running kernel.

Now enable Jailhouse with the QEMU ARM64 root-cell config:

# jailhouse enable /path/to/qemu-arm64.cell

Replace the path with the real path you found in the image.

Collect evidence:

# dmesg | tail -n 120
# jailhouse cell list

Expected concept:

Jailhouse enabled
root cell listed
Linux still responsive

If the guest freezes here, do not continue. Debug the enable step first.

137.15 What changed after enable¶

Before enable:

Linux kernel controls the whole virtual machine.

After enable:

Jailhouse controls the partition boundaries.
Linux is still running, but only as the root cell.

This distinction matters for debugging.

If Linux tries to use memory that no longer belongs to it, Jailhouse can block that access.

If an inmate tries to touch a device that does not belong to it, Jailhouse can block that access.

If the cell config is wrong, the wrong thing gets blocked or the wrong thing is allowed.

So the cell config is not a helper file. It is the security boundary.

137.16 Create the inmate cell¶

Create the demo inmate:

# jailhouse cell create /path/to/qemu-arm64-inmate-demo.cell

List cells:

# jailhouse cell list

Expected concept:

root cell exists
inmate cell exists
inmate not running yet

Creating a cell builds the partition. It does not mean code is running inside it.

137.17 Load and start the inmate¶

Load the demo binary:

# jailhouse cell load inmate-demo /path/to/gic-demo.bin

Start it:

# jailhouse cell start inmate-demo

Check the cell list:

# jailhouse cell list

Expected concept:

inmate-demo running
root Linux still responsive

Depending on the demo image, output may appear on the serial console, the Jailhouse console, or the QEMU terminal. Check:

# jailhouse console
# dmesg | tail -n 120

The exact demo output is less important than the state transition:

created -> loaded -> running

137.18 Stop and disable¶

Stop the inmate if the demo supports shutdown:

# jailhouse cell shutdown inmate-demo

Destroy the cell:

# jailhouse cell destroy inmate-demo

Disable Jailhouse:

# jailhouse disable

If a running cell prevents disable, stop or destroy the cell and run disable again.

Final evidence:

# jailhouse cell list
# dmesg | tail -n 120

The clean development loop is:

enable root-cell config
create inmate
load inmate binary
start inmate
stop inmate
destroy inmate
disable Jailhouse

137.19 Build the mental map¶

Draw this in your notes:

QEMU virt machine

RAM: 1 GiB total
  lower area: Linux root cell
  top area: Jailhouse plus inmates

CPUs: 16 virtual CPUs
  root cell: Linux management side
  inmate: CPU subset from inmate config

Interrupt controller:
  GICv3

Console:
  root Linux console
  Jailhouse debug console
  inmate console path

Then fill in real addresses from the configs.

This is the moment where Jailhouse stops being a command sequence and becomes an ownership table.

137.20 Failure case: overlapping memory ownership¶

The reference setup reserves memory outside Linux’s usable range for Jailhouse and its cells. If Linux’s memory limit and cell configuration disagree, Linux can use pages another owner expects to control. Enabling may fail, or the inconsistent setup may become unstable.

The important boundary is exclusive ownership of physical memory, not the particular text of one launch argument.

137.21 Failure case: a mismatched CPU layout¶

A root-cell configuration describes a particular CPU topology and ownership mask. A virtual machine exposing fewer or different CPUs cannot satisfy that description. Enable or later cell creation can fail.

A configuration file is hardware-specific policy. Its format being valid does not establish that its resources exist on the running machine.

137.22 Failure case: an incompatible executable¶

An inmate image is linked for its assigned memory and entry conditions. A Linux user-space executable instead expects a loader, ABI and process environment. Loading its bytes into cell memory does not supply those services.

Cell loading is not execve(). An image may be rejected or fail at runtime depending on what the loader checks and what the binary assumes.

137.23 Annotate the root-cell config¶

Create a note file:

$ cp src/jailhouse/configs/arm64/qemu-arm64.c notes/qemu-arm64-annotated.c

Add comments beside:

hypervisor_memory
debug_console
platform_info.arm.gic_version
platform_info.arm.gicd_base
platform_info.arm.gicr_base
root_cell.name
cpus
mem_regions
irqchips
pci_devices

Your comments should answer:

what is this resource?
who owns it?
why is it here?
which QEMU option must match it?

This is more valuable than memorizing commands.

137.24 Annotate the inmate config¶

Copy the inmate config:

$ cp src/jailhouse/configs/arm64/qemu-arm64-inmate-demo.c notes/qemu-arm64-inmate-annotated.c

Add comments beside:

cell name
CPU bitmap
memory regions
communication region
interrupt chip
console

For each memory region, classify it:

RAM
MMIO
communication
loadable area
read only
read write
executable

The flags tell you what the inmate is allowed to do with that region.

137.25 Why this is useful in real products¶

Jailhouse is useful when your product has a clear static split.

Good shape:

Linux UI and network stack on CPU 0..2
control workload on CPU 3
dedicated device for control workload
shared memory protocol between the two sides

Weak shape:

one CPU
many shared devices
unclear ownership
need to start and stop guests dynamically
need rich virtual hardware

For the weak shape, Xen, Linux processes, containers, PREEMPT_RT, or a separate MCU may be better.

The professional skill is not “use a hypervisor”. The professional skill is choosing the smallest boundary that solves the product problem.

137.26 Key Notes¶

Jailhouse partitions resources after Linux has booted. It does not provide Xen’s general domain-management model.
Root-cell and inmate descriptions must match actual CPUs, physical memory, MMIO and interrupts. Reservation and ownership must agree with Linux.
Resources removed from the root cell must no longer be used there. Two drivers cannot independently own the same controller because their processes have different names.
An inmate needs an executable and runtime contract consistent with its cell. A Linux application binary is not interchangeable with it.
The multicore QEMU ARM64 study does not make Jailhouse’s spare-core partitioning available on the single-core i.MX6ULL.
Successful cell startup demonstrates a particular ownership transition, not comprehensive platform or safety isolation.

137.27 Troubleshooting¶

modprobe jailhouse fails

The module may not match the running kernel.

Check:

# uname -r
# find /lib/modules/$(uname -r) -name "jailhouse.ko*"
# dmesg | tail -n 80

jailhouse enable freezes the guest

The root-cell config likely does not match the QEMU machine or Linux memory reservation.

Check:

QEMU RAM size
QEMU CPU count
QEMU GIC version
kernel command line
root-cell config

cell create fails

The inmate config may reference CPUs, memory, or IRQs not available to that cell.

Check the inmate config and the root-cell config together.

The inmate starts but prints nothing

The inmate may still be running. The console may be somewhere else.

Check:

# jailhouse cell list
# jailhouse console
# dmesg | tail -n 120

jailhouse disable fails

An inmate may still be running.

Stop or destroy non-root cells first:

# jailhouse cell list
# jailhouse cell shutdown inmate-demo
# jailhouse cell destroy inmate-demo
# jailhouse disable

137.28 What comes next¶

The reference example illustrates this ownership transition:

Linux root cell
Jailhouse module
root-cell config
inmate-cell config
inmate binary
cell lifecycle commands

Chapter 138 removes one more layer of magic. Instead of running the stock demo inmate, we will build and reason about a tiny non-Linux inmate workload.