Chapter 31: A root filesystem, by hand

udev: the user-space device manager that reacts to kernel device events and creates policy-driven /dev nodes.

What: build a working root filesystem from scratch: BusyBox compiled statically, the FHS directory tree, libraries copied from the cross-toolchain, /etc/inittab + /etc/init.d/rcS + /etc/fstab, and an NFS-export that the kernel mounts via root=/dev/nfs. NFS: Network File System, which lets the target mount a host directory over Ethernet during development.

Why: Chapter 29’s BusyBox initramfs was a toy. One cpio file, no persistence, no real /etc. It becomes a real rootfs once three things are in place. First, /etc/ holds config. Second, the shared libraries dynamically-linked tools need are present. Third, NFS exports it so you can iterate user space without reflashing. MCU bridge: Think of the rootfs as the firmware image’s file-backed runtime environment. On an MCU you link everything into flash. On Linux, programs and config live in this mounted tree. rootfs: root filesystem, the directory tree mounted at / that contains /bin, /etc, /dev, and libraries.

Focus: the handful of files in /etc/ that everything else depends on, inittab, init.d/rcS, fstab, passwd, group. Once you know these files, Buildroot, Yocto, and Ubuntu-base look like the same content generated by tools. Yocto: a metadata-driven build system for producing custom Linux distributions. Buildroot: a configuration-driven build system that produces a complete root filesystem and related images.

31.1 The FHS in 30 seconds

The Filesystem Hierarchy Standard says what each top-level directory is for. The embedded subset:

/                # root directory
├── bin/         # essential user binaries (ls, mv, sh, ...)
├── sbin/        # essential system binaries (init, mount, fsck, ...)
├── usr/         # non-essential user binaries; mirrors bin/ + sbin/ + lib/ structure
│   ├── bin/
│   ├── sbin/
│   └── lib/
├── lib/         # shared libraries that bin/ and sbin/ need
├── etc/         # configuration files (text, no binaries)
├── dev/         # device files (mostly auto-populated by devtmpfs)
├── proc/        # mount point for procfs (Ch 32)
├── sys/         # mount point for sysfs (Ch 32)
├── tmp/         # temporary files; usually tmpfs
├── var/         # variable data: logs, mailboxes, runtime state
│   ├── log/
│   └── run/
├── root/        # root user's home directory
├── home/        # user homes (often empty on embedded)
└── mnt/         # ad-hoc mount points

We create each of these directories. The ones that should contain files at boot (bin/, sbin/, lib/, etc/) we populate now. The ones the kernel or daemons fill (proc/, sys/, dev/, tmp/, var/run/) stay empty until mount time.

31.2 The plan

host                                     target
─────────────────────────────────────    ────────────────────────────────
~/imx6ull/rootfs/                        / (mounted via NFS from host)
  ├── bin/busybox + symlinks             same
  ├── sbin/init → /bin/busybox           same
  ├── etc/inittab, rcS, fstab            same
  ├── lib/*.so.* (from cross-toolchain)  same
  ├── proc/  (empty; mount point)        proc filesystem mounted here
  ├── sys/   (empty)                     sysfs mounted here
  └── dev/   (empty)                     devtmpfs mounted here

NFS server on host exports ~/imx6ull/rootfs/.
Kernel boots with bootargs: root=/dev/nfs nfsroot=192.168.7.1:...

By the end of this chapter, an ls on the target lists files the host owns. Edits made on the host are visible to the target after the next read.

31.3 Build BusyBox statically

$ cd ~/imx6ull/src
$ wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2
$ tar xjf busybox-1.36.1.tar.bz2
$ cd busybox-1.36.1

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- defconfig
$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- menuconfig

In menuconfig, enable two settings explicitly:

Settings  --->
    [*]   Build static binary (no shared libs)
    [*]   Enable Unicode support
    [*]     Check $LC_ALL, $LC_CTYPE and $LANG environment variables

Build:

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- -j$(nproc)
$ arm-none-linux-gnueabihf-strip busybox
$ ls -lh busybox
-rwxr-xr-x 1 you you 580K busybox

580 KB statically linked. (~450 KB with musl-gcc.)

Static or dynamic? Static is simpler to deploy, no libraries needed on the target. The drawback is DNS resolution over the network doesn’t work with static glibc, because glibc’s NSS (Name Service Switch, the pluggable backend behind gethostbyname / getaddrinfo) is dlopen’d at runtime even from a “static” binary. A static glibc binary can still resolve names listed in /etc/hosts. What it can’t do is real DNS over the network. If you need real DNS, build BusyBox dynamically and copy the toolchain libraries into the rootfs. This chapter does both: static for the first boot, dynamic in §31.10.

31.4 Populate the rootfs

$ cd ~/imx6ull
$ mkdir -p rootfs/{bin,sbin,etc/init.d,lib,proc,sys,dev,tmp,var/{log,run},root,home,mnt,usr/{bin,sbin,lib}}
$ cd rootfs

# Install BusyBox into the rootfs
$ make -C ~/imx6ull/src/busybox-1.36.1 \
       ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- \
       CONFIG_PREFIX=$PWD install

make install reads BusyBox’s config and creates symlinks for every applet:

$ ls bin/ | head
[
[[
ash
base32
base64
busybox
cat
chattr
chgrp
chmod
...
$ ls bin/ | wc -l
246

246 applets, all pointing at bin/busybox. The single 580 KB binary acts as 246 commands. Which one it acts as depends on argv[0].

$ ls -l sbin/init
lrwxrwxrwx 1 you you 14 ... sbin/init -> ../bin/busybox

sbin/init → busybox. When the kernel execs /sbin/init, busybox examines argv[0] = /sbin/init and acts as init.

31.5 Create /etc/inittab

BusyBox init reads /etc/inittab to know what to do. Format:

<id>:<runlevels>:<action>:<process>

For BusyBox, <id> is the controlling tty (empty for “use the system default console”), <runlevels> is ignored. The action determines when the process runs.

$ cat > etc/inittab <<'EOF'
# /etc/inittab — busybox init configuration

# System initialization. Runs once at boot, before anything else.
::sysinit:/etc/init.d/rcS

# Spawn a shell on the console. respawn = restart automatically when it exits.
console::respawn:-/bin/sh

# What to do on Ctrl-Alt-Del. (No keyboard? Harmless to keep.)
::ctrlaltdel:/sbin/reboot

# What to do on shutdown.
::shutdown:/bin/umount -a -r
::shutdown:/sbin/swapoff -a
EOF

The leading - on -/bin/sh makes it a login shell (reads /etc/profile and ~/.profile). The eight standard BusyBox init actions:

Action

When the process runs

sysinit

Once, at boot, before anything else

wait

Once, init waits for it to finish before continuing

once

Once, init does not wait

respawn

Started; whenever it exits, restart it

askfirst

Like respawn, but prints “Please press Enter to activate” and waits

restart

When init receives SIGHUP (i.e. asked to re-read inittab)

ctrlaltdel

When init receives SIGINT (Ctrl-Alt-Del on a real keyboard)

shutdown

On halt/poweroff/reboot

For a development board you typically need only sysinit, respawn, and shutdown.

31.6 Create /etc/init.d/rcS

The shell script /etc/inittab’s sysinit line runs. This is where most of the per-boot setup lives:

$ cat > etc/init.d/rcS <<'EOF'
#!/bin/sh

# Search paths for the rest of this session.
PATH=/sbin:/bin:/usr/sbin:/usr/bin
LD_LIBRARY_PATH=/lib:/usr/lib
export PATH LD_LIBRARY_PATH

# Mount the virtual filesystems described in /etc/fstab.
/bin/mount -a

# devtmpfs is mounted automatically by the kernel if CONFIG_DEVTMPFS_MOUNT=y.
# If yours isn't, mount it manually:
mountpoint -q /dev || /bin/mount -t devtmpfs none /dev

# devpts gives us pseudo-terminals for things like ssh, screen, etc.
mkdir -p /dev/pts
/bin/mount -t devpts none /dev/pts

# Hook mdev into the kernel's hotplug mechanism. (mdev is BusyBox's udev.)
echo /sbin/mdev > /proc/sys/kernel/hotplug

# Populate /dev/ from /sys/ for devices already enumerated before mdev started.
/sbin/mdev -s

# Set a hostname.
/bin/hostname pa-mini

# Start any background daemons here (sshd, ntpd, your app, ...).

echo "*** rootfs ready ***"
EOF
$ chmod +x etc/init.d/rcS

The chmod +x matters. Without execute permission, BusyBox init skips the script silently. No /proc, no /sys, no /dev/pts. This is the most common first-time-rootfs bug.

31.7 Create /etc/fstab

mount -a consults /etc/fstab to know what to mount. Five-column format:

<device> <mountpoint> <fs-type> <options> <dump-freq> <fsck-order>

$ cat > etc/fstab <<'EOF'
# device      mountpoint    type      options     dump  pass
proc          /proc         proc      defaults    0     0
sysfs         /sys          sysfs     defaults    0     0
tmpfs         /tmp          tmpfs     defaults    0     0
tmpfs         /run          tmpfs     defaults    0     0
EOF

Four virtual filesystems:

  • proc: the procfs (Ch 32) at /proc.

  • sysfs: the sysfs (Ch 32) at /sys.

  • tmpfs at /tmp: RAM-backed tmpfs for /tmp. Wipes on reboot, which is what /tmp is supposed to do.

  • tmpfs at /run: RAM-backed tmpfs for daemon PID/socket files.

No physical disk mounts, the rootfs itself is mounted by the kernel before this script runs, via the root= cmdline. Other partitions (data, log) would be additional /etc/fstab lines.

31.8 Create /etc/passwd and /etc/group

Even a single-user system needs minimal versions:

$ cat > etc/passwd <<'EOF'
root:x:0:0:root:/root:/bin/sh
EOF
$ cat > etc/group <<'EOF'
root:x:0:
EOF
$ cat > etc/shadow <<'EOF'
root::0:0:99999:7:::
EOF
$ chmod 600 etc/shadow

root:: (empty password field in shadow) means root logs in with no password. Insecure. Fine for an early dev image. Replace with a real hash before production.

31.9 Create /etc/profile

Read by login shells. Sets the prompt, default permissions, etc.:

$ cat > etc/profile <<'EOF'
# /etc/profile — login-shell configuration

export PATH=/sbin:/bin:/usr/sbin:/usr/bin
export LD_LIBRARY_PATH=/lib:/usr/lib

umask 022

# A nicer prompt: [user@host:cwd]#
PS1='[\u@\h:\w]\$ '
export PS1

# Useful aliases.
alias ll='ls -lh'
alias la='ls -la'
alias ..='cd ..'
EOF

After login, you see [root@pa-mini:~]# instead of just #. Small but pays back the first time you have two ssh sessions open.

31.10 Copy libraries (for dynamic binaries)

The static BusyBox we built in §31.3 doesn’t need libraries. But if you copy any other dynamically-linked binary into the rootfs, your own application, ssh, ping, anything from the toolchain, it needs the shared libraries to be present at the runtime path the dynamic linker expects.

The libraries live inside your cross-toolchain installation:

$ . ~/imx6ull/scripts/env.sh
$ TOOLCHAIN=$ARM_LINUX_TOOLCHAIN
$ cd ~/imx6ull/rootfs

# The "main" libraries (libc, libm, libpthread, libdl, ...) and the dynamic linker
$ cp -d $TOOLCHAIN/arm-none-linux-gnueabihf/libc/lib/*.so* lib/
$ cp -d $TOOLCHAIN/arm-none-linux-gnueabihf/libc/lib/*.a lib/

# The dynamic linker itself MUST be the real file, not a symlink.
$ rm lib/ld-linux-armhf.so.3
$ cp $TOOLCHAIN/arm-none-linux-gnueabihf/libc/lib/ld-linux-armhf.so.3 lib/

# usr/lib for less-common libraries
$ cp -d $TOOLCHAIN/arm-none-linux-gnueabihf/libc/usr/lib/*.so* usr/lib/

Two things to know:

  1. -d preserves symlinks. Most .so.N files are symlinks to .so.N.M. Without -d, cp follows the symlink and copies the target. Both files end up as identical full copies. With -d, the symlink stays a symlink and you save space.

  2. ld-linux-armhf.so.3 is the dynamic linker itself. In the toolchain it’s a symlink to the real ld-2.31.so. On the target it must be a real file at the path the ELF binaries’ INTERP section points to (/lib/ld-linux-armhf.so.3). The rm + cp (without -d) dance after the wildcard copy forces the real file.

ELF: Executable and Linkable Format, the standard Linux object and executable file format.

Total size: ~60 MB unstripped for glibc, ~5–10 MB after strip and removing the locales / NSS modules you don’t need. ~5 MB for a comparable musl install. The glibc bulk is mostly NSS modules, locale data, and unused libraries, all of which you’d strip in a production build.

31.11 Export over NFS, boot

Add to /etc/exports on the host:

/home/you/imx6ull/rootfs *(rw,sync,no_root_squash,no_subtree_check)

Restart NFS:

$ sudo exportfs -ar
$ showmount -e localhost
Export list for localhost:
/home/you/imx6ull/rootfs *

In U-Boot:

MCU bridge: Think of U-Boot like a much larger boot stub plus debug monitor: it initializes hardware, loads the next image, and gives you commands before Linux starts. U-Boot: the bootloader that initializes enough hardware to load and start the Linux kernel.

=> setenv bootargs 'console=ttymxc0,115200 earlycon \
                    root=/dev/nfs nfsroot=192.168.7.1:/home/you/imx6ull/rootfs,vers=3,nolock,tcp \
                    ip=192.168.7.2:192.168.7.1:192.168.7.1:255.255.255.0::eth0:off \
                    rw rootwait'
=> setenv bootcmd 'tftp 0x82000000 zImage; tftp 0x83000000 imx6ull.dtb; bootz 0x82000000 - 0x83000000'
=> saveenv
=> boot

After kernel boot:

[    3.412345] VFS: Mounted root (nfs filesystem) on device 0:16.
[    3.428901] Run /sbin/init as init process
*** rootfs ready ***

Please press Enter to activate this console.

[Enter]

[root@pa-mini:~]#

You have a Unix shell on the i.MX6ULL, with / provided by the host over NFS.

31.12 The development loop

Now the win:

# On the host:
$ echo 'echo "hello from the host"' > ~/imx6ull/rootfs/usr/bin/hello
$ chmod +x ~/imx6ull/rootfs/usr/bin/hello

# On the target — no reboot:
[root@pa-mini:~]# hello
hello from the host

The host’s filesystem is the target’s filesystem. You edit, the target sees. This is what makes embedded Linux iteration feel reasonable.

For installing new kernel modules:

# On the host after a kernel rebuild:
$ make -C ~/imx6ull/src/linux INSTALL_MOD_PATH=~/imx6ull/rootfs modules_install

# On the target (no reboot for already-loaded modules; modprobe sees them):
[root@pa-mini:~]# modprobe my-driver

31.13 Verify everything

A sanity check sweep:

[root@pa-mini:~]# uname -a
Linux pa-mini 6.6.0 #1 SMP ... armv7l GNU/Linux

[root@pa-mini:~]# cat /proc/cpuinfo | head -5
processor   : 0
model name  : ARMv7 Processor rev 5 (v7l)
BogoMIPS    : 24.00
Features    : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 ...

[root@pa-mini:~]# ls /dev/ | head
console
fb0
mem
mmcblk0
null
random
ttymxc0
urandom
zero

[root@pa-mini:~]# mount
/dev/nfs on / type nfs (rw,...)
proc on /proc type proc (rw,...)
sysfs on /sys type sysfs (rw,...)
tmpfs on /tmp type tmpfs (rw,...)
devtmpfs on /dev type devtmpfs (rw,...)

[root@pa-mini:~]# free -h
              total        used        free      shared
Mem:           448M         12M        434M          0

All five check out: kernel running, CPU detected, devices enumerated, mounts active, memory roughly what we expect (512 MiB total minus kernel reservation).

31.14 Lab

  1. Build the BusyBox rootfs. Get to a shell over NFS.

  2. Write your own /etc/profile with a colored prompt. (PS1='\033[01;32m[\u@\h:\w]\$\033[00m ' for green.)

  3. Add a second user. Edit /etc/passwd, /etc/shadow, /etc/group to add a user dev with uid 1000. Log in as that user on a second console (or by su - dev).

  4. Persist /var/log/. Currently nothing writes to it. Modify rcS to redirect dmesg > /var/log/dmesg.txt so each boot logs to a host-visible file. (Since /var is on NFS, this works.)

  5. Add ntpd from BusyBox. Edit /etc/init.d/rcS to start /usr/sbin/ntpd -p pool.ntp.org in the background. Verify time syncs after boot.

  6. Compare with a dynamic BusyBox. Rebuild BusyBox without “Build static binary”. Confirm DNS lookups (nslookup, ping example.com) work, they wouldn’t with the static build.

31.15 Pitfalls

  • Forgetting chmod +x on rcS. Silent failure. Nothing in /proc or /sys.

  • /sbin/init not pointing at BusyBox. Kernel finds no init, kernel panics, you scratch your head. Verify with ls -l rootfs/sbin/init.

  • /etc/fstab has tabs vs spaces inconsistency. mount -a is lenient but some tools aren’t. Use either consistently.

  • NFS root_squash. Without no_root_squash, the target’s root user maps to nobody and chmod 600 /etc/shadow fails. The export option matters for embedded development.

  • /dev/console missing. If CONFIG_DEVTMPFS_MOUNT is off and rcS doesn’t manually mount devtmpfs, the console itself may be missing and kernel_init prints a warning. Always keep devtmpfs auto-mounted unless you have a specific reason.

  • Library version mismatch. If your cross-toolchain uses glibc 2.31 but your application was built against glibc 2.34, the application’s required symbols (memcpy@GLIBC_2.34) won’t resolve. Fix: rebuild the app against the rootfs’s libc, or upgrade the rootfs’s libc to match the app.

  • NFS over Wi-Fi. Sometimes works. Often drops packets and freezes the target. Always NFS over wired Ethernet.

31.16 Going deeper

  • The FHS specification at refspecs.linuxfoundation.org/FHS_3.0/. Concise and prescriptive.

  • BusyBox manual at busybox.net/about.html and the per-applet --help.

  • /etc/inittab documentation in examples/inittab inside the BusyBox source tree.

  • man 8 mount, man 5 fstab, man 5 inittab, the canonical references.

  • The nss-musl and nss-static trick, if you really need static binaries AND DNS, musl static + a simple resolver works where glibc static doesn’t.

Next chapter: Chapter 32: /proc, /sys, devtmpfs. With the rootfs running, we look at the three virtual filesystems that are how user-space sees the kernel.