Chapter 31: A root filesystem, by hand¶

udev: the user-space device manager that reacts to kernel device events and creates policy-driven /dev nodes.

What: the composition of a small root filesystem: BusyBox, the FHS directory tree, runtime libraries, /etc/inittab, /etc/init.d/rcS, /etc/fstab and an NFS-backed root. NFS: Network File System, which lets the target mount a host directory over Ethernet during development.

Why: Chapter 29 isolated early user space in an initramfs. This case adds startup configuration, shared-library dependencies and a host-backed development root. These pieces reveal responsibilities that an automated image builder later assembles. MCU bridge: Think of the rootfs as the firmware image’s file-backed runtime environment. On an MCU you link everything into flash. On Linux, programs and config live in this mounted tree. rootfs: root filesystem, the directory tree mounted at / that contains /bin, /etc, /dev, and libraries.

Focus: the handful of files in /etc/ that everything else depends on, inittab, init.d/rcS, fstab, passwd, group. Once you know these files, Buildroot, Yocto, and Ubuntu-base look like the same content generated by tools. Yocto: a metadata-driven build system for producing custom Linux distributions. Buildroot: a configuration-driven build system that produces a complete root filesystem and related images.

31.1 The FHS in 30 seconds¶

The Filesystem Hierarchy Standard says what each top-level directory is for. The embedded subset:

/                # root directory
├── bin/         # essential user binaries (ls, mv, sh, ...)
├── sbin/        # essential system binaries (init, mount, fsck, ...)
├── usr/         # non-essential user binaries; mirrors bin/ + sbin/ + lib/ structure
│   ├── bin/
│   ├── sbin/
│   └── lib/
├── lib/         # shared libraries that bin/ and sbin/ need
├── etc/         # configuration files (text, no binaries)
├── dev/         # device files (mostly auto-populated by devtmpfs)
├── proc/        # mount point for procfs (Ch 32)
├── sys/         # mount point for sysfs (Ch 32)
├── tmp/         # temporary files; usually tmpfs
├── var/         # variable data: logs, mailboxes, runtime state
│   ├── log/
│   └── run/
├── root/        # root user's home directory
├── home/        # user homes (often empty on embedded)
└── mnt/         # ad-hoc mount points

We create each of these directories. The ones that should contain files at boot (bin/, sbin/, lib/, etc/) we populate now. The ones the kernel or daemons fill (proc/, sys/, dev/, tmp/, var/run/) stay empty until mount time.

31.2 The plan¶

host                                     target
─────────────────────────────────────    ────────────────────────────────
~/imx6ull/rootfs/                        / (mounted via NFS from host)
  ├── bin/busybox + symlinks             same
  ├── sbin/init → /bin/busybox           same
  ├── etc/inittab, rcS, fstab            same
  ├── lib/*.so.* (from cross-toolchain)  same
  ├── proc/  (empty; mount point)        proc filesystem mounted here
  ├── sys/   (empty)                     sysfs mounted here
  └── dev/   (empty)                     devtmpfs mounted here

NFS server on host exports ~/imx6ull/rootfs/.
Kernel boots with bootargs: root=/dev/nfs nfsroot=192.168.7.1:...

In this arrangement, the target sees a host-exported filesystem as its root. Understanding that relationship does not require setting up an NFS server or building the example.

31.3 Build BusyBox statically¶

$ cd ~/imx6ull/src
$ wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2
$ tar xjf busybox-1.36.1.tar.bz2
$ cd busybox-1.36.1

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- defconfig
$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- menuconfig

In menuconfig, enable two settings explicitly:

Settings  --->
    [*]   Build static binary (no shared libs)
    [*]   Enable Unicode support
    [*]     Check $LC_ALL, $LC_CTYPE and $LANG environment variables

Build:

$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- -j$(nproc)
$ arm-none-linux-gnueabihf-strip busybox
$ ls -lh busybox
-rwxr-xr-x 1 you you 580K busybox

580 KB statically linked. (~450 KB with musl-gcc.)

Static or dynamic? Static is simpler to deploy, no libraries needed on the target. The drawback is DNS resolution over the network doesn’t work with static glibc, because glibc’s NSS (Name Service Switch, the pluggable backend behind gethostbyname / getaddrinfo) is dlopen’d at runtime even from a “static” binary. A static glibc binary can still resolve names listed in /etc/hosts. What it can’t do is real DNS over the network. If you need real DNS, build BusyBox dynamically and copy the toolchain libraries into the rootfs. This chapter does both: static for the first boot, dynamic in §31.10.

31.4 Populate the rootfs¶

$ cd ~/imx6ull
$ mkdir -p rootfs/{bin,sbin,etc/init.d,lib,proc,sys,dev,tmp,var/{log,run},root,home,mnt,usr/{bin,sbin,lib}}
$ cd rootfs

# Install BusyBox into the rootfs
$ make -C ~/imx6ull/src/busybox-1.36.1 \
       ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- \
       CONFIG_PREFIX=$PWD install

make install reads BusyBox’s config and creates symlinks for every applet:

$ ls bin/ | head
[
[[
ash
base32
base64
busybox
cat
chattr
chgrp
chmod
...
$ ls bin/ | wc -l
246

246 applets, all pointing at bin/busybox. The single 580 KB binary acts as 246 commands. Which one it acts as depends on argv[0].

$ ls -l sbin/init
lrwxrwxrwx 1 you you 14 ... sbin/init -> ../bin/busybox

sbin/init → busybox. When the kernel execs /sbin/init, busybox examines argv[0] = /sbin/init and acts as init.

31.5 Create `/etc/inittab`¶

BusyBox init reads /etc/inittab to know what to do. Format:

<id>:<runlevels>:<action>:<process>

For BusyBox, <id> is the controlling tty (empty for “use the system default console”), <runlevels> is ignored. The action determines when the process runs.

$ cat > etc/inittab <<'EOF'
# /etc/inittab — busybox init configuration

# System initialization. Runs once at boot, before anything else.
::sysinit:/etc/init.d/rcS

# Spawn a shell on the console. respawn = restart automatically when it exits.
console::respawn:-/bin/sh

# What to do on Ctrl-Alt-Del. (No keyboard? Harmless to keep.)
::ctrlaltdel:/sbin/reboot

# What to do on shutdown.
::shutdown:/bin/umount -a -r
::shutdown:/sbin/swapoff -a
EOF

The leading - on -/bin/sh makes it a login shell (reads /etc/profile and ~/.profile). The eight standard BusyBox init actions:

Action	When the process runs
`sysinit`	Once, at boot, before anything else
`wait`	Once, init waits for it to finish before continuing
`once`	Once, init does not wait
`respawn`	Started; whenever it exits, restart it
`askfirst`	Like respawn, but prints “Please press Enter to activate” and waits
`restart`	When init receives `SIGHUP` (i.e. asked to re-read inittab)
`ctrlaltdel`	When init receives `SIGINT` (Ctrl-Alt-Del on a real keyboard)
`shutdown`	On halt/poweroff/reboot

For a development board you typically need only sysinit, respawn, and shutdown.

31.6 Create `/etc/init.d/rcS`¶

The shell script /etc/inittab’s sysinit line runs. This is where most of the per-boot setup lives:

$ cat > etc/init.d/rcS <<'EOF'
#!/bin/sh

# Search paths for the rest of this session.
PATH=/sbin:/bin:/usr/sbin:/usr/bin
LD_LIBRARY_PATH=/lib:/usr/lib
export PATH LD_LIBRARY_PATH

# Mount the virtual filesystems described in /etc/fstab.
/bin/mount -a

# devtmpfs is mounted automatically by the kernel if CONFIG_DEVTMPFS_MOUNT=y.
# If yours isn't, mount it manually:
mountpoint -q /dev || /bin/mount -t devtmpfs none /dev

# devpts gives us pseudo-terminals for things like ssh, screen, etc.
mkdir -p /dev/pts
/bin/mount -t devpts none /dev/pts

# Hook mdev into the kernel's hotplug mechanism. (mdev is BusyBox's udev.)
echo /sbin/mdev > /proc/sys/kernel/hotplug

# Populate /dev/ from /sys/ for devices already enumerated before mdev started.
/sbin/mdev -s

# Set a hostname.
/bin/hostname pa-mini

# Start any background daemons here (sshd, ntpd, your app, ...).

echo "*** rootfs ready ***"
EOF
$ chmod +x etc/init.d/rcS

The chmod +x matters. Without execute permission, BusyBox init skips the script silently. No /proc, no /sys, no /dev/pts. This is the most common first-time-rootfs bug.

31.7 Create `/etc/fstab`¶

mount -a consults /etc/fstab to know what to mount. Five-column format:

<device> <mountpoint> <fs-type> <options> <dump-freq> <fsck-order>

$ cat > etc/fstab <<'EOF'
# device      mountpoint    type      options     dump  pass
proc          /proc         proc      defaults    0     0
sysfs         /sys          sysfs     defaults    0     0
tmpfs         /tmp          tmpfs     defaults    0     0
tmpfs         /run          tmpfs     defaults    0     0
EOF

Four virtual filesystems:

proc: the procfs (Ch 32) at /proc.
sysfs: the sysfs (Ch 32) at /sys.
tmpfs at /tmp: RAM-backed tmpfs for /tmp. Wipes on reboot, which is what /tmp is supposed to do.
tmpfs at /run: RAM-backed tmpfs for daemon PID/socket files.

No physical disk mounts, the rootfs itself is mounted by the kernel before this script runs, via the root= cmdline. Other partitions (data, log) would be additional /etc/fstab lines.

31.8 Create `/etc/passwd` and `/etc/group`¶

Even a single-user system needs minimal versions:

$ cat > etc/passwd <<'EOF'
root:x:0:0:root:/root:/bin/sh
EOF
$ cat > etc/group <<'EOF'
root:x:0:
EOF
$ cat > etc/shadow <<'EOF'
root::0:0:99999:7:::
EOF
$ chmod 600 etc/shadow

root:: (empty password field in shadow) means root logs in with no password. Insecure. Fine for an early dev image. Replace with a real hash before production.

31.9 Create `/etc/profile`¶

Read by login shells. Sets the prompt, default permissions, etc.:

$ cat > etc/profile <<'EOF'
# /etc/profile — login-shell configuration

export PATH=/sbin:/bin:/usr/sbin:/usr/bin
export LD_LIBRARY_PATH=/lib:/usr/lib

umask 022

# A nicer prompt: [user@host:cwd]#
PS1='[\u@\h:\w]\$ '
export PS1

# Useful aliases.
alias ll='ls -lh'
alias la='ls -la'
alias ..='cd ..'
EOF

After login, you see [root@pa-mini:~]# instead of just #. Small but pays back the first time you have two ssh sessions open.

31.10 Copy libraries (for dynamic binaries)¶

The static BusyBox example in §31.3 avoids ordinary shared-library dependencies. A dynamically linked application, ssh or another tool needs the shared libraries present at the runtime paths the loader expects.

The libraries live inside your cross-toolchain installation:

$ . ~/imx6ull/scripts/env.sh
$ TOOLCHAIN=$ARM_LINUX_TOOLCHAIN
$ cd ~/imx6ull/rootfs

# The "main" libraries (libc, libm, libpthread, libdl, ...) and the dynamic linker
$ cp -d $TOOLCHAIN/arm-none-linux-gnueabihf/libc/lib/*.so* lib/
$ cp -d $TOOLCHAIN/arm-none-linux-gnueabihf/libc/lib/*.a lib/

# The dynamic linker itself MUST be the real file, not a symlink.
$ rm lib/ld-linux-armhf.so.3
$ cp $TOOLCHAIN/arm-none-linux-gnueabihf/libc/lib/ld-linux-armhf.so.3 lib/

# usr/lib for less-common libraries
$ cp -d $TOOLCHAIN/arm-none-linux-gnueabihf/libc/usr/lib/*.so* usr/lib/

Two things to know:

-d preserves symlinks. Most .so.N files are symlinks to .so.N.M. Without -d, cp follows the symlink and copies the target. Both files end up as identical full copies. With -d, the symlink stays a symlink and you save space.
ld-linux-armhf.so.3 is the dynamic linker itself. In the toolchain it’s a symlink to the real ld-2.31.so. On the target it must be a real file at the path the ELF binaries’ INTERP section points to (/lib/ld-linux-armhf.so.3). The rm + cp (without -d) dance after the wildcard copy forces the real file.

ELF: Executable and Linkable Format, the standard Linux object and executable file format.

Total size: ~60 MB unstripped for glibc, ~5–10 MB after strip and removing the locales / NSS modules you don’t need. ~5 MB for a comparable musl install. The glibc bulk is mostly NSS modules, locale data, and unused libraries, all of which you’d strip in a production build.

31.11 Export over NFS, boot¶

Add to /etc/exports on the host:

/home/you/imx6ull/rootfs *(rw,sync,no_root_squash,no_subtree_check)

Restart NFS:

$ sudo exportfs -ar
$ showmount -e localhost
Export list for localhost:
/home/you/imx6ull/rootfs *

In U-Boot:

MCU bridge: Think of U-Boot like a much larger boot stub plus debug monitor: it initializes hardware, loads the next image, and gives you commands before Linux starts. U-Boot: the bootloader that initializes enough hardware to load and start the Linux kernel.

=> setenv bootargs 'console=ttymxc0,115200 earlycon \
                    root=/dev/nfs nfsroot=192.168.7.1:/home/you/imx6ull/rootfs,vers=3,nolock,tcp \
                    ip=192.168.7.2:192.168.7.1:192.168.7.1:255.255.255.0::eth0:off \
                    rw rootwait'
=> setenv bootcmd 'tftp 0x82000000 zImage && tftp 0x83000000 imx6ull.dtb && bootz 0x82000000 - 0x83000000'
=> saveenv
=> boot

After kernel boot:

[    3.412345] VFS: Mounted root (nfs filesystem) on device 0:16.
[    3.428901] Run /sbin/init as init process
*** rootfs ready ***

Please press Enter to activate this console.

[Enter]

[root@pa-mini:~]#

The example’s shell uses / provided by the host over NFS, rather than a local root partition.

31.12 The development loop¶

Now the win:

# On the host:
$ echo 'echo "hello from the host"' > ~/imx6ull/rootfs/usr/bin/hello
$ chmod +x ~/imx6ull/rootfs/usr/bin/hello

# On the target — no reboot:
[root@pa-mini:~]# hello
hello from the host

The host’s filesystem is the target’s filesystem. You edit, the target sees. This is what makes embedded Linux iteration feel reasonable.

For installing new kernel modules:

# On the host after a kernel rebuild:
$ make -C ~/imx6ull/src/linux INSTALL_MOD_PATH=~/imx6ull/rootfs modules_install

# On the target (no reboot for already-loaded modules; modprobe sees them):
[root@pa-mini:~]# modprobe my-driver

31.13 Verify everything¶

A sanity check sweep:

[root@pa-mini:~]# uname -a
Linux pa-mini 6.6.0 #1 SMP ... armv7l GNU/Linux

[root@pa-mini:~]# cat /proc/cpuinfo | head -5
processor   : 0
model name  : ARMv7 Processor rev 5 (v7l)
BogoMIPS    : 24.00
Features    : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 ...

[root@pa-mini:~]# ls /dev/ | head
console
fb0
mem
mmcblk0
null
random
ttymxc0
urandom
zero

[root@pa-mini:~]# mount
/dev/nfs on / type nfs (rw,...)
proc on /proc type proc (rw,...)
sysfs on /sys type sysfs (rw,...)
tmpfs on /tmp type tmpfs (rw,...)
devtmpfs on /dev type devtmpfs (rw,...)

[root@pa-mini:~]# free -h
              total        used        free      shared
Mem:           448M         12M        434M          0

All five check out: kernel running, CPU detected, devices enumerated, mounts active, memory roughly what we expect (512 MiB total minus kernel reservation).

31.14 Key Notes¶

A rootfs supplies executables, libraries and configuration. Directories such as /proc, /sys and /dev are mount points whose runtime contents are not produced by copying ordinary files.
BusyBox selects applets through invocation names, so its binary, enabled applets and installed links jointly define the available commands and init behavior.
inittab selects startup and respawn actions, rcS performs per-boot setup, and fstab describes mounts. Their responsibilities are related but not interchangeable.
Account files and file modes define access policy. An empty root password and an unrestricted development shell are not production authentication designs.
Dynamic executables depend on a matching target loader and libraries, not the host’s library tree. Copying a binary alone does not establish a usable runtime.
NFS makes the host-exported tree part of the target’s runtime dependency. Export permissions, Linux networking and persistent host configuration remain independent concerns.
Storage location determines persistence: RAM-backed /tmp and /run disappear at reboot, while data on an NFS export belongs to the host-backed tree. Paths alone do not guarantee durable logging.

31.15 Pitfalls¶

Forgetting chmod +x on rcS. Silent failure. Nothing in /proc or /sys.
/sbin/init not pointing at BusyBox. Kernel finds no init, kernel panics, you scratch your head. Verify with ls -l rootfs/sbin/init.
/etc/fstab has tabs vs spaces inconsistency. mount -a is lenient but some tools aren’t. Use either consistently.
NFS root_squash. Without no_root_squash, the target’s root user maps to nobody and chmod 600 /etc/shadow fails. The export option matters for embedded development.
/dev/console missing. If CONFIG_DEVTMPFS_MOUNT is off and rcS doesn’t manually mount devtmpfs, the console itself may be missing and kernel_init prints a warning. Always keep devtmpfs auto-mounted unless you have a specific reason.
Library version mismatch. If your cross-toolchain uses glibc 2.31 but your application was built against glibc 2.34, the application’s required symbols (memcpy@GLIBC_2.34) won’t resolve. Fix: rebuild the app against the rootfs’s libc, or upgrade the rootfs’s libc to match the app.
NFS over Wi-Fi. Sometimes works. Often drops packets and freezes the target. Always NFS over wired Ethernet.

31.16 Going deeper¶

The FHS specification at refspecs.linuxfoundation.org/FHS_3.0/. Concise and prescriptive.
BusyBox manual at busybox.net/about.html and the per-applet --help.
/etc/inittab documentation in examples/inittab inside the BusyBox source tree.
man 8 mount, man 5 fstab, man 5 inittab, the canonical references.
The nss-musl and nss-static trick, if you really need static binaries AND DNS, musl static + a simple resolver works where glibc static doesn’t.

Next chapter: Chapter 32: /proc, /sys, devtmpfs. These virtual filesystems expose kernel state and device interfaces within the user-space filesystem view.