Rustbelt

Rustbelt is Tyler's hobby x86_64 kernel written in Rust. It started from the 2nd edition of Philipp Oppermann's Writing an OS in Rust and keeps going past the end of it. It boots in QEMU straight into its own shell: a user-mode program that has a Unix feel with DOS touches. It offers ls, cat, pipes and redirection, plus dir, type, ver, and drive letters from A: to E:.

Under the shell:

• A framebuffer console and serial output. The kernel draws its text console with a built-in 8x16 bitmap font on a 1024x768 virtio-gpu framebuffer (falling back to the old VGA text screen when there's no virtio-gpu), and mirrors everything to the serial port (COM1) and a second console, hvc0.
• CPU exceptions. It handles breakpoints, page faults, and double faults (on their own stack).
• Several CPUs. It starts every CPU core QEMU gives it (-smp 4) through the local APIC, and its scheduler runs threads and programs on all of them at once.
• Timers and keyboard. The PIT drives an uptime clock and wakes sleeping threads; each CPU's local APIC timer drives preemption. The PS/2 keyboard feeds a tty with cooked and raw modes.
• Memory. It walks and edits its own page tables. It has a kernel heap with a fixed-size-block allocator, and frees every frame a process or thread used when it exits. Every stack has an unmapped guard page under it, so an overflow is caught and reported instead of silently corrupting memory.
• Multitasking. It multitasks two ways: an async/await executor and preemptive kernel threads. Threads can sleep and wait for events (wait queues) without using any CPU while they wait.
• User programs. Small Rust ELF binaries run in ring 3 as processes, each with its own address space. Their memory is freed when they exit. They talk to the kernel through 101 syscalls, numbered like Linux's: read, write, open, stat, getdents64, pipe2, uname, socket, connect, getrandom... A few are our own: spawn, mounts, sysinfo, resolve, netinfo and trace_me.
• The Unix process model. fork with copy-on-write, execve with Linux's argv/envp/auxiliary-vector stack layout, wait4, mmap (anonymous, shared, and file-backed private mappings), mprotect, munmap and brk with demand paging, and signals with real signal frames: handlers, masks, sigreturn, SIGSEGV on bad accesses, SIGCHLD, SIGPIPE, and Ctrl+C to the foreground process group. The shell runs programs with fork + exec and has job control (kill, fg, bg, Ctrl+Z).
• Files. Files live behind a Linux-style VFS: a ramfs at / and /tmp, a FAT32 partition mounted at /mnt/disk0, an ext2 disk at /mnt/disk1, an exFAT disk at /mnt/disk2 and a btrfs disk at /mnt/disk3. The ext2 and exFAT drivers are hand-written and read and write their disks (ext2: permissions, symlinks, indirect blocks; exFAT: long Unicode names, case-insensitive lookup, FAT chains). The real e2fsck and fsck.exfat check what they write. The btrfs driver, also hand-written, is read-only: copy-on-write B-trees, chunk mapping (single and DUP), crc32c checksums on every node and data block, zlib/lzo/zstd compression, holes, symlinks and subvolumes. The disks are virtio-blk devices found by PCI enumeration, with their GPT/MBR partitions parsed. The shell and its programs are loaded from the FAT disk.
• Networking. A virtio-net card (interrupt-driven) carries Ethernet frames; the smoltcp crate does ARP, IPv4, ICMP, UDP, TCP, DHCP and DNS on top. User programs get BSD-style sockets (socket, connect, bind, listen, accept, sendto, recvfrom) and come with ifconfig, ping, host, fetch (a tiny wget) and httpd (a tiny web server you can reach from the host).
• Virtual devices. Hand-written drivers for the modern (virtio 1.0) virtio-rng (entropy: the getrandom syscall, /dev/random, smoltcp's seed and random TCP ports), virtio-console (/dev/hvc0, a second terminal the host reaches through a socket; getty runs a shell on it) and virtio-gpu in 2D mode (/dev/fb0 with ioctls; tile draws a tiling-window-manager mock-up on it).
• What a C library needs (stage 15). poll, select and epoll over pipes, sockets, terminals and files; O_NONBLOCK with EAGAIN; a loopback interface (127.0.0.1) so programs talk TCP/UDP to each other without a network card; threads: clone(CLONE_VM | CLONE_THREAD ...), futexes, thread-local storage (arch_prctl), gettid, tgkill, exit vs exit_group, running on all 4 CPUs at once; real ttys and ptys with termios, canonical/raw modes, echo, ^C/^Z/^\ as signals on every terminal, /dev/tty, /dev/ptmx and /dev/pts/N; plus nanosleep, clock_gettime(CLOCK_REALTIME), readv/writev, pread/pwrite and readlink /proc/self/exe.
• Ready for musl (stage 15.1). Every thread has its own FPU/SSE/AVX registers (XSAVE or FXSAVE, also in signal frames, cleared by execve); the file calls take C-string paths with Linux's signatures (openat and the other *at() calls, Linux's struct stat, rename, access), and rawabi, a C program built by gcc with no libc at all, runs unmodified.
• libc batch 1 (stage 16). The calls musl and the dash shell make once a program is running: rt_sigsuspend (so dash's wait sleeps instead of spinning), sigtimedwait, sigpending, queued real-time signals, sigaltstack; a per-process umask and rlimits; fcntl record locks; truncate, fsync, chmod, utimensat and real file times; prctl, sched_getaffinity, madvise and robust futex lists. Orphaned children are adopted by a subreaper or clean themselves up.
• Unix sockets and /proc/self (stage 18). AF_UNIX stream, datagram and seqpacket sockets with file-system, abstract and unnamed addresses, socketpair, sendmsg/recvmsg that can pass open files between programs (SCM_RIGHTS) and say who sent them (SCM_CREDENTIALS, SO_PEERCRED), real buffer limits, and /proc/self/fd and /proc/self/exe. That's what Wayland, D-Bus and X11 talk over.

It has a cargo test suite that runs inside QEMU, and a headless self-test that drives the shell with a script.

Most code follows the blog_os sources (MIT / Apache-2.0) closely, with extra comments. Everything else is new: the serial mirroring, uptime clock, demo features and scripts, and everything past chapter 12. That covers preemptive threads, user mode, syscalls, fork/exec/mmap/signals, ELF loading, the VFS, PCI, virtio-blk, partitions, FAT32, the shell, the ext2, exFAT and btrfs drivers, multi-core (SMP) support, the virtio-net driver and socket layer, the virtio-rng, virtio-console and virtio-gpu drivers with the framebuffer console, stage 15's poll/epoll, loopback, threads, futexes and ptys, and stage 15.1's FPU state switching and Linux path calls, and stage 16's libc batch 1. The 8x16 font is Spleen by Frederic Cambus (BSD 2-clause, license in src/font.rs).

New to Rust and C? Start here

Rustbelt is meant to be read. You don't need to know Rust or C well to follow it; here is a path through it, and the handful of ideas that make kernel code look strange at first.

A reading order. Each file starts with a comment explaining what it is for. Read them roughly in the order the kernel uses them:

1. src/main.rs -- kernel_main is the first Rust code that runs. It calls the init functions below in order, then starts the shell.
2. src/vga_buffer.rs, src/serial.rs -- printing text: the screen is just memory at address 0xb8000, the serial port is a few I/O ports.
3. src/interrupts.rs, src/gdt.rs -- what happens when the CPU is interrupted (a key press, a timer, a crash).
4. src/memory.rs, src/allocator.rs -- page tables (how addresses are translated) and the heap (Box, Vec).
5. src/thread.rs -- threads, the scheduler and sleeping. src/smp.rs and src/apic.rs -- how the other CPUs get started.
6. src/process.rs, src/syscall.rs, src/elf.rs -- user programs.
7. user/hello.rs, then user/sh.rs -- what a program on Rustbelt looks like.
8. src/fs/ -- the file systems, once the rest makes sense.
9. src/virtio/ -- devices: mod.rs and queue.rs (how a driver talks to a virtio device), then rng.rs (the smallest driver), console.rs and gpu.rs; src/fbcon.rs draws text on the framebuffer.

Rust in kernel code, in five ideas:

• #![no_std]: there is no operating system underneath, so Rust's standard library (files, threads, println!) isn't available. Only core (basic types) and, once there's a heap, alloc (Box, Vec, String). Rustbelt defines its own println!.
• unsafe: Rust normally guarantees memory safety; inside an unsafe {} block you promise it instead. Kernels need it to touch hardware (raw addresses, port I/O, CPU registers). Rustbelt keeps these blocks small and says in a comment why each one is OK.
• Ownership and Drop: every value has one owner, and when the owner goes away Rust runs its Drop. Rustbelt uses that for cleanup: dropping a KernelStack unmaps it and frees its memory (src/stack.rs); dropping the last Arc<Process> frees the whole address space.
• Mutex and Arc: spin::Mutex<T> is a lock: .lock() gives you the T and unlocks automatically when you're done. Arc<T> is a shared pointer with a reference count, so several threads can hold the same process.
• global_asm! / asm!: a few things (switching stacks, entering ring 3, starting a CPU) need raw assembly. Each block is commented line by line.

C ideas you'll see (the hardware docs and OSDev wiki are written in C terms): a pointer is just an address; #[repr(C)] lays out a struct the way C would, so it matches what the CPU or a disk format expects; volatile reads and writes tell the compiler a memory access really has to happen (it's a device, not ordinary memory).

Kernel words, briefly: a page is 4 KiB of memory; a frame is the physical memory behind it. An interrupt stops the CPU and runs a handler. Ring 0 is the kernel, ring 3 user programs. A syscall is a user program asking the kernel for something. SMP means several CPUs sharing memory; the first one to boot is the BSP, the others APs.

Try things. cargo run, then in the shell: help, ver, sysinfo, spin 200 > A:/tmp/spin.txt & followed by sysinfo (watch one CPU go to 100%), spin 1000 and then Ctrl+C, sigtest, overflow (a stack overflow, caught), tile (the console moves into a window; tile -r moves it back), random -d 6 5. Change something small -- the message in user/overflow.rs, the bar width in user/sysinfo.rs -- and run cargo test to check nothing broke. When something crashes, the [kernel] lines on the serial output tell you which handler caught it and where.

What's installed (on this box)

Tool	Version	Why
rustup	1.29.2	toolchain manager (in ~/.cargo/bin, ahead of Debian's /usr/bin/cargo)
Rust nightly	rustc 1.101.0-nightly (2026-10-29)	custom JSON targets + build-std are nightly-only
rust-src component	–	source of core, recompiled for our bare-metal target
llvm-tools-preview component	–	llvm-objcopy etc., used by bootimage
bootimage (cargo subcommand)	0.10.5	links kernel + bootloader into a bootable disk image
QEMU (qemu-system-x86, qemu-system-gui)	10.0.13	the emulator we boot in (-gui adds the GTK/SDL window)
dosfstools, mtools, fdisk	Debian packages	build.rs makes the FAT32 test disk with sfdisk + mkfs.vfat + mmd/mcopy
e2fsprogs, exfatprogs	Debian packages	the ext2 and exFAT images, and e2fsck / fsck.exfat to check Rustbelt's writes
btrfs-progs	6.14	mkfs.btrfs --rootdir and btrfs-convert make the btrfs images; btrfs check --readonly checks them

rust-toolchain.toml pins the project to nightly with the two components, so cargo inside this directory picks them up automatically. It also installs the built-in x86_64-unknown-none target, used to compile the user programs.

Crates: bootloader 0.9.35 (BIOS bootloader used by the 2nd edition), volatile 0.2, spin 0.5, x86_64 0.15, uart_16550 0.3, lazy_static 1.5, pic8259 0.11 (interrupt controller), pc-keyboard 0.7 (scancode decoding), linked_list_allocator 0.9 (fallback heap allocator), fatfs (git, no_std + alloc; FAT12/16/32 reading and writing), miniz_oxide 0.8 (zlib inflate) and ruzstd 0.8 (zstd decoding) for btrfs's compressed extents; LZO is hand-written (src/fs/lzo.rs).

Fresh setup on another machine:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh   # rustup
rustup toolchain install nightly -c rust-src -c llvm-tools-preview -t x86_64-unknown-none
cargo install bootimage
sudo apt install qemu-system-x86 qemu-system-gui dosfstools mtools fdisk e2fsprogs exfatprogs btrfs-progs

Build and run

cargo bootimage                     # build -> target/x86_64-rust_kernel/debug/bootimage-rustbelt.bin
cargo run                           # build + boot in a QEMU window; serial output also prints in the terminal

Using it: cargo run opens a QEMU window. The kernel prints a few boot lines, then you're at the shell prompt A:/ $. Click into the window and type, for example help, cd B:, ls or cat readme.txt | wc, or ls -l C:/ for the ext2 disk, ls -l D:/ for the exFAT one, or ls -l E:/ for the read-only btrfs one. See The shell for everything it can do. Everything on the screen is also mirrored to your terminal over serial. Typing exit ends the shell, and the kernel starts a fresh one.

Networking in QEMU: cargo run gives the guest QEMU's user-mode network (-netdev user, also called slirp). It needs no root and no setup on the host: QEMU itself plays a little router with a DHCP server (the guest gets 10.0.2.15), a gateway at 10.0.2.2 that answers pings and forwards TCP/UDP out through the host's own sockets, and a DNS forwarder at 10.0.2.3. Two extras are set up in Cargo.toml:

• guestfwd: connections from the guest to 10.0.2.100:80 and :7 start scripts/http-reply.sh (a 20-line HTTP responder) and cat (an echo server) on the host, so fetch http://10.0.2.100/ always works, even with no internet.
• hostfwd=tcp:127.0.0.1:5580-:80: run httpd in the shell, then curl http://127.0.0.1:5580/ on the host reaches it.

Try ifconfig, ping 10.0.2.2, host example.com or wget -o D:/page.html http://example.com/.

The second console (hvc0): cargo run connects the guest's virtio-console to a UNIX socket, target/hvc0.sock. The kernel log is mirrored there; to get a shell on it, type getty /dev/hvc0 & in the QEMU window, then on the host:

socat - UNIX-CONNECT:target/hvc0.sock     # or: nc -U target/hvc0.sock

Type commands and press Enter (that shell reads whole lines, like a script, and echoes each one after its prompt; exit ends it and getty starts a new one). The headless runs use a file instead of the socket (-chardev file,...,input-path=), see below.

The display is virtio-gpu (-vga none -device virtio-vga): the guest sets up a 1024x768 framebuffer and QEMU shows it scaled in its window. Leave those two options out and the kernel falls back to the VGA text console.

To get the mouse and keyboard back from QEMU, press Ctrl+Alt+G. To quit, close the window or press Ctrl+C in the terminal. Over SSH without X, use cargo run -- -display curses -vga std to see the screen right in your terminal (curses can only show text mode, so this switches back to the VGA console), and quit from another terminal with pkill qemu-system.

Headless (no window, good for SSH / scripts):

./scripts/run-headless.sh                 # boot, the shell runs disk/scripts/selftest.sh, QEMU exits;
                                          # passes if the shell exits 0 AND the expected lines appear
                                          # (and the host's curl got a page from the guest's httpd,
                                          # and the shell on hvc0 answered in target/headless-hvc0.txt)
./scripts/run-headless.sh stack-overflow  # deliberately overflow the stack; passes if the double-fault handler catches it
./scripts/run-headless.sh page-fault      # write to unmapped 0xdeadbeaf; passes if the page-fault handler catches it
EXT2_IMG=copy.img ./scripts/run-headless.sh  # use copy.img as the ext2 disk, writable (no snapshot)
EXFAT_IMG=copy.img ./scripts/run-headless.sh # the same for the exFAT disk
BTRFS_IMG=other.img ./scripts/run-headless.sh # another btrfs image as E: (always snapshot=on)
./scripts/ext2-check.sh                   # Rustbelt writes to copies of the ext2 images, then the
                                          # host's e2fsck -fn checks them (see "ext2 (stage 8)")
./scripts/exfat-check.sh                  # the same for exFAT with fsck.exfat -n (see "exFAT (stage 9)")
./scripts/btrfs-check.sh                  # btrfs check --readonly on the images, then every file Rustbelt
                                          # reads is compared with the host's copy (see "btrfs (stage 10)")
cargo run --features headless -- -display none                 # the self-test via cargo
                                          # (QEMU's exit status is 33 = success; cargo just reports it as non-zero)
cargo run --features demos                # the earlier stages' demos first, then the shell
./scripts/screenshot.sh [out.png] [session.txt] [expected...]
                                          # boot, type a shell session via QEMU monitor `sendkey`
                                          # (one command per line; {BS} = Backspace, {UP} = up arrow),
                                          # check the expected texts appear, save the screen (.png, or .ppm)
                                          # (default: the session in boot.png); exits 1 if text is missing
./scripts/gpu-check.sh                    # type `tile`, dump the virtio-gpu screen, compare 12 pixels
                                          # against the colours tile.rs draws (see "Virtual drivers")

Cargo features (in Cargo.toml):

Feature	Effect
(none)	boot, start the shell on the console (the default)
headless	the shell runs /mnt/disk0/scripts/selftest.sh instead of reading the keyboard; its exit code becomes QEMU's (0 -> success); a 60 s timer safety net fails the run if it hangs; panics exit with failure
demos	before the shell: the breakpoint, paging and heap demos, threads A/B/C, the async example, and the hello/evil/spin/demo programs of stages 2-6
stack-overflow	after boot, recurse forever to test the double-fault handler (with headless: exit success when caught)
page-fault	after boot, write to unmapped address 0xdeadbeaf (with headless: exit success when caught)

Expected serial output of ./scripts/run-headless.sh. Boot is quick now that the demos are behind a feature; most of the output is the shell running disk/scripts/selftest.sh. Each command is echoed after its prompt. The [process] forked/exec/exited lines are the kernel's own log (every program is started with fork + exec since stage 14; the ... lines are cut here). They go to serial only, so the VGA screen shows just the shell session (and kernel messages such as a killed process).

Rustbelt 0.15.0 -- a hobby x86_64 kernel in Rust
[smp] 4 CPUs online
[pci] 00:00.0 8086:1237 host bridge
... (12 PCI devices: the network card, the GPU, the rng, the console, 4 disks)
[pci] 00:03.0 1af4:1050 display controller
[pci] 00:04.0 1af4:1005 unclassified
[pci] 00:05.0 1af4:1003 communication controller
...
[pci] 12 devices on the PCI bus
[disk] vda: virtio-blk, 131072 sectors (64 MiB)
[disk] vda: GPT partition table, 1 partition(s)
[disk] vda1: start 2048 sectors 126976 type Microsoft basic data (FAT/exFAT/NTFS) name "fat32"
[disk] vdb: virtio-blk, 65536 sectors (32 MiB)
[disk] vdb: no partition table (NoSignature)
[disk] vdc: virtio-blk, 65536 sectors (32 MiB)
[disk] vdc: no partition table (WholeDiskExfat)
[disk] vdd: virtio-blk, 262144 sectors (128 MiB)
[disk] vdd: no partition table (NoSignature)
[disk] vda1: FAT32 volume "TYLERDISK" mounted at /mnt/disk0
[disk] vdb: ext2 volume "RUSTBELT" (rev 1, 1 KiB blocks, 4 groups) mounted at /mnt/disk1
[disk] vdc: exFAT volume "TYLEREXFAT" (4 KiB clusters, 7680 clusters) mounted at /mnt/disk2
[disk] vdd: btrfs volume "TYLERBTRFS" (16 KiB nodes, metadata DUP, data single, 2 subvolumes, read-only) mounted at /mnt/disk3
[vfs] mounted ramfs at /
[vfs] mounted ramfs at /tmp
[vfs] mounted devfs at /dev
[vfs] mounted fat32 at /mnt/disk0
[vfs] mounted ext2 at /mnt/disk1
[vfs] mounted exfat at /mnt/disk2
[vfs] mounted btrfs at /mnt/disk3
[virtio] rng: virtio-rng at 00:04.0
[virtio] hvc0: virtio-console at 00:05.0, IRQ 10
[virtio] gpu: virtio-gpu 2D at 00:03.0, display 1280x800, framebuffer 1024x768 (3072 KiB)
[virtio] fbcon: text console on the framebuffer, 128x48 characters
[net] lo: loopback, 127.0.0.1/8
[net] eth0: virtio-net, MAC 52:54:00:12:34:56, IRQ 10
[net] DHCP lease: 10.0.2.15/24, gateway 10.0.2.2, DNS 10.0.2.3
[process] started pid 1 (sh), entry 0x800040ada5
A:/ $ # Rustbelt shell self-test: `cargo run --features headless` runs this with
A:/ $ # `sh /mnt/disk0/scripts/selftest.sh` (B:/scripts/selftest.sh) and scripts/run-headless.sh checks the output.
A:/ $ ver
Rustbelt 0.15.0 (stage 15: libc plumbing) x86_64
A:/ $ drives
A:  /            ramfs
B:  /mnt/disk0   fat32
C:  /mnt/disk1   ext2
D:  /mnt/disk2   exfat
E:  /mnt/disk3   btrfs
A:/ $ pwd
/
A:/ $ cd B:
B:/ $ ls
[process] pid 1 forked pid 2
[process] pid 2 exec /mnt/disk0/bin/ls
bin/        notes/      readme.txt  scripts/
[process] pid 2 (ls) exited with code 0
B:/ $ ls -l notes
[process] pid 1 forked pid 3
[process] pid 3 exec /mnt/disk0/bin/ls
       125  todo.txt
[process] pid 3 (ls) exited with code 0
B:/ $ type B:/readme.txt
[process] pid 1 forked pid 4
[process] pid 4 exec /mnt/disk0/bin/cat
Hello from the FAT32 disk! This file lives on a virtio-blk disk, in a GPT
partition, in a FAT32 file system -- read through the kernel's VFS.
In the Rustbelt shell this disk is drive B: -- try `cd B:` and `ls`.
[process] pid 4 (cat) exited with code 0
B:/ $ cat b:notes/todo.txt | grep -i EXT | wc -l
[process] pid 1 forked pid 5
[process] pid 1 forked pid 6
[process] pid 5 exec /mnt/disk0/bin/cat
[process] pid 1 forked pid 7
[process] pid 5 (cat) exited with code 0
[process] pid 6 exec /mnt/disk0/bin/grep
[process] pid 7 exec /mnt/disk0/bin/wc
[process] pid 6 (grep) exited with code 0
1
[process] pid 7 (wc) exited with code 0
B:/ $ dir bin | wc -l
[process] pid 1 forked pid 8
[process] pid 1 forked pid 9
[process] pid 8 exec /mnt/disk0/bin/ls
[process] pid 9 exec /mnt/disk0/bin/wc
[process] pid 8 (ls) exited with code 0
29
[process] pid 9 (wc) exited with code 0
B:/ $ echo first line > A:/tmp/out.txt
B:/ $ echo second line >> /tmp/out.txt
B:/ $ cat < /tmp/out.txt
[process] pid 1 forked pid 10
[process] pid 10 exec /mnt/disk0/bin/cat
first line
second line
[process] pid 10 (cat) exited with code 0
B:/ $ md /tmp/work
[process] pid 1 forked pid 11
[process] pid 11 exec /mnt/disk0/bin/mkdir
[process] pid 11 (mkdir) exited with code 0
B:/ $ copy B:/notes/todo.txt /tmp/work
[process] pid 1 forked pid 12
[process] pid 12 exec /mnt/disk0/bin/cp
[process] pid 12 (cp) exited with code 0
B:/ $ touch /tmp/work/empty.txt
[process] pid 1 forked pid 13
[process] pid 13 exec /mnt/disk0/bin/touch
[process] pid 13 (touch) exited with code 0
B:/ $ ls -l A:/tmp/work
[process] pid 1 forked pid 14
[process] pid 14 exec /mnt/disk0/bin/ls
         0  empty.txt
       125  todo.txt
[process] pid 14 (ls) exited with code 0
B:/ $ del /tmp/work/todo.txt /tmp/work/empty.txt /tmp/work
[process] pid 1 forked pid 15
[process] pid 15 exec /mnt/disk0/bin/rm
[process] pid 15 (rm) exited with code 0
B:/ $ grep "B:" readme.txt
[process] pid 1 forked pid 16
[process] pid 16 exec /mnt/disk0/bin/grep
In the Rustbelt shell this disk is drive B: -- try `cd B:` and `ls`.
[process] pid 16 (grep) exited with code 0
B:/ $ alias ll="ls -l"
B:/ $ ll A:/tmp
[process] pid 1 forked pid 17
[process] pid 17 exec /mnt/disk0/bin/ls
        23  out.txt
[process] pid 17 (ls) exited with code 0
B:/ $ run --trace cat B:/notes/todo.txt
[process] pid 1 forked pid 18
[process] pid 18 exec /mnt/disk0/bin/cat
[trace 18] execve("/mnt/disk0/bin/cat", ...) = 0
[trace 18] open("/mnt/disk0/notes/todo.txt", 0o0) = 3
[trace 18] read(3, "Done: the shell (cd, ls, cat, run, start"..., 512) = 125
Done: the shell (cd, ls, cat, run, start, pipes, redirection, drive letters).
Next: more file systems -- ext2, exFAT, btrfs.
[trace 18] write(1, "Done: the shell (cd, ls, cat, run, start"..., 125) = 125
[trace 18] read(3, "", 512) = 0
[trace 18] close(3) = 0
[trace 18] exit(0)
[process] pid 18 (cat) exited with code 0
B:/ $ start sleep 1
[process] pid 1 forked pid 19
[pid 19] sleep started in the background
B:/ $ jobs
[1] [pid 19] [process] pid 19 exec /mnt/disk0/bin/sleep
running  start sleep 1
B:/ $ wait
[process] pid 19 (sleep) exited with code 0
[done] pid 19 (start sleep 1), exit code 0
B:/ $ cd C:
C:/ $ ls
[process] pid 1 forked pid 20
[process] pid 20 exec /mnt/disk0/bin/ls
big.txt      docs/        hello-link@  hello.txt    lost+found/  readme.txt
sparse.bin
[process] pid 20 (ls) exited with code 0
C:/ $ cat hello-link
[process] pid 1 forked pid 21
[process] pid 21 exec /mnt/disk0/bin/cat
Hello from the ext2 disk! This file lives on a second virtio disk, in an
ext2 file system that Rustbelt reads and writes with its own driver.
[process] pid 21 (cat) exited with code 0
C:/ $ grep "line 12345" big.txt
[process] pid 1 forked pid 22
[process] pid 22 exec /mnt/disk0/bin/grep
big.txt line 12345
[process] pid 22 (grep) exited with code 0
C:/ $ cp B:/notes/todo.txt C:/todo.txt
[process] pid 1 forked pid 23
[process] pid 23 exec /mnt/disk0/bin/cp
[process] pid 23 (cp) exited with code 0
C:/ $ cat C:/todo.txt | grep -i ext2
[process] pid 1 forked pid 24
[process] pid 1 forked pid 25
[process] pid 24 exec /mnt/disk0/bin/cat
[process] pid 24 (cat) exited with code 0
[process] pid 25 exec /mnt/disk0/bin/grep
Next: more file systems -- ext2, exFAT, btrfs.
[process] pid 25 (grep) exited with code 0
C:/ $ md C:/made-here
[process] pid 1 forked pid 26
[process] pid 26 exec /mnt/disk0/bin/mkdir
[process] pid 26 (mkdir) exited with code 0
C:/ $ echo written by Rustbelt > C:/made-here/note.txt
C:/ $ type C:/made-here/note.txt
[process] pid 1 forked pid 27
[process] pid 27 exec /mnt/disk0/bin/cat
written by Rustbelt
[process] pid 27 (cat) exited with code 0
C:/ $ ls -l C:/
[process] pid 1 forked pid 28
[process] pid 28 exec /mnt/disk0/bin/ls
-rw-r--r--     304000  big.txt
drwxr-xr-x      <DIR>  docs/
lrwxrwxrwx             hello-link -> hello.txt
-rw-r--r--        142  hello.txt
drwx------      <DIR>  lost+found/
drwxr-xr-x      <DIR>  made-here/
-rw-r--r--        284  readme.txt
-rw-r--r--   69206016  sparse.bin
-rw-r--r--        125  todo.txt
[process] pid 28 (ls) exited with code 0
C:/ $ cd D:
D:/ $ ls
[process] pid 1 forked pid 29
[process] pid 29 exec /mnt/disk0/bin/ls
Long File Name With Spaces.txt
MixedCase.TXT
Photos/
a-really-long-file-name-that-takes-several-file-name-directory-entries-in-exfat-xxxxxxxxxxxxxxxxxxxx.txt
frag-chain.txt
numbers.txt
readme.txt
Ünïcödé café.txt
[process] pid 29 (ls) exited with code 0
D:/ $ cat "long file name with spaces.txt"
[process] pid 1 forked pid 30
[process] pid 30 exec /mnt/disk0/bin/cat
This file has a long name with spaces in it. exFAT stores names of up
to 255 UTF-16 characters, 15 per "File Name" directory entry.
[process] pid 30 (cat) exited with code 0
D:/ $ cp B:/notes/todo.txt D:/
[process] pid 1 forked pid 31
[process] pid 31 exec /mnt/disk0/bin/cp
[process] pid 31 (cp) exited with code 0
D:/ $ cp C:/hello.txt "Copied From Ext2.txt"
[process] pid 1 forked pid 32
[process] pid 32 exec /mnt/disk0/bin/cp
[process] pid 32 (cp) exited with code 0
D:/ $ cp mixedcase.txt C:/from-exfat.txt
[process] pid 1 forked pid 33
[process] pid 33 exec /mnt/disk0/bin/cp
[process] pid 33 (cp) exited with code 0
D:/ $ cat C:/from-exfat.txt
[process] pid 1 forked pid 34
[process] pid 34 exec /mnt/disk0/bin/cat
MixedCase.TXT: exFAT keeps the case you wrote but ignores it when looking names up.
[process] pid 34 (cat) exited with code 0
D:/ $ md "New Folder"
[process] pid 1 forked pid 35
[process] pid 35 exec /mnt/disk0/bin/mkdir
[process] pid 35 (mkdir) exited with code 0
D:/ $ echo hi from exFAT > "New Folder/Note.txt"
D:/ $ type "new folder/NOTE.TXT"
[process] pid 1 forked pid 36
[process] pid 36 exec /mnt/disk0/bin/cat
hi from exFAT
[process] pid 36 (cat) exited with code 0
D:/ $ ls -l D:/
[process] pid 1 forked pid 37
[process] pid 37 exec /mnt/disk0/bin/ls
       142  Copied From Ext2.txt
       132  Long File Name With Spaces.txt
        84  MixedCase.TXT
     <DIR>  New Folder/
     <DIR>  Photos/
        88  a-really-long-file-name-that-takes-several-file-name-directory-entries-in-exfat-xxxxxxxxxxxxxxxxxxxx.txt
     38000  frag-chain.txt
    340000  numbers.txt
       363  readme.txt
       125  todo.txt
        61  Ünïcödé café.txt
[process] pid 37 (ls) exited with code 0
D:/ $ cd E:
E:/ $ ls -l
[process] pid 1 forked pid 38
[process] pid 38 exec /mnt/disk0/bin/ls
drwxr-xr-x      <DIR>  Music/
drwxr-xr-x      <DIR>  archive/
-rw-r--r--    2160000  big.txt
drwxr-xr-x      <DIR>  docs/
-rw-r--r--          0  empty.txt
lrwxrwxrwx             hello-link -> hello.txt
-rw-r--r--         78  hello.txt
drwxr-xr-x      <DIR>  projects/
-rw-r--r--    1500000  random.bin
-rw-r--r--        637  readme.txt
-rw-r--r--    1048576  zeros.bin
[process] pid 38 (ls) exited with code 0
E:/ $ cat hello.txt
[process] pid 1 forked pid 39
[process] pid 39 exec /mnt/disk0/bin/cat
Hello from btrfs! (an inline extent: these bytes live inside the B-tree leaf)
[process] pid 39 (cat) exited with code 0
E:/ $ cat hello-link
[process] pid 1 forked pid 40
[process] pid 40 exec /mnt/disk0/bin/cat
Hello from btrfs! (an inline extent: these bytes live inside the B-tree leaf)
[process] pid 40 (cat) exited with code 0
E:/ $ cat docs/btrfs/cow.txt
[process] pid 1 forked pid 41
[process] pid 41 exec /mnt/disk0/bin/cat
Copy-on-write: btrfs never overwrites a block in place. To change a leaf
it writes a new copy somewhere free, then a new copy of its parent that
points to it, and so on up to the root. The superblock is the last thing
written; until then the old tree is still complete on disk.
[process] pid 41 (cat) exited with code 0
E:/ $ wc big.txt
[process] pid 1 forked pid 42
[process] pid 42 exec /mnt/disk0/bin/wc
 120000  360000 2160000 big.txt
[process] pid 42 (wc) exited with code 0
E:/ $ grep "line 119999" big.txt
[process] pid 1 forked pid 43
[process] pid 43 exec /mnt/disk0/bin/grep
btrfs line 119999
[process] pid 43 (grep) exited with code 0
E:/ $ cat projects/rustbelt/stage10.txt
[process] pid 1 forked pid 44
[process] pid 44 exec /mnt/disk0/bin/cat
Stage 10: a read-only btrfs driver. Next: core polish (SMP, memory
cleanup, guard pages, a proper scheduler), then networking.
[process] pid 44 (cat) exited with code 0
E:/ $ cp docs/btrfs/cow.txt D:/cow.txt
[process] pid 1 forked pid 45
[process] pid 45 exec /mnt/disk0/bin/cp
[process] pid 45 (cp) exited with code 0
E:/ $ type D:/COW.TXT | wc -l
[process] pid 1 forked pid 46
[process] pid 1 forked pid 47
[process] pid 46 exec /mnt/disk0/bin/cat
[process] pid 46 (cat) exited with code 0
[process] pid 47 exec /mnt/disk0/bin/wc
4
[process] pid 47 (wc) exited with code 0
E:/ $ cp docs/btrfs/chunks.txt C:/chunks.txt
[process] pid 1 forked pid 48
[process] pid 48 exec /mnt/disk0/bin/cp
[process] pid 48 (cp) exited with code 0
E:/ $ cat C:/chunks.txt | wc -l
[process] pid 1 forked pid 49
[process] pid 1 forked pid 50
[process] pid 49 exec /mnt/disk0/bin/cat
[process] pid 49 (cat) exited with code 0
[process] pid 50 exec /mnt/disk0/bin/wc
3
[process] pid 50 (wc) exited with code 0
E:/ $ echo hi > new.txt
sh: new.txt: Read-only file system
E:/ $ rm hello.txt
[process] pid 1 forked pid 51
[process] pid 51 exec /mnt/disk0/bin/rm
rm: hello.txt: Read-only file system
[process] pid 51 (rm) exited with code 1
[exit code 1]
E:/ $ mkdir newdir
[process] pid 1 forked pid 52
[process] pid 52 exec /mnt/disk0/bin/mkdir
mkdir: newdir: Read-only file system
[process] pid 52 (mkdir) exited with code 1
[exit code 1]
E:/ $ run evil
[process] pid 1 forked pid 53
[process] pid 53 exec /mnt/disk0/bin/evil
[evil] pid 53: reading kernel memory at 0x200000...
[kernel] user page fault: address 0x200000, rip 0x8000400558, PageFaultErrorCode(PROTECTION_VIOLATION | USER_MODE), Unmapped
[process] pid 53 (evil) killed: page fault (SIGSEGV)
[exit code 139] (killed by SIGSEGV)
E:/ $ # Stage 11: a stack overflow hits the guard zone and is killed cleanly;
E:/ $ # then two CPU hogs in the background, and sysinfo shows the load per CPU.
E:/ $ run overflow
[process] pid 1 forked pid 54
[process] pid 54 exec /mnt/disk0/bin/overflow
[overflow] pid 54: recursing until the stack runs out...
[kernel] user page fault: address 0xffffefed38, rip 0x80004000e1, PageFaultErrorCode(CAUSED_BY_WRITE | USER_MODE), Unmapped
[process] pid 54 (overflow) killed: stack overflow (SIGSEGV)
[exit code 139] (killed by SIGSEGV)
E:/ $ start spin
[process] pid 1 forked pid 55
[pid 55] spin started in the background
E:/ $ start spin
[process] pid 55 exec /mnt/disk0/bin/spin
[process] pid 1 forked pid 56
[spin] pid 55: spinning without syscalls...
[pid 56] spin started in the background
[process] pid 56 exec /mnt/disk0/bin/spin
E:/ $ sysinfo
[spin] pid 56: spinning without syscalls...
[process] pid 1 forked pid 57
[process] pid 57 exec /mnt/disk0/bin/sysinfo
[spin] round 1 done (count = 30000000)
[spin] round 1 done (count = 30000000)
[spin] round 2 done (count = 60000000)
[spin] round 2 done (count = 60000000)
4 CPUs online (this program ran on CPU 2), up 15 s
11 threads, 4 processes, 4572 KiB of memory frames in use
CPU load over the last quarter second:
  CPU0 [....................]   0%
  CPU1 [....................]   0%
[spin] round 3 done (count = 90000000)
[process] pid 55 (spin) exited with code 0
  CPU2 [####################] 100%
[spin] round 3 done (count = 90000000)
[process] pid 56 (spin) exited with code 0
  CPU3 [####################] 100%
[process] pid 57 (sysinfo) exited with code 0
[done] pid 55 (start spin), exit code 0
[done] pid 56 (start spin), exit code 0
E:/ $ wait
E:/ $ # Stage 12: networking. DHCP gave us an address; ping QEMU's gateway; fetch
E:/ $ # from the test web server QEMU runs for 10.0.2.100 (guestfwd), and save a
E:/ $ # 108 KB file to D:; look up a name (only works if the host has DNS); then
E:/ $ # serve one request with httpd (run-headless.sh curls it through hostfwd).
E:/ $ ifconfig
[process] pid 1 forked pid 58
[process] pid 58 exec /mnt/disk0/bin/ifconfig
eth0: virtio-net  MAC 52:54:00:12:34:56  IRQ 10
      inet 10.0.2.15/24  gateway 10.0.2.2  DNS 10.0.2.3  (DHCP)
      RX 15 packets (1 KiB)  TX 14 packets (1 KiB)
[process] pid 58 (ifconfig) exited with code 0
E:/ $ ping -c 3 -i 200 10.0.2.2
[process] pid 1 forked pid 59
[process] pid 59 exec /mnt/disk0/bin/ping
PING 10.0.2.2 (10.0.2.2): 56 data bytes
64 bytes from 10.0.2.2: icmp_seq=1 time=8.97 ms
64 bytes from 10.0.2.2: icmp_seq=2 time=2.15 ms
64 bytes from 10.0.2.2: icmp_seq=3 time=2.19 ms
--- 10.0.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss
rtt min/avg/max = 2.15/4.43/8.97 ms
[process] pid 59 (ping) exited with code 0
E:/ $ fetch http://10.0.2.100/
[process] pid 1 forked pid 60
[process] pid 60 exec /mnt/disk0/bin/fetch
Hello from the host, via QEMU guestfwd!
[process] pid 60 (fetch) exited with code 0
E:/ $ wget -o D:/seq.txt http://10.0.2.100/big
[process] pid 1 forked pid 61
[process] pid 61 exec /mnt/disk0/bin/fetch
fetch: HTTP/1.0 200 OK, saved 108894 bytes to /mnt/disk2/seq.txt in 126 ms (843 KiB/s)
[process] pid 61 (fetch) exited with code 0
E:/ $ cat D:/seq.txt | wc -l
[process] pid 1 forked pid 62
[process] pid 1 forked pid 63
[process] pid 62 exec /mnt/disk0/bin/cat
[process] pid 63 exec /mnt/disk0/bin/wc
[process] pid 62 (cat) exited with code 0
20000
[process] pid 63 (wc) exited with code 0
E:/ $ host example.com
[process] pid 1 forked pid 64
[process] pid 64 exec /mnt/disk0/bin/host
example.com has address 198.18.0.1
[process] pid 64 (host) exited with code 0
E:/ $ httpd -n 1 -t 20
[process] pid 1 forked pid 65
[process] pid 65 exec /mnt/disk0/bin/httpd
[httpd] listening on port 80
[httpd] 10.0.2.2:48206 GET / -> 200 (133 bytes)
[httpd] served 1 request, bye
[process] pid 65 (httpd) exited with code 0
E:/ $ # Stage 13: virtual drivers. The devices in /dev; random bytes from
E:/ $ # virtio-rng; a shell on the second console (getty -1 runs the commands
E:/ $ # QEMU feeds into hvc0 from a file; its output goes to another file that
E:/ $ # run-headless.sh checks); the tiling demo on the virtio-gpu framebuffer.
E:/ $ ls -l /dev
[process] pid 1 forked pid 66
[process] pid 66 exec /mnt/disk0/bin/ls
crw-rw-rw-      <DEV>  fb0
crw-rw-rw-      <DEV>  hvc0
crw-rw-rw-      <DEV>  null
crw-rw-rw-      <DEV>  random
crw-rw-rw-      <DEV>  urandom
[process] pid 66 (ls) exited with code 0
E:/ $ random -n 8
[process] pid 1 forked pid 67
[process] pid 67 exec /mnt/disk0/bin/random
a83abb283794a4d6
[process] pid 67 (random) exited with code 0
E:/ $ random -d 6 5
[process] pid 1 forked pid 68
[process] pid 68 exec /mnt/disk0/bin/random
3 3 6 6 5
[process] pid 68 (random) exited with code 0
E:/ $ getty -1 /dev/hvc0
[process] pid 1 forked pid 69
[process] pid 69 exec /mnt/disk0/bin/getty
[process] started pid 70 (sh), entry 0x800040ada5
[process] pid 70 forked pid 71
[process] pid 71 exec /mnt/disk0/bin/random
[process] pid 71 (random) exited with code 0
[process] pid 70 forked pid 72
[process] pid 72 exec /mnt/disk0/bin/ls
[process] pid 72 (ls) exited with code 0
[process] pid 70 (sh) exited with code 0
[process] pid 69 (getty) exited with code 0
E:/ $ tile
[process] pid 1 forked pid 73
[process] pid 73 exec /mnt/disk0/bin/tile
tile: 4 windows on 1024x768, console 61x46
[process] pid 73 (tile) exited with code 0
E:/ $ tile -r
[process] pid 1 forked pid 74
[process] pid 74 exec /mnt/disk0/bin/tile
console: full screen, 128x48 characters
[process] pid 74 (tile) exited with code 0
E:/ $ # Stage 14: processes. Every program above was started with fork + execve;
E:/ $ # these check the process model itself (each prints "all checks passed").
E:/ $ # Then a background job ended with the kill builtin (SIGTERM: 128+15).
E:/ $ forktest
[process] pid 1 forked pid 75
[process] pid 75 exec /mnt/disk0/bin/forktest
[process] pid 75 forked pid 76
[process] pid 76 (forktest) exited with code 7
[forktest] ok: fork returns the child's pid in the parent
[forktest] ok: child saw getppid() == parent; wait4 gives its exit code 7
...
[forktest] all checks passed
[process] pid 75 (forktest) exited with code 0
E:/ $ cowtest
[process] pid 1 forked pid 92
[process] pid 92 exec /mnt/disk0/bin/cowtest
[process] pid 92 forked pid 93
[process] pid 93 (cowtest) exited with code 0
[cowtest] ok: the child starts with the parent's data and can change its copy
[cowtest] ok: the child's writes to global data stay in the child
...
[cowtest] all checks passed
[process] pid 92 (cowtest) exited with code 0
E:/ $ mmaptest
[process] pid 1 forked pid 98
[process] pid 98 exec /mnt/disk0/bin/mmaptest
[mmaptest] ok: mmap 3 pages of anonymous memory
[mmaptest] ok: it starts zeroed
[mmaptest] ok: it keeps what we write
[mmaptest] ok: mmap 64 MiB: no memory used yet (pages come on first touch)
...
[mmaptest] all checks passed
[process] pid 98 (mmaptest) exited with code 0
E:/ $ sigtest
[process] pid 1 forked pid 99
[process] pid 99 exec /mnt/disk0/bin/sigtest
[sigtest] ok: SIG_IGN: an ignored signal does nothing
[sigtest] ok: rt_sigaction installs a handler
[sigtest] ok: kill(getpid(), SIGUSR1) runs the handler once, with sig = 10
[sigtest] ok: after rt_sigreturn the interrupted code computes the same result
...
[sigtest] all checks passed
[process] pid 99 (sigtest) exited with code 0
E:/ $ start spin 1000
[process] pid 1 forked pid 110
[pid 110] spin started in the background
E:/ $ kill %1
[process] pid 110 exec /mnt/disk0/bin/spin
[process] pid 110 (spin) killed by SIGTERM
E:/ $ wait
[done] pid 110 (start spin 1000), exit code 143
E:/ $ kill -l
 1) SIGHUP      2) SIGINT      3) SIGQUIT     4) SIGILL      5) SIGTRAP   
 6) SIGABRT     7) SIGBUS      8) SIGFPE      9) SIGKILL    10) SIGUSR1   
11) SIGSEGV    12) SIGUSR2    13) SIGPIPE    14) SIGALRM    15) SIGTERM   
16) SIGSTKFLT  17) SIGCHLD    18) SIGCONT    19) SIGSTOP    20) SIGTSTP   
21) SIGTTIN    22) SIGTTOU    23) SIGURG     24) SIGXCPU    25) SIGXFSZ   
26) SIGVTALRM  27) SIGPROF    28) SIGWINCH   29) SIGIO      30) SIGPWR    
31) SIGSYS     
E:/ $ nosuchprogram
sh: nosuchprogram: command not found (try `help`)
E:/ $ cd Q:
cd: there is no drive Q: (see `drives`)
E:/ $ cd A:/tmp
A:/tmp $ pwd
/tmp
A:/tmp $ exit 0
[process] pid 1 (sh) exited with code 0
[ok] shell self-test finished: Exited(0)
[memory] 1015 frames in use before the shell, 1015 after it and all its children exited
[run-headless] frames in use returned to the baseline
[run-headless] host curl got httpd's page: Hello from Rustbelt! Rustbelt 0.15.0 (stage 15: libc plumbing)
[run-headless] all 67+12 expected shell and hvc0 lines found
[run-headless] kernel exited OK

And target/headless-hvc0.txt, what the host got from hvc0: the kernel log up to the moment getty opened the port, then the shell running the commands in scripts/hvc0-session.txt:

...
E:/ $ getty -1 /dev/hvc0
[process] pid 1 forked pid 69
[process] pid 69 exec /mnt/disk0/bin/getty

Rustbelt getty: shell on this terminal (type `exit` to end it)
E:/ $ ver
Rustbelt 0.15.0 (stage 15: libc plumbing) x86_64
E:/ $ echo hi over hvc0
hi over hvc0
E:/ $ random -d 20 3
3 12 8
E:/ $ ls /dev
fb0
hvc0
null
random
urandom
E:/ $ exit 0
[process] pid 69 (getty) exited with code 0
E:/ $ tile
...

(Which CPU each program lands on, the exact order of lines printed by programs running at the same time, ping times and download speeds change from run to run, and so do the random numbers. example.com's address is whatever the host's DNS says; without DNS that line is host: example.com: ... and the run still passes.)

The run ends with the same number of frames in use before and after the shell, even though about 110 processes have come and gone (forked, exec'd, sharing pages copy-on-write); run-headless.sh fails if they differ. Exited processes give back their page tables, code, user stacks and kernel stacks. (Of the 1015 at boot, 768 are the 3 MiB framebuffer; most of the rest are the virtio queues and buffers of the four disks, the network card, the rng, the console and the GPU, the kernel stack area and the other CPUs' stacks.)

The kernel boots from the bootimage on the first (IDE) disk; the scripts and Cargo.toml attach a second disk, target/disk.img (GPT + FAT32), a third, target/ext2.img (raw ext2), a fourth, target/exfat.img (raw exFAT), and a fifth, target/btrfs.img (raw btrfs), as -drive file=target/disk.img,format=raw,if=virtio,snapshot=on and so on. snapshot=on makes QEMU throw away writes when it exits, so every boot and every test sees the same fresh image even though the kernel writes to it. The tests attach more images after those five, so the drive letters B: to E: are the same in cargo run and cargo test: two more ext2 images (4 KiB blocks, revision 0, /mnt/disk4 and disk5), one more exFAT image (512-byte clusters, disk6), and four more btrfs images (lzo, zstd, converted from ext2: disk7 to disk9; and one with sha256 checksums that is refused, so it gets no /mnt/diskN). Drive letters follow mount order, so a new user-visible disk always goes right after the last one cargo run attaches, never after the test-only ones.

Tests

cargo test                        # everything (lib unit tests + tests/*.rs), each booted in QEMU headless
cargo test --lib                  # just the unit tests inside src/
cargo test --test heap_allocation # one integration test

Each test binary is its own little kernel: bootimage runner boots it with the test-args from Cargo.toml (no display, serial to stdout, isa-debug-exit) and treats QEMU exit code 33 as a pass. Output looks like:

Running 5 tests
heap_allocation::large_allocation_uses_fallback...  [ok]
heap_allocation::large_vec...                       [ok]
heap_allocation::many_boxes...                      [ok]
heap_allocation::many_boxes_long_lived...           [ok]
heap_allocation::simple_allocation...               [ok]

Test	What it checks
lib (src/*.rs #[test_case]s)	trivial assertion, breakpoint handler returns, VGA println! writes the right cells; btrfs: crc32c check value and the "default" name hash, a hand-built prealloc extent and an explicit hole, an LZO literal run + overlapping match
tests/basic_boot.rs	printing works straight after boot, before any init()
tests/heap_allocation.rs	Box, big Vec, 100 K short-lived boxes (memory reuse), same with one long-lived box, >2 KiB fallback allocation
tests/should_panic.rs	a failing assert_eq! reaches the panic handler (harness = false)
tests/stack_overflow.rs	stack overflow lands in the double-fault handler on the IST stack (harness = false)
tests/executor.rs	SimpleExecutor runs an async fn to completion; the waker-based Executor interleaves two tasks that yield_now().await (a1 b1 a2 b2 a3 b3)
tests/shell.rs	the shell: tokenizer (operators without spaces, quotes, comments), parser (pipelines, < > >>, &, error cases), drive-letter assignment from a mount table, B:/x -> /mnt/disk0/x translation and back for the prompt; then the real /bin/sh runs a script (cd with a drive letter, ls, a pipe, >/>>/<, ver, exit 3) with its stdout in a pipe and the output is checked; background jobs (start, jobs, wait); 20 processes started and reaped leave frames_in_use unchanged
tests/user_mode.rs	programs loaded from the FAT disk: hello exits 0, demo passes all its checks (exit 0), evil is killed by SIGSEGV (code 139), spin is preempted; spawn with arguments + trace + exit code 7, then reaped; a child's stdout into a pipe; spawn errors (ENOENT, ENOEXEC, EISDIR, ECHILD); wait with WNOHANG returns EAGAIN while the child runs; ELF loader rejects garbage; user pointers give EFAULT (length 0 is fine); sleep blocks for the right number of ticks; a WaitQueue wakes its waiter
tests/vfs.rs	the root holds dev, etc, hello.txt, mnt, proc (stage 18), tmp; read /hello.txt to EOF, path normalisation (., .., relative to a cwd), ENOENT/ENOTDIR/EINVAL/EISDIR/EEXIST, /tmp is a separate ramfs, longest-prefix mounts (5 with /dev and /proc), sparse writes zero-fill, listing, remove (ENOTEMPTY, EBUSY on mount points), seek/stat/truncate, getdents64 record layout, pipes pass bytes and report EOF / EPIPE
tests/block.rs	PCI scan finds the virtio disk; sector 1 of the image is the GPT header; multi-sector reads equal single-sector reads; write + read back; out-of-range and partial-sector requests fail; the GPT partition is registered as vda1 (with a FAT32 boot sector); MBR parsing on a RAM disk
tests/fat.rs	the FAT32 volume is mounted at /mnt/disk0 with the expected files (bin, notes, readme.txt, scripts); read readme.txt; every program in bin/ is a valid ELF; create/write/read/truncate/remove a 10 KB file; directories with long names, ENOTEMPTY; the BlockIo adapter handles unaligned reads and writes
tests/ext2.rs	the three ext2 images are mounted (/mnt/disk1, disk4, disk5); permissions and case-sensitive names; big.txt line by line and in one read (direct, single and double indirect blocks); sparse.bin's data at 0 / 40 MiB / 65 MiB (triple indirect) and its holes; fast and slow symlinks, readlink, removing a link keeps its target; 4 KiB blocks and revision 0; on each image: mkdir, create, a 300 KB write, append, overwrite across a block boundary, a write at 70 MiB (hole + triple indirect), truncate, ENOTEMPTY/EISDIR/ENOTDIR, unlink, rmdir, and the superblock's free counts are back where they started; 200 files grow a directory; copy from FAT to ext2. scripts/ext2-check.sh then checks the same kernel's writes with the host's e2fsck
tests/exfat.rs	both exFAT images are mounted (/mnt/disk2, /mnt/disk6), label and cluster size; long, mixed-case and Unicode names listed as written; case-insensitive lookup (ÜNÏCÖDÉ CAFÉ.TXT, nested paths); numbers.txt at a cluster boundary and at its end; the FAT-chained frag-chain.txt on 4 KiB and 512-byte clusters; bad names (EINVAL, ENAMETOOLONG, EEXIST for a name in another case); on each image: mkdir, create, write across clusters, append, truncate, a write past the end (zero gap), ENOTEMPTY, unlink, rmdir, and the bitmap and PercentInUse are back where they started; a contiguous file turns into a FAT chain; 200 files grow a directory; the root grows; copies between FAT, ext2 and exFAT; a last test leaves files for scripts/exfat-check.sh's fsck.exfat
tests/btrfs.rs	the four btrfs images are mounted (/mnt/disk3, disk7-disk9) and described (label, node size, DUP/single, subvolume count); every file on every image is read and its size and CRC-32 compared with build.rs's manifest of the host's tree (zlib, lzo with a two-level tree, zstd with 64 KiB nodes, and the converted image); reads at odd offsets across compressed and plain extent boundaries; holes in the converted sparse.bin; ext2_saved/image starts like an ext2 disk; hash lookups are exact (HELLO.TXT is ENOENT); symlinks, including one leaving a subvolume; subvolumes listed with ids and read-only flags; the sha256 image is refused with a reason; create/write/truncate/mkdir/remove give EROFS; and, last, damage on purpose: copy 0 of a DUP node (copy 1 is used), a data sector (EIO for that block only), then both copies (EIO). scripts/btrfs-check.sh runs btrfs check --readonly and compares the same kernel's output with the host's files
tests/threads.rs	two busy-looping threads that never yield both make progress while the main thread busy-waits too (only possible with preemption); a thread can exit and the others keep running
tests/smp.rs	multi-core and the scheduler: all 4 CPUs are online and take timer ticks; 4 busy threads run on 2+ CPUs at the same time (per-CPU busy ticks); 3 spin processes run on 2+ CPUs; 4 threads on 4 CPUs add 200,000 to a locked counter without losing one; sleep_ticks(10) lasts 10-12 ticks; sleepers wake in deadline order (3, 6, 9); while 4 threads sleep the CPUs are idle (<10% busy, so no busy-polling); two threads ping-pong 1,000 times through a WaitQueue; leak checks: 200 threads and 40 processes spawned and exited leave frames_in_use, the thread count, the process count and the number of kernel stacks exactly where they were
tests/net.rs	networking, all against things QEMU itself provides, so it's deterministic and works offline: DHCP gives 10.0.2.15/24, gateway 10.0.2.2, DNS 10.0.2.3 and the MAC is QEMU's default; 3 ICMP echoes to the gateway come back with their payload; a UDP TFTP read of readme.txt from slirp's TFTP server (tftp=disk in test-args); a 64 KiB TCP round trip through the cat echo server (guestfwd 10.0.2.100:7); an HTTP GET of 108,894 bytes through the socket API; ECONNREFUSED from a closed host port; a UDP recv with a 300 ms timeout returns EAGAIN after 250-1000 ms; the ping and fetch programs (saving to a file, and exit 1 on a 404); 20 TCP + 20 UDP sockets opened and closed leave the socket count and frames_in_use unchanged; DNS for example.com, skipped (not failed) if the host has no DNS
tests/virtio.rs	the virtual drivers: 2 x 4 KiB from virtio-rng differ, aren't constant, have 15384-17384 one bits out of 32768 and contain all 256 byte values; the random program (hex, dice, a usage error); hvc0 receives exactly the bytes QEMU feeds in from tests/hvc0-input.txt, and 8 KiB written to it go out with nothing dropped; virtio-gpu answers GET_DISPLAY_INFO, refuses a flush of a resource that doesn't exist with ERR_INVALID_RESOURCE_ID, full and clipped flushes work, a pattern written into the framebuffer reads back; the text console is on the framebuffer (128x48) and printing lights up pixels; /dev lists fb0 hvc0 null random urandom; tile draws the expected colours at known pixels, moves the console to 61x46 and tile -r back, with no frames leaked. scripts/gpu-check.sh checks what QEMU actually displays
tests/guard_pages.rs	the guard page sits right below each kernel stack, and the user stack's 64 KiB guard zone has the right bounds; a kernel thread that recurses forever is killed with "kernel stack overflow" three times in a row (the rest of the kernel carries on, and its stack is freed); the user overflow program ends with Killed(SIGSEGV, "stack overflow") and leaves no frames behind
tests/process.rs	the process model (stage 14), mostly by running the user test programs: cowtest (globals, heap and stack stay separate after fork in both directions, three generations, MAP_SHARED vs MAP_PRIVATE); forktest (fork/wait4/getppid, 10 children reaped with wait4(-1), ECHILD, execve errors, vfork + exec, pipe2(O_CLOEXEC), fork + dup2 + exec through a pipe, a child dying of SIGSEGV); exec argv/envp/auxv (forktest args "two words" "" with FOO=bar EMPTY=: argc, every argv and envp string, AT_PAGESZ, AT_ENTRY = _start, AT_PHDR covering the entry, AT_RANDOM, AT_EXECFN, stack position); mmaptest (zeroed anonymous memory, 64 MiB that costs nothing until touched, SIGSEGV handlers that see si_addr and SEGV_ACCERR/SEGV_MAPERR and fix the fault so the access is retried, PROT_NONE, file-backed private mappings, brk, EINVALs, memory back to the start); an unhandled fault is Killed(SIGSEGV) with status 11 and exit code 139; sigtest (handler round trip through sigreturn with registers intact, masks and pending signals, SIGKILL can't be caught, default actions, SIGCHLD with si_pid/si_status, SIGPIPE, stop/continue seen by wait4(WUNTRACED/WCONTINUED), EINTR vs SA_RESTART, kill errors, signal 0, kill(-pgid)); wait-status encoding; a fork bomb stops at EAGAIN, is killed and reaped, and leaves frames and the process count unchanged; 20 fork/exec/exit loops leak no frames and leave no shared frames; Ctrl+C (tty::input_char('\x03')) kills spin in the foreground group with SIGINT (exit code 130), and does nothing when no group is in the foreground
tests/libc.rs	libc plumbing (stage 15), by running four user test programs twice each (a warm-up, then a run that must leave the thread, process, futex-waiter, pty, socket and frame counts exactly as they were): threads (4 threads sum 1..=40,000,000 on several CPUs; a futex mutex keeps a shared counter exact while an unlocked one loses updates; each thread sees its own TLS value through fs:; gettid; FUTEX_WAIT with a stale value is EAGAIN, times out, FUTEX_WAKE with no waiters; CLONE_CHILD_CLEARTID + futex wake for join; exit ends one thread, exit_group all; tgkill and ESRCH); echoserver (an epoll server and a poll server on 127.0.0.1, three forked clients each, plus UDP over loopback); ptytest (canonical mode holds a half line back, backspace editing, CR -> NL, echo, ONLCR; raw mode passes bytes through without echo; TCGETS/TCSETS; ^C and ^\ kill the child that made the pty its controlling terminal with SIGINT/SIGQUIT and echo "^C"; /dev/tty with and without a controlling terminal; closing the master gives the slave EOF; stage 19: every /dev node's device number, mode and one inode for stat and fstat, ttyname of a pty slave and the console, /proc/self/cwd); nbtest (EAGAIN from pipes, sockets and ptys; 64 KiB pipe capacity; poll bits; fcntl(F_SETFL, O_NONBLOCK) and FIONBIO; nonblocking connect is EINPROGRESS then POLLOUT; select, ppoll, readv/writev, pread/pwrite, getcwd, uname, CLOCK_REALTIME, nanosleep, readlink /proc/self/exe)
tests/musl_prereq.rs	musl prerequisites (stage 15.1), each program twice with no leaks: the FPU is set up (XSAVE with x87 + SSE (+ AVX) under -cpu max); fputest (16 xmm/ymm registers with a per-thread pattern in 4 threads x 3 processes at once, fork copies them, exec clears them, a signal handler gets clean registers and Linux's fpstate in its frame and may edit it, 40 async signals mid-check, SIGFPE when TCG raises it); pathtest (EFAULT for NULL/kernel/unmapped/straddling path pointers, ENAMETOOLONG, struct stat fields and types, lstat, AT_EMPTY_PATH, every *at() call, O_EXCL/O_DIRECTORY/O_NOFOLLOW, fchdir, rename); rawabi (a gcc-built no-libc C program, unmodified)
tests/libc_batch1.rs	libc batch 1 (stage 16): libctest runs one group per test, twice, and nothing (threads, processes, frames, file locks) may be left over: wait (dash's wait: SIGCHLD blocked, fork, rt_sigsuspend returns EINTR once the handler ran and puts the old mask back; a SIGCHLD already pending ends it at once); signals (rt_sigpending, rt_sigtimedwait with and without a timeout and from another process, real-time signals queued 3 times with their values, a standard signal only once, EPERM/ESRCH for rt_sigqueueinfo, sigaltstack + SA_ONSTACK, EPERM on the stack, unknown sa_flags dropped); locks (F_GETFL access modes, F_SETFL, F_GETLK/F_SETLK/F_SETLKW between two processes, EDEADLK, locks dropped on close and on exit, RLIMIT_NOFILE vs dup and F_DUPFD); files (truncate/ftruncate/fsync on ramfs, ext2 and FAT, umask on open and mkdir and across fork, chmod/fchmod/fchmodat, utimensat with UTIME_NOW/UTIME_OMIT, EROFS on btrfs); misc (rlimits, prctl, sched_getaffinity, madvise(MADV_DONTNEED) reads zeros, a robust futex gets FUTEX_OWNER_DIED); orphans (running and zombie orphans are freed, a PR_SET_CHILD_SUBREAPER process adopts and reaps a grandchild); --trace names unknown calls syscall_N
tests/coreutils_batch2.rs	coreutils batch 2 (stage 17): coretest runs one group per test, twice, and nothing (threads, processes, frames, file locks, timers) may be left over: links (link/linkat/symlink/symlinkat on ramfs and ext2 with st_nlink and st_ino, a dangling symlink, renaming a full directory and over a second name, EPERM on FAT/exFAT, EROFS on btrfs, EXDEV); owners (the chown family on ramfs and ext2, -1, lchown vs chown on a symlink, FAT's EPERM unless nothing changes); statfs (statfs/fstatfs magic numbers and sizes per mount, ST_RDONLY, pipes, statx vs stat, st_dev per mount); info (sysinfo 99 and 502, times, getrusage for self and children, the CPU-time clocks, clock_getres); sockets (getsockname/getpeername, SO_REUSEADDR & co., SO_TYPE, SO_ERROR after a refused connect, SHUT_WR = end of file for the peer, EPIPE after it, socketpair refused for AF_INET and unknown families; since stage 18 it works for AF_UNIX); timers (alarm, one-shot and periodic setitimer, pause, POSIX timers with SI_TIMER + si_value, TIMER_ABSTIME, SIGEV_NONE, not inherited by fork); ids (setuid & co. with the saved id, groups, new files owned by their creator, nice values, sethostname -> uname); misc (flock between open files and processes, preadv/pwritev, sendfile, mask sizes for ppoll/pselect6, a ppoll mask letting a signal in, getdents64 while deleting on ramfs and ext2); --trace names (link, statx, rb_sysinfo)
tests/unix_sockets.rs	Unix sockets and /proc/self (stage 18): unixtest runs one group per test, twice, and threads, processes, frames (so no open file stuck "in flight") and bound socket names must be back where they were: stream (a server on a path in /tmp and on ext2 echoes for a forked client; S_IFSOCK with mode 0777 minus the umask, EADDRINUSE, ENXIO from open, unlink, ENOENT/ECONNREFUSED, EPERM on FAT; abstract names and autobind (two sockets, two different names); ECONNRESET for unread data and for a connection the server closed before accept; EPIPE; SO_PEERCRED, getsockname/getpeername); pair (socketpair both ways and across fork, SHUT_WR, MSG_PEEK, seqpacket message borders, dgram pairs, SOCK_CLOEXEC/SOCK_NONBLOCK); dgram (sender names, MSG_TRUNC, connect and EPERM for strangers, AF_UNSPEC, EPROTOTYPE, EMSGSIZE, ECONNREFUSED); rights (a file and a pipe end read through received fds, MSG_CTRUNC, MSG_CMSG_CLOEXEC, EBADF for a bad fd, fds still in flight closed with their socket, a stream read taking the bytes before an fd message together with it and stopping after it (stage 19), SCM_CREDENTIALS); nonblock (EAGAIN, SO_SNDBUF 4096 reading back 8192, the buffer limit, poll bits, a blocked writer finishing in one write, EPOLLRDHUP/EPOLLHUP, nonblocking accept/accept4, a full backlog, SO_RCVTIMEO, FIONREAD); proc (/proc/self/fd listing and links, pipe:[N]/socket:[N] matching st_ino, opening an entry, /proc/self/exe); --trace names (sendmsg, recvmsg)
tests/threads_stress.rs	the flake fix (stage 19): the threads program 40 times in a row; a watchdog prints every thread and futex waiter if a run hangs (it used to, about once in 40 runs)
tests/vdso.rs	the vDSO (stage 19): the embedded image is a one-page ET_DYN with __vdso_* symbols; the vvar page is published (even seq, TSC flag set); the kernel clock counts in nanoseconds; vdsotest passes twice with no leaks (auxv entries, vDSO vs syscall ordering, 100,000 monotonic reads, sub-microsecond steps, gettimeofday/time/sched_getcpu, the vvar page can't be made writable or written, it all works after fork and after execve, a dynamic program is ENOEXEC); --trace shows no clock syscalls for 1000 rounds of clock calls, and with the fast path switched off (vdso::force_syscall) they're all there; --trace names (gettimeofday, time, getcpu)
tests/static_pie.rs	static PIE (stage 19): piehello (a musl-gcc -static-pie program: relocations applied, AT_PHDR/AT_ENTRY/AT_BASE, RELRO, no INTERP), a different page-aligned base on every run, piethreads (pthreads, __thread variables, stdio to a file, ttyname) twice with no leaks, dynhello (PT_INTERP) refused with ENOEXEC, ET_EXEC programs still load
tests/mremap.rs	mremap (stage 19): remaptest twice with no leaks: grow in place, shrink, same size, ENOMEM without MREMAP_MAYMOVE, a move keeps the bytes and empties the old place, MREMAP_FIXED, the EINVAL/EFAULT cases, a moved file mapping still reads the file, copy-on-write after a move in a forked child, musl's realloc of big blocks
tests/static_pie.rs (stage 20 part)	plain static programs (stage 20): exechello and execthreads (stock musl-gcc -static, ET_EXEC at 0x400000) run, their heap and mmap memory stay out of the image, nothing leaks
tests/timerfd.rs	timerfd (stage 20): timerfdtest (one-shot, interval counts, EAGAIN, EINVAL, poll/select/epoll wake-ups, TFD_TIMER_ABSTIME) twice; closing leaves no timer and no memory behind
tests/boot_v2.rs	boot contract v2 (stage 21; only this test gets target/rootfs-gpt.img): the ext2 root partition is /, the exFAT share is /mnt/share (read-write), /tmp and drive B: unchanged; /sbin/init runs as pid 1 and its exit status comes back; the /bin/init, /bin/sh fallbacks; init's argv/env; the exit-code mapping; msync (inside inittest)
tests/fat.rs (stage 21 part)	no root partition (stage 21): the old layout, / the ramfs and no /mnt/share

How it works: #![feature(custom_test_frameworks)] collects every #[test_case] function and passes them to rustbelt::test_runner (in src/lib.rs), which prints each name to serial, runs it, prints [ok], then exits QEMU with success. A panic goes to test_panic_handler, which prints [failed] and exits with failure.

Extra args after -- in cargo run -- ... are appended to the QEMU command line, e.g. cargo run -- -d int -no-reboot to log every interrupt and stop on triple faults, or cargo run -- -s -S to wait for gdb on port 1234.

Project layout

.cargo/config.toml        build-std (core, alloc) + custom target + scripts/runner.sh as the cargo runner
rust-toolchain.toml       pin nightly + rust-src + llvm-tools-preview
x86_64-rust_kernel.json   custom target: no OS, no red zone, soft-float, abort on panic, rust-lld
Cargo.toml                deps, features, test settings, QEMU run/test args
src/lib.rs                the kernel library: module list, init(), kprintln!, test framework, exit_qemu
src/main.rs               the bootable binary: kernel_main (init, then a thread that runs the shell and
                          restarts it when it exits), the `demos` feature's demos, panic handler
src/gdt.rs                GDT (kernel + user segments) + TSS (double-fault IST stack, rsp0 per thread);
                          one GDT + TSS per CPU
src/interrupts.rs         IDT; breakpoint / page fault / GPF / invalid opcode / double fault handlers
                          (faults from ring 3 kill the process; guard-page hits kill with
                          "stack overflow"); 8259 PIC; local APIC timer, reschedule IPI;
                          PIT handler (uptime, ticks, wakes sleeping threads),
                          keyboard handler (just queues the scancode)
src/memory.rs             OffsetPageTable setup from CR3, example mapping, BootInfoFrameAllocator,
                          global frame allocator + free list (free_frame), per-process address spaces
                          (and freeing them), user pointer checks, kernel stack mappings
src/allocator.rs          heap location/size, init_heap (maps heap pages), #[global_allocator], Locked<A>
src/allocator/bump.rs     bump allocator (not active, for learning)
src/allocator/fixed_size_block.rs  fixed-size-block allocator (ACTIVE), falls back to linked_list_allocator
src/task/mod.rs           async: Task (pinned boxed future + id), yield_now()
src/task/simple_executor.rs  first executor: polls every task in a loop (no wakers)
src/task/executor.rs      real executor: task queue + wakers, sleeps with hlt when idle
src/task/keyboard.rs      lock-free scancode queue, ScancodeStream, async keyboard task (keys -> tty)
src/thread.rs             threads + the multi-core scheduler: spawn, one shared run queue, per-CPU idle
                          threads, asm context switch, sleep/wake, WaitQueue, join, stack-overflow kills
src/stack.rs              kernel stacks: 64 KiB each with an unmapped guard page below, in their own
                          region (L4 slot 200), slots reused when threads exit
src/smp.rs                multi-core bring-up: AP trampoline (real mode -> long mode, in assembly),
                          INIT/SIPI, per-CPU data (swapgs), CPU numbering, waking idle CPUs
src/apic.rs               local APIC: enable, timer (calibrated against the PIT), EOI, IPIs
src/acpi.rs               finds the ACPI RSDP/RSDT/MADT to list the CPUs
src/syscall.rs            syscall MSR setup, asm entry stub, dispatch + all syscalls (table at the top)
src/trap.rs               TrapFrame (all user registers) + the common iretq return to user mode
src/process.rs            Process (pid, parent, pgid, cwd, env, fd table + close-on-exec, exit status),
                          spawn, fork, execve (argv/envp/auxv stack), exit/kill, wait4 (reaping), groups
src/mm.rs                 a process's memory map: regions (VMAs), demand paging, copy-on-write faults,
                          mmap/munmap/mprotect/brk, fork of an address space
src/signal.rs             signals: actions, masks, pending sets, default actions, rt_sigframe setup and
                          sigreturn, SIGCHLD/stop/continue, delivery on the way back to user mode
src/pci.rs                PCI bus scan through config ports 0xCF8/0xCFC, device names
src/block/mod.rs          BlockDevice trait, device registry (vda, vda1...), RamDisk (for tests)
src/block/virtio_blk.rs   legacy virtio-blk driver: one virtqueue, polled, bounce buffer
src/block/partition.rs    MBR + GPT parsing, Partition = a BlockDevice window onto its disk
src/elf.rs                minimal ELF64 loader (PT_LOAD segments -> regions; program headers for AT_PHDR)
src/errno.rs              Linux errno numbers
src/tty.rs                ttys (stage 15): the console, hvc0 and ptys share one line discipline
                          (termios, canonical/raw, echo, ^C/^Z/^\ -> signals), sessions and
                          controlling terminals, /dev/tty, /dev/ptmx + /dev/pts/N
src/poll.rs               poll, ppoll, select, pselect6 and epoll (one global "something changed" counter)
src/futex.rs              futex WAIT/WAKE (+ bitsets, private flag, timeouts): what libc mutexes and
                          pthread_join sleep on
src/fs/mod.rs             VFS: Inode + FileSystem traits, mount table, path normalize/lookup/create/remove,
                          symlink following (ELOOP after 8), init (boot files),
                          mount_disks (ext2, exFAT, btrfs or FAT partitions/disks -> /mnt/diskN)
src/fs/file.rs            File trait (what an fd points to), InodeFile (inode + offset + path), Stat,
                          LinuxStat (Linux's struct stat), getdents64
src/fs/ramfs.rs           in-memory filesystem
src/fs/fat.rs             FAT32 via the fatfs crate: BlockIo adapter + FatFs/FatInode for the VFS
src/fs/ext2.rs            hand-written ext2: superblock, groups, bitmaps, inodes, indirect blocks,
                          directories, symlinks; read + write (the layout is explained at the top)
src/fs/exfat.rs           hand-written exFAT: boot checksum, bitmap, FAT chains + NoFatChain, up-case
                          table, entry sets + SetChecksum; read + write (layout explained at the top)
src/fs/btrfs.rs           hand-written read-only btrfs: superblock, chunk tree (single/DUP), root and FS
                          trees, crc32c on nodes and data, inline/regular/compressed extents, holes,
                          symlinks, subvolumes (the format is explained at the top)
src/fs/lzo.rs             LZO1X decompressor for btrfs's lzo extents
src/fs/pipe.rs            pipes: a byte queue with a blocking read end and a write end
build.rs                  compiles user/*.rs to ELF files, then builds target/disk.img (GPT + FAT32)
                          and the ext2 images target/ext2.img, ext2-4k.img, ext2-rev0.img (mke2fs -d),
                          and the exFAT images target/exfat.img, exfat-512.img (mkfs.exfat + populator),
                          and the btrfs images target/btrfs*.img (mkfs.btrfs --rootdir, btrfs-convert)
                          with manifests of their files (target/btrfs-*.manifest)
build/exfat_populate.rs   build.rs's own little exFAT writer: copies a tree into a fresh mkfs.exfat image
disk/                     files copied onto the FAT32 disk (readme.txt, notes/todo.txt,
                          scripts/selftest.sh = the headless shell session)
disk2/                    files copied onto the ext2 disk (hello.txt, readme.txt, docs/...);
                          build.rs adds big.txt, sparse.bin and two symlinks
disk3/                    files copied onto the exFAT disk (long, mixed-case, Unicode names, Photos/...);
                          build.rs adds numbers.txt and frag-chain.txt
disk4/                    files copied onto the btrfs disks (hello.txt, docs/, Music/, projects/ and
                          archive/ become subvolumes); build.rs adds big.txt, random.bin, zeros.bin,
                          empty.txt and symlinks
user/ulib.rs              tiny user-space "libc": syscall wrappers (fork, execve, wait4 + W* macros, mmap,
                          signals with a restorer), println!/eprintln!, a brk-based heap (Vec/String),
                          line reading, strerror, _start reading argv/envp/auxv, panic handler
user/link.ld              linker script for user programs (base 0x80_0040_0000)
user/sh.rs                the shell: line editor, builtins, aliases, PATH, pipes, redirection, fork + exec,
                          job control (process groups, Ctrl+C, kill, fg, bg)
user/shparse.rs           the shell's parser + drive letters (also compiled into tests/shell.rs)
user/{ls,cat,wc,grep,mkdir,rm,touch,cp,sleep}.rs   the shell's tools
user/{hello,evil,spin,demo,child}.rs  test/demo programs (all programs go to /mnt/disk0/bin)
user/sysinfo.rs           CPUs, per-CPU load bars, threads, processes, memory (the sysinfo syscall)
user/overflow.rs          recurses forever: killed with "stack overflow" by the user stack guard zone
src/time.rs               the clock: the TSC, calibrated against the PIT at boot (nanoseconds since stage 19)
src/net/mod.rs            networking: the smoltcp Interface, DHCP + DNS, the `netd` thread, IRQ handling
src/net/virtio_net.rs     legacy virtio-net driver: RX/TX virtqueues, MAC
src/net/device.rs         smoltcp's Device: frames for our own MAC/IP (127.x) loop back, the rest go to the NIC
src/net/socket.rs         BSD-style sockets (TCP, UDP, ICMP) on smoltcp, blocking on a wait queue
src/net/unix.rs           stage 18: AF_UNIX sockets (stream, dgram, seqpacket), SCM_RIGHTS, sendmsg/recvmsg
user/{ifconfig,ping,host,fetch,httpd}.rs  the network programs
user/{random,getty,tile}.rs  stage 13: getrandom, a shell on hvc0, the framebuffer tiling demo
user/{forktest,cowtest,mmaptest,sigtest}.rs  stage 14: the process-model checks (each prints ok/FAILED
                          lines and "all checks passed"; tests/process.rs runs them)
user/{threads,echoserver,ptytest,nbtest}.rs  stage 15: threads + futexes, poll/epoll over loopback,
                          ptys, nonblocking I/O (tests/libc.rs runs them)
user/{fputest,pathtest}.rs  stage 15.1: per-thread FPU/SSE/AVX state + signal fpstate; the Linux
                          C-string path calls (tests/musl_prereq.rs runs them, and rawabi)
user/rawabi.c             stage 15.1: a no-libc Linux C program (gcc -nostdlib): raw syscalls, C paths, SSE
user/libctest.rs          stage 16: libc batch 1 checks, one group per argument (tests/libc_batch1.rs)
user/coretest.rs          stage 17: coreutils batch 2 checks, one group per argument (tests/coreutils_batch2.rs)
src/syscall_ext.rs        stage 17: links, chown, statfs/statx, CPU times, socket names/options, timers, ids, flock... (table at the top)
src/timer.rs              stage 17: alarm, setitimer and POSIX timers; the timerd thread that sends their signals
user/unixtest.rs          stage 18: Unix socket and /proc/self checks, one group per argument (tests/unix_sockets.rs)
src/fs/procfs.rs          stage 18: /proc/self/fd and /proc/self/exe, made up from the asking process
vdso/vdso.c, vdso/vdso.lds  stage 19: the vDSO (clock_gettime, gettimeofday, time, getcpu without a
                          system call) and its linker script; build.rs compiles it into the kernel
src/vdso.rs               stage 19: the vDSO's pages, the vvar page and its seqlock, the TSC trust rule
user/{vdsotest,piehello,piethreads,dynhello,remaptest}.c  stage 19: C programs built with musl-gcc
                          (needs musl-tools; tests/vdso.rs, static_pie.rs and mremap.rs run them)
user/{exechello,timerfdtest}.c  stage 20: plain `musl-gcc -static` programs (execthreads = piethreads.c
                          built that way too); tests/static_pie.rs and tests/timerfd.rs run them
src/boot.rs               stage 21: boot contract v2: /sbin/init -> /bin/init -> /bin/sh as pid 1, its argv/env, exit status -> QEMU
user/inittest.c           stage 21: the test init on target/rootfs-gpt.img (tests/boot_v2.rs, scripts/boot-v2-check.sh)
scripts/qemu.sh           stage 21: QEMU wrapper for cargo run/test: honours CARGO_TARGET_DIR, adds rootfs-gpt.img for boot_v2
scripts/boot-v2-check.sh  stage 21: boots the headless kernel with the GPT root image, wants QEMU exit 47, fscks both partitions
src/fs/lock.rs            stage 16: fcntl record locks (F_GETLK/F_SETLK/F_SETLKW), deadlock detection
src/fpu.rs                stage 15.1: FPU/SSE/AVX on per CPU (CR0/CR4/XCR0), XSAVE or FXSAVE save/restore
scripts/hvc0-session.txt  the commands run-headless.sh feeds into hvc0 (for `getty -1 /dev/hvc0`)
scripts/gpu-check.sh      types `tile`, dumps the virtio-gpu screen (PPM), compares pixels
scripts/http-reply.sh     the host side of `fetch http://10.0.2.100/`: QEMU runs it per connection
src/vga_buffer.rs         the console writer, print!/println!, print_at (status line), backspace: VGA text
                          mode, or an FbConsole once virtio-gpu is up (use_framebuffer, set_viewport)
src/virtio/mod.rs         modern (1.0) virtio over PCI: capabilities, feature negotiation, the
                          Transport (common config, notify, ISR, device config), IRQ fan-out
src/virtio/queue.rs       split virtqueue: descriptor table, available and used rings
src/virtio/rng.rs         virtio-rng: fill(buf) for getrandom, /dev/random and seeds (+ fallback)
src/virtio/console.rs     virtio-console (hvc0): 8 RX + 8 TX buffers, IRQ-driven input, log mirroring
src/virtio/gpu.rs         virtio-gpu 2D: display info, resource, backing, scanout, transfer + flush
src/fbcon.rs              text console on the framebuffer: cells, cursor, scrolling, dirty rows, a viewport
src/font.rs               the 8x16 bitmap font (Spleen, BSD-2), Latin-1 + box drawing
src/dev.rs                devfs at /dev: null, random, urandom, console, tty, ptmx, pts/, hvc0, fb0
src/serial.rs             COM1 UART writer + serial_print!/serial_println!
tests/                    integration tests, one QEMU-booted kernel each
scripts/runner.sh         `cargo run`/`cargo test` runner: `bootimage runner` + stale-image fix
scripts/run-headless.sh   headless boot + shell self-test, timeout, exit-code and output checks
scripts/screenshot.sh     headless boot + a typed shell session + PNG/PPM screenshot via QEMU monitor
scripts/ext2-check.sh     Rustbelt writes to copies of the ext2 images, then host e2fsck -fn + debugfs
scripts/exfat-check.sh    Rustbelt writes to copies of the exFAT images, then host fsck.exfat -n + dump.exfat
scripts/btrfs-check.sh    btrfs check --readonly, then tests/btrfs.rs's file CRCs vs. the host's files
boot.png                  screenshot of the virtio-gpu screen (tile + a shell session)

What the interrupt code does (chapters 5-7)

• CPU exceptions / IDT (interrupts.rs): the IDT maps vector numbers to handler functions written with extern "x86-interrupt" (nightly abi_x86_interrupt), so the compiler saves all registers and returns with iretq. _start fires an int3 on purpose; the breakpoint handler prints the stack frame and execution continues. A page-fault handler prints the faulting address from CR2.
• Double faults (gdt.rs): a stack overflow hits the guard page -> page fault -> the CPU can't push the exception frame on the broken stack -> double fault -> same again -> triple fault = machine reset. We load our own GDT containing a TSS whose Interrupt Stack Table entry 0 points at a separate 20 KiB static stack, and mark the double-fault IDT entry to use it. Try ./scripts/run-headless.sh stack-overflow.
• Hardware interrupts: the two 8259 PICs are remapped to vectors 32-47 (0-31 are CPU exceptions) and then sti enables interrupts. The timer (IRQ 0, ~18.2 Hz) bumps an atomic tick counter and redraws the uptime clock; the keyboard (IRQ 1) reads the scancode from port 0x60 and queues it; an async task decodes it with pc-keyboard (US layout, scancode set 1) and echoes it (see Multitasking below). Each handler must send End-Of-Interrupt to the PIC or no further interrupts arrive. print!/serial_print! disable interrupts while holding their lock, otherwise an interrupt handler printing while _start holds the lock would deadlock. The idle loop uses hlt, so QEMU doesn't burn a host CPU core.

Memory (chapters 8-11)

Paging. Every address the kernel uses is virtual. The CPU translates it with a 4-level page table tree (level 4 -> 3 -> 2 -> 1 -> 4 KiB page); 9 bits of the address index each level, the low 12 bits are the offset in the page. CR3 holds the physical address of the level-4 table (here 0x1000). The page-fault feature writes to 0xdeadbeaf, which no table maps, and the page-fault handler reports the address (from CR2) and the reason (CAUSED_BY_WRITE, not PROTECTION_VIOLATION -> the page simply isn't present).

Reaching page tables. Page tables contain physical addresses, but we can only dereference virtual ones. With the bootloader's map_physical_memory feature, all of physical RAM is also mapped at physical_memory_offset (0x18000000000 here), so virt = offset + phys. memory::init wraps that in the x86_64 crate's OffsetPageTable, which can translate_addr (walk the tables) and map_to (add entries).

Virtual memory layout (as set up by bootloader 0.9 in this build):

Virtual address	What
0xb8000	VGA text buffer (identity-mapped: virt = phys)
0x200_0000_0000...	kernel code + data (the ELF; maps to phys around 0x400000). Until stage 20 it was at 0x200000
0x0100_0000_0000...	kernel stack (80 pages, guard page below it)
0x0180_0000_0000...	all physical memory (physical_memory_offset)
0x4444_4444_0000	kernel heap, 4 MiB (also holds thread stacks) (we map it in allocator::init_heap)
0xdead_beaf_000	demo page we map to the VGA frame (writes "New!" on screen)

The "New!" demo writes to screen row 20 through the new mapping; by the time you look, later lines have scrolled it up, so in boot.png it overwrote the start of the first line.

Physical frames. BootInfoFrameAllocator hands out 4 KiB frames from the regions the BIOS memory map marks Usable (~121 MiB of QEMU's default 128 MiB). It never frees; fine for now.

Heap. allocator::init_heap maps the 1024 pages (4 MiB) of HEAP_START..+HEAP_SIZE to fresh frames, then gives the range to the #[global_allocator]. From then on the alloc crate (Box, Vec, Rc, String, BTreeMap...) works. alloc is compiled for our target by build-std in .cargo/config.toml.

Allocator designs (src/allocator/): the bump allocator just moves a pointer forward and can only reclaim memory when every allocation is freed (it fails the many_boxes_long_lived test). The linked_list_allocator crate keeps a list of free regions (first fit, slower, but reuses any freed memory). The active fixed-size-block allocator rounds each request up to 8/16/.../2048 bytes and keeps one free list per size, so alloc/free are O(1) pops/pushes; bigger requests and empty lists fall back to the linked-list allocator. Switch allocators by editing one line in src/allocator.rs.

Multitasking (async/await + threads)

Cooperative: async/await (src/task/, phil-opp chapter 12)

An async fn compiles into a state machine (a Future). An executor calls poll on it; the future runs until it has to wait, then returns Pending and remembers a Waker. Whoever makes progress possible later (here: the keyboard interrupt) calls waker.wake(), which puts the task's id back into the executor's queue. When the queue is empty, the executor hlts -- carefully: it disables interrupts, checks the queue, and then uses sti; hlt as one step (enable_and_hlt), so a wakeup arriving in between can't be missed.

The keyboard interrupt handler now does almost nothing: read the scancode from port 0x60 and push it into a fixed-size lock-free queue (crossbeam_queue::ArrayQueue), then wake the keyboard task (futures_util::task::AtomicWaker). No locks in the interrupt handler means no deadlock if the interrupt arrives while the task holds one. The task (print_keypresses) is an async loop over a ScancodeStream that decodes and echoes characters.

"Cooperative" means a task only gives up the CPU at an .await. A task that loops forever without awaiting freezes every other task. Upside: switching is cheap (just return from poll), and all tasks share one stack.

Preemptive: kernel threads (src/thread.rs, beyond the blog series)

Each thread has its own 64 KiB stack (a KernelStack, with a guard page under it). On every local APIC timer interrupt (~100 times a second, on each CPU) the handler sends the EOI and then calls thread::schedule(), which picks the next ready thread round-robin and calls switch_context(&mut old.rsp, new.rsp) (a dozen lines of global_asm!):

push rbp, rbx, r12, r13, r14, r15   ; callee-saved regs of the OLD thread, onto its stack
mov  [rdi], rsp                     ; remember where the old stack ends
mov  rsp, rsi                       ; ---- now on the NEW thread's stack ----
pop  r15, r14, r13, r12, rbx, rbp   ; its callee-saved regs
ret                                 ; return to wherever IT called switch_context

Why only those six registers? switch_context is called like a normal function, so the compiler already saved the caller-saved registers, and the x86-interrupt handler around it saved everything the interrupted code was using (plus the CPU pushed RIP/RSP/RFLAGS). So a paused thread's complete state sits on its own stack; the scheduler only stores its rsp. The new thread "returns" from schedule(), from its timer handler, and iretqs back to where it was interrupted.

A brand-new thread was never interrupted, so spawn hand-builds a stack that looks as if switch_context had saved it: six register slots (r12 = the thread's function) and a return address pointing to thread_trampoline, which moves r12 into rdi and calls thread_start(entry). That re-enables interrupts (we arrive from inside the timer interrupt, where they're off), runs the function, and marks the thread finished when it returns. The code that booted the kernel becomes thread 0, "main", which runs the async executor.

Preemption means even loop {} can't hog the CPU (thread C in the demo is a pure busy loop). The price: any code can be interrupted anywhere, so shared data needs locks, and a lock that the timer handler might also want must be taken with interrupts disabled. Here: the print macros and print_at already hold their spinlocks with interrupts off (without_interrupts), so a thread can't be switched away while holding them; the scheduler lock is only taken with interrupts off and is released before switching (the next thread would otherwise wait forever for a lock held by a paused thread).

The locking rule: a lock that is ever taken with interrupts off must always be taken with interrupts off. Otherwise: thread T takes the lock with interrupts on, gets preempted, and some code running with interrupts off (so it can never be preempted back to T) spins on that lock forever. Stage 6 hit exactly this with the heap allocator: wait-queue code runs with interrupts off and allocates, so GlobalAlloc::alloc/dealloc now take the heap lock inside without_interrupts.

Blocking (stage 6). A thread can be blocked: the scheduler skips it until someone wakes it. Two ways to be woken:

• sleep_ticks(n) marks the thread sleeping until tick now + n; the PIT handler wakes it once that tick has passed (nanosleep sleeps whole 55 ms ticks this way, then yields until the TSC clock says the time is really up).
• A WaitQueue (a list of waiting thread ids). wait_until(|| condition) turns interrupts off, checks the condition, and if it's false puts the thread on the queue, marks it blocked and switches away; when woken it checks again. wake_all() unblocks everyone on the queue. Checking with interrupts off is what prevents a lost wakeup (the event firing between "check" and "sleep"). Users: the console (read(0) waits for Enter; the keyboard path wakes it), pipes (reader waits for data or EOF) and wait(pid) (parent waits on the child's exited queue).

If every thread is blocked, the scheduler idles with sti; hlt until an interrupt makes something runnable.

Stack size. Stage 6 first used 16 KiB stacks and crashed with a ring-0 jump to address 0: the syscall -> VFS -> FAT -> ELF loader path in an unoptimized debug build used more than that, overflowed into the heap and smashed a neighbour. Stacks are 64 KiB now, and since stage 11 each has a real guard page under it (see "Multiple CPUs" below), which replaced the old canary check.

Stage 11 rewrote the scheduler for several CPUs; the next section explains what changed.

Multiple CPUs, sleep/wake and guard pages (stage 11)

Until stage 10, Rustbelt used one CPU. QEMU now runs with -smp 4, and the kernel starts all four.

Starting the other CPUs (src/smp.rs, src/apic.rs, src/acpi.rs)

At power-on only one CPU runs: the bootstrap processor (BSP, CPU 0). The others (application processors, APs) wait, halted, until the BSP sends them messages through its local APIC, a small interrupt controller built into every CPU core:

1. Which CPUs exist? The firmware describes them in ACPI tables. acpi.rs finds the RSDP (a signature in the BIOS area), follows it to the RSDT, then to the MADT, which lists one entry per CPU with its APIC id.
2. Trampoline. An AP starts in 16-bit real mode, like an 8086, at an address below 1 MiB. So smp.rs copies a small assembly trampoline to physical address 0x8000, plus parameters: the page table (CR3), the control registers, a fresh kernel stack, the CPU number and the Rust function to call. The trampoline jumps straight from real mode into 64-bit long mode using the BSP's settings.
3. INIT, then STARTUP. The BSP sends an INIT inter-processor interrupt (IPI), waits ~110 ms (two PIT ticks), then a STARTUP IPI whose vector (0x08) means "start at 0x8000"; a second STARTUP if the AP didn't answer. Then it waits for the AP to say it's ready, one CPU at a time.
4. AP setup (ap_entry): its own GDT and TSS (with its own double-fault stack), the shared IDT, the syscall MSRs, its local APIC and timer, and then it becomes an idle thread in the scheduler.

Per-CPU data. Some things must exist once per CPU: the TSS (it holds the stack used when an interrupt arrives from ring 3), the "current thread", the syscall entry's scratch slots. cpu_id() reads the local APIC id and maps it to 0..3. The syscall entry can't call Rust before it has a stack, so it uses swapgs: the KERNEL_GS_BASE MSR of each CPU points at that CPU's PerCpu { syscall_kernel_rsp, syscall_user_rsp }, and swapgs makes gs:[0]/gs:[8] reach them.

Timers. The old PIT (on the 8259 PIC) still runs, on CPU 0 only: it keeps the uptime and wakes sleeping threads. Each CPU's local APIC timer, calibrated against the PIT to ~10 ms, preempts whatever that CPU runs. No I/O APIC: it routes device interrupts to chosen CPUs, but Rustbelt's devices are polled (virtio-blk) or happy on CPU 0 (keyboard, PIT), so the 8259 is kept and wired to CPU 0's local APIC (LINT0 in "ExtINT" mode). Moving to the I/O APIC is a good exercise once interrupt-driven virtio needs it.

The scheduler (src/thread.rs)

• One shared ready queue (a VecDeque) under one lock. Each CPU has its own current thread and its own idle thread, which runs sti; hlt when there's nothing to do. Simple and fair; per-CPU queues with work stealing would scale better, but four CPUs don't need them.
• Thread states: Ready, Running, Blocked (on a wait queue), Sleeping(until tick), Finished.
• Handing a thread to another CPU safely. When CPU 0 switches away from thread T, T stays marked on_cpu until CPU 0 has fully left T's stack (finish_switch, right after the switch). Until then no other CPU may pick T, or two CPUs would run on the same stack.
• Sleeping without polling. sleep_ticks(n) marks the thread Sleeping(now + n) and switches away; it's not on the ready queue, so it costs nothing. The PIT handler wakes threads whose time has come (wake_sleepers) and, if some CPU is idle, sends it a reschedule IPI (vector 0x41) so it picks the thread up at once instead of at its next timer tick.
• Wait queues. queue.wait_until(|| condition) blocks until the closure returns Some; queue.wake_all() wakes the waiters. The classic lost wake-up (the event happens between "check" and "block", possibly on another CPU) is prevented by joining the queue before checking, plus a woken flag: a wake that arrives early makes the block return immediately. Console input, pipes, wait, join and the async executor (when it has nothing to do) all use wait queues.
• Statistics. Each local APIC tick counts as busy or idle for that CPU; the sysinfo syscall (502) returns these, and user/sysinfo.rs turns two samples into load bars.

The locking rule now matters more: a lock that is ever taken with interrupts off must always be taken with interrupts off. On one CPU breaking it deadlocks only sometimes; on four, a thread spinning with interrupts off on CPU 1 can wait forever for a lock held by a preempted thread. The scheduler, frame allocator, heap, process table and kernel page-table locks all follow it.

Memory cleanup

Everything a thread or process had is given back when it ends:

• A finished thread can't free the stack it's running on, so the next thread on that CPU frees it (finish_switch), before removing the thread from the table. So when join returns, the memory really is free.
• A process's address space (its page tables at every level, code, data, heap, user stack) is freed in Drop for Process, when the last Arc to it goes away. wait joins the process's thread before reaping, so the parent holds the last reference.
• Children whose parent exits first are orphaned and clean themselves up when they exit.

memory::frames_in_use() counts frames; the headless run and tests/smp.rs check it returns to exactly the same number after hundreds of threads and processes.

Guard pages (src/stack.rs, src/process.rs)

A stack grows down. If it grows past its end it would silently overwrite whatever lies below. A guard page is an address range under the stack that is deliberately not mapped: touching it causes a page fault, so the overflow is caught at the first byte.

• Kernel stacks live in their own region of the address space (L4 slot 200, 0x0000_6400_0000_0000): 4 KiB guard page, 64 KiB stack, guard page, stack, ... When a kernel stack overflows, the CPU can't even push the page-fault frame (it would land in the guard page too), so it raises a double fault, which runs on its own IST stack. That handler sees the faulting address is a guard page and prints [kernel] kernel stack overflow in thread 'NAME' (hit guard page 0x...), then kills that thread (or its process) and keeps going.
• Stack probes. A function with a big local array could jump over a 4 KiB guard page. "stack-probes": {"kind": "inline"} in the target JSON makes the compiler touch each page of a big frame in order, so it can't.
• User stacks (1 MiB since stage 14, pages appearing as it grows) get a 64 KiB never-mapped zone underneath, and the ELF loader refuses programs whose segments reach into it. A fault there is a SIGSEGV, reported as "stack overflow" (exit code 139). Try overflow in the shell.

Kernel stacks are reused: a freed stack's slot goes on a free list and its page tables stay, so they never need freeing. Why no TLB shootdown? When a stack is freed, other CPUs may still have its old addresses cached in their TLB (the CPU's cache of page translations). Normally the freeing CPU would send them all an IPI to flush it. Here nothing can use a freed slot until a new thread gets it, so each thread instead flushes its stack's pages (invlpg) on each CPU the first time it runs there. Cheaper and simpler, and only possible because nobody else ever touches a thread's stack.

Known simplifications (good exercises): no priorities; one shared run queue; no CPU affinity; device interrupts all go to CPU 0.

User mode, syscalls and processes

The user programs in user/ run in ring 3 as separate processes, each with its own address space and file-descriptor table. The shell and its tools are described in The shell. These are the test and demo programs from stages 5 and 6 (run them from the shell: run hello, run evil...):

Program	What it does
hello	prints "Hello from user mode!", getpid, reads /hello.txt from the ramfs, creates and reads back /tmp/note.txt, checks error cases (ENOENT, EBADF, EFAULT, ENOSYS), exits with the number of failed checks (0)
evil	reads kernel memory at 0x200_0000_0000 (0x200000 before stage 20) -> page fault in ring 3 -> the kernel kills it and keeps going
spin	busy-loops ~2 s with no syscalls; it still gets preempted (threads A/B/C keep printing)
demo	stage 6 checks: cwd, reads /mnt/disk0/readme.txt, stat/fstat/lseek, lists bin/ with getdents64, chdir errors, mkdir/unlink/rmdir on ramfs and FAT, a pipe from a child, arguments, wait exit codes (42, 139 for killed evil), ECHILD/ENOENT/ENOEXEC, sleep_ms; exits with the number of failed checks
child	helper for demo and tests: child pipe writes a line to stdout, child cat copies stdin to stdout, child 42 exits with 42, otherwise prints its arguments

All of them are read from /mnt/disk0/bin/, on the FAT32 partition of the virtio disk; they're not compiled into the kernel any more.

Rings in one paragraph

The CPU's current privilege level (CPL) is the low 2 bits of the CS register: 0 = kernel, 3 = user. In ring 3 the CPU refuses privileged instructions (hlt, cli, writing CR3, in/out...) and refuses to touch any page whose page-table entries lack the USER_ACCESSIBLE bit. User code can only get back into ring 0 through doors the kernel set up: interrupts/exceptions (via the IDT) and the syscall instruction (via the LSTAR MSR). Every such door switches to a kernel stack, so a user program can't corrupt kernel state through its own stack pointer.

GDT and TSS (src/gdt.rs)

0x00 null
0x08 kernel code (ring 0)       0x10 kernel data (ring 0)
0x18 user data   (ring 3) |3    0x20 user code   (ring 3) |3
0x28 TSS (16 bytes: two slots)

The odd order (user data before user code) is forced by sysret: it loads SS = STAR[63:48] + 8 and CS = STAR[63:48] + 16, so the two user segments must be adjacent in that order. The TSS has two jobs: IST[0] is the double-fault stack (stage 2), and rsp0 is the stack the CPU switches to when an interrupt arrives while in ring 3. Each thread has its own kernel stack, so the scheduler writes the next thread's stack top into TSS.rsp0 on every switch (gdt::set_kernel_stack). Since stage 11 every CPU has its own GDT and TSS (the TSS is per CPU by nature: it says where this CPU's interrupts land).

The syscall path (src/syscall.rs)

syscall was chosen over int 0x80: it's what Linux uses on x86_64, it's faster, and the setup is only four MSRs. The catch is that, unlike an interrupt, syscall does not switch stacks -- the entry code must.

Setup (syscall::init): EFER.SCE = 1 (enable), STAR = the selectors above, LSTAR = address of syscall_entry, SFMASK = IF|DF|TF (so we enter with interrupts off until we're on a kernel stack).

One call, step by step:

1. User code puts the number in rax, args in rdi, rsi, rdx (Linux ABI; we also save r10, r8, r9) and executes syscall.
2. CPU: rcx = user RIP, r11 = user RFLAGS, CS/SS = kernel, RFLAGS &= ~SFMASK, RIP = LSTAR. RSP is still the user stack.
3. syscall_entry (asm): swapgs to reach this CPU's PerCpu data, stash user RSP there, load this thread's kernel stack top (set by the scheduler), push user RSP, swapgs back, push rcx, r11 and the args.
4. Shuffle into the C ABI and call dispatch(nr, a1..a5) in Rust with interrupts re-enabled (so a long syscall, e.g. a blocking read, can be preempted).
5. dispatch validates user pointers (user_slice: the range must be inside the user half and inside the process's memory regions; pages not there yet are faulted in, copy-on-write pages copied before a write), runs the call, and returns a value or -errno in rax.
6. Since stage 14 the entry saves all registers as a TrapFrame (src/trap.rs) and returns through the same exit as interrupts: deliver any pending signal (which may rewrite the frame to enter a handler), pop everything back and iretq. Stage 13 used cli + sysretq (RIP = rcx, RFLAGS = r11), which is faster but can't restore rcx and r11, and a return from a signal handler needs every register exactly as it was.

With several CPUs, the "current kernel stack" and "saved user RSP" can't be globals (two CPUs can be in a syscall at once), so they're per-CPU, reached with swapgs (see "Multiple CPUs" below).

Numbers are Linux's wherever Linux has the same call, so man pages and strace habits mostly apply. Args 4 and 5 go in r10 and r8 like Linux.

nr	name	args	notes
0	read	fd, buf, len	console: blocks until Enter; pipe: blocks until data or EOF; socket: like recv
1	write	fd, buf, len	console = screen + serial; pipe: EPIPE and SIGPIPE once the reader is gone; socket: like send
2	open	path, flags, mode	C string since stage 15.1 (= openat(AT_FDCWD, ...)). O_RDONLY/O_WRONLY/O_RDWR 0/1/2, O_CREAT 0o100, O_EXCL 0o200 (EEXIST), O_TRUNC 0o1000, O_APPEND 0o2000 (every write goes to the end: >>), O_DIRECTORY 0o200000 (ENOTDIR), O_NOFOLLOW 0o400000 (ELOOP on a symlink), O_CLOEXEC 0o2000000 (closed by execve); directories can be opened (for getdents64), but not for writing (EISDIR). Since stage 16 a new file gets mode & ~umask (ramfs, ext2), and F_GETFL reports the access mode
3	close	fd
4	stat	path, *stat	stage 15.1: Linux's 144-byte struct stat; follows symlinks
5	fstat	fd, *stat
6	lstat	path, *stat	stage 15.1: doesn't follow a final symlink
7	poll	*pollfd, n, timeout_ms	stage 15: -1 = forever; pipes, sockets, ttys, files, epoll fds
8	lseek	fd, offset, whence	SEEK_SET/CUR/END = 0/1/2; ESPIPE on console/pipes
9	mmap	addr, len, prot, flags, fd, offset	MAP_PRIVATE/SHARED, MAP_ANONYMOUS, MAP_FIXED; files: private only
10	mprotect	addr, len, prot	PROT_READ 1, PROT_WRITE 2, PROT_EXEC 4; stage 19: the vvar page can't be made writable (EACCES)
11	munmap	addr, len	stage 19: also above the mmap area (the stack, the vDSO), like mprotect
12	brk	addr	returns the (new) end of the heap; brk(0) = current
13	rt_sigaction	sig, *new, *old, 8	kernel struct sigaction; handlers need SA_RESTORER
14	rt_sigprocmask	how, *set, *old, 8	SIG_BLOCK 0 / SIG_UNBLOCK 1 / SIG_SETMASK 2
15	rt_sigreturn		only from a signal handler's restorer
16	ioctl	fd, request, arg	any fd, stage 15: FIONBIO (0x5421). Ttys (console, hvc0, ptys), stage 15: TCGETS/TCSETS/TCSETSW/TCSETSF (0x5401-0x5404, the 36-byte kernel struct termios), TIOCSCTTY, TIOCNOTTY, TIOCSWINSZ, TIOCGPTN/TIOCSPTLCK on /dev/ptmx. Console: TTY_SET_RAW (1): raw mode on (arg 1) / off (0); returns the old mode. /dev/fb0: FB_GET_INFO (0x4600) writes [width, height, bytes per row, 32] to *arg; FB_FLUSH (0x46f0) shows *[x, y, w, h] (arg 0: the rows this fd wrote); FB_FILL (0x46f1) fills *[x, y, w, h, 0xRRGGBB]; FB_SET_CONSOLE (0x46f2) moves the text console into *[x, y, w, h] (arg 0: full screen) and returns cols << 16 | rows. Console, stage 14: TIOCGPGRP/TIOCSPGRP (0x540f/0x5410) get/set the foreground process group (Ctrl+C goes to it), TIOCGWINSZ (0x5413) the size in characters. Others: ENOTTY
17	pread64	fd, buf, len, offset	stage 15: regular files; leaves the file position alone
18	pwrite64	fd, buf, len, offset	stage 15
19	readv	fd, *iovec, n	stage 15: struct iovec {base, len}, up to 1024
20	writev	fd, *iovec, n	stage 15
21	access	path, mode	stage 15.1: 0 if the path exists (no permissions to check)
22	pipe	*[u32; 2]	[read end, write end]; pipe2 (293) also takes O_CLOEXEC
23	select	n, *rd, *wr, *ex, *timeval	stage 15: fd bitmaps (n <= 1024)
24	sched_yield
25	mremap	addr, old_len, new_len, flags, new_addr	stage 19: grow (in place or, with MREMAP_MAYMOVE 1, moved), shrink, MREMAP_FIXED 2; mmap regions only; MREMAP_DONTUNMAP and old_len 0 are EINVAL
26	msync	addr, len, flags	stage 21: only checks its arguments (nothing to write back: no shared file mappings); EINVAL for an unaligned addr, unknown flags or MS_SYNC+MS_ASYNC, ENOMEM for an unmapped page
28	madvise	addr, len, advice	stage 16: MADV_DONTNEED 4 / MADV_FREE 8 free private anonymous pages (they read back as zeros); other known hints return 0; unknown ones EINVAL, unmapped memory ENOMEM
32	dup	fd	lowest free fd
33	dup2	old, new	closes new first if open
34	pause		sleeps until a signal; returns EINTR
35	nanosleep	*timespec, *remaining	Linux's nanosleep since stage 15 (stages 2-14 had our own sleep_ms(ms) here). Sleeps whole PIT ticks, then yields until the TSC clock says the time is up; a signal ends it early with EINTR and the time left in *remaining
36	getitimer	which, *itimerval	stage 17: ITIMER_REAL only (others EINVAL)
37	alarm	seconds	stage 17: SIGALRM after N seconds; returns the seconds left on the old one; 0 cancels
38	setitimer	which, *new, *old	stage 17: ITIMER_REAL, one-shot or periodic, in microseconds; fires up to ~110 ms late, never early
39	getpid
40	sendfile	out_fd, in_fd, *offset, count	stage 17: in-kernel copy; the input must be seekable (EINVAL for a pipe); with *offset the input position doesn't move
41	socket	domain, type, protocol	Stage 18: AF_UNIX (1) too, with SOCK_STREAM, SOCK_DGRAM and SOCK_SEQPACKET (5), plus SOCK_NONBLOCK/SOCK_CLOEXEC; see "Unix sockets". AF_INET (2): SOCK_STREAM (1) = TCP, SOCK_DGRAM (2) = UDP, or SOCK_DGRAM + IPPROTO_ICMP (1) = ping socket (like Linux's unprivileged ping sockets). Returns an fd; close closes it
42	connect	fd, *sockaddr_in, len	Stage 18: AF_UNIX takes a sockaddr_un (path, abstract, or AF_UNSPEC to disconnect a datagram socket); ECONNREFUSED if nobody listens, EAGAIN for a full backlog when nonblocking. TCP: blocks for the handshake (10 s, ETIMEDOUT; ECONNREFUSED on a reset). UDP: sets the default peer
43	accept	fd, *sockaddr_in, *len	(also AF_UNIX, stage 18) blocks for a connection on a listening socket; returns a new fd. Since 15.1 *len is read and updated like Linux
44	sendto	fd, buf, len, flags, *sockaddr_in, len	address 0 = the connected peer. ICMP: you write the echo header, the kernel fills in the id and checksum. flags (so MSG_NOSIGNAL) and len are ignored
45	recvfrom	fd, buf, len, flags, *sockaddr_in, *len	blocks (wait queue) until data; 0 = the peer closed (TCP); EAGAIN after SO_RCVTIMEO. Since 15.1 the address honours *len
46	sendmsg	fd, *msghdr, flags	stage 18: struct msghdr (56 bytes): a name, iovecs and control messages. AF_UNIX: SCM_RIGHTS (up to 253 fds) and SCM_CREDENTIALS; flags MSG_DONTWAIT, MSG_NOSIGNAL. INET sockets: name + iovecs
47	recvmsg	fd, *msghdr, flags	stage 18: fills the iovecs, the sender's name, control messages (fds become new fds; MSG_CMSG_CLOEXEC; MSG_CTRUNC if they don't fit, the rest are closed) and msg_flags (MSG_TRUNC); flags MSG_PEEK, MSG_WAITALL, MSG_TRUNC, MSG_DONTWAIT
48	shutdown	fd, how	stage 17: SHUT_RD/SHUT_WR/SHUT_RDWR; SHUT_WR sends a FIN (the peer reads 0); ENOTCONN if not connected
49	bind	fd, *sockaddr_in, len	local port (address must be 0 or ours). Stage 18 AF_UNIX: makes a socket file (EADDRINUSE if the name exists, EPERM on FAT/exFAT), an abstract name, or with only the family an automatic abstract name
50	listen	fd, backlog	TCP; up to 4 connections can be waiting
51	getsockname	fd, *sockaddr_in, *len	stage 17: my address and port (after bind, connect or accept)
52	getpeername	fd, *sockaddr_in, *len	stage 17: the other end; ENOTCONN if none
53	socketpair	domain, type, protocol, *[u32; 2]	stage 18: AF_UNIX, any of the three types: two connected sockets (stage 17 had no AF_UNIX: EAFNOSUPPORT); EOPNOTSUPP for AF_INET
54	setsockopt	fd, level, name, *val, len	SO_RCVTIMEO (a timeval) is real; stage 17: SO_REUSEADDR/SO_REUSEPORT/SO_KEEPALIVE, SO_SNDTIMEO, buffer sizes and TCP_NODELAY are accepted and remembered; anything else ENOPROTOOPT, like Linux
55	getsockopt	fd, level, name, *val, *len	stage 17: SO_TYPE, SO_ERROR (once), SO_ACCEPTCONN, SO_DOMAIN, SO_PROTOCOL, the timeouts, buffer sizes and remembered flags. Stage 18 AF_UNIX: SO_PEERCRED (pid, uid, gid of the other end), SO_PASSCRED, and buffer sizes that really limit the queue (setsockopt doubles them, like Linux)
56	clone	flags, stack, *ptid, *ctid, tls	fork-like (SIGCHLD, optional CLONE_VFORK), or since stage 15 a thread: `CLONE_VM
57	fork		copy-on-write; parent gets the pid, child 0
58	vfork		fork + the parent waits until the child execs or exits
59	execve	path, argv, envp (C strings)	Linux's stack layout: argc, argv, envp, auxv
60	exit	code	since stage 15 ends only the calling thread; exit_group (231) ends the whole process
61	wait4	pid, *status, options, *rusage	Linux's wait4 since stage 14 (stage 13 had a simpler wait): pid > 0 one child, -1 any, 0 / < -1 a process group; *status gets the packed status (WIFEXITED, WEXITSTATUS, WIFSIGNALED, WTERMSIG, WIFSTOPPED...); options WNOHANG 1 (returns 0 if nobody is done), WUNTRACED 2, WCONTINUED 8; reaps the child; ECHILD if none; since 15.1 a *rusage is zero-filled (no usage accounting yet)
62	kill	pid, sig	pid 0 = my group, -1 = all, < -1 = group -pid
63	uname	*utsname	Linux's struct: 6 NUL-padded 65-byte fields: "Rustbelt", "qemu", "0.16.0", "stage 16: libc batch 1", "x86_64", "(none)"
72	fcntl	fd, cmd, arg	F_DUPFD 0 (below RLIMIT_NOFILE, else EINVAL), F_GETFD 1, F_SETFD 2, F_GETFL 3 (since stage 16 the real access mode, O_APPEND, O_NONBLOCK), F_SETFL 4 (O_NONBLOCK; O_APPEND since 16), F_DUPFD_CLOEXEC 1030. Stage 16: record locks F_GETLK 5, F_SETLK 6 (EAGAIN on a conflict), F_SETLKW 7 (sleeps; EDEADLK, EINTR), Linux's struct flock; dropped when the process closes any fd of the file, or exits
73	flock	fd, op	stage 17: whole-file LOCK_SH/LOCK_EX, LOCK_NB, LOCK_UN; owned by the open file, dropped when its last fd closes
74	fsync	fd	stage 16: 0 for files and directories (writes already go straight to the disk); fdatasync 75 the same; EINVAL for pipes, sockets, ttys
76	truncate	path, length	stage 16: grow (zeros) or shrink a file; EISDIR, EINVAL (negative), EFBIG (over 4 GiB), EROFS on btrfs
77	ftruncate	fd, length	stage 16: the same through an fd opened for writing (else EINVAL)
79	getcwd	buf, len	like Linux since stage 15: writes a NUL and returns the length including it; ERANGE if too small
80	chdir	path	per-process cwd, inherited by forked and spawned children
81	fchdir	fd	stage 15.1: an open directory
82	rename	old, new	stage 15.1: files (also over an existing file) and empty directories, within one file system (else EXDEV). Not atomic: see "Linux paths". Stage 17: ramfs and ext2 move the entry itself (full directories, files with several names); renameat2 316 adds RENAME_NOREPLACE
83	mkdir	path, mode	since stage 16 mode & ~umask (ramfs, ext2)
84	rmdir	path	empty directories only (ENOTEMPTY); EBUSY on a mount point
86	link	old, new	stage 17: a hard link on ramfs and ext2 (EMLINK past 32000); EPERM on FAT/exFAT, EROFS on btrfs, EXDEV across file systems
87	unlink	path	files and symlinks (the link, not its target); directories give EISDIR
88	symlink	target, path	stage 17: ramfs and ext2; the target needn't exist
89	readlink	path, buf, len	a symlink's target (no NUL); returns its length, cut to len like Linux; EINVAL if not a symlink. Stage 15: /proc/self/exe is the running program's path. Stage 18: /proc/self/fd/N too (pipe:[N], socket:[N] for things without a path)
90	chmod	path, mode	stage 16: the 12 permission bits, stored on ramfs and ext2; EPERM on FAT/exFAT (nowhere to keep them), EROFS on btrfs. Never checked: everyone is root
91	fchmod	fd, mode	stage 16
92	chown	path, uid, gid	stage 17: -1 = unchanged; stored on ramfs and ext2 (FAT/exFAT: EPERM unless nothing changes); not enforced; fchown 93, lchown 94
95	umask	mask	stage 16: per process (default 022), inherited by fork; returns the old one
96	gettimeofday	*timeval, *timezone	stage 19: microseconds of CLOCK_REALTIME; the time zone is always 0 (UTC). Usually answered by the vDSO
97	getrlimit	resource, *[cur, max]	stage 16: = prlimit64(0, ...); setrlimit 160 too
98	getrusage	who, *rusage	stage 17: user/system CPU time (10 ms samples) for RUSAGE_SELF, RUSAGE_CHILDREN, RUSAGE_THREAD; ru_maxrss
99	sysinfo	*sysinfo	stage 17: Linux's 112-byte struct: uptime, memory, process count, "load" = busy fraction since boot
100	times	*tms	stage 17: CPU times of me and my waited-for children in 1/100 s; returns ticks since boot
102	getuid		stage 17: the real user id (0 = root unless changed); getgid 104, geteuid 107, getegid 108 likewise
105	setuid	uid	stage 17: root sets all three ids; others only the effective one, to their real or saved id (EPERM); setgid 106 likewise
109	setpgid	pid, pgid	0 = myself / a new group
110	getppid		0 if started by the kernel or orphaned
111	getpgrp
112	setsid		stage 15: new session + group, no controlling tty
113	setreuid	ruid, euid	stage 17: Linux's rules for the saved id; setregid 114
115	getgroups	size, *list	stage 17: supplementary groups; setgroups 116 (root only)
117	setresuid	ruid, euid, suid	stage 17: not root: each new value must be one of the current three; getresuid 118, setresgid/getresgid 119/120
121	getpgid	pid
122	setfsuid	uid	stage 17: the id new files get; returns the old one; setfsgid 123
124	getsid	pid	stage 15
127	rt_sigpending	*set, size	stage 16: signals waiting because they're blocked
128	rt_sigtimedwait	*set, *siginfo, *timespec, 8	stage 16: take one of set without running a handler (sleeps; EAGAIN after the timeout, {0, 0} = just look)
129	rt_sigqueueinfo	pid, sig, *siginfo	stage 16: send with your own siginfo (sigqueue); si_code must be negative for another process (EPERM). Real-time signals (32-64) queue up, each with its value; rt_tgsigqueueinfo 297 for one thread
130	rt_sigsuspend	*mask, 8	stage 16: swap in mask, sleep until a handler runs, put the old mask back, EINTR (dash's wait)
131	sigaltstack	*new, *old	stage 16: a separate stack for SA_ONSTACK handlers; SS_DISABLE 2, SS_ONSTACK 1 (read only), SS_AUTODISARM; ENOMEM below 2 KiB, EPERM while on it
137	statfs	path, *statfs	stage 17: Linux's magic numbers (ramfs, msdos, ext2, exfat, btrfs), sizes, free blocks and inodes, ST_RDONLY
138	fstatfs	fd, *statfs	stage 17; pipes and sockets get PIPEFS/SOCKFS
140	getpriority	which, who	stage 17: returns 20 - nice (the raw call; libc subtracts)
141	setpriority	which, who, nice	stage 17: clamped to -20..19; lowering it needs root (EACCES); the scheduler ignores it
157	prctl	option, arg2...	stage 16: PR_SET_NAME/PR_GET_NAME 15/16 (thread name, 15 bytes), PR_SET_PDEATHSIG/GET 1/2, PR_SET_DUMPABLE/GET 4/3, PR_SET_CHILD_SUBREAPER/GET 36/37, PR_SET_NO_NEW_PRIVS/GET 38/39; others EINVAL
158	arch_prctl	code, addr	stage 15: ARCH_SET_FS 0x1002 / ARCH_GET_FS 0x1003: the TLS pointer
160	setrlimit	resource, *[cur, max]	stage 16: see prlimit64
170	sethostname	name, len	stage 17: what uname reports; root only, at most 64 bytes; setdomainname 171
186	gettid		stage 15: thread id (= pid for the first thread)
200	tkill	tid, sig	stage 15
201	time	*time_t or 0	stage 19: seconds since 1970. Usually answered by the vDSO
202	futex	*u32, op, val, *timespec, *u32, val3	stage 15: FUTEX_WAIT 0, WAKE 1, WAIT_BITSET 9, WAKE_BITSET 10 (+ PRIVATE 128)
204	sched_getaffinity	tid, len, *mask	stage 16: one bit per CPU (all of them: we don't pin threads); returns the bytes written (8). sched_setaffinity 203 accepts any mask that includes a CPU
213	epoll_create	size	stage 15: = epoll_create1(0)
217	getdents64	fd, buf, len	real Linux struct linux_dirent64 records (ino, off, reclen, type, name\0)
218	set_tid_address	*u32	stage 15: cleared + futex-woken when this thread exits; returns the tid
222	timer_create	clock, *sigevent, *id	stage 17: CLOCK_REALTIME/MONOTONIC/BOOTTIME; SIGEV_SIGNAL, SIGEV_NONE, SIGEV_THREAD_ID; signals carry SI_TIMER and si_value
223	timer_settime	id, flags, *new, *old	stage 17: TIMER_ABSTIME; periodic; timer_gettime 224, timer_getoverrun 225, timer_delete 226
228	clock_gettime	clock, *timespec	microsecond resolution (stage 19: nanoseconds, and usually answered by the vDSO without a system call) (TSC calibrated against the PIT). CLOCK_MONOTONIC (1) counts from boot; since stage 15 CLOCK_REALTIME (0) is real Unix time (the CMOS clock read once at boot, plus uptime); stage 17: CLOCK_PROCESS_CPUTIME_ID (2) and CLOCK_THREAD_CPUTIME_ID (3), in 10 ms steps
229	clock_getres	clock, *timespec	stage 17: 1 µs for the wall clocks (1 ns since stage 19), 10 ms for the CPU-time ones
230	clock_nanosleep	clock, flags, *timespec, *remaining	stage 15.1: CLOCK_REALTIME, MONOTONIC or BOOTTIME; flag TIMER_ABSTIME (1) sleeps until that time. musl's nanosleep and sleep use this
231	exit_group	code	ends the process: every thread (stage 15; exit, 60, ends one thread)
232	epoll_wait	epfd, *epoll_event, max, timeout_ms	stage 15: level-triggered; EPOLLONESHOT supported
233	epoll_ctl	epfd, op, fd, *epoll_event	stage 15: ADD 1, DEL 2, MOD 3
234	tgkill	tgid, tid, sig	stage 15: a signal for one thread
257	openat	dirfd, path, flags, mode	stage 15.1: like open, relative to dirfd or AT_FDCWD (-100); musl's open calls this. Stage 18: /proc/self/fd/N opens fd N's file again (pipes and ttys: the same open file); a socket file is ENXIO
258	mkdirat	dirfd, path, mode	stage 15.1
260	fchownat	dirfd, path, uid, gid, flags	stage 17: AT_SYMLINK_NOFOLLOW, AT_EMPTY_PATH
262	newfstatat	dirfd, path, *stat, flags	stage 15.1: AT_SYMLINK_NOFOLLOW 0x100 (= lstat), AT_EMPTY_PATH 0x1000 (= fstat(dirfd)); musl's stat/fstat/lstat all end up here
263	unlinkat	dirfd, path, flags	stage 15.1: AT_REMOVEDIR 0x200 = rmdir
264	renameat	olddirfd, old, newdirfd, new	stage 15.1; renameat2 (316) too, with flags 0 (stage 17: or RENAME_NOREPLACE)
265	linkat	olddirfd, old, newdirfd, new, flags	stage 17: AT_SYMLINK_FOLLOW, AT_EMPTY_PATH
266	symlinkat	target, dirfd, path	stage 17
267	readlinkat	dirfd, path, buf, len	stage 15.1
268	fchmodat	dirfd, path, mode	stage 16; fchmodat2 (452) also takes AT_SYMLINK_NOFOLLOW (EOPNOTSUPP on a symlink, like Linux) and AT_EMPTY_PATH
269	faccessat	dirfd, path, mode	stage 15.1; faccessat2 (439) too, its flags ignored
270	pselect6	n, *rd, *wr, *ex, *timespec, *{*mask, 8}	stage 15
271	ppoll	*pollfd, n, *timespec, *mask, 8	stage 15: sets the mask while waiting
273	set_robust_list	*head, 24	stage 16: pthreads' robust mutexes: when the thread dies, each lock it still holds gets FUTEX_OWNER_DIED and a waiter is woken. get_robust_list 274
280	utimensat	dirfd, path, *[atime, mtime], flags	stage 16: set file times (UTIME_NOW, UTIME_OMIT; NULL = now); path NULL = the fd itself (futimens). ext2 keeps whole seconds; btrfs EROFS
281	epoll_pwait	epfd, *events, max, timeout, *mask, 8	stage 15
283	timerfd_create	clock, flags	stage 20: a timer you read; clocks REALTIME 0, MONOTONIC 1, BOOTTIME 7 (= MONOTONIC); flags TFD_NONBLOCK, TFD_CLOEXEC
286	timerfd_settime	fd, flags, *new, *old	stage 20: like timer_settime; TFD_TIMER_ABSTIME 1; TFD_TIMER_CANCEL_ON_SET 2 is EINVAL
287	timerfd_gettime	fd, *itimerspec	stage 20: time left and the interval
288	accept4	fd, *addr, *len, flags	stage 15: flags: SOCK_NONBLOCK, SOCK_CLOEXEC (others EINVAL for AF_UNIX, stage 18)
291	epoll_create1	flags	stage 15: EPOLL_CLOEXEC
292	dup3	old, new, flags	flags: O_CLOEXEC
293	pipe2	*[u32; 2], flags	flags: O_CLOEXEC
295	preadv	fd, *iovec, n, offset	stage 17: the file position doesn't move; pwritev 296
302	prlimit64	pid, resource, *new, *old	stage 16: limits are per process and inherited. Real numbers where the kernel has a real limit: RLIMIT_NOFILE 64 (the fd table), RLIMIT_STACK (the user stack), RLIMIT_NPROC; the rest unlimited. Lowering works; raising a hard limit past the real one is EPERM
309	getcpu	*cpu, *node, 0	stage 19: the CPU we're on (node 0); sched_getcpu(). Usually answered by the vDSO with rdtscp
318	getrandom	buf, len, flags	random bytes from virtio-rng (the host's /dev/urandom), up to 64 KiB per call; never blocks, flags ignored. Without virtio-rng: a TSC-seeded xorshift (not secure)
332	statx	dirfd, path, flags, mask, *statx	stage 17: the 256-byte struct statx, from the same data as stat
500	spawn	path, argv, argv_len, *SpawnOptions	ours: start a program, returns its pid (fork + exec in one step, like posix_spawn; the kernel's first shell and getty use it, the shell uses fork + exec)
501	mounts	buf, len	ours (Linux reads /proc/mounts): the mount table as text, one path fstype line per mount, in mount order; returns the length, ERANGE if too small. The shell assigns drive letters from it
502	rb_sysinfo	*SysInfo	ours (called sysinfo until stage 17; Linux's sysinfo is 99): CPU count, the CPU you're on, thread/process/frame counts, uptime ticks, and per-CPU timer ticks and busy ticks (sample twice for the load)
503	resolve	name, name_len, *addrs, max	ours (Linux does DNS in libc, in user space): IPv4 addresses for a name via the DHCP-given DNS server; returns how many; ENOENT / ETIMEDOUT
504	netinfo	*NetInfo	ours (Linux has ioctls and netlink): MAC, IRQ, address/prefix, gateway, DNS, packet and byte counters
505	trace_me	on	ours: print my syscalls from now on (survives execve; run --trace)

Deviations from Linux worth knowing:

• Paths are C strings since stage 15.1 (stages 5-15 passed pointer + length, which was easier to validate and matches Rust's &str). The kernel copies at most PATH_MAX (4096) bytes, NUL included, a page at a time, so a bad pointer is EFAULT and a missing NUL ENAMETOOLONG. Relative paths start at the process's cwd or at an *at() call's dirfd; . and .. are resolved in the kernel (fs::normalize).
• stat is Linux's (since 15.1): the 144-byte x86_64 struct stat. st_mode has the real type bits (S_IFREG, S_IFDIR, S_IFLNK, S_IFCHR for the console and /dev/*, S_IFIFO, S_IFSOCK) and the file system's permission bits, or a sensible default (0644 files, 0755 directories) on ramfs and FAT. st_ino is a hash of the path (no file system here exposes stable inode numbers through the VFS yet), uid/gid are 0, and (since stage 16) the times are real: ext2 and btrfs keep them on disk, ramfs in memory; FAT and exFAT still report 0.
• Permissions are ignored: everything runs as root (uid 0) and access only checks that a path exists. Since stage 16 open's and mkdir's modes (minus the umask) and chmod are stored on ramfs and ext2, so ls -l and stat show them, but nothing checks them.
• Stage 14's calls are exactly Linux's. execve takes NUL-terminated C strings and char ** arrays, rt_sigaction takes the kernel's struct sigaction, wait4 writes Linux's status word, siginfo and the signal frame have Linux's layout: a C library can use them unchanged (see "Processes" below for how they work).
• spawn as well as fork + exec. spawn (500) does both steps at once, like Windows CreateProcess or POSIX posix_spawn. What Unix shells do between fork and exec -- pointing fds 0/1/2 at a pipe or a file -- is spawn's SpawnOptions: { stdin, stdout, stderr: i64 (one of *my* fds, -1 = same as mine), flags }. flags = SPAWN_TRACE (1) prints every syscall the child makes, as [trace pid] open("notes/todo.txt", 0o0) = 3 in strace style (strings shown, errors as -2 ENOENT). The shell now uses fork + exec, and trace_me (505) to get the same tracing for run --trace. spawn's argv is one buffer with NUL-separated arguments ("child\0with\0three args"); either way the program starts with Linux's argc/argv/envp/auxv stack, which ulib reads as arg(i)/getenv/getauxval.

Errors are negative errno values like Linux (-2 ENOENT, -9 EBADF, -14 EFAULT, -38 ENOSYS...). Unknown numbers return -ENOSYS.

Address spaces (src/memory.rs, src/process.rs)

Each process gets a fresh level-4 page table. The kernel's L4 entries are copied into it (so the kernel, heap, physical-memory map and thread stacks are mapped in every address space and the kernel keeps working after a CR3 switch), and L4 slot 1 is the process's own:

0x0000_0000_0040_0000   stage 20: USER_START; plain `gcc -static` programs per process
0x0000_0080_0000_0000   L4 slot 1 (USER_START until stage 20)             per process
0x0000_0080_0040_0000     program (.text R-X, .rodata R--, .data/.bss RW-), then the brk heap
                          ... mmap regions, handed out downwards (stage 14, see below)
0x0000_00ff_ffef_f000     user stack (1 MiB, RW-, grows down; a 64 KiB guard zone under it)
0x0000_0100_0000_0000   USER_END
0x0000_0100_0020_0000   boot stack (L4 slot 2)                           shared
0x0000_0180_0000_0000   all physical memory (phys_mem_offset, slot 3)    shared
0x0000_0200_0000_0000   kernel code (stage 20; was 0x200000)             shared, ring 0 only
0x0000_4444_4444_0000   kernel heap (thread kernel stacks live here)     shared

Real kernels put the kernel in the upper half (0xffff_8000_...) and give user space the whole lower half. With bootloader 0.9 the kernel sits low, so here user space is one 512 GiB slot that the kernel doesn't use. Only user pages have USER_ACCESSIBLE, which is why evil faults: the page at 0x200000 is mapped, just not for ring 3 (PROTECTION_VIOLATION | USER_MODE). (Stage 20 moved the kernel to 0x200_0000_0000 and made user space start at 4 MiB; see "Plain static programs and timerfd (stage 20)".) .text is not writable and everything else is NO_EXECUTE (when EFER.NXE is on).

A process = a kernel thread (thread.rs, it already has a kernel stack and gets preempted) + a CR3 value + a Process struct (pid, name, fd table, exit status). The scheduler, on each switch, sets TSS.rsp0 and the syscall kernel stack to the next thread's stack and writes CR3 if it differs. A new process's thread starts in the kernel, then iretqs into ring 3 with a hand-built frame (SS=user data, RSP=user stack, RFLAGS=IF, CS=user code, RIP=ELF entry).

When ring-3 code faults (page fault with USER_MODE, general protection fault or invalid opcode with CS RPL 3), the kernel doesn't panic. A page fault may just be demand paging or copy-on-write (see stage 14); otherwise the process gets a signal (SIGSEGV, SIGILL, SIGFPE...). If it has no handler for it, it is killed (Killed(SIGSEGV, ...), exit code 139), its thread exits and the scheduler picks someone else. A fault in ring 0 still panics -- that's a kernel bug.

ELF loading and building user programs (src/elf.rs, build.rs, user/)

build.rs compiles each user/*.rs with plain rustc for the built-in x86_64-unknown-none target (no build-std needed: it ships a precompiled core and uses soft-float, so no SSE state to save). Flags worth knowing: -C relocation-model=static (no dynamic loader), -C code-model=large (the program lives above 4 GiB; the default model assumes 32-bit addresses and fails with "relocation overflow"), -T user/link.ld (one PT_LOAD segment per permission, page-aligned). The large code model names its sections .ltext/.lrodata/.ldata/.lbss, so the linker script lists those too -- forgetting that put the code into a non-executable segment and every program died at its first instruction. The ELFs (~15-20 KB) used to be embedded in the kernel with include_bytes!; since stage 6 build.rs copies them onto the FAT32 disk image instead and process::spawn_path reads them through the VFS (see "Disks" below).

user/ulib.rs is a tiny libc-equivalent included by each program: the syscall inline asm, wrappers, print!/println!/eprintln!, _start (calls main, then exit with its return value) and a panic handler (exit(101)). Since stage 7 it also has a heap with a power-of-two size-class allocator (free list per size) as the #[global_allocator]; since stage 14 its memory comes from brk (up to 1 MiB blocks) instead of a fixed 64 KiB array. A program that says extern crate alloc; can use Vec, String and format!; the x86_64-unknown-none target ships a precompiled alloc too.

The loader (~160 lines, no crate) checks the header (magic, 64-bit, little endian, x86_64, ET_EXEC), and for every PT_LOAD segment: checks it's inside the user range, maps zeroed frames with flags from the segment (W -> writable, no X -> NX), copies filesz bytes through the physical-memory map and leaves the rest (.bss) zero. It refuses (ENOEXEC) a file where two segments with different permissions share a page. A page has one set of permissions, so the second segment would silently get the first one's. This happened for real in stage 7: sleep has no .data, so the linker put its .bss on the read-only .rodata page, and sleep died on its first write to a static. user/link.ld now aligns the location counter before .data, not just the (empty, hence dropped) section.

VFS (src/fs/) and file descriptors

Designed like Linux's so real filesystems can plug in later:

• trait Inode (fs/mod.rs): a file, directory or symlink: kind, size, mode, readlink, read_at(offset, buf), write_at, truncate, lookup(name), create(name, kind), list (names + kinds), remove(name). Methods have default impls returning an error, so a read-only fs implements only what it supports.
• trait FileSystem: name() and root() -> Arc<dyn Inode>. A disk filesystem will hold its block device and parsed superblock.
• Mount table: mount("/tmp", fs); lookup(path) picks the longest mount point that prefixes the path, then walks the rest with Inode::lookup. Boot mounts a ramfs on / (/hello.txt, /etc/motd, /tmp, /mnt), a second, separate ramfs on /tmp, and each FAT partition at /mnt/diskN. fs::normalize(cwd, path) turns any path into a clean absolute one first.
• trait File (fs/file.rs): what a file descriptor points to: read(buf), write(buf), seek, stat, getdents. InodeFile = an inode + its own offset (two opens of the same file have separate offsets, like Linux's struct file). tty::Console is another File: writes go to VGA+serial, reads come from the keyboard line buffer. fs/pipe.rs gives two more: a pipe's read and write ends.
• fd table: Vec<Option<Arc<dyn File>>> per process (max 16), fds 0-2 = Console. open = fs::lookup (or fs::create with O_CREAT) + InodeFile::open
  • first free slot.

Keyboard input for read(0) goes through a small tty line discipline (src/tty.rs), like Linux's termios:

• Cooked mode (the default): the kernel echoes the characters and Backspace edits the current line. Enter makes the line, with its \n, available to readers. Programs like cat with no arguments see this.
• Raw mode (ioctl(0, TTY_SET_RAW, 1)): every key goes to the reader at once and nothing is echoed. Backspace is \x08, and the arrows, Home, End and Delete arrive as ANSI escape sequences (ESC [ A = up...). The shell uses this while you edit a line, so it can do the echo and history itself. It switches back to cooked before running a program. Raw mode is also switched off automatically if its owner process exits.

The console understands two control characters on output. \x08 moves back one column and blanks it, and \x0c (form feed) clears the screen, which is what cls writes.

Process cleanup. When wait reaps a process, memory::free_address_space walks L4 slot 1 (the user half) and frees every page, page table and the L4 table. Freed frames go on a free list that alloc_frame uses first. The list is an intrusive linked list: each free frame stores the address of the next one, written through the physical-memory map, so it needs no heap. The exiting process disables interrupts before publishing its exit status. That way the parent can't free the address space while the dying thread might still run on it.

Known simplifications (exercises): a process nobody waits for keeps its memory; orphans aren't reparented; no signals; user pointer checks walk the page table but don't guard against another thread unmapping the page mid-syscall (there's no munmap yet, so that's fine for now).

Disks: PCI, virtio-blk, partitions, FAT32 (stage 6)

/mnt/disk0/readme.txt
   VFS (fs/mod.rs)          longest mount prefix -> FatFs, rest "readme.txt"
   FatInode (fs/fat.rs)     fatfs crate: clusters, FAT chains, directory entries
   BlockIo adapter          byte offsets -> whole 512-byte sectors
   Partition vda1           sector + 2048, bounds-checked
   VirtioBlk vda            virtqueue request to QEMU
   PCI                      found the device and its I/O port at boot

PCI (src/pci.rs). Every PCI device has a 256-byte configuration space (vendor/device id, class code, BARs = where its registers live). On a PC you read it by writing bus/device/function/offset to port 0xCF8 and reading port 0xCFC. pci::scan() tries every bus/device/function, keeps the ones whose vendor id isn't 0xFFFF, and prints them with a class name. QEMU's default machine has 7: host bridge, ISA/IDE/ACPI parts of the PIIX chipset, VGA, the e1000 network card, and our disk, 1af4:1001.

Why virtio-blk and not ATA PIO? ATA PIO is simpler (a few port writes, then 256 in instructions per sector) but it's a 1980s interface that only emulators and old hardware have. virtio is what QEMU/KVM and cloud VMs actually use, and it teaches the pattern every modern device uses: the driver and device share queues of descriptors in RAM. The legacy (virtio 0.9.5) interface is used, through I/O port BAR0, because it needs no MMIO mapping or PCI capability parsing; QEMU's device is "transitional" and speaks both.

virtio-blk in one paragraph (src/block/virtio_blk.rs, ~280 lines). Reset the device, set ACKNOWLEDGE + DRIVER, accept no optional features, read the queue size, allocate the virtqueue (descriptor table + available ring + used ring, physically contiguous: memory::alloc_contiguous_frames) and give its page number to the device, set DRIVER_OK. A request is a chain of three descriptors: a 16-byte header (IN = read or OUT = write, sector number), the data buffer, and a 1-byte status the device fills in. Put the chain's head in the available ring, write the queue number to the notify port, then poll the used ring until the device hands it back. Data goes through a one-page bounce buffer whose physical address we know (heap buffers are virtually contiguous but not physically). Interrupts are switched off for the device (VRING_AVAIL_F_NO_INTERRUPT); polling is fine because QEMU completes requests almost instantly. Exercises: IRQ-driven completion (block the thread on a wait queue), several requests in flight, virtio 1.0 "modern" MMIO.

BlockDevice (src/block/mod.rs): sector_count(), read_sectors(lba, buf), write_sectors(lba, buf), buffers a whole number of 512-byte sectors. Devices get Linux-style names in a registry (block::devices(), block::get("vda1")). RamDisk is a Vec<u8> disk used by tests.

Partitions (src/block/partition.rs). Sector 0 is the MBR: four 16-byte entries at offset 446 and the 55 AA signature. A single entry of type 0xEE is a "protective MBR" meaning look for a GPT: sector 1 has the EFI PART header, which says where the partition entry array is (128 bytes each: type GUID, unique GUID, first/last LBA, UTF-16 name). Each partition becomes a Partition -- itself a BlockDevice that adds its start sector and refuses to go past its end -- registered as vda1, vda2... CRCs aren't checked yet.

FAT32 via the fatfs crate (src/fs/fat.rs). fatfs does the filesystem logic; it wants something implementing its Read + Write + Seek traits on bytes. BlockIo is that adapter: it keeps a byte position, sends whole aligned sectors straight to the device, and handles partial sectors through a one-sector buffer (read the sector, change the bytes, write it back). Writes go straight through, so flush has nothing to do. (fatfs 0.3 from crates.io depends on the abandoned core_io crate, which doesn't build on current nightlies, so Cargo.toml pins a git revision of the 0.4 branch with its own no_std IO traits.) For the VFS, FatFs holds the fatfs::FileSystem in an Arc<Mutex<..>>, and a FatInode is just (that filesystem, a path, a kind): each call reopens its path from the root. Slower than caching directory entries, but simple and never stale. Lookups are case-insensitive, like FAT itself. No timestamps (NullTimeProvider) and long file names on.

The test disk (build.rs). After compiling the user programs, build.rs makes target/disk.img (64 MiB): sfdisk writes a GPT with one "Microsoft basic data" partition at sector 2048, mkfs.vfat -F 32 -s 1 -n TYLERDISK formats a separate partition image (1 sector per cluster so 62 MiB is enough clusters for FAT32), mmd/mcopy (mtools, which edit FAT images without mounting, so no root) copy in disk/ and bin/<programs>, and the partition image is spliced into the disk at sector 2048. If a tool is missing the build stops with sudo apt install dosfstools mtools fdisk (mkfs.vfat lives in /usr/sbin, which isn't in a normal user's PATH on Debian, so build.rs looks there too). Look inside it from Linux:

/usr/sbin/sfdisk -l target/disk.img             # not in a normal user's PATH on Debian
export MTOOLS_SKIP_CHECK=1                     # mtools: don't complain about the small-cluster FAT32
mdir -i target/disk.img@@1M ::/bin          # @@1M = partition offset (2048 * 512)
mtype -i target/disk.img@@1M ::/readme.txt

The shell (stage 7)

/bin/sh on the disk (user/sh.rs, about 550 lines) is an ordinary ring-3 program. It uses nothing but the syscalls above. Its design is a hybrid: Unix navigation and pipes, with DOS-style program running and drive letters on top. At boot a kernel thread starts it on the console and waits for it. When it exits, the thread starts a new one, the way init respawns a login shell.

Rustbelt shell. Type `help` for commands. Drives: A: = /, B: = /mnt/disk0, C: = /mnt/disk1
A:/ $ cd b:
B:/ $ ls
bin/        notes/      readme.txt  scripts/
B:/ $ type readme.txt | grep -i rustbelt
In the Rustbelt shell this disk is drive B: -- try `cd B:` and `ls`.
B:/ $ ls bin | wc -l
15
B:/ $ echo first > A:/tmp/a.txt
B:/ $ echo second >> A:/tmp/a.txt
B:/ $ wc < A:/tmp/a.txt
      2       2      13
B:/ $ run --trace cat notes/todo.txt
[trace 8] open("notes/todo.txt", 0o0) = 3
[trace 8] read(3, "Done: the shell (cd, ls, cat, run, start"..., 512) = 125
Done: the shell (cd, ls, cat, run, start, pipes, redirection, drive letters).
Next: more file systems -- ext2, exFAT, btrfs.
[trace 8] write(1, "Done: the shell (cd, ls, cat, run, start"..., 125) = 125
[trace 8] read(3, "", 512) = 0
[trace 8] close(3) = 0
[trace 8] exit(0)
B:/ $ start sleep 5
[pid 9] sleep started in the background
B:/ $ jobs
[pid 9] running  start sleep 5
B:/ $ run evil
[evil] pid 10: reading kernel memory at 0x200000...
[process] pid 10 (evil) killed: page fault
[exit code 139]
B:/ $ ver
Rustbelt 0.8.0 (stage 8: ext2) x86_64
B:/ $                      <- Enter on an empty line, a few seconds later
[done] pid 9 (start sleep 5), exit code 0
B:/ $

Commands

Command	What it does
cd [dir]	change directory (cd alone = A:/); cd B: goes to the root of drive B
pwd	print the current directory as a Unix path (the prompt shows the drive form)
echo text...	print the arguments
ver	kernel name and version, from the uname syscall
drives	the drive letters and what they map to
alias [name=value]	list aliases or define one: alias ll="ls -l"
history	the commands typed so far
jobs / wait [pid]	list background jobs ([1] [pid 5] running ...) / wait for them (all, or one pid)
kill [-SIG] pid|%job... / kill -l	send a signal (default SIGTERM; -9, -KILL, -INT...); %1 = job 1, a whole process group; -l lists the signals
fg [%job] / bg [%job]	continue a stopped job (Ctrl+Z) in the foreground / background
cls, clear	clear the screen
help	the command list
exit [code]	leave the shell (with the last command's exit code, or code)
run [--trace] prog args	run a program in the foreground; --trace prints every syscall it makes
start prog args	run it in the background and print its pid (same as prog args &)
prog args	a bare name is looked up in each drive's bin directory, in order: A:/bin, B:/bin...

Programs in B:/bin (user/*.rs, each 30-150 lines):

Program	What it does
ls [-l] [path...]	list a directory, sorted; in columns on the console, one name per line into a pipe or file (like real ls); symlinks get an @; -l shows sizes and <DIR>, plus permissions (-rw-r--r--) and link -> target on ext2
cat [file...]	print files, or stdin (also for -)
wc [-l] [file...]	lines, words, bytes
grep [-i] [-v] [-n] text [file...]	lines containing text (plain text, no regexes); exit code 0 = found, 1 = not found
mkdir dir...	create directories
rm path...	delete files or empty directories
touch file...	create empty files
cp src dest	copy a file (into dest if it's a directory)
sleep N / sleep 250ms	wait (blocks in the kernel, no busy loop)
ifconfig (ip)	the network card, its address from DHCP, gateway, DNS, packet counters
ping [-c N] [-i ms] host	ICMP echo, with times and a summary like the real one
host name (nslookup)	look up a name's IPv4 addresses
fetch [-o file] http://host[:port]/path (wget)	HTTP/1.0 GET; prints the body, or saves it (-o D:/page.html) with a size and speed report; exit 1 on non-200
httpd [-p port] [-n count] [-t secs]	a tiny web server: / is a greeting with a visitor count, other paths serve files from the current directory
random [-n bytes] / random -d sides [count]	random bytes as hex (default 16), or fair dice rolls; uses getrandom
getty [-1] device	run a shell on another terminal, e.g. getty /dev/hvc0 &; restarts it when it exits (-1: once)
tile [1-6] / tile -r	draw a tiling-window-manager mock-up on /dev/fb0 and move the console into the first window; -r gives the console the whole screen back
hello, demo, evil, spin, child	the stage 5/6 test programs: run demo runs all the stage 6 syscall checks
forktest, cowtest, mmaptest, sigtest	the stage 14 checks of fork/exec/wait4, copy-on-write, mmap and signals; each ends with "all checks passed" (forktest bomb is a bounded fork bomb, mmaptest crash dies of SIGSEGV)
threads [n], echoserver [port], ptytest, nbtest	the stage 15 checks: a parallel sum on n threads with a futex mutex; an epoll + poll echo server over 127.0.0.1 (echoserver 7070 just serves forever); ptys and the line discipline; nonblocking I/O and the libc extras. Each ends with "all checks passed"
fputest, pathtest, rawabi [dir]	the stage 15.1 checks: per-thread FPU/SSE/AVX state and signal frames; the Linux path calls; a C program with no libc doing file calls in dir (default /tmp), SSE maths, a signal and fork + wait4. Each ends with "all checks passed"
libctest wait|signals|locks|files|misc|orphans	the stage 16 checks (libc batch 1), one group at a time; each ends with "all checks passed"
coretest links|owners|statfs|info|sockets|timers|ids|misc	the stage 17 checks (coreutils batch 2), one group at a time; each ends with "all checks passed"

Built-in aliases are the DOS names: dir = ls, type = cat, del = rm, md = mkdir, copy = cp; and for networking, ip = ifconfig, nslookup = host, wget = fetch.

Ctrl+C interrupts the program in the foreground (SIGINT to its process group; the shell shows [exit code 130] (interrupted: SIGINT)), and on an empty or half-typed line it just starts a new one. Ctrl+Z stops it (fg continues it). Since stage 14 the shell starts every program with fork + execve, in its own process group (see "Processes" below).

Line editing: Backspace, and Up/Down to browse the history. Left/Right and Tab aren't supported yet. Scripts: sh file runs the commands in a file and echoes each one after its prompt. If stdin isn't the console (sh < file, cat file | sh), the shell reads commands from it the same way. # starts a comment.

Pipes and redirection

a | b | c, > file (create or truncate), >> file (append) and < file can be combined: cat < in.txt | grep x | wc -l > count.txt. The operators don't need spaces around them (ls|wc). Quotes ('...' or "...") group words and hide operators. There are no variables or escapes.

How a pipeline runs (Shell::run_pipeline): for each stage, the shell picks its stdin and stdout and spawns it.

• stdin: the < file, else the read end of the previous stage's pipe, else the shell's own stdin.
• stdout: a new pipe() to the next stage, overridden by > file, else the shell's own stdout.
• Spawning: the stage starts with spawn(path, argv, SpawnOptions { stdin, stdout, ... }).

Then the shell closes its own copies of all pipe ends and files. A reader only sees end-of-file once every write end is closed, and the shell holds one. Last, it waits for every stage; the last stage's exit code is the pipeline's. A builtin can be a stage too (help | grep run): it just writes to the pipe. Pipes have no size limit here, so that can't block.

Drive letters

Drive letters are only shell syntax. The kernel keeps one Unix tree:

• Why start at A:? In DOS and Windows the system drive is C: only for historical reasons. A: and B: were reserved for floppy drives, so the first hard disk got C:, and Windows still keeps that habit. Rustbelt has no floppies, so its letters simply start at the front of the alphabet. (The first version of the shell used C: for /.)

• Mapping: A: is /. B:, C:... go to the mounts under /mnt, in the order they were mounted. The shell reads them from the mounts syscall before every prompt, so a newly mounted disk would get a letter right away. Other mounts, such as /tmp, stay under A: (A:/tmp).

• Translation: every unquoted word that starts with a known letter is translated before any syscall sees it (shparse::translate):

  • B:/notes/todo.txt becomes /mnt/disk0/notes/todo.txt.
  • b: becomes /mnt/disk0 (any case).
  • B:notes becomes /mnt/disk0/notes, since drives don't have their own current directory.

  Quote a word to keep it literal: grep "B:" readme.txt.

• Prompt: the prompt goes the other way (shparse::display). The longest matching drive path wins, so /mnt/disk0/bin shows as B:/bin.

• Unknown letters: these are left alone, so cd Q: says there is no drive Q:.

• Stable letters: since letters follow mount order, Cargo.toml and the scripts attach the user-visible disks first (B: FAT32, C: ext2, D: exFAT, E: btrfs) and the test-only images after them. A disk that fails to mount (like the sha256 btrfs test image) gets no /mnt/diskN and no letter, so it can't shift the others either.

How the pieces fit

• user/shparse.rs: the tokenizer, the parser (Pipeline of Commands, each with words and redirections) and the drive-letter functions. All of them are pure functions with no syscalls. The same file is compiled into the shell and into tests/shell.rs (#[path = "../user/shparse.rs"]), so the parser is unit-tested inside the kernel test suite.
• user/sh.rs: the main loop runs these steps: refresh the drives, report finished background jobs (wait with WNOHANG), print the prompt, read a line, then run it. Before running, each command goes through prepare(): it translates drive letters, expands aliases, and strips run / run --trace / start prefixes. The command then runs as a builtin or is spawned with the right fds.
• Kernel support added for the shell:
  • ioctl (raw mode), uname, mounts, wait flags and O_APPEND.
  • Freeing process memory, so a long session doesn't run out of frames.
  • strace-style --trace output.
  • The \x08 and \x0c console controls.
  • A fix for the | key (see Notes).

ext2 (stage 8)

The third disk, target/ext2.img, holds an ext2 file system that Rustbelt reads and writes with its own driver, src/fs/ext2.rs (about 1100 lines including a long explanation at the top, no crates). It is mounted at /mnt/disk1, which is drive C: in the shell:

A:/ $ drives
A:  /            ramfs
B:  /mnt/disk0   fat32
C:  /mnt/disk1   ext2
A:/ $ cd c:
C:/ $ ls -l
-rw-r--r--     304000  big.txt
drwxr-xr-x      <DIR>  docs/
lrwxrwxrwx             hello-link -> hello.txt
-rw-r--r--        142  hello.txt
drwx------      <DIR>  lost+found/
-rw-r--r--        284  readme.txt
-rw-r--r--   69206016  sparse.bin
C:/ $ cat hello-link
Hello from the ext2 disk! This file lives on a second virtio disk, in an
ext2 file system that Rustbelt reads and writes with its own driver.
C:/ $ cp B:/notes/todo.txt C:/
C:/ $ cat todo.txt | grep -i ext2
Next: more file systems -- ext2, exFAT, btrfs.

The on-disk layout, for a learner

ext2 (1993) was Linux's main file system through the 1990s, and ext3 and ext4 grew out of it. It is the classic file system to learn from, because everything on disk is an array of fixed-size records that you find with plain arithmetic. You can follow along with dumpe2fs target/ext2.img and debugfs target/ext2.img (both in /usr/sbin).

• Blocks and groups. The disk is cut into blocks of 1, 2 or 4 KiB, and the blocks into block groups (8192 blocks each with 1 KiB blocks). Our 32 MiB disk has 1 KiB blocks, so it has 4 groups. Each group keeps its own bitmaps and inode table, so a file's inode and its data usually sit close together.

  byte 0       1024          2048 (block 2)
  | boot block | superblock  | group descriptors | reserved | block bitmap | inode bitmap | inode table | data...   <- group 0
  | superblock + descriptor backups | block bitmap | inode bitmap | inode table | data...                         <- group 1
  

• Superblock (always at byte 1024, whatever the block size). It holds the totals: block size (1024 << s_log_block_size), blocks and inodes per group, the free counts, the magic number 0xEF53, the revision and three sets of feature flags. probe() reads just that magic, and mount_disks tries ext2 first, then FAT.

• Feature flags. There are three sets, and each set tells an old driver what to do with a flag it doesn't know:

  • compat: ignore it.
  • ro_compat: mount read-only (Rustbelt then says read-only in the boot log, and writes give EROFS).
  • incompat: don't mount at all. ext4's extents are one, so Rustbelt refuses ext4 instead of misreading it.

• Group descriptors (32 bytes per group, in the block after the superblock). Each says where that group's block bitmap, inode bitmap and inode table are, and how many blocks and inodes are free there.

• Bitmaps. One bit per block or per inode, 1 = in use. Allocating is "find a 0 bit, set it, decrement the free count in the group descriptor and the superblock". Rustbelt allocates a file's blocks in the group of its inode first.

• Inodes. Inode N is at index (N-1) % inodes_per_group of group (N-1) / inodes_per_group's table. Inode 2 is the root directory, and 11 is lost+found. An inode has the mode (type + permissions), owner, size, times, link count and 15 block pointers:

  i_block[0..12]  data blocks 0-11              1 KiB blocks: the first 12 KiB
  i_block[12]     -> 256 block numbers          single indirect, up to 268 KiB
  i_block[13]     -> 256 single-indirect blocks double indirect, up to ~64 MiB
  i_block[14]     -> 256 double-indirect blocks triple indirect, up to ~16 GiB
  

  Volume::bmap turns "block 40 960 of this file" into a disk block by working out which level it falls in and walking down the pointers. A 0 pointer is a hole, which reads as zeros. That is how sparse.bin is 66 MiB while using only a few blocks. With 4 KiB blocks, each indirect block holds 1024 numbers.

• Directories are files full of variable-length entries: inode (u32), rec_len (u16), name_len (u8), file_type (u8), name. rec_len is the distance to the next entry. To delete an entry, Rustbelt adds its length to the previous entry's rec_len. To add one, it looks for an entry with spare room behind it, and if there is none, it gives the directory a new block. Revision 0 has no file_type byte, so the driver reads the inode to tell files from directories.

• Symlinks. A target shorter than 60 bytes is stored right inside i_block (a fast symlink, like hello-link). Longer targets get a data block (a slow symlink, like docs/long-link, 74 bytes). The VFS follows up to 8 links in a path, then gives ELOOP. stat follows them, rm doesn't, and the new readlink syscall (89) returns the target, which ls -l prints as name -> target.

What the driver does and doesn't do

• Reads:
  • The revision 0 and revision 1 formats, with 1 KiB and 4 KiB blocks (2 KiB should work too, but isn't tested).
  • Direct, single, double and triple indirect blocks, and holes.
  • Fast and slow symlinks.
  • Directories with dir_index hash trees. The index hides inside entries that plain ext2 code skips.
• Writes:
  • Creating files and directories (mode 0644 and 0755), writing, appending, writing past the end (which leaves a hole), truncating to 0, unlink and rmdir.
  • The bitmaps, the free counts and the parent's link count stay consistent, so e2fsck is happy. Changing a hash-tree directory clears its index flag, like Linux's ext2 does.
• Design:
  • No cache: every operation re-reads what it needs, and every write goes straight to the disk.
  • One Mutex per volume.
  • An Ext2Inode is just (volume, inode number, kind).
• Not yet:
  • Creating symlinks, hard links, rename, and chmod/chown.
  • Deleting a file that has an extended-attribute block (gives EOPNOTSUPP).
  • Writes past 2 GiB.
  • Real timestamps: there is no clock yet, so new files get the time the image was made.
  • The backup superblocks are never updated (Linux doesn't either), and there is no journal: a crash in the middle of an operation leaves work for e2fsck, just like real ext2.

The test images (build.rs)

build.rs copies disk2/ (text files, nested directories) to target/ext2-tree/ and adds a few generated files:

• big.txt: 16 000 numbered lines, 304 000 bytes, which reaches the double indirect blocks.
• sparse.bin: 66 MiB with data only at 0, 40 MiB and 65 MiB, so the last one sits in a triple indirect block.
• The two symlinks.

Then mke2fs -t ext2 -d <tree> builds the images without root or mounting:

Image	Size	Block size	Notes	Attached as
target/ext2.img	32 MiB	1 KiB	revision 1, label RUSTBELT, the full tree	vdb -> /mnt/disk1 = C: (run and tests)
target/ext2-4k.img	8 MiB	4 KiB	label FOURK, small tree	vdd -> /mnt/disk3, tests only
target/ext2-rev0.img	4 MiB	1 KiB	revision 0 (-E revision=0), small tree	vde -> /mnt/disk4, tests only

Raw, not partitioned. The FAT disk has a GPT because that's what real disks have. The ext2 image is the bare file system, with no partition table, so the host's e2fsck, debugfs and dumpe2fs work on the file directly. mount_disks mounts a whole disk if it has no partitions, and its partitions if it does. The disks are numbered in device order: vda1 = disk0, vdb = disk1, vdc (exFAT) = disk2.

Checking Rustbelt's writes with the real e2fsck

All QEMU disks normally use snapshot=on, so nothing reaches the image files. scripts/ext2-check.sh does the opposite. It copies the images to target/fsck/ and boots Rustbelt on the writable copies twice:

• once for the shell self-test (EXT2_IMG=... ./scripts/run-headless.sh), which does cp B: -> C:, md and echo > on C:.
• once for the tests/ext2.rs kernel, which creates, writes, truncates and deletes on all three images.

Then it runs e2fsck -fn on every copy (-f = check even if marked clean, -n = read-only, change nothing) and shows the root directory with debugfs:

cargo test --test ext2 && ./scripts/ext2-check.sh

== e2fsck -fn target/fsck/selftest.img (after Rustbelt wrote to it)
e2fsck 1.47.2 (1-Jan-2025)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
RUSTBELT: 25/8192 files (0.0% non-contiguous), 2777/32768 blocks
...
== debugfs: ls -l / on the self-test copy
     23  100644 (1)      0      0     125 30-Oct-2026 01:45 todo.txt
     24   40755 (2)      0      0    1024 30-Oct-2026 01:45 made-here
== debugfs: cat /made-here/note.txt
written by Rustbelt
== all images are clean

Pass 5 is the one that catches allocation bugs: it recounts every bitmap and compares the result with the free counts in the group descriptors and the superblock.

exFAT (stage 9)

The fourth disk, target/exfat.img, holds an exFAT file system: the format on SD cards over 32 GB and most USB sticks, which Windows, macOS, Linux, cameras and phones all understand. Rustbelt reads and writes it with its own driver, src/fs/exfat.rs: about 1000 lines including a long explanation at the top, written from Microsoft's exFAT specification, with no crates. It is mounted at /mnt/disk2, which is drive D::

A:/ $ cd d:
D:/ $ ls -l
       132  Long File Name With Spaces.txt
        84  MixedCase.TXT
     <DIR>  Photos/
        88  a-really-long-file-name-that-takes-several-file-name-directory-entries-in-exfat-xxxxxxxxxxxxxxxxxxxx.txt
     38000  frag-chain.txt
    340000  numbers.txt
       363  readme.txt
        61  Ünïcödé café.txt
D:/ $ cat "long file name with spaces.txt"
This file has a long name with spaces in it. exFAT stores names of up
to 255 UTF-16 characters, 15 per "File Name" directory entry.
D:/ $ cp C:/hello.txt "From Ext2.txt"
D:/ $ type "FROM EXT2.TXT"
Hello from the ext2 disk! This file lives on a second virtio disk, in an
ext2 file system that Rustbelt reads and writes with its own driver.

Names keep the case you typed but are found in any case, like on Windows. Quote names with spaces. Inside quotes the shell doesn't translate drive letters, so cd D: first (or write /mnt/disk2/...). exFAT has no permissions, so ls -l shows sizes only, like on the FAT disk.

The on-disk layout, for a learner

exFAT is FAT's successor, redesigned for big flash cards. If you know FAT32, three things are new: a bitmap says which clusters are free, contiguous files can skip the FAT entirely, and a file's directory entry is a set of 32-byte entries with a checksum.

sector 0      1-8        9-10   11        12-23          FatOffset    ClusterHeapOffset
| boot sector | ext. boot | OEM | checksum | backup of 0-11 | ... | FAT | ... | cluster 2 | cluster 3 | ...

• Boot sector. Says EXFAT at byte 3; that's how mount_disks recognizes it. Then the geometry, all in sectors: where the FAT and the cluster heap start, how many clusters there are, the root directory's first cluster, and the sector and cluster sizes as powers of two (9 = 512 bytes). Byte 112 is PercentInUse, which we keep up to date.
• Boot checksum. Sector 11 is the 32-bit "rotate right, add the next byte" checksum of sectors 0-10, repeated to fill the sector. It skips the three bytes that change while the volume is in use (VolumeFlags and PercentInUse), so updating those doesn't mean rewriting the checksum. We check it at mount.
• FAT. One 32-bit entry per cluster: the next cluster of the file, or 0xFFFFFFFF for the end. Clusters are numbered from 2, and cluster N starts at ClusterHeapOffset + (N - 2) * cluster size.
• Everything else lives in clusters, even the "system files":
  • Allocation bitmap: 1 bit per cluster. This, not the FAT, says what's free. We keep it in memory (960 bytes for our disk) and write each changed byte straight through.
  • Up-case table: the upper-case form of every UTF-16 character. Name lookups compare up-cased names. Because the table is on the disk, every implementation agrees on what "the same name" means, even for é/É. It's stored compressed (0xFFFF, n = "the next n characters map to themselves") and has its own checksum, which we verify. We keep only the characters that change, in a sorted Vec, and binary-search it.
  • Root directory: like any directory, except that it has no entry of its own, so its size is the length of its FAT chain.

Directories are files of 32-byte entries. Byte 0 is the type, and bit 7 of it means "in use". Deleting just clears that bit, and type 0x00 marks the end. The root also holds a volume label entry (0x83), the bitmap's (0x81) and the up-case table's (0x82). A file or directory is an entry set:

0x85 File              SecondaryCount, SetChecksum, attributes (0x10 = directory), 3 timestamps
0xC0 Stream Extension  flags, NameLength, NameHash, ValidDataLength, FirstCluster, DataLength
0xC1 File Name         15 UTF-16 characters  <- repeated: a 104-character name needs 7

• SetChecksum is a 16-bit "rotate right, add" over every byte of the set except itself. Change a file's size and you must recompute it, or other systems treat the file as corrupt.
• NameHash is the same checksum over the up-cased name, so a lookup can skip most entries without comparing names.
• NoFatChain (stream flag 0x02): "my clusters are consecutive; don't read the FAT". Most files are contiguous, so most files never touch the FAT. When a file must grow and the next cluster is taken, the driver writes FAT links for the clusters it already has, clears the flag, and carries on wherever there's room (Volume::grow). Directories work the same way.
• ValidDataLength <= DataLength: bytes past it read as zeros (Windows uses this to create big files quickly). We read it that way and always write the two equal.
• Directories have no . or .. entries, and their size is always whole clusters.

What the driver does and doesn't do

• Reading: contiguous and FAT-chained files, long UTF-16 names, case-insensitive lookup through the volume's own up-case table, nested directories, the volume label, and both cluster sizes we test (4 KiB and 512 bytes). Damaged entry sets (bad SetChecksum) are skipped.
• Writing: create, write, append, truncate to 0, mkdir, unlink and rmdir (ENOTEMPTY if it isn't empty). That includes:
  • allocating from the bitmap, preferring the cluster right after the file's last one;
  • growing directories when a new entry set doesn't fit, the root too;
  • converting NoFatChain files to FAT chains, freeing chains, and recomputing SetChecksums, NameHashes and PercentInUse.
  • Names are checked: 1-255 UTF-16 characters, no control characters or " * / : < > ? \ |. A name that differs only in case is EEXIST.
• Not supported:
  • rename and symlinks (exFAT has no symlinks);
  • real timestamps: there's no clock driver, so everything is 1 Jan 2026;
  • a second FAT (TexFAT): such a volume mounts read-only;
  • the VolumeDirty flag (writes go straight through);
  • shrinking directories (nobody does).
• Like the ext2 driver, there's no block cache: only the bitmap and one FAT sector are cached. An inode is just the path from the root, resolved again on every call, because exFAT has no inode numbers (a file is its directory entry).

The test images, and why there's a populator (build.rs)

mkfs.exfat (exfatprogs) formats an image, but unlike mke2fs -d it can't copy files in, and mtools doesn't speak exFAT. The usual way is to mount the image, but that needs root and a loop device (or FUSE, which doesn't work on this box either). So build/exfat_populate.rs is a small exFAT writer of its own (~230 lines) that build.rs runs after mkfs.exfat. It only handles a freshly formatted volume, which makes it simple: clusters are handed out in order, each file is one contiguous NoFatChain run, and files named frag-* get every other cluster, linked through the FAT, so the kernel has a fragmented file to read. It deliberately shares no code with the kernel driver, so a misreading of the spec in one doesn't hide in the other, and the host's fsck.exfat checks its output on every run of scripts/exfat-check.sh.

The tree is disk3/ (long names with spaces, mixed case, non-ASCII names, a 104-character name, Photos/2026 Trip/trip notes.txt) plus two generated files: numbers.txt (20 000 lines, 340 000 bytes, contiguous) and frag-chain.txt (38 000 bytes, fragmented).

Image	Size	Clusters	Notes	Attached as
target/exfat.img	32 MiB	4 KiB	label TYLEREXFAT, the full tree	vdc -> /mnt/disk2 = D: (run and tests)
target/exfat-512.img	8 MiB	512 bytes	label SMALLCLUST, small tree: every file spans many clusters	vdf -> /mnt/disk5, tests only

Like the ext2 disk, it's a bare file system with no partition table, so fsck.exfat and dump.exfat work on the file directly. An exFAT boot sector also ends in 55 AA, like an MBR, so the partition scanner checks for EXFAT and reports no partition table (WholeDiskExfat) instead of "MBR, 0 partitions".

Checking Rustbelt's writes with the real fsck.exfat

scripts/exfat-check.sh works like ext2-check.sh. It copies the images to target/fsck/ and boots Rustbelt on the writable copies: once for the shell self-test (EXFAT_IMG=... ./scripts/run-headless.sh), and once for the tests/exfat.rs kernel. Its last test, written_files_stay_for_fsck, deletes nothing, so fsck sees the tricky cases on disk: 300 files in a grown directory that became a FAT chain, deleted entries between live ones, and a file converted from NoFatChain to a FAT chain. Then it runs fsck.exfat -n (report, change nothing) on every copy and reads a new file back with dump.exfat and dd:

cargo test --test exfat && ./scripts/exfat-check.sh

== fsck.exfat -n target/exfat.img (pristine, as built)
target/exfat.img: clean. directories 3, files 8
== fsck.exfat -n target/exfat-512.img (pristine, as built)
target/exfat-512.img: clean. directories 3, files 4
13 exFAT tests ok
== fsck.exfat -n target/fsck/selftest-exfat.img (after Rustbelt wrote to it)
target/fsck/selftest-exfat.img: clean. directories 4, files 11
== fsck.exfat -n target/fsck/tests-exfat.img (after Rustbelt wrote to it)
target/fsck/tests-exfat.img: clean. directories 4, files 212
== fsck.exfat -n target/fsck/tests-exfat-512.img (after Rustbelt wrote to it)
target/fsck/tests-exfat-512.img: clean. directories 4, files 32
== dump.exfat -s "/New Folder": the entry set Rustbelt wrote for Note.txt
  Path: /New Folder/Note.txt
  Entrytype:                      0x85
  SetChecksum:                    0xAF76
  Entrytype:                      0xC0
  GeneralSecondaryFlags:          0x03
  NameLength:                     8
  NameHash:                       0x2C4C
  ValidDataLength:                14
  FirstCluster:                   19
  DataLength:                     14
  ...
== its 14 bytes at cluster 19, read with dd:
hi from exFAT
== all images are clean

fsck.exfat recomputes the bitmap from every file's clusters, and checks each SetChecksum and the boot checksum. A flipped bitmap bit or one changed byte in an entry set gives "files corrupted 1". It doesn't check PercentInUse, so the tests check that themselves: their usage() helper reads the bitmap and byte 112 straight off the disk with a few lines of independent parsing.

btrfs (stage 10)

E: is a btrfs volume, read by a hand-written, read-only driver (src/fs/btrfs.rs, about 1 200 lines with comments; src/fs/lzo.rs for LZO). btrfs is what most of Tyler's Linux machines will use, so this is the file system Rustbelt most needs to read. Writing comes much later (see below).

E:/ $ ls -l
drwxr-xr-x      <DIR>  Music/
drwxr-xr-x      <DIR>  archive/            <- a read-only subvolume (id 256)
-rw-r--r--    2160000  big.txt             <- 17 zlib-compressed extents
drwxr-xr-x      <DIR>  docs/
-rw-r--r--          0  empty.txt
lrwxrwxrwx             hello-link -> hello.txt
-rw-r--r--         78  hello.txt           <- an inline extent, inside a leaf
drwxr-xr-x      <DIR>  projects/           <- a subvolume (id 257)
-rw-r--r--    1500000  random.bin          <- incompressible: stored as is
-rw-r--r--        637  readme.txt
-rw-r--r--    1048576  zeros.bin
E:/ $ echo hi > new.txt
sh: new.txt: Read-only file system

The ideas, for a learner

Everything is a B-tree item. ext2 and exFAT have fixed tables (inode tables, a FAT, bitmaps) at fixed places. btrfs has almost none: just the superblock at 64 KiB (magic _BHRfS_M). Everything else is an item in a B-tree, sorted by a 17-byte key (objectid, type, offset). A file is a handful of items that sit next to each other because they share an objectid (the inode number):

(257 INODE_ITEM 0)         size, mode, link count, times, flags
(257 INODE_REF 256)        "my name in directory 256 is docs"
(258 DIR_ITEM <hash>)      a directory entry, found by crc32c(name)
(258 DIR_INDEX <n>)        the same entry again, in creation order (readdir)
(259 EXTENT_DATA 0)        file bytes 0.. : inline data, or "see disk extent X"
(259 EXTENT_DATA 131072)   file bytes 131072.. : the next extent

Leaves hold items; internal nodes hold (key, child pointer) pairs. Looking something up is "walk down from the root to the last key <= the one you want" (Volume::search_le); listing a directory or a file's extents is a range walk (Volume::range). lookup uses DIR_ITEM (one search by name hash), and ls uses DIR_INDEX (a range, in creation order).

Copy-on-write B-trees. btrfs never overwrites a live block. To change a leaf it writes a new copy elsewhere, then a new copy of the parent pointing to it, and so on up to the root; the superblock is written last and switches everything over at once. Until then the old tree is still complete, so a crash just leaves the old version. The same trick makes snapshots nearly free: a snapshot is a second root pointing at the same nodes. For a reader it means every node is immutable once written, and each one records the transaction (generation) that wrote it.

Chunks: logical addresses. Tree pointers and extent addresses are logical byte numbers, not disk offsets. The chunk tree maps logical ranges to (device, physical offset) "stripes". With single there is one stripe; with DUP (the default for metadata on one disk) there are two copies on the same disk, and the driver reads the second when the first fails its checksum. RAID profiles would spread stripes over several devices; the driver refuses those with a message. But the chunk tree is itself stored at logical addresses, so how do you read it? The superblock carries a small bootstrap copy of the SYSTEM chunks (sys_chunk_array), just enough to find the chunk tree. Open order: superblock, then sys_chunk_array, then chunk tree, then root tree, then FS tree.

Trees of trees. The root tree lists every other tree with a ROOT_ITEM (where its root node is, its level). Tree 5 is the top-level FS tree (files and directories); tree 7 is the csum tree (checksums of data blocks); others (extent tree, device tree...) only matter for writing. The "default" entry in the root tree's directory 6 says which subvolume to show; the driver honours it.

Subvolumes are separate FS trees (ids 256 and up) with their own inode numbers, so (subvolume, inode) names a file (Loc in the driver). A directory entry that points at a subvolume has a ROOT_ITEM key instead of an INODE_ITEM key; following it moves to that tree's top directory. The driver lists them from the root tree's ROOT_REF items with their read-only flag (BtrfsFs::subvolumes; on E: archive is 256, read-only, and projects is 257), and cd E:/projects just walks into one. Snapshots are subvolumes too, so they would work the same way.

Checksums everywhere. Every node starts with a checksum of the rest of the node, plus its own logical address and the file system's id, so a misplaced or stale block is caught as well as a corrupt one. Data blocks are checksummed per sector in the csum tree, keyed by logical address. The driver checks both (crc32c: the Castagnoli CRC, a few lines in software); a bad block is EIO with a log line, and with DUP it tries the other copy first. Newer btrfs can use xxhash64, sha256 or blake2b instead; those are refused at mount time: btrfs, but can't mount it: checksum type 2 (sha256) is not supported, only crc32c. Directory names are found by crc32c(name) too, with seed ~1 and no final inversion (the "default" hash 2378154706 is a unit test).

Extents. A file's data comes as EXTENT_DATA items, each covering a range of the file:

Kind	What it means	Driver
inline	the bytes are inside the item, in the leaf (small files, symlink targets)	copied, or decompressed
regular	disk_bytenr + offset + num_bytes: a slice of an on-disk extent, which may be shared with other files or snapshots	read + checksummed
compressed	a regular or inline extent with zlib, lzo or zstd: the whole extent is decompressed, then sliced	miniz_oxide, src/fs/lzo.rs, ruzstd
prealloc	reserved by fallocate, never written	zeros
hole	disk_bytenr 0, or simply no item (the no_holes feature)	zeros

btrfs's lzo is not plain LZO: it's a 4-byte total length, then per segment a 4-byte length plus an LZO1X block, and a segment header never crosses a 4 KiB sector boundary (padding instead). zstd extents can be followed by zero padding, so the driver decodes frame by frame instead of "all input".

Why read-only first, and what writing would take

Reading btrfs only needs the trees as they are on disk. Writing needs everything copy-on-write implies:

• Allocation: the extent tree and free-space tree say which logical ranges are used, with reference counts (an extent can be shared by files, snapshots and reflinks). Every new node or data block needs a free range, and every freed one a decrement.
• COW updates: changing one item means new copies of the leaf and of every node up to the root, in each affected tree (FS tree, extent tree, csum tree, root tree...), plus splitting and merging nodes to stay balanced.
• Transactions: batch all of that, write it, flush, then write the superblock (all copies) with the new generation, in the right order for crash safety. Also the log tree for fsync.
• Checksums and compression on the way out, inode/dir item updates, back-references (INODE_REF, ROOT_BACKREF...).

That's a few thousand careful lines where a bug corrupts the volume, versus about a thousand for reading. Read-only first gets Tyler's disks readable, and it builds the parts writing needs anyway (key order, tree walking, chunk mapping, checksums). A first write step could be overwriting existing data blocks in place on a NODATACOW file.

What the driver does and doesn't do

• Does: superblock (crc32c, feature flags, single device), sys_chunk_array
  • chunk tree with single and DUP, root tree, the default subvolume, FS and csum trees; node checks (checksum, bytenr, fsid, level); inode items, INODE_REF, DIR_ITEM (hash lookup), DIR_INDEX (readdir); inline, regular, prealloc and hole extents, zlib/lzo/zstd, shared extents (slices into a bigger extent, as btrfs-convert makes them); data checksums; symlinks; subvolumes (listed, entered). A 256 KiB node cache and a one-extent decompression cache keep cat big.txt from re-reading and re-inflating.
• Doesn't: write anything (EROFS for write, truncate, create, mkdir, rm); RAID profiles or multiple devices; checksum types other than crc32c; encryption; extended attributes, hard-link counts and timestamps aren't shown; the extent tree isn't read (not needed to read files).

The test images (build.rs)

build.rs copies disk4/ to a staging tree, adds big.txt (120 000 lines, 2.16 MB), random.bin (1.5 MB of noise), zeros.bin, empty.txt and three symlinks, and makes 128 MiB images with mkfs.btrfs --rootdir:

Image	Options	Why
btrfs.img (E:)	--compress zlib -u projects -u ro:archive, defaults: 16 KiB nodes, metadata DUP	the main one
btrfs-lzo.img	--compress lzo -n 4096 -m single	lzo; small nodes make a 2-level FS tree
btrfs-zstd.img	--compress zstd -n 65536	zstd; 64 KiB nodes
btrfs-convert.img	mke2fs -d (4 KiB blocks), then btrfs-convert	real holes and shared extents, an ext2_saved subvolume
btrfs-sha256.img	--csum sha256	must be refused

mkfs.btrfs --rootdir never makes holes or prealloc extents (sparse files are stored as data), so holes come from the converted image and prealloc from a hand-made item in a unit test. Each tree gets a manifest (target/btrfs-tree.manifest, btrfs-convert.manifest): one line per directory, file (size + CRC-32) and symlink (target).

Checking against the host

./scripts/btrfs-check.sh (after cargo test --test btrfs):

== btrfs check --readonly target/btrfs.img
  found 2056192 bytes used, no error found
... (all five images)
== run the tests/btrfs.rs kernel
  11 btrfs tests ok
== compare what Rustbelt read with the host's files (zlib.crc32)
  /mnt/disk3: 22 entries (12 files, 4710275 bytes) match
  /mnt/disk7: 22 entries (12 files, 4710275 bytes) match
  /mnt/disk8: 22 entries (12 files, 4710275 bytes) match
  /mnt/disk9: 7 entries (4 files, 5442941 bytes) match
== all good

The test kernel itself already compares every file with the manifests; the script checks again independently, from the host's files with Python's zlib.crc32, using the [btrfs-file] lines the kernel prints.

Networking (stage 12)

Rustbelt can now talk to the world: get an address by DHCP, ping, look up names, download web pages and serve them. The card is QEMU's virtio-net, the protocols come from the smoltcp crate, and user programs see BSD sockets as file descriptors.

Packets and layers, for a learner

A network moves packets: chunks of at most about 1500 bytes. Every bit of data, from a ping to a 100 MB download, gets cut into packets, and each one travels on its own. A packet is built like a set of envelopes inside each other. Each layer wraps the one inside it with its own header, and each header answers one question:

+-------------------+-------------+--------------+---------------------+
| Ethernet header   | IPv4 header | TCP header   | data: "GET / HTTP.."|
| to which card?    | to which    | to which     |                     |
| (MAC 52:54:00:..) | computer?   | program?     |                     |
|                   | (10.0.2.2)  | (port 80)    |                     |
+-------------------+-------------+--------------+---------------------+
   14 bytes            20 bytes      20 bytes

• Ethernet (link layer) delivers a frame to a network card on the same wire, by its 6-byte MAC address. That's all the card itself understands. Our virtio-net driver just moves whole frames in and out.
• ARP answers "which MAC has IP 10.0.2.2?" To send to an IP address on the local network, the kernel broadcasts an ARP question to every card and caches the answer.
• IPv4 (network layer) delivers a packet to a computer by its 4-byte IP address, across many networks. Each hop is a router. If the address isn't on our own network (10.0.2.0/24 = the first 24 bits match), the packet goes to the gateway (10.0.2.2), which passes it on.
• ICMP is IP's own control messages. ping sends an echo request and times the echo reply.
• UDP delivers a datagram to a port, i.e. a program, on that computer. It's one packet, which may get lost, with no connection. DNS (names to addresses), DHCP (getting an address at all) and TFTP run over UDP.
• TCP turns lossy packets into a reliable byte stream between two programs. It numbers every byte, resends what isn't acknowledged, keeps things in order, and slows down when the network is busy. A connection starts with a three-way handshake (SYN, SYN-ACK, ACK) and ends with FINs. HTTP runs over TCP. A web request is literally the text GET /path HTTP/1.0 + headers + an empty line, and the answer is a status line, headers, an empty line, then the file.

How we get an address: at boot nobody knows us, so the kernel broadcasts a DHCP "discover". QEMU's built-in DHCP server offers 10.0.2.15/24 with gateway 10.0.2.2 and DNS server 10.0.2.3. The kernel requests it, the server acknowledges, and the lease is ours ([net] DHCP lease: ...).

Why smoltcp

Writing ARP, IP and ICMP by hand is a fun weekend. TCP is not: getting retransmission timers, windows, the state machine (11 states), out-of-order segments and congestion control right takes months, and a subtly wrong TCP mostly seems to work. smoltcp is a TCP/IP stack written for exactly this situation:

• no_std: it needs only alloc, no OS. It never allocates on its own and never spawns threads; it doesn't even read a clock. We hand it Instants.
• It's driven by us. We call iface.poll(now, device, sockets). It reads the frames that arrived, updates every socket, and sends whatever is due (data, ACKs, retransmissions, ARP, DHCP). poll_delay says when it next needs a poll (e.g. a retransmit timer).
• It talks to the card through a two-method Device trait (receive, transmit), which src/net/virtio_net.rs implements in about 40 lines.
• It's small and readable, and used in real embedded and OS projects (Redox, Hermit, ESP32 firmware).

The same thinking as using fatfs for FAT32: write by hand what teaches something (the virtio driver, the socket layer, ext2/exFAT/btrfs), and use a good crate where the hard part is decades of edge cases.

The virtio-net driver (src/net/virtio_net.rs)

It uses the same legacy virtio interface as virtio-blk. PCI finds vendor 1af4 device 1000; we reset it, accept only the MAC feature, and set up two virtqueues:

• RX (queue 0): we give the card 32 empty 2 KiB buffers up front. When a frame arrives the card fills one, puts it on the used ring, and raises an interrupt. We copy the frame out and hand the buffer straight back.
• TX (queue 1): to send, we copy the frame into a free buffer behind a 10-byte virtio-net header (all zeros: no checksum offload) and notify the card. Finished buffers are collected the next time we send; this queue has interrupts turned off because nobody needs to wait for them.

Interrupts. The card's PCI interrupt line (config byte 0x3C) is 11 on QEMU's PC. It arrives through the old 8259 PIC, which still feeds CPU 0's local APIC (LINT0, "ExtINT"). The handler reads the card's ISR register (which also acknowledges it) and wakes the netd kernel thread. netd calls smoltcp's poll, then sleeps on a wait queue until the next interrupt or until poll_delay says a timer is due. No busy polling: an idle network costs no CPU.

Sockets (src/net/socket.rs, syscalls 41-54)

User programs use the BSD socket API that every OS copied from 4.2BSD in 1983:

// user/fetch.rs, roughly (ulib returns a negative errno on failure)
let fd = socket(SOCK_STREAM, 0);                   // a TCP socket, as an fd (AF_INET is implied)
connect(fd, ip, 80);                               // handshake (blocks)
write(fd, b"GET / HTTP/1.0\r\n\r\n");               // send bytes
let n = read(fd, &mut buf);                        // receive (blocks); 0 = the server closed
close(fd);                                         // FIN; the kernel finishes the goodbye
// a server: socket, bind(port), listen(backlog), then accept() per client

Each socket is a File in the fd table, so read/write/close and the shell's redirections work on it like on a pipe. Inside, it holds a smoltcp socket handle. Everything shared (interface, card, all sockets) sits in one NET lock, taken with interrupts off.

Blocking reuses stage 11's wait queues. A recv with nothing to read sleeps on the CHANGED queue, which poll wakes whenever packets have been processed. Then it looks again. SO_RCVTIMEO turns into a deadline on the same wait (wait_until_deadline), and EAGAIN when it passes, so ping can give up on a lost reply. A TCP connect sleeps until the handshake is done (or ECONNREFUSED / ETIMEDOUT after 10 s).

Closing TCP is a conversation, not an instant (FIN, ACK, maybe a last bit of data). close starts it, and the socket stays in a "closing" list that poll reaps once smoltcp says it's done.

Two calls are Rustbelt's own: resolve (Linux does DNS inside libc, by sending UDP packets to the server in /etc/resolv.conf; we have smoltcp's DNS socket, so the kernel does it) and netinfo (instead of Linux's ioctls and netlink). clock_gettime came along for ping's millisecond times: it reads the TSC, calibrated against the PIT at boot (src/time.rs).

QEMU's user-mode network, and the test servers

-netdev user (slirp) is a whole tiny network inside QEMU: DHCP at 10.0.2.2, DNS at 10.0.2.3, and NAT through the host's ordinary sockets, so nothing on the host needs root, a bridge or a TAP device. That makes it perfect for a headless box and for tests. The test setup uses three of its extras:

What	Where	Used by
guestfwd=tcp:10.0.2.100:80-cmd:sh scripts/http-reply.sh	a fake web server: each connection runs the script, with the socket as its stdin/stdout. / = a greeting, /big = seq 1 20000 (108,894 bytes)	fetch, the self-test, tests/net.rs
guestfwd=tcp:10.0.2.100:7-cmd:cat	an echo server (cat copies its stdin to its stdout)	tests/net.rs
tftp=disk (tests only)	slirp's TFTP server at 10.0.2.2:69, serving disk/	the UDP test
hostfwd=tcp:127.0.0.1:5580-:80 (cargo run; run-headless picks a free port)	host port -> guest port 80	httpd, reached with curl

Because those servers live inside QEMU and on the box itself, the network tests pass with no internet at all. Only host example.com needs the host's DNS, and the tests skip it when that's missing.

What it does and doesn't do

Does: DHCP, ARP, IPv4, ICMP echo, UDP, TCP client and server (up to 4 waiting connections per listener), DNS A records, blocking with timeouts, and an interrupt-driven RX path. Doesn't (yet): IPv6, non-blocking sockets / poll/select, shutdown, getsockname, checksum offload or multi-queue, more than one card, a loopback interface (127.0.0.1), or TLS (so fetch is plain http:// only).

Virtual drivers (stage 13)

Stage 13 adds the last three virtual devices Rustbelt wants from QEMU, with no physical hardware involved:

• virtio-rng (src/virtio/rng.rs): random bytes from the host. They feed the getrandom syscall (Linux number 318), /dev/random and /dev/urandom, the random program, smoltcp's random seed (which it uses for things like TCP initial sequence numbers) and the first local TCP/UDP port, which used to be a constant 49152.
• virtio-console (src/virtio/console.rs): a second terminal, hvc0, like Linux's /dev/hvc0 in a VM. The kernel log is mirrored to it, and getty /dev/hvc0 runs a shell on it. On the host it's a QEMU chardev: a UNIX socket for cargo run, files for the tests.
• virtio-gpu, 2D (src/virtio/gpu.rs): a 1024x768 framebuffer. The text console moves onto it (src/fbcon.rs, with a built-in bitmap font, src/font.rs), and user programs get /dev/fb0 (src/dev.rs). Without a virtio-gpu, the VGA text console carries on as before.

A:/ $ ls -l /dev
crw-rw-rw-      <DEV>  fb0
crw-rw-rw-      <DEV>  hvc0
crw-rw-rw-      <DEV>  null
crw-rw-rw-      <DEV>  random
crw-rw-rw-      <DEV>  urandom
A:/ $ random
82461427e1b4012e64cd289b0db180fe
A:/ $ random -d 6 5
6 1 2 6 1
A:/ $ tile
tile: 4 windows on 1024x768, console 61x46

What virtio is, for a learner

Real hardware is complicated because it's real: a network chip has registers, timing rules and errata. An emulator like QEMU can copy such a chip register by register (that's how it emulates the e1000 network card or the old VGA card), but it's slow and both sides get complicated. Virtio is the alternative: device types designed for virtual machines, where guest and host just agree on a simple protocol in shared memory. The same drivers work in QEMU, KVM, cloud VMs and Firecracker.

Every virtio device has the same three parts:

1. Feature bits. The device offers a set of features and the driver accepts the ones it understands. Rustbelt only ever accepts VERSION_1 (bit 32: "I speak virtio 1.0"), which the modern interface requires.
2. Virtqueues. Ring buffers in guest memory (src/virtio/queue.rs). A descriptor table lists buffers (physical address, length, and whether the device writes it). The driver puts the index of a request on the available ring and writes the queue number to a notify register (the "doorbell"). The device does the work, then puts the same index on the used ring together with how many bytes it wrote, and maybe raises an interrupt. The driver either sleeps until that interrupt or polls the used ring. That's the whole protocol: what a request means is up to the device type.
3. Config space. A few device-specific registers: the GPU's number of displays, the console's size, a network card's MAC address.

The three drivers show three ways of using queues:

Device	Queues	A request is	Waiting
rng	1	one empty buffer; the device fills it with random bytes	polled (it answers at once)
console	2 (receive, transmit)	receive: 8 empty 512-byte buffers always posted; transmit: bytes to print	receive: interrupt (input can come at any time); transmit: polled
gpu	2 (control, cursor; only control used)	a command (24-byte header + arguments) plus a buffer for the response	polled

Legacy and modern. The virtio-blk and virtio-net drivers from stages 6 and 12 speak the legacy (pre-1.0) interface: every register at a fixed offset in one I/O-port BAR. That's simpler, but virtio-gpu doesn't support it. The modern interface (src/virtio/mod.rs) puts register blocks (common config, notify, interrupt status, device config) in memory BARs. PCI capabilities (a linked list in PCI config space, vendor id 0x09) say which BAR and offset each block is at. Transport::new walks that list, maps the BARs uncached (memory::map_mmio) and does the start-up dance: reset, ACKNOWLEDGE, DRIVER, features, FEATURES_OK, set up the queues, DRIVER_OK. All three new drivers share it. The rng and console cards are "transitional" (they'd speak both), so the modern code could replace the legacy one in virtio-blk and virtio-net later.

Interrupts. The console's receive queue uses the card's PCI interrupt line, which it shares with the network card (IRQ 10 here). PCI interrupts are level-triggered and can be shared, so interrupts::pci_irq asks every driver "was it you?" (net::handle_irq, then virtio::handle_irq). Reading a device's interrupt-status register says yes or no and lowers its line. A driver that forgot that read would leave the line high, and the CPU would take the same interrupt forever.

Framebuffers and scanout

A framebuffer is memory with one number per pixel: here a 32-bit 0x00RRGGBB for each of 1024 x 768 pixels, row after row (3 MiB). Drawing is just writing to memory. Scanout is the display hardware reading that memory 60 times a second and sending it to the monitor. A real GPU scans out straight from video memory. virtio-gpu adds a step, because the image the host shows lives on the host:

  guest RAM (our framebuffer)          host (QEMU)                 window
  +-------------------------+  TRANSFER_TO_HOST_2D  +-----------+  RESOURCE_FLUSH
  | 0x00RRGGBB pixels       | -------------------> | resource 1 | --------------> screen
  +-------------------------+   (copy a rectangle)  +-----------+   (show it)
         ^ ATTACH_BACKING ties these pages to resource 1; SET_SCANOUT shows resource 1

So setting up (gpu::init) is: GET_DISPLAY_INFO (QEMU prefers 1280x800; we ask for 1024x768 and QEMU scales the window), RESOURCE_CREATE_2D (an image with id 1, format B8G8R8X8, which makes a u32 read as 0x00RRGGBB), RESOURCE_ATTACH_BACKING (its pixels are these 768 contiguous guest pages) and SET_SCANOUT (show resource 1 on display 0). After drawing, gpu::flush(x, y, w, h) sends TRANSFER_TO_HOST_2D and RESOURCE_FLUSH together. Nothing reaches the screen without a flush, so the console keeps a dirty range: the pixel rows it changed since the last flush. It flushes once per print!, not per character.

The framebuffer console (src/fbcon.rs) keeps the old console's model: a grid of cells, each a character and a VGA colour attribute, and the same 16-colour palette. Drawing a cell copies a glyph from src/font.rs (8x16 pixels, one byte per row, one bit per pixel) into the framebuffer. 1024x768 gives 128x48 characters instead of VGA's 80x25. Scrolling moves the pixel rows up with one memmove (row by row when the console is in a viewport narrower than the screen). The font is Spleen (BSD-2): ASCII and Latin-1, so é now shows up instead of a ■, plus box-drawing characters. The console can be confined to a viewport (FB_SET_CONSOLE): that's how tile puts it in a window.

/dev/fb0 for programs. Linux programs mmap /dev/fb0 and draw into it directly. Rustbelt has no mmap yet (the next stage), so /dev/fb0 is a file you lseek + write pixels into, plus ioctls for what doesn't fit in a write:

ioctl	arg	does
FB_GET_INFO (0x4600)	*[u32; 4]	width, height, bytes per row, bits per pixel
FB_FLUSH (0x46f0)	*[x, y, w, h], or 0	show a rectangle (0: the rows this fd wrote since the last flush)
FB_FILL (0x46f1)	*[x, y, w, h, color]	fill a rectangle in the kernel (much faster than writing it)
FB_SET_CONSOLE (0x46f2)	*[x, y, w, h], or 0	move the text console into the rectangle (0: the whole screen); returns cols << 16 | rows

user/tile.rs uses all of them. It writes a gradient wallpaper 8 rows at a time, splits the screen "dwindle"-style (each window takes half of what's left, alternating left/right and top/bottom), fills borders and windows with FB_FILL, flushes, then moves the console into the first window. That's a first sketch of the tiling compositor on the roadmap; the compositor itself will need mmap (to draw without a syscall per rectangle), input events and windows owned by other processes.

Why 2D first, and what 3D would take

2D is all a console, a tiling compositor and a software renderer need: the CPU draws the pixels, the GPU just shows them. It's six commands, and every later step builds on it, because even 3D output ends with SET_SCANOUT and RESOURCE_FLUSH.

3D through virgl is a different scale of project. The guest doesn't send pixels but rendering commands. virtio-gpu adds 3D contexts, 3D resources (textures, buffers) and SUBMIT_3D, which carries a command stream in Gallium's format. On the host, virglrenderer turns that stream into OpenGL calls on the real GPU. In the guest, the stream comes from Mesa, the open-source OpenGL/Vulkan implementation, through its virgl driver. So Rustbelt would need to run Mesa: that means a C library (the musl port), mmap, threads in user space, a way to load shared libraries or a big static build, and kernel support for 3D resources and fences. Newer QEMU versions also offer Venus (Vulkan over virtio-gpu) and virtio-gpu rutabaga/gfxstream; they need the same user-space groundwork. For now, 2D and software rendering are plenty for a tiling compositor.

The second console, and testing it headless

virtconsole is a port on a virtio-serial-pci card. Rustbelt doesn't negotiate the multi-port feature, so there's exactly one port, port 0, which needs no control messages. Bytes the host sends land in the receive buffers; the interrupt handler copies them into a 4 KiB queue and wakes any reader. read on /dev/hvc0 blocks until there's input. Writes go straight to the transmit queue. While a program has /dev/hvc0 open, the kernel stops mirroring its log there, so a shell on hvc0 isn't interrupted by [process] lines.

QEMU decides where the other end goes: -chardev socket,...,server=on for cargo run (connect with socat), and -chardev file,path=OUT,input-path=IN for tests. The file chardev sends all of IN once at start-up and writes everything the guest sends into OUT, so a headless run can feed hvc0 a script and check the reply afterwards. run-headless.sh feeds scripts/hvc0-session.txt to getty -1 /dev/hvc0 and checks target/headless-hvc0.txt; tests/virtio.rs checks the exact bytes of tests/hvc0-input.txt arrive.

Testing the GPU

Inside the guest, tests/virtio.rs checks the protocol (an error comes back for a bad resource id) and reads pixels back from the framebuffer. That only proves the guest's memory is right, though. To check what QEMU actually displays, scripts/gpu-check.sh boots the kernel, types tile through the QEMU monitor, saves the screen with the monitor's screendump command as a PPM file (a tiny header, then raw RGB bytes, so a few lines of Python can read it) and compares 12 pixels with the colours tile.rs draws: the wallpaper gradient at three heights, the gap between windows, the cyan and gray borders, three window fills, a "text" bar, and the console window's black background with text on it.

What it does and doesn't do

• virtio-rng: 4 KiB per request, a 1 s timeout; if the device is missing or silent, a TSC-seeded xorshift64* takes over (fine for port numbers, not for cryptography).
• virtio-console: one port, no resize events, no multiport; TX waits at most 10 ms for a free buffer and counts what it drops. The shell on it isn't interactive (no line editing); the host terminal does the echo.
• virtio-gpu: one display, one resource, a fixed 1024x768 (the host's preferred size is read and logged, not used), no hardware cursor, no resize, no 3D. Flushes are synchronous (polling for each answer).
• /dev/fb0: write/lseek/ioctls; mmap comes with the next stage.
• Every new user program's heap is still 64 KiB, so tile writes the wallpaper in 32 KiB pieces.

Processes: fork, exec, memory maps and signals (stage 14)

Until stage 13 the kernel had one way to start a program, spawn, and one way to stop it: exit, or be killed for a bad memory access. Stage 14 adds the classic Unix process model, with Linux's own syscall numbers and structs, so a C library such as musl could later run on it unchanged:

• fork (57), vfork (58) and fork-like clone (56): copy a running process, using copy-on-write.
• execve (59): replace a process's program with another. It sets up argv, envp and the auxiliary vector exactly like Linux.
• mmap/munmap/mprotect (9/11/10) and brk (12): memory regions created on request and filled lazily (demand paging).
• Signals: rt_sigaction, rt_sigprocmask, rt_sigreturn, kill, pause, with real signal frames on the user stack and default actions. A bad memory access is now a SIGSEGV that a program can catch. Children that exit send SIGCHLD, a broken pipe sends SIGPIPE, and Ctrl+C sends SIGINT.
• wait4 (61) with Linux's status encoding, getppid, exit_group, dup/dup2/dup3, pipe2, fcntl, close-on-exec, and process groups plus a terminal "foreground group" for job control.

The shell now starts every program with fork + exec. Ctrl+C interrupts the foreground program. kill, fg and bg are builtins, and Ctrl+Z stops a program.

A:/ $ spin 1000 > /tmp/hog.txt
^C
[exit code 130] (interrupted: SIGINT)
A:/ $ start spin 1000 > /tmp/bg.txt
[pid 5] spin started in the background
A:/ $ jobs
[1] [pid 5] running  start spin 1000 > /tmp/bg.txt
A:/ $ kill %1
A:/ $ wait
[done] pid 5 (start spin 1000 > /tmp/bg.txt), exit code 143
A:/ $ run overflow
[overflow] pid 6: recursing until the stack runs out...
[process] pid 6 (overflow) killed: stack overflow (SIGSEGV)
[exit code 139] (killed by SIGSEGV)
A:/ $ kill -l
 1) SIGHUP      2) SIGINT      3) SIGQUIT     4) SIGILL      5) SIGTRAP
...

(130 = 128 + 2, 143 = 128 + 15 (SIGTERM), 139 = 128 + 11 (SIGSEGV): a shell reports "killed by signal N" as exit code 128 + N.)

fork and exec, and why Unix splits them

A process is a running program: its memory (the address space), its registers, its open files, its current directory. Unix creates processes in two steps:

1. fork() makes an almost exact copy of the calling process. Both copies return from the same fork() call: the parent gets the child's pid, the child gets 0. That return value is the only difference.
2. execve(path, argv, envp) throws away the calling process's program and memory and loads a new program in their place. The pid, the open files, the current directory and the parent stay the same.

A shell running ls -l > out.txt does:

pid = fork()
if pid == 0:                       # the child (still running the shell's code)
    fd = open("out.txt", O_CREAT|O_TRUNC|O_WRONLY)
    dup2(fd, 1)                    # make stdout point at the file
    close(fd)
    execve("/bin/ls", ["ls", "-l"], environ)   # becomes ls; never returns on success
    exit(127)                      # only reached if execve failed
else:                              # the parent: the shell
    wait4(pid, &status, 0, 0)      # sleep until ls is done

Why two steps? The gap between fork and exec is the point. The child is still running the shell's code, so it can set up the new program's world with the ordinary system calls: redirect fds with dup2, change directory, join a process group, reset signal handlers. execve then keeps all of that. Without the split, every one of those options would need a parameter of one big "create process" call. Windows' CreateProcess has ten parameters, and our old spawn had its SpawnOptions struct for exactly this reason. The cost is that fork seems to copy a whole address space only for exec to throw it away a moment later, which is why the next section matters. (posix_spawn and vfork exist for the same reason. Rustbelt keeps spawn (500) as a convenience. The kernel itself uses it to start the first shell, and getty uses it.)

What execve hands the new program. Linux starts a program with everything on its stack, and Rustbelt now does exactly the same:

            (higher addresses)
            "PATH=/bin:..."  "ls"  "-l"   16 random bytes   ... the strings themselves
            AT_NULL 0                                         auxiliary vector:
            AT_RANDOM -> the 16 bytes                         (key, value) pairs that tell
            AT_ENTRY, AT_PHDR, AT_PHNUM, AT_PAGESZ, ...       the C library about the program
            NULL
            envp[0] -> "PATH=/bin:..."                        environment pointers
            NULL
            argv[1] -> "-l"
            argv[0] -> "ls"
  rsp ->    argc = 2

The auxiliary vector is how the kernel tells the C library things only the kernel knows: where the program's headers are (AT_PHDR, so musl can find its thread-local storage template), the page size (AT_PAGESZ), the entry point (AT_ENTRY), 16 random bytes for stack canaries and hash seeds (AT_RANDOM, from virtio-rng), the path it was run as (AT_EXECFN), and the user and group ids. user/ulib.rs's _start now reads this layout (arg(i), getenv, getauxval), and forktest args checks every field.

Copy-on-write

Copying all of a process's memory on fork would be slow and usually wasted (an exec follows). Instead, fork copies only the page tables. The child's table points at the same physical frames as the parent's, and every writable private page is made read-only in both:

  parent page table          physical frames          child page table
  0x...1000  R  ----------->  [frame 42, refs 2] <-----------  0x...1000  R
  0x...2000  R  ----------->  [frame 77, refs 2] <-----------  0x...2000  R

Reading works as before. The first write to such a page causes a page fault. The fault handler (mm::Mm::handle_fault) looks at the region the address belongs to: it is writable, so the fault is "copy-on-write", not a bug. If the frame's reference count is above 1, the handler copies the page into a fresh frame, maps the copy writable for the writer and drops one reference. If the count is 1, the other process has already made its own copy (or exited), so the writer gets write access back with no copy at all. Code pages are never written, so they stay shared forever.

The reference counts live in memory.rs (a map from frame to count, kept only for frames that are shared). A frame goes back to the allocator when its last reference goes away. The leak tests fork, exec and exit hundreds of times and check that the number of frames in use and the number of shared frames return to where they started.

One subtle consequence: the kernel also writes into user pages (a read() copying file data into a buffer, for example). If that buffer is a copy-on-write page, the kernel would write into the frame the parent still uses. Since stage 14 the CPU's CR0.WP bit is on, so even ring 0 faults on read-only pages, and every syscall first runs the buffer through the same fault logic (mm::user_access_ok). The pages are copied before the kernel touches them.

Virtual memory maps: mmap, brk and demand paging

Every process sees the same addresses (0x80_0000_0000 and up), but each has its own page tables, so the same address leads to different physical frames in different processes. That is virtual memory. Stage 14 describes a process's memory as a list of regions (Linux calls them VMAs) instead of a list of mapped pages:

0x0000_0000_0040_0000   (stage 20) a plain `gcc -static` program goes here instead
0x0000_0080_0040_0000   program: .text R-X, .rodata R--, .data/.bss RW-     (loaded at exec)
                        heap: from the end of the program up to brk       (grows with brk)
                        ...  free  ...
                        mmap regions, handed out downwards from MMAP_TOP   (mmap)
0x0000_00ff_ffee_f000   64 KiB guard zone (never mapped: overflow -> SIGSEGV)
0x0000_00ff_ffef_f000   user stack, 1 MiB, pages appear as it grows
0x0000_00ff_ffff_f000   one unmapped page, then USER_END

A region is a promise, not memory. mmap(64 MiB) just records the range. The first touch of a page in it causes a page fault, and the handler finds the region, allocates a zeroed frame (or reads the file's bytes into it), maps it, and lets the instruction run again. This is demand paging, and mmaptest checks it: a 64 MiB mapping costs no memory until pages are touched, then about one frame per touched page. Programs never notice any of this.

What's supported:

• MAP_PRIVATE | MAP_ANONYMOUS: plain memory (what malloc uses for big blocks). It is copy-on-write across fork.
• MAP_SHARED | MAP_ANONYMOUS: memory that stays shared after fork. Parent and child see each other's writes (cowtest checks this). Its frames are allocated at mmap time.
• MAP_PRIVATE of a file: the file's bytes are read in page by page on first touch, with zeros past the end of the file. Writes (after mprotect) go to a private copy and never reach the file. MAP_SHARED of a file returns ENODEV, an honest error rather than silently lost writes.
• MAP_FIXED, munmap of any page range (it can split a region), mprotect to PROT_NONE/READ/WRITE/EXEC, and brk to grow and shrink the heap. user/ulib.rs's allocator now gets its memory from brk, so programs aren't stuck with stage 13's fixed 64 KiB heap.

An access outside every region, or one the region forbids (writing a read-only page, reading PROT_NONE), is a real error. The process gets SIGSEGV, with si_addr set to the faulting address and si_code SEGV_MAPERR (nothing mapped there) or SEGV_ACCERR (not allowed).

Signals

A signal is a small number delivered to a process to say "something happened": SIGINT (2) when you press Ctrl+C, SIGSEGV (11) for a bad memory access, SIGCHLD (17) when a child exits, SIGPIPE (13) when it writes to a pipe nobody reads, SIGTERM (15) for a polite "please exit", SIGKILL (9) for a final one. For each signal a process chooses (rt_sigaction) between:

• the default action: terminate (most), terminate "with a core dump" (SIGQUIT, SIGSEGV... We write no core files, so WCOREDUMP stays 0, as on Linux with core dumps turned off), ignore (SIGCHLD, SIGURG, SIGWINCH), stop (SIGSTOP, SIGTSTP from Ctrl+Z) or continue (SIGCONT);
• ignore it (SIG_IGN);
• a handler: a function in the program that runs when the signal arrives, interrupting whatever the program was doing.

A process can also block signals for a while (rt_sigprocmask). They then stay pending until it unblocks them. SIGKILL and SIGSTOP can't be caught, blocked or ignored, so there is always a way to stop any process.

How a handler runs. Sending a signal only sets a bit in the target's pending set and wakes it if it is sleeping. The target acts on the signal on its way back to user mode: after a syscall, a page fault or a timer interrupt. To run a handler, the kernel pushes a signal frame onto the user stack: all the user registers at the moment of interruption, the old signal mask, and a siginfo (which signal, who sent it, which address faulted). Then it edits the saved registers so that the return to user mode lands in the handler, with a return address pointing at a tiny "restorer" function (mov rax, 15; syscall). When the handler returns, the restorer calls rt_sigreturn, which copies the saved registers back from the frame, and the program carries on exactly where it was. sigtest checks this by interrupting a calculation and comparing the result. The frame is Linux's struct rt_sigframe, so musl's handlers will work unchanged.

This is also why syscalls now return to user mode with iretq through a common TrapFrame (src/trap.rs) rather than sysretq. Returning into a handler, or back from one, has to restore every register, including the rcx and r11 that sysretq overwrites.

Interrupted syscalls. A process sleeping in the kernel (reading the keyboard or a pipe, sleep, wait4, pause) is woken when a signal arrives. The call then fails with EINTR, or is restarted transparently if the handler was installed with SA_RESTART. sigtest checks both.

Process groups and Ctrl+C. A pipeline like cat x | grep y is two processes but one job. The shell puts them in their own process group (setpgid) and tells the terminal that this group is in the foreground (ioctl(TIOCSPGRP)). When you press Ctrl+C, the console (src/tty.rs) sends SIGINT to the whole foreground group. The shell itself ignores SIGINT, so only the job dies. kill -TERM %1 sends a signal to a background job's group, Ctrl+Z (SIGTSTP) stops the foreground job, and fg and bg continue it with SIGCONT.

wait4 and status codes. A parent learns how a child ended from wait4, which returns a packed status word decoded with Linux's macros: WIFEXITED/WEXITSTATUS (exited with a code), WIFSIGNALED/WTERMSIG/ WCOREDUMP (killed by a signal), WIFSTOPPED/WSTOPSIG (with WUNTRACED) and WIFCONTINUED (with WCONTINUED). The parent also gets SIGCHLD each time a child exits, stops or continues.

What it does and doesn't do

• One thread per process: no CLONE_VM/CLONE_THREAD, no futexes yet (they are next on the roadmap, because libc threads need them). Stage 15 added both; see the next chapter. clone accepts only fork-style flags (SIGCHLD, optionally CLONE_VFORK).
• vfork is fork + "the parent waits until the child execs or exits". It is correct for every legal use of vfork, just not faster than fork.
• At most 64 processes (MAX_PROCESSES). A fork bomb gets EAGAIN at the limit, and the test kills it and checks that nothing leaked.
• No MAP_SHARED file mappings (no page cache), no swap, no sigaltstack, no real-time signal queueing (a pending signal is one bit, as for Linux's standard signals), no users or permissions (uid 0).
• Ctrl+C and the foreground group work on the main console. hvc0's getty shell runs programs without a foreground group, so there Ctrl+C is just a character. Proper ttys and ptys are on the roadmap. Stage 15: hvc0 is a real tty now, and getty makes it the shell's controlling terminal, so Ctrl+C works there too.

Libc plumbing (stage 15)

Stage 14 gave Rustbelt processes. A real C library (musl, glibc) expects quite a lot more from a kernel before ordinary programs work: a way to wait on many files at once, files that say "not now" instead of blocking, threads, a way for threads to sleep on a lock, per-thread variables, and terminals that behave like terminals. Stage 15 adds all of that, still with Rustbelt's own Rust programs as the users. This chapter explains each idea from scratch.

Event loops: poll, select and epoll

Imagine a chat server with three clients. If it calls read on client 1, it blocks until client 1 says something, while clients 2 and 3 wait. One fix is a thread per client (below). The other, used by nearly every server, GUI toolkit and terminal emulator, is an event loop:

loop {
    ready = wait_until_any_is_ready([listener, client1, client2, ...])
    for fd in ready { accept / read / write -- none of these block now }
}

The kernel offers three versions of wait_until_any_is_ready:

• poll(fds, n, timeout_ms): an array of {fd, events, revents}. You set events (POLLIN = "tell me when I can read", POLLOUT = "...write"), the kernel fills in revents, including POLLHUP ("the other end hung up") and POLLERR. ppoll is the same with a nanosecond timeout and a signal mask.
• select(n, readfds, writefds, exceptfds, timeout): the 1983 BSD original, with bitmaps instead of an array (fd_set: bit fd set means "watch this fd"). pselect6 is its ppoll.
• epoll: epoll_create1 makes an epoll instance (itself an fd); epoll_ctl(ADD/MOD/DEL, fd, events, data) registers fds once; epoll_wait returns only the ready ones, each with your data (usually the fd or a pointer to your client struct). With 10 000 connections, poll copies and checks 10 000 entries per call; epoll doesn't.

How Rustbelt does it (src/poll.rs): every open file can answer "what's ready right now?" (File::poll). Whatever might change that (bytes into a pipe, a key, a packet, a pty write, a close) calls poll::notify(), which bumps one global counter and wakes every sleeping poller. A poller checks its fds; if none is ready it sleeps until the counter moves or the timeout ends. Simple and it can't miss an event, at the cost of waking pollers that didn't need waking. epoll here is level-triggered: a fd is reported as long as it stays ready (Linux's default; EPOLLET, "only report changes", is accepted but behaves the same).

Try echoserver (it runs the same echo test with an epoll server and a poll server) or echoserver 7070 to keep one running.

Nonblocking I/O and EAGAIN

An event loop must never block, so its fds are put in nonblocking mode: fcntl(fd, F_SETFL, O_NONBLOCK), or ioctl(fd, FIONBIO, &1), or O_NONBLOCK at open/pipe2/socket time. A read that would have to wait now returns at once with EAGAIN ("try again later"; also called EWOULDBLOCK), and so does a write to a full pipe (Rustbelt's pipes hold 64 KiB, like Linux's). A nonblocking connect returns EINPROGRESS; poll for POLLOUT to learn when it's connected. nbtest checks all of these.

Loopback

127.0.0.1 is "this machine". Programs use it to talk TCP/UDP to each other (a database and its client, a compositor and its apps, X11) with no network card involved. Rustbelt now has an lo interface: frames sent to 127.x (or to our own eth0 address) are handed straight back to smoltcp's receive side instead of the virtio-net card (src/net/device.rs). It is there even with no network card at all, so echoserver works offline.

Threads vs processes

A process is an address space plus open files, a cwd, signal handlers and so on. A thread is "a place where the CPU is executing": registers, a stack, a signal mask. Stage 14 had exactly one thread per process. Linux has one call for both, clone(flags, ...), and the flags say what to share with the parent:

Flag	Shares
CLONE_VM	the address space (same memory)
CLONE_FILES, CLONE_FS	the fd table; the cwd
CLONE_SIGHAND	signal handlers
CLONE_THREAD	the process itself: same pid, the new thread gets a new tid
CLONE_SETTLS	start the new thread with this TLS pointer (below)
CLONE_PARENT_SETTID / CLONE_CHILD_SETTID	write the new tid here
CLONE_CHILD_CLEARTID	when the thread exits, write 0 here and futex-wake it (that's pthread_join)

fork is just clone with none of them. pthread_create is all of them. getpid gives the process id, gettid the thread id (the first thread's tid equals the pid). exit (60) ends one thread; exit_group (231, what libc's exit() calls) ends them all. tgkill sends a signal to one thread. Threads of one process run on different CPUs at the same time: threads sums 1..=40,000,000 on 4 threads and prints how many CPUs took part.

Sharing memory means two threads can update the same variable at once. counter += 1 is a load, an add and a store; two threads interleaving them lose updates. threads shows exactly that, then fixes it with a lock.

Futexes: how locks sleep

A lock is just an integer. Taking a free lock is one atomic compare-and-swap in user space, no kernel at all. The kernel is only needed when a thread must wait, and that's futex ("fast userspace mutex"):

• futex(&word, FUTEX_WAIT, expected): "sleep, but only if word still equals expected"
• futex(&word, FUTEX_WAKE, n): "wake up to n threads sleeping on word"

The "only if still equal" check, done under the kernel's lock, closes the race where the holder unlocks between our check and our sleep: then WAIT sees the new value and returns EAGAIN immediately. The mutex in user/threads.rs is the classic one from Ulrich Drepper's paper Futexes Are Tricky: the word is 0 (free), 1 (locked) or 2 (locked and someone may be waiting). Lock: CAS 0 -> 1; if that fails, swap in 2 and FUTEX_WAIT while it isn't 0. Unlock: swap in 0; only if the old value was 2 call FUTEX_WAKE. With no contention, neither side ever enters the kernel.

The kernel keeps no per-lock state, just a list of sleepers, each tagged with the word it waits on: (address space, virtual address), or the physical address for MAP_SHARED memory (src/futex.rs). Timeouts, FUTEX_WAIT_BITSET/WAKE_BITSET, REQUEUE and the private flag are there too, because libc uses them.

Thread-local storage

errno is a global variable... but each thread needs its own, or one thread's failing read would overwrite another's. Such per-thread variables are thread-local storage (TLS). On x86_64 the trick is a segment register: every thread has its own FS base address, and code reads its TLS at fs:offset. libc allocates a TLS block per thread and sets FS with arch_prctl(ARCH_SET_FS, addr) (or CLONE_SETTLS). The kernel's whole job is to save and restore that address on each thread switch (it lives in the IA32_FS_BASE MSR, src/thread.rs). set_tid_address registers the CLONE_CHILD_CLEARTID word for the first thread.

ttys and ptys, and why terminals are weird

Terminals are strange because they're a 1970s design that never went away. A terminal was a keyboard and screen at the end of a serial line; the kernel sat between the line and the programs, and that middle layer, the line discipline, is still in every Unix:

• Canonical mode (the default): what you type collects in a line buffer you can edit (Backspace, ^U erases the line) and read gets nothing until Enter. ^D at the start of a line is end-of-file. Programs that want every key (editors, Rustbelt's shell line editor) switch to raw (non-canonical) mode.
• Echo: the kernel prints what you type, not the program. That's why cat shows your typing before it prints the line back, and why a password prompt turns ECHO off.
• Signals: ^C sends SIGINT, ^\ SIGQUIT, ^Z SIGTSTP, not to "the program" but to the terminal's foreground process group, the job the shell put in front (TIOCSPGRP). That's why ^C kills cat and not the shell waiting for it.
• Output processing: ONLCR turns \n into \r\n, because a teletype needed a carriage return to move back to column 0.

All of it is configured by one struct, termios, read and written with the TCGETS/TCSETS ioctls (libc: tcgetattr/tcsetattr).

Sessions. setsid() starts a session (roughly, a login). The first terminal it opens becomes its controlling terminal; /dev/tty always means "my controlling terminal", whatever its real name. When the session leader exits, the foreground job gets SIGHUP ("hang up", from the modem days). getty on hvc0 is now a classic one: it forks, the child calls setsid, opens /dev/hvc0 (which becomes its controlling terminal) and execs the shell, so ^C on hvc0 interrupts programs there just like on the console.

Pseudo-terminals. A terminal window has no serial line; a program plays the terminal instead. Opening /dev/ptmx gives that program the master end and creates /dev/pts/N, the slave end, which the shell gets as its stdin/out/err. Bytes written to the master are "typed" into the slave, through the line discipline (so echo and ^C work); what the shell writes to the slave comes out of the master to be drawn. xterm, ssh, tmux and the future compositor's terminal all work this way. In Rustbelt the console, hvc0 and every pty share one Tty (src/tty.rs). ptytest drives one from both ends.

What stage 15 does and doesn't do

Does: everything above, at -smp 4, checked by tests/libc.rs, which runs each test program twice and requires the thread, process, futex, pty, socket and memory-frame counts to come back exactly where they were (no leaks). Also nanosleep (now exact to the microsecond on top of the 55 ms PIT tick), clock_gettime(CLOCK_REALTIME) (the CMOS clock), uname, getcwd, readlink /proc/self/exe, readv/writev, pread/pwrite.

Doesn't (yet):

• No SIGTTIN/SIGTTOU: a background job may read and write the terminal.
• epoll is level-triggered only; no EPOLLONESHOT, no signalfd, eventfd or timerfd.
• One global "something changed" counter wakes every poller.
• Timeouts are rounded up to 55 ms PIT ticks (never early).
• No FPU/SSE state saving (Rustbelt's programs are soft-float), and paths passed as pointer + length: both fixed in stage 15.1, next.

musl prerequisites (stage 15.1)

Stage 15 gave Rustbelt what a C library needs from a kernel's behaviour. Stage 15.1 fixes what a C library needs from its interface. Up to now Rustbelt's own Rust programs were the only users, so a few of its calls could take shortcuts that musl would trip over. This stage removes the three that matter most. It does not port musl; that is the next step.

1. The FPU/SSE/AVX registers belong to each thread. They are saved and restored on every switch and put into signal frames.
2. Paths are C strings. The file calls take Linux's arguments in Linux's order, including the *at() family, and fill Linux's struct stat.
3. An audit of every syscall number and argument order. A plain Linux program, rawabi, is built by gcc with no libc and runs unmodified.

Floating point is part of a thread's registers

Rustbelt's kernel and its Rust programs are built soft-float: they do floating-point maths with integer instructions and never touch the FPU. A C compiler is different. It keeps every double in an SSE register (xmm0-xmm15), and musl's memcpy and strlen use them too. For a C program those 16 registers (32 bytes each with AVX) are as much "its registers" as rax is. Picture two threads sharing a CPU. If the kernel switches between them without saving these registers, one thread's x * y quietly comes out with the other thread's x.

The CPU can save all of this state with one instruction (see src/fpu.rs):

• FXSAVE/FXRSTOR exist on every x86_64 CPU. They save the x87 stack, xmm0-15 and the MXCSR control register into a 512-byte area.
• XSAVE/XRSTOR are on newer CPUs. They save the same 512 bytes, a 64-byte header that says which parts are present, and then one block per extra feature, such as AVX's upper ymm halves (256 more bytes). The OS chooses which features it saves by writing XCR0, and CPUID then reports how big the area is (832 bytes for x87 + SSE + AVX).

Before user code may run an SSE instruction, the OS has to promise that it saves this state. It does that with control-register bits, set on every CPU in fpu::init_cpu:

• CR4.OSFXSR: the OS uses FXSAVE, so SSE is allowed. Without it SSE instructions raise #UD.
• CR4.OSXMMEXCPT: unmasked SSE maths errors raise #XM.
• CR4.OSXSAVE: XSAVE and XCR0 may be used.
• In CR0, EM and TS are cleared, and MP and NE are set.

Rustbelt then uses eager switching. When schedule() leaves a user thread it saves that thread's state into Task::fpu, and it loads the next thread's state before returning to user mode. Lazy switching (set CR0.TS, take a #NM fault on the first FPU instruction, and save only then) used to be faster. It is also the source of a CPU side channel (LazyFP), and Linux dropped it in 2018. Eager switching costs about one kilobyte copied per switch, and it is ten lines of code.

The state follows a thread through the process model the same way the other registers do:

event	FPU state
fork, clone (process or thread)	the child starts with a copy of the parent's
execve	reset: zeros, x87 control word 0x37f, MXCSR 0x1f80
signal delivery	saved into the signal frame, then reset for the handler
rt_sigreturn	loaded back from the frame, so a handler may change it
x87 error / SSE error (#MF 16 / #XM 19)	SIGFPE; for #XM the si_code (FPE_FLTDIV, FPE_FLTINV...) comes from MXCSR's flags

FPU state in signal frames

A signal handler is ordinary C code, so it uses xmm registers. The interrupted code must find its own values again afterwards. Linux puts the FPU state in the signal frame, and musl and glibc never look at it. GDB, and programs that edit the registers in a handler, do look, and use ucontext->uc_mcontext.fpregs. Rustbelt now writes the same layout as Linux (signal.rs, setup_frame):

higher addresses
  [ 128-byte red zone of the interrupted code ]
  fpstate (64-byte aligned): the XSAVE (or FXSAVE) image
      bytes 464..512 ("sw_bytes", unused by the CPU):
          magic1 0x46505853 "FPXS", extended size, XCR0, XSAVE size
      ...followed by magic2 0x46505845 "FPXE" (XSAVE only)
  rt_sigframe: restorer address, siginfo, ucontext
      ucontext.uc_mcontext.fpregs -> fpstate
lower addresses: the handler's rsp

The magic numbers tell a reader that the area is XSAVE format and how big it is. Without them it is a plain FXSAVE image. Because the area comes from user memory, rt_sigreturn checks it before loading it. An MXCSR with reserved bits set would make the CPU fault inside the kernel, so the process gets SIGSEGV ("bad sigreturn FPU state") instead. A null fpregs means "start clean", as on Linux.

Linux paths: C strings, struct stat and the *at() calls

Since stage 5 the file calls took a path as pointer + length. That was easy to check and matched Rust's &str, but it is not what Linux does. musl's open(path, flags) puts one pointer in rdi, so on the old ABI it would have read flags as the path length. Every path call now takes a NUL-terminated string.

The kernel copies the string in page by page, with user_cstr. Each page is checked against the process's memory regions before it is read, so a string that runs into an unmapped page is EFAULT, never a kernel crash. The copy stops at PATH_MAX (4096 bytes, NUL included), so a path with no NUL is ENAMETOOLONG rather than an endless read. pathtest tries NULL, a kernel address, an unmapped address and a string that straddles into an unmapped page, on every path call.

The *at() calls (openat, newfstatat, mkdirat, unlinkat, renameat, readlinkat, faccessat) resolve a relative path against an open directory fd instead of the cwd, or against the cwd when dirfd = AT_FDCWD (-100). They exist so a program can work inside a directory that another process might rename meanwhile. musl calls some of them even for the plain functions: stat() is newfstatat(AT_FDCWD, path, &st, 0), and fstat() may become newfstatat(fd, "", &st, AT_EMPTY_PATH). The old calls (open, stat, mkdir...) are now one-line wrappers that pass AT_FDCWD. The kernel remembers each open file's path (File::path), so a directory fd can serve as a base.

struct stat is now Linux's 144-byte x86_64 layout. The kernel keeps its own small Stat internally and converts it at the syscall boundary (LinuxStat::new in fs/file.rs):

• st_mode combines the type (S_IFREG, S_IFDIR, S_IFLNK, S_IFCHR, S_IFIFO, S_IFSOCK) with the permission bits. File systems without permissions (ramfs, FAT) get 0644 for files and 0755 for directories.
• st_ino is a hash of the path. The VFS has no stable inode numbers yet, but musl only compares them (for example in realpath), and a hash is stable and almost always unique.

ulib now has Stat::is_dir(), is_char_device() and perm() in place of the old kind numbers, and isatty() works the way libc's does, with ioctl(TCGETS).

rename is new. The VFS has no "move this directory entry" operation, so fs::rename copies a file, or recreates an empty directory, and then removes the old name. It refuses what it can't do safely:

case	result
a directory into itself	EINVAL
a mount point	EBUSY
a non-empty directory, a symlink, or across file systems	EXDEV

EXDEV is the same error Linux gives across file systems, so mv falls back to copying. The rename is not atomic: a crash in the middle can leave both names, or a reader can briefly see neither. Real per-file-system rename (an ext2 directory-entry move, for example) is future work.

A few calls changed their argument order to match Linux:

• recvfrom gets its sixth argument, *addrlen, and accept honours *addrlen (the address is cut to the given length, and the full length is written back).
• wait4 accepts a struct rusage pointer and zero-fills it.
• clock_nanosleep arrived; musl's nanosleep and sleep use it.
• spawn, Rustbelt's own call, now takes a C-string path too.

The audit

A script compared every SYS_* constant in src/syscall.rs and user/ulib.rs against the host's <asm/unistd_64.h>. The only numbers that differ from Linux are Rustbelt's own 500-505 (spawn, mounts, sysinfo, resolve, netinfo, trace_me). These are not Linux calls, so they can't collide with one. The one catch is the name sysinfo: on Linux it is call 99 with a different struct, and musl's sysinfo() would get ENOSYS.

The argument orders that differed were fixed above. The ones that still differ in meaning, not in order, are:

• sendto ignores flags, so MSG_NOSIGNAL does not stop a SIGPIPE.
• getrandom ignores its flags.
• mode arguments (open, mkdir) are dropped, and all uids are 0.
• setsockopt knows only SO_RCVTIMEO. Anything else now gives Linux's ENOPROTOOPT.

rawabi: a Linux program with no libc

user/rawabi.c is the test that matters most for musl. It is ordinary C written against Linux, not against Rustbelt:

• its own _start
• a syscall wrapper with Linux's numbers
• C-string paths and Linux's struct stat
• double maths (SSE instructions, plus an explicit sqrtsd)
• a signal handler that clobbers xmm registers while the interrupted code keeps a double live in one

build.rs compiles it with the host's gcc:

cc -O2 -static -nostdlib -ffreestanding -fno-stack-protector -fpie -no-pie \
   -Wl,-Ttext-segment=0x8000400000 -o rawabi rawabi.c

-nostdlib means there is no libc and no startup files. -fpie -no-pie looks contradictory, but it isn't. -fpie makes gcc address everything RIP-relative, so the code works at any address. -no-pie still links an ordinary fixed-address ET_EXEC file, the only kind Rustbelt's ELF loader accepts. -Ttext-segment puts that file at Rustbelt's user base (0x80_0000_0000). That address is above 4 GiB, where gcc's default "absolute 32-bit addresses" code would not link. A normal Linux machine runs the same file without complaint. Try ./rawabi /tmp on the host: it prints the same checks.

What it checks:

• the startup calls musl makes (arch_prctl + %fs:0, set_tid_address, rt_sigprocmask)
• uname, brk and an anonymous mmap
• mkdir, openat(AT_FDCWD, O_CREAT) + writev, stat, newfstatat, access and stat(NULL) = EFAULT
• rename, then read, fstat and lseek
• getdents64, chdir + getcwd, unlinkat with and without AT_REMOVEDIR
• clock_nanosleep on CLOCK_MONOTONIC
• π from a series, plus the signal test above
• fork + wait4 with a rusage buffer

It runs from the shell (rawabi), in the boot self-test, and in tests/musl_prereq.rs.

Testing it

tests/musl_prereq.rs runs the three programs twice each, at -smp 4, and fails if a thread, process or memory frame leaks:

• fputest: three processes, each with three threads plus main, load all 16 xmm (or ymm) registers with their own pattern and spin, checking it after every yield. The test also covers:
  • a fork child inherits the parent's registers
  • execve starts clean
  • a handler starts with clean registers and sees the interrupted values in fpregs (magic numbers, sizes, xmm0 and the upper ymm half)
  • a handler that edits the saved xmm15 changes what the interrupted code sees after sigreturn
  • 40 asynchronous tgkills land in the middle of the check loop
  • division by zero with exceptions unmasked gives SIGFPE (skipped under QEMU's TCG, which never raises it; see below)
• pathtest: EFAULT for NULL, kernel and unmapped pointers on 13 calls, and for a path straddling into an unmapped page (the same path ending at the page boundary is fine). It also tests:
  • ENAMETOOLONG at 4096 bytes, a 4095-byte path accepted, and the empty path giving ENOENT
  • the struct stat fields and file types, lstat vs stat, and AT_EMPTY_PATH
  • every *at() call with a real directory fd and with a bad one
  • O_EXCL, O_DIRECTORY, O_NOFOLLOW, fchdir, and every rename case
• rawabi, as above.

QEMU's default CPU model (qemu64) has no XSAVE, so Cargo.toml now passes -cpu max to QEMU. That gives XSAVE and AVX under QEMU's software emulation (TCG). The FXSAVE fallback is tested by running the same tests with -cpu qemu64.

What stage 15.1 does and doesn't do

Does: everything above, at -smp 4, with XSAVE (x87 + SSE + AVX) or FXSAVE. The boot log says which, for example [fpu] XSAVE: x87 SSE AVX, 832 bytes per thread.

Doesn't (yet):

• AVX-512 and other XSAVE features stay off in XCR0. The area would be about 2.7 KiB per thread, and QEMU's TCG doesn't emulate AVX-512 anyway.
• #XM under TCG. QEMU's TCG never raises the SIMD exception: an unmasked divide by zero just sets the flag in MXCSR and gives infinity. fputest notices this and skips that check. The #XM → SIGFPE path is written but untested here, because KVM (where the real CPU would raise it) isn't available on the build machine.
• No XSAVEOPT/XSAVES (which skip unchanged state) and no "init-optimization" tricks: every switch saves the whole area.
• Rename isn't atomic, and can't move non-empty directories or symlinks.
• No permissions: mode, umask, chmod and uids.
• ELF loading accepts only static ET_EXEC files at or above 0x80_0000_0000. A stock musl-gcc -static binary is linked at 0x400000, so it needs -Wl,-Ttext-segment=0x8000400000, or the loader needs to learn static-PIE (ET_DYN, loaded at a base of its choosing).

Syscalls musl will call that Rustbelt still lacks. These all return -ENOSYS today. Most matter only for the functions that use them. (Stage 16, below, crossed off the rows for signals, rlimits, umask and chmod, truncate, fsync, utimensat, record locks, prctl, robust lists, sched_getaffinity and madvise.) (Stage 17 crossed off chown, link/symlink, statfs/flock, sysinfo, getrusage/times/clock(), clock_getres, the timers, the setuid family, nice, preadv/pwritev/sendfile, and most of the socket row.)

musl function(s)	missing syscall (x86_64 number)
malloc (mallocng)	mremap 25 (failure is tolerated)
dlclose barriers	membarrier 324
sockets	socketpair 53 (needs AF_UNIX; EAFNOSUPPORT now), sendmsg 46, recvmsg 47 (the DNS resolver's TCP fallback)
timerfd_create	timerfd_create 283, timerfd_settime 286, timerfd_gettime 287

Not syscalls, but missing just the same:

• AF_UNIX sockets (syslog, many daemons), and AF_INET6. musl's resolver falls back to IPv4.
• /proc/self/fd/N, which musl's ttyname, realpath and fchmodat read. Only /proc/self/exe exists.
• The vDSO. musl looks up AT_SYSINFO_EHDR and falls back to the real clock_gettime syscall when it is missing. That fallback is fine, only slower.
• /etc/passwd, /etc/resolv.conf and the other files musl reads. Rustbelt does DNS in the kernel (resolve, 503).

libc batch 1 (stage 16)

Stage 15.1 got a musl program to start. Stage 16 is about what it does next. Clarke ran real musl 1.2.5 programs and the dash shell with run --trace and collected every call that came back ENOSYS. This stage is the first batch of them, in the order they hurt:

1. rt_sigsuspend, without which dash's wait spun forever;
2. umask and the resource limits;
3. sched_getaffinity (how many CPUs do I have?);
4. orphaned children that were never freed;
5. then the rest: sigaltstack, sigtimedwait, fcntl locks, truncate, chmod, utimensat, prctl and friends.

Every call uses Linux's number, arguments and error codes, so man 2 <name> describes it. Where we can't do the real thing we say so with an honest error (ENOSYS, EINVAL, EROFS), never a fake success.

Waiting for a signal without missing it

A shell runs sleep 5 & and then wait. It must sleep until a SIGCHLD says a child finished. The obvious code has a bug:

if (!child_done)    // (1) no, not yet...
    pause();        // (2) ...so sleep until a signal

If the SIGCHLD arrives between (1) and (2), its handler sets child_done before we sleep, and pause then sleeps forever: nobody will send a second signal. This is called a lost wake-up.

The fix is to keep SIGCHLD blocked (held back, "pending") while you check, and let one system call unblock it and go to sleep in a single step:

sigprocmask(SIG_BLOCK, {SIGCHLD}, &old);
while (!child_done)
    sigsuspend(&old);   // unblock + sleep, atomically; then block again

rt_sigsuspend (130) swaps in the given mask, sleeps until a handler has run, puts the old mask back and returns EINTR (it always "fails"; that is its success). If the signal was already pending it doesn't sleep at all: the handler runs at once. In the kernel this was short, because ppoll (stage 15) already does "a temporary mask while waiting": set_temporary_mask, then the same sleep as pause, and the signal delivery code restores the mask when the handler is done.

Before stage 16 the call returned ENOSYS right away, so dash's loop ran round and round at 100% CPU. libctest wait is that exact loop with a limit on the rounds, so a broken kernel fails the test instead of hanging it.

Its relatives:

• rt_sigpending (127): which blocked signals are waiting.
• rt_sigtimedwait (128): take a signal from a set without running its handler, and get its siginfo. It is like read for signals. The caller blocks the signals first; the kernel unblocks them only for the wait. A timeout gives EAGAIN, and {0, 0} means "just look".
• rt_sigqueueinfo (129) and rt_tgsigqueueinfo (297): send a signal with your own siginfo, which is how sigqueue() attaches a number to a signal. The kernel won't let you pretend to be the kernel or kill to someone else: si_code must be negative (EPERM).

Standard vs real-time signals. Signals 1-31 are flags: SIGUSR1 sent three times while blocked is delivered once. Signals 32-64 (the "real-time" ones) are a queue: sent three times, they arrive three times, in order, each with its own value. Stage 16 added that queue (up to 1024 per process; then EAGAIN).

A spare stack for signal handlers

A handler normally runs on the thread's own stack, below where the interrupted code was. What if the signal is SIGSEGV because that stack overflowed? There is no room left to run the handler. sigaltstack (131) registers a second stack, and a handler installed with SA_ONSTACK runs there instead. Rust's standard library uses it to print "thread main has overflowed its stack" instead of dying silently.

The kernel only has to pick a different place for the signal frame (stage 14's "Signals" section explains frames). Some rules, as on Linux: the stack must be at least 2 KiB (ENOMEM); while a handler is running on it you can't change it (EPERM); and asking about it from inside the handler reports SS_ONSTACK. A handler that interrupts another handler already on the alternate stack just continues below it.

umask and resource limits

umask (95) is a per-process mask of permission bits that new files must not get. A program asks for open("x", O_CREAT, 0666); with the usual umask 022 the file gets 0666 & ~022 = 0644 (rw-r--r--). It is inherited across fork, so a shell's umask 077 affects every command it starts. Rustbelt now stores the result on ramfs and ext2, and chmod changes it, but nothing checks permissions yet: everyone is still root.

Resource limits (getrlimit 97, setrlimit 160, prlimit64 302) are pairs: a soft limit (what applies now) and a hard limit (the most the soft one may be raised to). Programs ask, for example, "how many fds may I open?" and size tables from the answer. Lying would break them, so where Rustbelt has a real limit it reports it:

resource	value	why
RLIMIT_NOFILE 7	64 / 64	the fd table has 64 slots
RLIMIT_STACK 3	the user stack size	fixed when the stack is mapped
RLIMIT_NPROC 6	the process table size
RLIMIT_CORE 4	0 / unlimited	no core dumps
the rest	unlimited	nothing enforces them

Limits are per process and inherited by fork. Lowering one works and is enforced for RLIMIT_NOFILE (dup and open then fail with EMFILE, F_DUPFD with EINVAL). Raising a hard limit past the real one is EPERM.

File locks between processes

fcntl(F_SETLK) locks a byte range of a file: a read (shared) lock or a write (exclusive) one. Two processes that write the same file use it to take turns. The kernel keeps a list of (file, process, range, type) in src/fs/lock.rs:

• F_SETLK takes or releases a lock, or fails at once with EAGAIN if another process holds a conflicting one.
• F_SETLKW ("wait") sleeps until it can take it. If A waits for B while B waits for A, both would sleep forever, so the kernel follows the chain of "who waits for whom" and refuses the second wait with EDEADLK.
• F_GETLK asks "would this lock work?" and if not, which process is in the way.

They are POSIX record locks, with POSIX's famous quirk: closing any fd of a file drops all of that process's locks on it, even ones taken through another fd. Exiting drops them too. libctest locks checks both.

fcntl also learned to tell the truth with F_GETFL: the real access mode (O_RDONLY/O_WRONLY/O_RDWR, it used to always say O_RDWR), O_APPEND and O_NONBLOCK.

truncate, fsync, chmod and file times

• truncate/ftruncate (76/77) set a file's size: shorter cuts off the end, longer adds zero bytes. ramfs and FAT do it directly; ext2 through a generic "copy what stays" fallback. ftruncate needs an fd opened for writing.
• fsync/fdatasync (74/75) say "make sure this is on the disk". Rustbelt writes to the disk straight away (there is no write-back cache), so for files they just return 0.
• chmod/fchmod/fchmodat (90/91/268) store permission bits on ramfs and ext2. FAT and exFAT have nowhere to keep them (EPERM).
• utimensat (280) sets the access and modification times (touch does this). stat used to report every time as 0 (1970); now ramfs, ext2 and btrfs report real ones, and new files are stamped with the current time. ext2 keeps whole seconds.
• On btrfs, which Rustbelt only reads, all of these are EROFS ("read-only file system"), as on Linux for a read-only mount.

Processes: prctl, CPUs, madvise, robust locks

• prctl (157) is Linux's drawer of odds and ends. We do the ones libc uses: the thread's name (pthread_setname_np, 15 characters), "send me this signal when my parent dies" (PR_SET_PDEATHSIG), PR_SET_DUMPABLE, PR_SET_NO_NEW_PRIVS, and PR_SET_CHILD_SUBREAPER (below). Others return EINVAL.
• sched_getaffinity (204) returns a bit mask of the CPUs a thread may run on. musl's sysconf(_SC_NPROCESSORS_ONLN) counts its bits, so this is how a C program learns it has 4 CPUs. Every thread may run anywhere, so it's simply one bit per CPU. Its return value is the number of bytes written (8), not 0. That is a classic gotcha: the libc wrapper zeroes the rest of your buffer and then returns 0.
• madvise (28) gives hints about memory. Most can be ignored, and we do. MADV_DONTNEED can't be ignored: it promises the pages read as zeros afterwards, and musl's malloc relies on that. So we free those pages and the next touch gets a fresh zeroed page (stage 14's demand paging).
• Robust futexes (set_robust_list, 273). A pthread mutex lives in user memory. If the thread holding it dies, normally nobody ever unlocks it. A robust mutex is on a list the thread gives the kernel; when the thread exits, the kernel walks the list, marks each lock it still holds FUTEX_OWNER_DIED and wakes one waiter, who can then repair things.

Orphans: who cleans up?

When a process exits before its children, they become orphans. On Unix, someone must eventually wait for every process, or its exit status (a "zombie") stays in the process table forever. Linux hands orphans to pid 1, init, whose job is to wait for anything.

Rustbelt's pid 1 is usually the shell, which only waits for its own jobs. Handing it orphans would just collect zombies. So:

• An orphan is adopted by the nearest living ancestor that called prctl(PR_SET_CHILD_SUBREAPER, 1), as on Linux. That's how service managers keep track of the grandchildren of what they start.
• Failing that, by pid 1, if pid 1 is a real init: named init, or a subreaper itself.
• Otherwise it has no parent, and when it exits it reaps itself. Its memory is freed straight away.

libctest orphans leaves a running orphan and a zombie behind three times, and the kernel test checks that the process count and the number of memory frames come back to where they were.

Tracing calls we don't have

run --trace prog printed a call it didn't know as ?(...), so you had to guess which one was missing. Now it prints the number, syscall_436(...), which you can look up in Linux's arch/x86/entry/syscalls/syscall_64.tbl.

Testing it

user/libctest.rs has one group of checks per argument, and tests/libc_batch1.rs runs each group twice. The second run must leave the thread, process, memory-frame and file-lock counts exactly where they were. The groups are wait, signals, locks, files, misc and orphans (the Tests table above lists what each checks). You can run them from the shell too, for example libctest locks.

What stage 16 does and doesn't do

Does: everything above, with Linux's numbers and error codes.

Doesn't (yet):

• Permissions are stored, not checked. Everyone is uid 0, and read/write don't check the fd's access mode either (only F_GETFL, ftruncate and locks look at it). There is no chown.
• flock (73, whole-file locks), the F_OFD_* locks and F_GETOWN aren't here; nor are utime/utimes/futimesat (musl uses utimensat anyway).
• FAT and exFAT times: they have time fields, but our drivers don't write them, so utimensat is accepted and forgotten, and stat shows 0.
• sigtimedwait's timeout is rounded up to whole timer ticks (about 55 ms each), so a short timeout can take noticeably longer than asked.
• madvise: MADV_DONTNEED frees only private memory (shared and file mappings keep their pages), and MADV_REMOVE/MADV_WIPEONFORK are EINVAL.
• The rest of Clarke's list (static-PIE loading, a hook for running disk images) are separate stages.

coreutils batch 2 (stage 17)

Stage 16 was what a C library needs to run. Stage 17 is what the everyday tools need: ln, chown, df, uptime, time, sleep with an alarm, id, nice, hostname, rm -r, cp, and the socket calls a small server makes. Clarke ran coreutils and busybox (tier 3) with run --trace, wrote down every ENOSYS, and put them in order:

1. link and symlink (and their *at forms);
2. setsockopt options that may be ignored, times(), CPU-time clocks;
3. the chown family;
4. statfs / fstatfs (df);
5. Linux's sysinfo (99), getrusage, clock_getres;
6. getsockopt, getsockname, shutdown, socketpair;
7. alarm, interval timers and POSIX timers;
8. user and group ids, priority, sethostname;
9. then, if there was room: flock, preadv/pwritev, sendfile (and mremap, which didn't make it).

As before, every call uses Linux's number, arguments and error codes, so man 2 <name> describes it. The new calls live in their own file, src/syscall_ext.rs, with a table at the top, because syscall.rs had grown past 2000 lines. The dispatcher in syscall.rs tries its own table first and then asks syscall_ext::dispatch.

Hard links and symbolic links

A file's name and the file itself are different things. On ext2 (and on Linux's ramfs) the file is an inode, with a number, and a directory is just a list of name -> inode number. So one file can have two names:

ln a.txt b.txt      # link(86): a second name for the same inode
ls -i a.txt b.txt   # the same inode number twice
stat a.txt          # "Links: 2"

Change the file through one name and the other sees it; delete one name and the file lives on until the last name (and the last open fd) is gone. That count is st_nlink, and stat now reports the real one.

A symbolic link (symlink 88, ln -s) is different: a tiny file whose content is a path. Opening it follows the path; the target doesn't even have to exist (a dangling link). readlink (stage 15.1) reads it.

Where it works and where it says no:

file system	link	symlink
ramfs (/, /tmp)	yes	yes
ext2 (/mnt/disk1)	yes, up to 32000 links (EMLINK)	yes
FAT, exFAT	EPERM (no inodes: a name is the file)	EPERM
btrfs (read-only)	EROFS	EROFS
two different file systems	EXDEV (an inode number only means something on its own disk)	works (it's just a path)

linkat (265) adds AT_SYMLINK_FOLLOW (link to what a symlink points at, not the symlink) and AT_EMPTY_PATH (link an open fd). symlinkat (266) is symlink relative to a directory fd.

rename learned to move things. Before, rename copied and deleted, which can't work for a file with two names (the copy would be a third, separate file) or for a full directory. ramfs and ext2 now move the directory entry itself (Inode::rename_entry), so mv dir1 dir2/ works, and so does renaming over a second name of the same file. renameat2's RENAME_NOREPLACE ("fail with EEXIST rather than overwrite") is supported too.

Owners: chown

Every file has an owner (a user id) and a group (a group id). chown (92), fchown (93), lchown (94, the symlink itself) and fchownat (260) change them; -1 means "leave this one alone". stat reports them. ramfs and ext2 store them; FAT and exFAT have nowhere to put them, so a real change is EPERM, but "change it to what it already is" (root:root) succeeds, as Linux's vfat does. That matters: cp -p and tar call chown on every file they copy.

New files belong to their creator (the process's fs uid and gid, see below).

How big is the disk? statfs

df asks statfs (137) or fstatfs (138) for each mount: the block size, how many blocks there are and how many are free, how many inodes, the longest name, and a magic number saying which kind of file system it is. Linux uses the same magic numbers, so stat -f prints the right name:

mount	magic	type
/, /tmp	0x858458f6	ramfs
/mnt/disk0	0x4d44	msdos (FAT)
/mnt/disk1	0xEF53	ext2
/mnt/disk2	0x2011BAB0	exfat
/mnt/disk3	0x9123683E	btrfs (flagged read-only, ST_RDONLY)

A pipe or socket fd gets PIPEFS/SOCKFS, like Linux. Each mount also has its own st_dev now (it used to be 0 everywhere), which is how cp -r and find -xdev tell "same file system" from "crossed a mount".

statx (332) is the newer stat: a 256-byte struct with a mask saying which fields are filled in. coreutils' ls and stat try it first, so it was cheaper to implement it (from the same data as stat) than to send them down a fallback path.

What time is it for me? CPU time

time ls prints three numbers: real (wall clock), user (CPU time running the program's own code) and sys (CPU time in the kernel on its behalf). The kernel learns those by looking at what each CPU was doing at every timer interrupt (every 10 ms): if a thread was running, it gets 10 ms of user or system time, depending on whether the interrupt arrived in ring 3 or ring 0. It's sampling, not exact, but it's what Linux did for decades.

From those counters:

• times (100) gives user and system time of the process and of its waited-for children, in "clock ticks" of 1/100 s.
• getrusage (98) gives the same as timevals, for RUSAGE_SELF, RUSAGE_CHILDREN or RUSAGE_THREAD, plus the memory high-water mark (here: the resident memory now).
• clock_gettime(CLOCK_PROCESS_CPUTIME_ID) (clock 2) and CLOCK_THREAD_CPUTIME_ID (3) used to be EINVAL. C's clock() uses clock 2.
• clock_getres (229) says how fine each clock is: 1 µs for the wall clocks, 10 ms for the CPU-time ones. Honest numbers, not 1 ns.

sysinfo, the Linux one. Rustbelt has had its own sysinfo since stage 11, but it took number 502 and a different struct (per-CPU load bars for user/sysinfo.rs). Linux's is 99, and musl's sysinfo(), uptime and free use it. Both exist now: 99 fills Linux's struct sysinfo (uptime, total/free memory, process count, load averages), and ours stays at 502, renamed rb_sysinfo in --trace so the two can't be confused. The "load average" is the busy fraction of the CPUs since boot (Rustbelt doesn't keep a 1/5/15-minute history yet).

Sockets: names, options, half-closing

• getsockname (51) and getpeername (52): "which address and port am I?" and "who am I talking to?". A server that binds to port 0 uses the first to learn which port it got.
• setsockopt (54) used to know only SO_RCVTIMEO. Servers set SO_REUSEADDR (and SO_KEEPALIVE, TCP_NODELAY, buffer sizes) before bind and treat an error as fatal, so these are now accepted and remembered (smoltcp doesn't need them). Unknown options are still ENOPROTOOPT, not a fake success.
• getsockopt (55) reads them back, plus SO_TYPE, SO_ERROR (a failed nonblocking connect reports ECONNREFUSED here, once) and SO_ACCEPTCONN.
• shutdown (48) closes one direction: SHUT_WR sends the "no more data from me" (a TCP FIN) while you can still read the answer; the other side's read returns 0. Writing after it is EPIPE (and a SIGPIPE), reading after SHUT_RD returns 0.
• socketpair (53) makes two connected sockets, which needs AF_UNIX sockets. Rustbelt doesn't have those yet, so it says so: EAFNOSUPPORT (EOPNOTSUPP for AF_INET, as on Linux).

Alarms and timers

Three generations of "send me a signal later":

• alarm(seconds) (37): SIGALRM once, after N seconds. Returns how many seconds were left on the previous alarm; alarm(0) cancels.
• setitimer/getitimer (38/36): the same timer (ITIMER_REAL) in microseconds, optionally repeating every interval.
• POSIX timers (timer_create 222 to timer_delete 226): as many as you like, on CLOCK_REALTIME, CLOCK_MONOTONIC or CLOCK_BOOTTIME, any signal, an absolute or relative start, and a value (si_value) that arrives in the handler's siginfo with si_code = SI_TIMER. SIGEV_NONE makes a timer you only read with timer_gettime. If a periodic timer fires while its last signal is still pending, the missed periods are counted as overruns (timer_getoverrun).

They all live in src/timer.rs: one table, and one kernel thread (timerd) that sleeps until the nearest deadline and sends what's due. Why not send signals from the timer interrupt? Sending one takes locks and may wake threads; doing that in a normal thread keeps the interrupt handler tiny.

As on Linux: fork doesn't copy timers, execve keeps the alarm but deletes POSIX timers, and exiting deletes them all.

Who am I? User ids

A process has three user ids, and the same three for groups:

• real: who started it;
• effective: whose permissions it has right now (0 = root);
• saved: a spare, so a program that dropped privileges can get them back.

Plus an fs id (what new files get) and a list of supplementary groups. setuid, setreuid, setresuid, setfsuid, the group versions, getgroups/setgroups and the getters (102-123) follow Linux's rules: root may set anything; anyone else may only switch between the ids they already have. So a root daemon that does setuid(1000) has given up root for good, while setresuid(1000, 1000, 0) keeps 0 in the saved id and can come back with setuid(0). Ids are inherited by fork and kept by execve.

They are stored, not enforced: nothing checks permissions yet (stage 16 said the same about modes). id prints the truth, chown uses them, and new files get the fs ids, but uid 1000 can still read everything.

Priority. getpriority/setpriority (140/141) and nice keep a nice value per process, -20 (most important) to 19. Only root may lower it (EACCES). The scheduler doesn't look at it yet: every thread gets the same share.

Host name. sethostname/setdomainname (170/171) change what uname reports (the node name was always "qemu"); root only, at most 64 bytes.

Whole-file locks, vectors and sendfile

• flock (73) is the other kind of file lock (stage 16 did fcntl's). It locks the whole file, shared (LOCK_SH) or exclusive (LOCK_EX), with LOCK_NB for "don't wait" and LOCK_UN to let go. Its big difference from fcntl locks: it belongs to the open file (what one open() made, shared by dup and fork), not to the process, and it goes away when the last fd of that open file is closed. Two open()s of the same file by the same process conflict. This is what flock(1) and many package managers use.
• preadv/pwritev (295/296): readv/writev at an offset, without moving the file position.
• sendfile (40) copies from one fd to another inside the kernel, no user buffer in between. cp and web servers use it. The input must be something with a position (a file); the output can be anything.

Small fixes Clarke's traces found

• rm -r left files behind. getdents64 re-listed the directory on every call and skipped the entries it had already returned by count. rm -r deletes entries between calls, so the count was off and every other file was skipped. Now an open directory takes its listing once (until it's rewound) and rm sees every name. getdents64 also reports the same inode number stat does.
• --trace and interrupted calls. A call cut short by a signal used to be traced as = -512 or -514 (ERESTARTSYS, ERESTARTNOHAND), kernel-internal codes no program ever sees. It's now traced after the signal is delivered, with what the program really gets: = -4 EINTR, or = ? (interrupted, restarted) when SA_RESTART makes the kernel run it again.
• ppoll, pselect6, epoll_pwait now check the signal-mask size (must be 8, else EINVAL), as Linux does.
• The headless frame report (run-headless.sh) is printed after every process and thread has finished, not while the last ones are still exiting, so the numbers no longer jump around.

unlinkat, mkdirat and readlinkat were audited and were already right.

Testing it

user/coretest.rs has one group of checks per argument: links, owners, statfs, info, sockets, timers, ids and misc. tests/coreutils_batch2.rs runs each group twice; the second run must leave threads, processes, memory frames, file locks (flock ones too) and timers exactly where they were. From the shell: coretest timers.

What stage 17 does and doesn't do

Does: everything above, with Linux's numbers and error codes.

Doesn't (yet):

• Ids, modes and priorities are stored, not enforced. No permission checks, and the scheduler ignores nice values.
• Timers are coarse. timerd sleeps in PIT ticks (~55 ms), so a timer fires up to ~110 ms late (never early). A 10 ms periodic timer will mostly count overruns.
• Only ITIMER_REAL. ITIMER_VIRTUAL and ITIMER_PROF (CPU-time interval timers) are EINVAL, and so is timer_create on a CPU-time clock.
• socketpair, which needs AF_UNIX sockets; timerfd; and mremap (musl's malloc copes with ENOSYS) are left for later.
• link/symlink on FAT and exFAT are EPERM: those formats have nowhere to keep them.
• Rustbelt's own sysinfo stays at 502 (as rb_sysinfo), next to Linux's 99.

Unix sockets and /proc/self (stage 18)

Stage 18 is about programs talking to each other on the same machine. A Wayland compositor and its windows, D-Bus, X11 and most local servers do that through Unix domain sockets (AF_UNIX), so this stage is the plumbing a compositor will need. It also adds a small /proc: /proc/self/fd and /proc/self/exe, which musl's ttyname and realpath read. The code is in src/net/unix.rs and src/fs/procfs.rs.

Before that, a small first commit fixed what Clarke's review of 0.17.0 found; see "Small fixes from Clarke's review" below.

What a Unix socket is

You've met two ways for programs to talk already: pipes (stage 7) and network sockets (stage 12). A Unix socket sits between them. It uses the same calls as a network socket (socket, bind, listen, accept, connect, send, recv), but the "network" is just the kernel: sending means "copy the bytes into the other socket's queue and wake whoever is waiting for them". Compared with a pipe, a Unix socket

• goes both ways (a pipe is one-way);
• has a name other programs can find (see below), so one server can accept many clients, like a web server does;
• can carry open files from one program to another (SCM_RIGHTS);
• can tell you who is on the other end (SO_PEERCRED).

There are three kinds, picked by socket()'s second argument:

Type	Like	Message borders
SOCK_STREAM (1)	TCP, or a two-way pipe	not kept: two writes can come out as one read
SOCK_DGRAM (2)	UDP, but reliable and in order	kept: one send, one recv; no connection needed
SOCK_SEQPACKET (5)	a stream that keeps borders	kept, and connected like a stream

Wayland uses SOCK_STREAM; SOCK_SEQPACKET is what you pick when you want a connection and messages that stay whole.

Names: a file, an abstract name, or none

A Unix socket's address is a struct sockaddr_un: the family (1) and up to 108 bytes of name. There are three kinds of name:

• A path, like /tmp/echo.sock or Wayland's $XDG_RUNTIME_DIR/wayland-0. bind creates a socket file there (ls -l would show an s; stat says S_IFSOCK). The file is only a signpost: the data never touches the disk, and connect uses the file to find the listening socket. That works on the ramfs and on ext2, which both have an inode type for it. FAT and exFAT have no such type, so bind there is EPERM. You can't open a socket file (ENXIO), and it stays behind after the server exits until someone unlinks it, just like on Linux (that's why servers delete the old one before bind).
• An abstract name: the path starts with a zero byte. It lives only in the kernel, so nothing has to be cleaned up; it disappears when its socket closes. Tools print it with a leading @, like @rustbelt-test. bind with only the family and no name at all ("autobind") picks a free abstract name of five hex digits.
• Unnamed: a socket that never called bind, such as both ends of a socketpair. getpeername on those returns just the family.

socketpair(AF_UNIX, type, 0, fds) makes two sockets already connected to each other, with no names at all. It's the two-way pipe a parent makes before fork so it can talk to its child.

Connecting, and the "embryo"

listen turns a bound stream (or seqpacket) socket into a server. connect then builds a brand-new socket for the server side right away, links it to the client, and puts it in the listener's queue of waiting connections (the backlog). accept just takes the next one out of that queue and gives it an fd. Linux calls the not-yet-accepted socket an "embryo". So a client can connect and even send before the server gets round to accept; if the server closes without accepting, the client gets ECONNRESET. When the backlog is full, connect waits (or fails with EAGAIN on a nonblocking socket), as on Linux.

Backpressure: why a full queue makes you wait

Every socket has a receive queue. If a fast writer could add to it forever, it would eat all the kernel's memory, so each queue has a limit: the smaller of the sender's SO_SNDBUF and the receiver's SO_RCVBUF (208 KiB by default, like Linux). When the queue is full, write sleeps on a wait queue until the reader takes something, or returns EAGAIN when the socket is nonblocking, and poll stops reporting POLLOUT. That's called backpressure. As on Linux, setsockopt(SO_SNDBUF, 4096) reads back as 8192: the kernel doubles what you ask for to leave room for its own bookkeeping.

For datagram and seqpacket sockets every message also costs 256 bytes on top of its data, so a flood of empty messages fills the queue too. (Linux instead also limits a datagram queue to a number of messages; we only count bytes.)

One rule keeps it deadlock-free: the kernel never holds two sockets' locks at the same time. Two programs writing to each other at once can't each be waiting for the other's lock.

Passing open files: SCM_RIGHTS

This is the surprising one. sendmsg takes, next to the data, some control messages ("ancillary data"). With SCM_RIGHTS you put fd numbers in there, and the receiver's recvmsg gets new fds in its own table for the same open files: the same file position, the same pipe, the same socket. An fd number only means something inside one process; what travels is the open file behind it. A Wayland client hands the compositor shared-memory buffers this way.

Some details that matter, all Linux's behaviour:

• While a message is in a queue its files are in flight: they stay open even if the sender closes its own fds. If the receiving socket is closed with the message still unread, the kernel closes them, so nothing leaks (the tests check memory frames to be sure).
• If the receiver's control buffer is too small for all the fds, it gets the ones that fit, MSG_CTRUNC is set in msg_flags, and the rest are closed.
• MSG_CMSG_CLOEXEC in recvmsg's flags marks the new fds close-on-exec, like O_CLOEXEC.
• On a stream socket a read takes the bytes before a message that carries fds together with that message, fds attached, and then stops: one read never gets two batches of fds. (Stage 18 stopped in front of the fd message instead; stage 19 does what Linux does: "AB", then "CD"+fd reads as "ABCD" plus the fd. A plain read closes the fd.)

The control buffer has a fixed layout: a struct cmsghdr (length, level, type: 16 bytes) and then the data, padded to 8 bytes. That's why room for one fd is 20 bytes (CMSG_LEN(4)) but CMSG_SPACE(4) is 24, which, since the padding counts, is also enough for two.

Who's there: credentials

• getsockopt(SO_PEERCRED) gives the pid, uid and gid of the program at the other end, as they were when it connected. A server can use it to decide whom to trust.
• SCM_CREDENTIALS is a control message with the same three numbers. The sender may send its own (the kernel checks); with SO_PASSCRED on, the receiver gets them with every message even if the sender sent none.

poll, epoll and shutdown

Unix sockets work with poll, select and epoll like every other fd. POLLIN means there's data (or, on a listener, a connection to accept); POLLOUT means there's room. shutdown(SHUT_WR) tells the other end "no more data": its reads return 0 (end of file) and its poll reports POLLRDHUP; when the other end closes completely it's POLLHUP. Writing to a socket whose peer is gone is EPIPE plus SIGPIPE (or just EPIPE with MSG_NOSIGNAL).

/proc/self

On Linux /proc is a file system whose files are made up on the spot by the kernel. Rustbelt now mounts a tiny one at /proc (statfs magic 0x9fa0, like Linux). Every answer depends on who asks: the same path shows each process its own files.

• /proc/self/exe is a symlink to the running program (it used to be a special case inside readlink).
• /proc/self/fd/ lists your open fds, and each /proc/self/fd/N is a symlink to what fd N is: a path for a file, /dev/null or /dev/pts/0 for a device, and for things with no path Linux's made-up names pipe:[4001], socket:[123] and anon_inode:[eventpoll]. The number in brackets is the same as st_ino from fstat, so two fds for the same pipe can be recognised.
• Opening /proc/self/fd/N opens fd N's file again: a regular file gets a fresh open (its own position); a pipe or a tty gives you the same open file again. A socket can't be opened that way (ENXIO).

There's no /proc/<pid> for other processes yet, and /proc/self is a real directory rather than Linux's symlink to /proc/<pid>.

Small fixes from Clarke's review

Clarke's probes against 0.17.0 found three things; they went in as a separate commit before the sockets:

• POSIX timer overruns. While a timer's signal was still waiting to be delivered, further expirations replaced the overrun count instead of adding to it. Now they add up, and the count is latched when the signal is delivered, so timer_getoverrun reports the overrun of the signal you just got.
• The headless frame report moved. The timerd thread (stage 17) was started by the first timer, so its 16 frames showed up as a difference between "before the shell" and "after". It now starts at boot and the two numbers match again.
• "/tmp has /'s st_dev" turned out not to be a kernel bug. Clarke's C probe printed st.st_dev in the same printf call that also ran stat("/", &st), and C doesn't say in which order a function's arguments are evaluated. The kernel was already right; coretest now checks stat, lstat and fstat of mount points so it stays that way.

A known flake, for the record: about 3% of runs of tests/libc.rs's threads_and_futexes fail. It happened before this stage too (at 0.17.0) and rerunning passes; it's on the list to track down.

Testing it

user/unixtest.rs has one group of checks per argument: stream, pair, dgram, rights, nonblock and proc. tests/unix_sockets.rs runs each group twice; the second run must leave threads, processes, memory frames and bound socket names exactly where they were (so an fd stuck "in flight" would show up as a leak). From the shell: unixtest rights.

What stage 18 does and doesn't do

Does: everything above, with Linux's numbers, structures and error codes.

Doesn't (yet):

• No garbage collector for socket cycles. If you send a Unix socket's fd through that same socket (or two sockets through each other) and then close everything, the messages keep the sockets alive and they're never freed. Linux has a special garbage collector for exactly this; real programs almost never do it.
• Datagram queues are limited by bytes only, not also by a message count like Linux's.
• SO_PASSCRED doesn't autobind an unnamed socket (Linux gives it an abstract name).
• Permissions on socket files aren't checked, like every other permission so far.
• sendmsg/recvmsg on network sockets carry the name and data, but no control messages.
• /proc has only self/fd and self/exe: no /proc/<pid>, no /proc/mounts (the mounts call 501 still does that job), and /proc/self is a directory, not a symlink.

The vDSO, static PIE and mremap (stage 19)

Stage 19 is about running ordinary C programs, built with a normal C library (musl), faster and with fewer special tricks. Three things:

1. the vDSO, so asking for the time doesn't need a system call;
2. static PIE programs, which is what musl-gcc -static-pie makes;
3. mremap, so realloc of a big block doesn't copy it.

Before those, two commits fixed old problems: the threads_and_futexes flake and a list from Clarke's review of 0.18.0. They're described at the end ("The flake" and "Small fixes from Clarke's probe").

The C programs are compiled by build.rs with musl-gcc, which comes in the musl-tools package: sudo apt install musl-tools.

Why a system call for the time is too slow

A program asks "what time is it?" all the time: for every log line, every timeout, every frame of a game. Until now each of those was a system call: switch to the kernel, save registers, look up the call, read the clock, switch back. Reading the clock is the cheap part.

The CPU has a counter that anyone may read with one instruction, rdtsc (the time stamp counter, TSC). The kernel already turns it into time (src/time.rs): at boot it measures how many TSC ticks there are per millisecond, and from then on

nanoseconds = (tsc_now - tsc_at_boot) * mult >> 32

where mult is "nanoseconds per tick" as a fixed-point number (a whole number scaled up by 2^32, because the kernel avoids floating point). A program could do this sum itself, if it knew tsc_at_boot and mult. That's what the vDSO is for.

The vDSO: a library the kernel gives every program

vDSO means "virtual dynamic shared object": a tiny shared library (.so) that isn't a file anywhere. Its source is vdso/vdso.c; build.rs compiles it (with the linker script vdso/vdso.lds) into a one-page ELF file, and the kernel embeds that with include_bytes! (src/vdso.rs). Every process gets it mapped into its address space, and the kernel tells the program where with an entry in the auxiliary vector (the list of facts on the initial stack, stage 14): AT_SYSINFO_EHDR = the address of the vDSO's ELF header.

musl looks there at start-up, finds the function __vdso_clock_gettime in the vDSO's symbol table (with the version name LINUX_2.6, which is what Linux uses), and from then on clock_gettime() calls it instead of making a system call. The vDSO has four functions, the same as Linux's:

Function	What it does	Without the fast path
__vdso_clock_gettime	CLOCK_REALTIME, CLOCK_MONOTONIC and their _COARSE/_RAW versions from the TSC	system call 228; other clocks (CPU time, BOOTTIME) always use it
__vdso_gettimeofday	the wall clock in microseconds	system call 96
__vdso_time	the wall clock in seconds	system call 201
__vdso_getcpu	which CPU am I on? (sched_getcpu()) via rdtscp	system call 309

The three system calls 96, 201 and 309 are new too: the vDSO needs something to fall back on, and old programs call them directly.

The vvar page and its seqlock

The numbers the vDSO needs (tsc_at_boot, mult, the wall-clock time at boot, and a "you may use the TSC" flag) live in the vvar page: one 4 KiB page that the kernel writes and every process can only read. It sits right in front of the vDSO's code, so the vDSO finds it at "my own address minus 4096" without any relocation. Its layout is struct vvar in vdso/vdso.c, and the same thing as VVar in src/vdso.rs; the two must match.

But what if the kernel changes the numbers (say, after calibrating the clock again) while a program is halfway through reading them? It could get a new mult with an old tsc_at_boot and compute nonsense. The fix is a seqlock ("sequence lock"), which costs readers almost nothing:

• The page starts with a counter, seq. The kernel makes it odd before it changes anything and even again afterwards.
• A reader reads seq, then the numbers, then seq again. If the first value was odd, or the two differ, a write happened in between: it simply tries again.

Readers never block and never write anything, which is why the page can be read-only. mprotect refuses to make it writable (EACCES), and a program that writes to it anyway gets SIGSEGV.

The vDSO's code and the vvar page are the same physical frames in every process. They're mapped at a fixed place just below the stack's guard zone (VVAR_ADDR, then VDSO_ADDR 4 KiB above it; 64 KiB are kept for them), and mmap never hands out that range. They survive fork like any other page and are mapped again by execve.

Can we trust the TSC?

On some older CPUs the TSC changes speed with the CPU's clock (to save power), so ticks don't mean time. Newer CPUs promise an invariant TSC, shown by a CPUID bit, and Linux only uses the vDSO fast path then.

Surprise: QEMU without KVM (TCG, what the tests run on) doesn't offer that bit, not even with -cpu max,+invtsc; it prints a warning and leaves it off. Its TSC is still steady, because it's just QEMU's own clock. So Rustbelt trusts the TSC when it's invariant or when CPUID says "you're running under a hypervisor". The rule is tsc_trusted() in src/vdso.rs; on a real old CPU without the bit the vDSO just makes the system call. For the tests, vdso::force_syscall(true) switches the fast path off, so the fallback gets tested too.

While at it, the kernel clock went from microseconds to nanoseconds (clock_getres now says 1 ns), so the vDSO and the system call give the same answer to the nanosecond.

Static PIE: programs that can live anywhere

Until now every program was an ET_EXEC file: linked to run at one fixed address, 0x80_0040_0000, which Rustbelt's linker script (user/link.ld) chooses. A PIE ("position-independent executable", ELF type ET_DYN) can be loaded anywhere. It carries a list of relocations: places in its data that hold addresses and must be adjusted by "where I actually got loaded". Normally the dynamic loader (ld.so) does that, but a static PIE fixes itself up: musl's start code (_dlstart) applies its own relocations before main.

What the kernel does for one (src/elf.rs):

• It picks a random base, a whole number of pages above USER_START + 4 MiB (up to 1 GiB higher), and loads every segment at "base + the address in the file". A different base each run makes attacks that guess addresses harder (ASLR).
• It tells the program where it ended up: AT_PHDR (its program headers), AT_ENTRY (its entry point) and AT_BASE = 0 (no dynamic loader, as on Linux).
• PT_GNU_STACK says whether the stack may be executable (almost never; respected now), PT_TLS is the template for thread-local variables (musl finds it through AT_PHDR), and PT_GNU_RELRO marks data a dynamic loader would make read-only after relocating; both are accepted.
• A program that needs a dynamic loader (PT_INTERP, made by plain musl-gcc without -static) can't run here: execve says ENOEXEC.

user/piehello.c checks its own relocations and auxv entries; user/piethreads.c runs pthreads, __thread variables, stdio and ttyname as a static PIE.

Why not a plain musl-gcc -static program? That makes an ET_EXEC linked at 0x400000, the usual Linux address. Rustbelt's user space is only one 512 GiB slot of the address space, starting at 0x80_0000_0000 (the rest is the kernel's), so 0x400000 is simply not available, and an ET_EXEC can't be moved. To build a classic ET_EXEC anyway, build.rs links it at our address: -static -no-pie -nostartfiles with musl's crt1.o/crti.o/crtn.o named by hand, -Wl,-Ttext-segment=0x8000400000, -Wl,--defsym=_DYNAMIC=0x8000400000 and -Wl,--no-relax. (Debian's musl start files are compiled as position-independent code, which is why the extra flags are needed.) vdsotest is built that way. For your own programs, -static-pie is much easier and just works.

mremap: growing memory without copying

mremap(old, old_len, new_len, flags, new_addr) makes an mmap region bigger or smaller. musl's realloc uses it for big blocks (which get their own mmap): without it, growing a 4 MiB block meant allocating a new one and copying all 4 MiB.

• Smaller: the end is unmapped.
• Bigger, with free space right behind it: the region just grows.
• Bigger, no room: with MREMAP_MAYMOVE the kernel finds a new place, otherwise it's ENOMEM. Moving copies nothing: the page-table entries are taken out at the old address and put back at the new one (Mm::remap in src/mm.rs), so the very same frames, still shared copy-on-write with a forked child if they were, just show up somewhere else.
• MREMAP_FIXED says exactly where to move it.

Only mmap regions can be resized (not the program, the heap, the stack, the vvar page or the vDSO). user/remaptest.c checks all of this, plus realloc up to 32 MiB.

The flake

For a few stages, about one run in 30 of tests/libc.rs's threads_and_futexes failed, and rerunning passed. Running the threads program 150 times per boot made it hang every time, which gave something to look at. It was three separate races between CPUs:

1. clone set up "wake my joiner when I exit" too late. The new thread was already running when clone stored that address (CLONE_CHILD_CLEARTID). A very short thread could finish first, so nobody woke the thread waiting in pthread_join.
2. A deadlock between a lock and a TLB shootdown. One CPU held a process's memory lock and waited for every other CPU to say "I've forgotten those pages" (a TLB shootdown, stage 14). Another CPU of the same process wanted that lock and spun for it with interrupts off, so it never heard the request. Each waited for the other. Now a CPU waiting for that lock answers shootdowns while it waits.
3. A lost futex wake. futex_wait read the futex value and then joined the waiting list; a wake in between was lost. It now reads the value while holding the list's lock, as Linux does.

tests/threads_stress.rs runs threads 40 times per boot and prints every thread and futex waiter if one hangs.

Small fixes from Clarke's probe

Clarke's tests of 0.18.0 found a few differences from Linux:

• Device numbers. Every /dev file now has Linux's major/minor numbers (st_rdev: /dev/null is 1:3, /dev/zero (new) 1:5, /dev/pts/N 136:N, ...), and fstat of an open device gives the same answer as stat of its /dev path. musl's ttyname() (and so tty) compares exactly those numbers, so it works now.
• Unix sockets and fds: a stream read takes the bytes in front of an fd-carrying message together with it, like Linux (see the note in the stage 18 chapter).
• /proc/self/cwd is a link to the current directory.
• Autobind starts at a random name, like Linux.

Testing it

Test file	Program it runs	What it shows
tests/vdso.rs	vdsotest	the vDSO and the system calls agree and never go backwards; --trace shows no clock system calls at all; it works after fork and exec
tests/static_pie.rs	piehello, piethreads, dynhello	static PIEs load at random bases, threads and stdio work, PT_INTERP is refused
tests/mremap.rs	remaptest	every mremap case, and realloc
tests/threads_stress.rs	threads	the flake stays fixed

From the shell: vdsotest, piehello, remaptest.

What stage 19 does and doesn't do

Does: everything above, with Linux's numbers and names.

Doesn't (yet):

• No dynamic loader: programs with PT_INTERP (dynamically linked) get ENOEXEC. Use -static-pie.
• Stock musl-gcc -static (ET_EXEC at 0x400000) can't run; see above for the flags that make an ET_EXEC that can. (Done in stage 20.)
• timerfd isn't there yet: it needs a new kind of file that poll and epoll can wait on, and the timers (stage 17) only tick every ~55 ms anyway. Next on the list. (Done in stage 20.)
• mremap: no MREMAP_DONTUNMAP, no old_len 0 (a second view of a shared mapping), and only mmap regions.
• The clock isn't adjusted: no NTP, no settimeofday; the vvar page is written once at boot.
• /dev/tty opened as a file fstats as the terminal it stands for (like /dev/pts/0), not as /dev/tty itself.

Plain static programs and timerfd (stage 20)

Stage 20 is two small follow-ups to stage 19, kept deliberately small: no new subsystems and no new dependencies.

1. A program built the most ordinary way, musl-gcc -static hello.c, now runs unmodified.
2. timerfd: a timer that is a file, so an event loop can wait for it with poll/select/epoll like any other fd.

Why gcc -static programs couldn't run

A classic Linux program (ELF type ET_EXEC) is linked to run at one fixed address, and every Linux linker picks 0x400000 (4 MiB) for it. It can't be moved: its code has that address baked in.

Until now that address belonged to the kernel. bootloader 0.9 loads the kernel wherever its ELF file says, and our kernel was linked at 0x200000 (2 MiB), spilling up to about 5.5 MiB. So user space had to live somewhere else: level-4 slot 1, starting at 512 GiB. Stage 19 got around that with special link flags (see "Static PIE" above).

Moving the kernel out of the way

Two small changes, the minimal way:

• The kernel is linked at 0x200_0000_0000 (2 TiB, level-4 slot 4) instead of 0x200000. That's one line in x86_64-rust_kernel.json: "pre-link-args": { "ld.lld": ["--image-base=0x20000000000"] }. The bootloader reads the address from the ELF file, so nothing else changes; the slots it picks for its own things (boot stack in slot 2, physical memory in slot 3) stay the same.
• User space now starts at 4 MiB: USER_START is 0x40_0000, and user space covers level-4 slots 0 and 1 (up to USER_END, 1 TiB, as before).

There's one catch: the first 2 MiB still belong to the kernel. The VGA text buffer at 0xb8000, the page where the other CPUs start (0x8000, see smp.rs) and the bootloader's own page tables are all there, identity-mapped (virtual address = physical address). So each process gets its own level-3 and level-2 table for slot 0, with one entry copied from the kernel: the one for 0..2 MiB. It isn't USER_ACCESSIBLE, so ring 3 still can't touch it. 2..4 MiB is left empty, so a NULL pointer plus a big offset still faults.

0x0000_0000_0000_0000   0..2 MiB: kernel only (VGA, SMP trampoline)     shared by everyone
0x0000_0000_0020_0000   2..4 MiB: nothing (a gap)
0x0000_0000_0040_0000   USER_START: a plain `gcc -static` program, then its brk heap
0x0000_0080_0040_0000   our Rust programs and static PIEs (unchanged)
                        ... mmap regions, handed out downwards
0x0000_00ff_ffef_f000   user stack (unchanged), vDSO and vvar just under it
0x0000_0100_0000_0000   USER_END
0x0000_0200_0000_0000   the kernel's code and data (was 0x200000)

The page-table code in memory.rs that walks a process's user pages (clone_user_space for fork, clear_user_space for exec and exit) now walks both slots and skips the kernel's 0..2 MiB entry. Each time it makes a new address space it also checks that the kernel really maps nothing else in slot 0, and panics with a clear message if it ever does.

Everything else stayed where it was: Rust programs (user/link.ld), static PIEs (random base just above 512 GiB), the stack, the vDSO, mmap from the top down. The heap of an ET_EXEC program starts right after its image (brk), and mmap memory comes from near 1 TiB, so the two can't run into the image or each other.

build.rs lost all its special ET_EXEC flags: vdsotest and the no-libc rawabi are now built with the stock recipe (musl-gcc -static, cc -static -nostdlib -no-pie). evil and hello now poke at the kernel's new address (0x200_0000_0000) to check that ring 3 can't read kernel memory.

timerfd: a timer you can read

Stage 17's timers deliver a signal when they expire. Signals are awkward in an event loop (they interrupt whatever you're doing). A timerfd is the friendlier version: it's an fd, and

• every expiration adds 1 to a counter;
• read(fd, &n, 8) returns that counter as an 8-byte number and resets it to 0 (it waits if it's 0, or says EAGAIN with TFD_NONBLOCK; a buffer smaller than 8 bytes is EINVAL);
• poll/select/epoll report it readable while the counter isn't 0.

int fd = timerfd_create(CLOCK_MONOTONIC, 0);
struct itimerspec its = { .it_value = {1, 0}, .it_interval = {1, 0} };
timerfd_settime(fd, 0, &its, NULL);   /* in 1 s, then every second */
uint64_t n;
read(fd, &n, 8);                      /* waits; n = expirations since last read */

How it's built, reusing what we had:

• The timer is an entry in the same table as the stage 17 timers (src/timer.rs), and the same kernel thread, timerd, fires it. Instead of sending a signal, firing adds to the entry's counter, wakes readers, and calls poll::notify() so poll/select/epoll look again.
• The fd is a small File (TimerFd, also in timer.rs). Its read and poll look at the counter. When the last fd for it is closed, the file is dropped and its timer is deleted too. A timerfd isn't tied to a process: it survives execve, and fork shares it, as on Linux.
• Clocks: CLOCK_REALTIME, CLOCK_MONOTONIC, and CLOCK_BOOTTIME (the same as MONOTONIC: Rustbelt never sleeps). TFD_TIMER_ABSTIME means "at this time" instead of "in this long". TFD_TIMER_CANCEL_ON_SET (wake me if someone changes the clock) is EINVAL: nobody can set our clock yet anyway.

Resolution: like all our timers, a timerfd is checked on PIT ticks, about every 55 ms. It never fires early, but it can fire up to ~110 ms late. A 10 ms interval timer still counts correctly: when timerd wakes up late, it adds all the periods that went by.

Testing it

Test file	Program it runs	What it shows
tests/static_pie.rs	exechello, execthreads	stock musl-gcc -static programs (ET_EXEC at 0x400000) run; brk, mmap and malloc memory stays out of the image; threads and stdio work; no leaks
tests/timerfd.rs	timerfdtest	one-shot, interval counts adding up, EAGAIN, EINVAL, poll/select/epoll wake-ups, TFD_TIMER_ABSTIME; closing leaves no timer and no memory behind

execthreads is the same source as piethreads (user/piethreads.c), just built as a plain ET_EXEC. From the shell: exechello, execthreads, timerfdtest.

What stage 20 does and doesn't do

Does: everything above.

Doesn't (yet):

• The kernel isn't in the upper half (0xffff_8000_...) like on Linux; it just moved to 2 TiB. User space is the lowest 1 TiB, minus the first 4 MiB.
• timerfd: no TFD_TIMER_CANCEL_ON_SET, no TFD_IOC_SET_TICKS ioctl, and the ~55 ms resolution above.
• Still no dynamic loader (PT_INTERP gets ENOEXEC).

Finding the root disk and starting init (stage 21)

Stage 21 is a small set of requests from Clarke's Foundry OS, the operating system he builds on top of this kernel. Same rule as stage 20: no new subsystems and no new dependencies. Four things:

1. The kernel can find its root filesystem on a disk and start the OS's own init program as process 1 ("boot contract v2").
2. An exFAT share partition is mounted at /mnt/share by itself.
3. msync (system call 26).
4. RUSTBELT_DISK0: build with somebody else's disk image as drive B:, and the scripts honour CARGO_TARGET_DIR.

What "root" and "init" mean

On Linux, the kernel's last job at boot is to find the root filesystem (the one that becomes /) and run one program from it: init, which gets process id 1. Everything else (shells, services, logins) is started by init or by its children. If init ever exits, the system is over.

Until now Rustbelt did all of that itself: / was a ramfs inside the kernel, and main.rs started our own shell (sh /scripts/selftest.sh in a headless build). That's still exactly what happens on a normal boot. Boot contract v2 only switches on when a disk says it should.

How the kernel finds the root

Every GPT partition has a type GUID: a 16-byte id that says what the partition is for. The Discoverable Partitions Specification reserves one for "the root filesystem of an x86-64 system":

4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709    Linux root (x86-64)

So at boot (mount_disks in src/fs/mod.rs):

• block/mod.rs now remembers the type of every partition it found.
• If a partition has the root type and holds an ext2 filesystem, it is mounted at / instead of the ramfs (replace_root). The kernel's own mounts stay: /tmp (ramfs), /dev and /proc are separate mounts, so they still work; the directories are made if the disk doesn't have them.
• The root partition doesn't get a /mnt/diskN name, so the other disks keep their numbers: the FAT32 data disk is still /mnt/disk0 (drive B:).

GUIDs are stored "mixed-endian" on disk: the first three groups are little-endian, so 4F68BCE3-E8CD-4DB1-... is the bytes e3 bc 68 4f cd e8 b1 4d 96 e7 .... That's how it's written in block/partition.rs.

How init starts (src/boot.rs)

With a root partition, main.rs doesn't start our shell. It tries, in order:

/sbin/init   ->   /bin/init   ->   /bin/sh

The first one that starts becomes pid 1 (it's the first process the kernel makes). A missing file is skipped quietly; a file that exists but can't run (not an ELF file, say) is logged and skipped too. If none of the three works the kernel prints [init] no init found and QEMU exits with "failed".

What init gets, by default:

argv	init (or sh for the /bin/sh fallback)
argv in a headless build	init selftest (or sh /scripts/selftest.sh, like before)
environment	PATH=/sbin:/bin:/usr/bin:/mnt/disk0/bin, HOME=/, TERM=linux
environment in a headless build	the same plus RUSTBELT_HEADLESS=1

Orphaned processes are adopted by pid 1 when it is called init (stage 16's rule), so init can reap them like on Linux.

The knobs

bootloader 0.9 has no kernel command line (there's nowhere to type init=/bin/foo like on Linux), so the knobs are set when the kernel is built:

Knob	What it does
cargo build --features headless	init gets selftest as argv[1] and RUSTBELT_HEADLESS=1
RUSTBELT_INIT_ARGS="a b c" (env var while building)	more arguments after those, split on spaces
RUSTBELT_INIT_ENV="K=V K2=V2" (env var while building)	more environment; replaces a default with the same name (so RUSTBELT_INIT_ENV="PATH=/bin" changes PATH)
RUSTBELT_DISK0=/path/to/disk.img (env var while building)	use that image as drive B: (see below)

For example:

RUSTBELT_INIT_ARGS="--verbose" RUSTBELT_INIT_ENV="LANG=C.UTF-8" cargo build --features headless

init's exit status becomes QEMU's

When pid 1 exits, the run is over (init isn't restarted). The kernel hands init's status to QEMU through the isa-debug-exit device, which makes QEMU exit with value * 2 + 1:

init did	value written	QEMU's exit code
exit(0)	0x10	33 (the usual "success")
exit(1)	0x11	35 (the usual "failure")
exit(n), n from 0 to 111	0x10 + n	33 + 2n
exit(n) with a bigger n, or killed by a signal	0x11	35

So a script can get init's code back with (code - 33) / 2. (Codes up to 111, because QEMU's exit code can't be more than 255.)

The exFAT share

A partition with the Microsoft basic data type (EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, the one Windows uses for FAT, exFAT and NTFS) that holds an exFAT filesystem (its boot sector says EXFAT ) is mounted at /mnt/share. That's handy for moving files between the host and the guest: make a partition, copy files in with exfatprogs or a file manager, boot. If there's no such partition, nothing happens and nothing is printed. Like the root, it doesn't use up a /mnt/diskN number.

It's mounted read-write. The exFAT driver (stage 9) creates, writes, appends, truncates, makes and removes directories and deletes files, keeping the cluster bitmap, FAT chains and checksums right, and fsck.exfat has found it clean after every write test since stage 9 (scripts/exfat-check.sh, and the stage 21 check below). What it doesn't do: rename, file times, and the "volume dirty" flag (a crash in the middle of a write could leave the volume needing an fsck). A TexFAT volume with two FATs is still mounted read-only.

msync

msync(addr, len, flags) asks the kernel to write a shared file mapping back to its file. In Rustbelt there's nothing to write back: mmap of a file with MAP_SHARED is refused (only private copies of files, and shared anonymous memory, exist). So msync only checks its arguments, like Linux does:

• EINVAL if addr isn't page-aligned, if flags has an unknown bit, or if it has both MS_SYNC and MS_ASYNC;
• ENOMEM if any page in the range isn't mapped (mm.rs's covers);
• otherwise 0, with len rounded up to whole pages (len 0 is fine).

RUSTBELT_DISK0 and CARGO_TARGET_DIR

Clarke builds his own drive B: image. With RUSTBELT_DISK0=/path/to/foundry.img set while building, build.rs copies that file to disk.img byte for byte instead of making our FAT32 disk, and skips building the programs that only live on our disk (sh, ls, cat and friends). Without it, nothing changes.

Cargo puts everything it builds in target/, unless CARGO_TARGET_DIR says somewhere else. The disk images used to be written to the right place but looked for in target/. Now cargo run/cargo test go through scripts/qemu.sh (a tiny wrapper around QEMU set as bootimage's run-command in Cargo.toml), which swaps target/ for $CARGO_TARGET_DIR/, and every script starts with T="${CARGO_TARGET_DIR:-target}".

Testing it

build.rs makes one more image, target/rootfs-gpt.img (22 MiB): a GPT disk with two partitions, made with the same host tools as before (sfdisk, mke2fs, mkfs.exfat):

Partition	Type	Contents
1 (12 MiB)	Linux root (x86-64)	ext2: /sbin/init, /bin/init and /bin/sh (all user/inittest.c), /etc/rootfs-v2, /etc/init-status = 7, and empty tmp, dev, proc, mnt
2 (8 MiB)	Microsoft basic data	exFAT "SHARE" with hello.txt

inittest looks at how it was started. As /sbin/init it checks that it is pid 1, its argv and PATH, that / is the ext2 partition, that /tmp, /dev, /proc and /mnt/disk0 are there, reads /mnt/share/hello.txt and writes a file next to it, and tries msync the right and wrong ways; then it exits with the number in /etc/init-status (7), or 1 if anything failed. As /bin/init it exits 21, as /bin/sh 22, so a test can tell which one ran.

Test	What it shows
tests/boot_v2.rs	(only this test kernel gets rootfs-gpt.img, from scripts/qemu.sh) / is the ext2 partition, /mnt/share is the exFAT one and writable, /tmp is still the ramfs, B: is still /mnt/disk0; /sbin/init runs as pid 1 and returns 7 (all of inittest's checks, msync included) with no leaks; delete /sbin/init and /bin/init runs, then /bin/sh, then ENOENT; the argv/env above; the exit-code table
tests/fat.rs (stage 21 part)	with no root partition: / is the ramfs, no /mnt/share, the disks are numbered as before
scripts/boot-v2-check.sh	the real headless kernel with a writable copy of the image: init runs, QEMU exits 47 (= exit(7)), and e2fsck/fsck.exfat find both partitions clean afterwards

scripts/run-headless.sh doesn't attach the image, so it checks that a normal boot is exactly what it was.

What stage 21 does and doesn't do

Does: everything above.

Doesn't (yet):

• No kernel command line: knobs are fixed when the kernel is built.
• The root must be ext2 (not exFAT, FAT or btrfs), and the first matching partition wins; no root= to pick another.
• pid 1 isn't restarted when it exits; the run ends.
• msync writes nothing back, because there are no shared file mappings to write.
• exFAT still has no rename, file times or dirty flag.

Roadmap

Done: all 12 chapters of the 2nd edition, plus preemptive kernel threads, user mode + syscalls, ELF loading, a VFS with a ramfs, PCI, virtio-blk, MBR/GPT partitions, FAT32, blocking threads, pipes, spawn/wait, process memory cleanup, the shell, hand-written ext2, exFAT and (read-only) btrfs drivers, multi-core support with a sleep/wake scheduler and guard pages, networking (virtio-net + smoltcp + sockets), and the virtual drivers (virtio-rng, virtio-console, virtio-gpu 2D with a framebuffer console), the process model (fork/exec/mmap/signals), and the kernel side of a libc (poll/epoll, nonblocking I/O, loopback, threads and futexes, ttys and ptys), musl's prerequisites (per-thread FPU/SSE/AVX state, Linux C-string path calls and struct stat), the first batch of the calls musl and dash use at run time, and a second batch for coreutils and busybox (links, owners, statfs, CPU times, timers, user ids), and Unix sockets with /proc/self (stage 18).

• Freestanding binary — phil-opp

• Minimal kernel booting in QEMU — phil-opp, OSDev: Bare Bones

• VGA text mode printing — phil-opp, OSDev: Printing to Screen

• Serial output + QEMU exit — phil-opp: Testing, OSDev: Serial Ports

• CPU exceptions / IDT — breakpoint & page fault handlers. phil-opp: CPU Exceptions, OSDev: IDT, OSDev: Exceptions

• GDT / TSS / double faults — separate IST stack so stack overflows don't triple-fault. phil-opp: Double Faults, OSDev: GDT, OSDev: TSS

• Testing — cargo test in QEMU, custom test framework, integration tests. phil-opp: Testing

• Hardware interrupts — 8259 PIC, timer, PS/2 keyboard echo, hlt idle loop. phil-opp: Hardware Interrupts, OSDev: 8259 PIC, OSDev: PIT, OSDev: PS/2 Keyboard (later: APIC)

• Introduction to paging — CR3, page fault demo. phil-opp, OSDev: Paging

• Paging implementation — OffsetPageTable, translation, frame allocator, example mapping. phil-opp, OSDev: Page Frame Allocation

• Heap allocation — alloc, heap mapping, global allocator. phil-opp, OSDev: Memory Allocation

• Allocator designs — bump, linked list (crate), fixed-size block (active). phil-opp

• Async/await — Task, simple executor, waker-based executor, lock-free async keyboard task (with Backspace). phil-opp: Async/Await, OSDev: Multitasking Systems

• Preemptive kernel threads — own stacks, timer-driven round-robin, asm context switch. OSDev: Brendan's Multi-tasking Tutorial, OSDev: Context Switching

• User mode (ring 3) & syscalls — GDT user segments, TSS.rsp0 per thread, syscall/sysret, per-process page tables, user faults kill the process. OSDev: Getting to Ring 3, OSDev: System Calls, OSDev: SYSENTER/SYSCALL

• ELF loading — hand-written PT_LOAD loader, user programs built by build.rs. OSDev: ELF, OSDev: ELF Tutorial

• VFS skeleton + ramfs — Inode/FileSystem/File traits, mount table, fd table, open/read/write/close. OSDev: VFS, Linux VFS overview

• PCI enumeration + virtio-blk driver + BlockDevice trait — OSDev: PCI, OSDev: Virtio, virtio spec (legacy interface)

• MBR + GPT partitions — OSDev: MBR, OSDev: GPT

• FAT32 through fatfs, mounted at /mnt/disk0; programs loaded from disk — OSDev: FAT

• Blocking threads + wait queues, pipes, spawn/wait, cwd, stat/lseek/getdents64 — OSDev: Blocking Process

• Shell as a user program: line editing, builtins, PATH, pipes, redirection, background jobs, drive letters, run --trace — OSDev: Shell

• Process cleanup: frame free list, address spaces freed on reap

• ext2, hand-written: read + write, indirect blocks, symlinks, 1/4 KiB blocks, revision 0/1, checked with e2fsck — OSDev: Ext2, The Second Extended File System (Dave Poirier)

• exFAT, hand-written: read + write, entry sets + checksums, up-case table, NoFatChain and FAT chains, checked with fsck.exfat — Microsoft exFAT specification

• SMP (stage 11): ACPI MADT, local APIC + timer, INIT/SIPI trampoline, per-CPU GDT/TSS/swapgs data, a scheduler for 4 CPUs with sleep/wake and wait queues, frames freed on every exit, guard pages under kernel and user stacks — OSDev: SMP, OSDev: APIC, OSDev: MADT, OSDev: APIC timer

• btrfs, hand-written, read-only: chunk tree (single/DUP), root/FS/csum trees, crc32c everywhere, zlib/lzo/zstd, holes, subvolumes, checked against btrfs check and the host's files — btrfs on-disk format

• Networking (stage 12): virtio-net (legacy, interrupt-driven), smoltcp for ARP/IPv4/ICMP/UDP/TCP/DHCP/DNS, BSD-style socket syscalls with blocking on wait queues, ifconfig/ping/host/fetch/httpd — OSDev: Virtio, smoltcp, OSDev: Networking, Beej's Guide to Network Programming

• Virtual drivers (stage 13): a modern (1.0) virtio transport shared by virtio-rng (getrandom, /dev/random, smoltcp's seed), virtio-console (/dev/hvc0, getty) and virtio-gpu 2D (a 1024x768 framebuffer, a bitmap-font console, /dev/fb0, tile) — virtio 1.2 spec, OSDev: Virtio, Linux virtio-gpu driver

• Processes (stage 14): fork with copy-on-write (refcounted frames), execve with Linux's argv/envp/auxv stack, mmap/munmap/mprotect/brk with demand paging, signals with Linux signal frames and sigreturn, wait4, dup2/pipe2/fcntl, process groups, Ctrl+C and job control in the shell — man 2 fork, man 2 execve, man 7 signal, LWN: getauxval() and the auxiliary vector, OSDev: Paging

• Libc plumbing (stage 15): poll/ppoll/select/pselect6/epoll, O_NONBLOCK + EAGAIN, a loopback interface, futex, clone threads with TLS (arch_prctl), set_tid_address, gettid/tgkill, exit vs exit_group, ttys and ptys with termios, /dev/tty, /dev/ptmx + /dev/pts/N, a fork+exec getty, nanosleep, CLOCK_REALTIME, readv/writev, pread/pwrite — man 7 epoll, man 2 futex, Drepper: Futexes Are Tricky, The TTY demystified, man 7 pty

• musl prerequisites (stage 15.1): per-thread FPU/SSE/AVX state with XSAVE/FXSAVE (eager switching, cleared on exec), Linux's fpstate in signal frames, #MF/#XM -> SIGFPE, C-string paths with PATH_MAX and EFAULT, Linux struct stat, the *at() calls, rename, a syscall-number audit, and rawabi (a no-libc gcc binary) — Intel SDM vol. 1 ch. 13: XSAVE, OSDev: SSE, man 2 openat, man 2 stat, LWN: LazyFP

• libc batch 1 (stage 16): rt_sigsuspend/rt_sigpending/rt_sigtimedwait/rt_sigqueueinfo, queued real-time signals, sigaltstack, per-process umask and rlimits (prlimit64), fcntl record locks with deadlock detection and honest F_GETFL, truncate/ftruncate/fsync/chmod/utimensat with real file times, prctl, sched_getaffinity, madvise, robust futex lists, subreapers and self-reaping orphans, --trace naming unknown calls by number — man 2 sigsuspend, man 2 fcntl, man 2 sigaltstack, man 2 prlimit, man 2 get_robust_list

• coreutils batch 2 (stage 17): link/symlink and their *at forms, native rename of full directories and RENAME_NOREPLACE, the chown family with owners on ramfs and ext2, statfs/fstatfs/statx with Linux's magic numbers and a real st_dev/st_nlink, Linux's sysinfo (99), times/getrusage/CPU-time clocks/clock_getres, getsockname/getpeername/getsockopt/shutdown and remembered socket options, alarm/setitimer/POSIX timers, real (unenforced) user and group ids, nice values, sethostname, flock, preadv/pwritev, sendfile, and a getdents64 that rm -r can trust — man 2 link, man 2 statfs, man 2 timer_create, man 7 credentials, man 2 flock

• Unix sockets (stage 18): AF_UNIX stream/dgram/seqpacket with path, abstract and unnamed names, socketpair, sendmsg/recvmsg with SCM_RIGHTS (in-flight fds, MSG_CTRUNC, MSG_CMSG_CLOEXEC) and SCM_CREDENTIALS, SO_PEERCRED/SO_PASSCRED, real buffer limits with backpressure, poll/epoll with POLLRDHUP/POLLHUP, and /proc/self/fd + /proc/self/exe — man 7 unix, man 3 cmsg, man 2 sendmsg, man 5 proc, Wayland protocol: wire format

• The vDSO, static PIE and mremap (stage 19): __vdso_clock_gettime/gettimeofday/time/getcpu with a seqlocked vvar page and AT_SYSINFO_EHDR, a nanosecond kernel clock, ET_DYN programs without PT_INTERP at a random base (AT_PHDR/AT_ENTRY/AT_BASE, PT_GNU_STACK/PT_TLS/PT_GNU_RELRO), musl-gcc C programs in the build, mremap with MREMAP_MAYMOVE/MREMAP_FIXED, the threads flake fixed, Linux device numbers and ttyname — man 7 vdso, man 2 mremap

• Plain static programs and timerfd (stage 20): the kernel moved to 0x200_0000_0000 so user space starts at 4 MiB and stock musl-gcc -static ET_EXEC binaries run unmodified; timerfd_create/settime/gettime on the existing timer table, readable with read/poll/select/epoll

• Finding the root and starting init (stage 21): the GPT "Linux root (x86-64)" partition mounted at /, /sbin/init -> /bin/init -> /bin/sh as pid 1 with build-time argv/env knobs and its exit status passed to QEMU, an exFAT "Microsoft basic data" partition at /mnt/share (read-write), msync, RUSTBELT_DISK0 and CARGO_TARGET_DIR everywhere — Discoverable Partitions Specification, man 2 msync

• The kernel/OS split and a musl port (next; waits for Tyler's go; see the missing-syscall list in "musl prerequisites")

• A Hyprland-style tiling compositor on the framebuffer (later)

Filesystem roadmap (Tyler's plan)

Everything below plugs into the VFS above: a filesystem is a FileSystem whose root() returns Inodes; mounting it is one fs::mount call.

1. ramfs / tmpfs — done (src/fs/ramfs.rs).
2. Block device driver — done: BlockDevice trait + legacy virtio-blk (stage 6). Still to do: a block cache, IRQ-driven completion, maybe an ATA PIO driver for comparison (OSDev: ATA PIO Mode).
3. Partition tables — done: MBR + GPT (no CRC checks yet).
4. FAT32 — done via the fatfs crate (src/fs/fat.rs). A good exercise later: a hand-written read-only FAT32 to see what the crate does.
5. ext2 — done (stage 8, src/fs/ext2.rs): written by hand, read and write: superblock, block groups, bitmaps, inode table, direct/indirect blocks, directory entries, symlinks. Still to do: symlink creation, rename, hard links, a block cache, real timestamps once there's an RTC driver. Later, ext4's extents would be a nice follow-up. OSDev: Ext2, The Second Extended File System (Dave Poirier)
6. exFAT — done (stage 9, src/fs/exfat.rs): written by hand from Microsoft's spec, read and write: boot checksum, allocation bitmap, FAT chains and NoFatChain runs, up-case table, entry sets with SetChecksum and NameHash. Chosen because it works on literally anything (cameras, phones, Windows, macOS, Linux), so Rustbelt can share USB sticks and SD cards with every other machine. Still to do: rename, timestamps, TexFAT. Microsoft exFAT specification
7. btrfs, read-only — done (stage 10, src/fs/btrfs.rs): because most of Tyler's Linux systems will use btrfs going forward, so this kernel can read their disks. Copy-on-write B-trees, the chunk tree (logical -> physical, single and DUP), root/FS/csum trees, crc32c checksums, zlib/lzo/zstd, holes, subvolumes. Still to do: writing (see "Why read-only first"), other checksum types, RAID/multi-device, xattrs. btrfs on-disk format, btrfs docs
8. NTFS — last, maybe never.

Next, in this order:

1. Core polish — done (stage 11): SMP on 4 CPUs, a scheduler with sleep/wake and wait queues, memory freed on every exit, guard pages. Left over for later: priorities, per-CPU run queues, an I/O APIC, a bigger or growable heap.
2. Networking — done (stage 12): virtio-net plus smoltcp, sockets, and ifconfig, ping, host, fetch, httpd. Left over for later: non-blocking sockets and poll, IPv6, a loopback interface, TLS.
3. More virtio devices — done (stage 13): virtio-rng, virtio-console (hvc0) and virtio-gpu in 2D mode, with the text console on a framebuffer and /dev/fb0 for programs. Left over for later: moving virtio-blk and virtio-net to the modern transport, resizing the display, a hardware cursor, virtio-input (keyboard/mouse events for the compositor), 3D (virgl, see above).
4. The process model — done (stage 14): fork (copy-on-write), execve (with the auxiliary vector a libc needs), mmap and friends with demand paging, signals, wait4, process groups and job control. Left over for later: MAP_SHARED file mappings (needs a page cache), sigaltstack, queued real-time signals, users and permissions.
5. The rest of the kernel side of a libc — done (stage 15): all of the list below except openat with C strings, the bigger stat and getuid, which come with the musl port (they are easier to get right with musl's own test programs). Left over: SIGTTIN/SIGTTOU (background jobs touching the terminal), edge-triggered epoll, IPv6 loopback, per-file wait queues instead of one global poll counter. The plan as it was:
   • poll/select (and later epoll), so one program can wait on several fds: the keyboard, a pipe and a socket at once;
   • non-blocking sockets (O_NONBLOCK, EAGAIN, connect in progress) and a loopback interface (127.0.0.1), so programs can talk to each other over TCP on one machine;
   • futexes and threads: clone(CLONE_VM | CLONE_THREAD ...), set_tid_address, arch_prctl(ARCH_SET_FS) for thread-local storage, and futex wait/wake, which is what libc mutexes and pthread_join are built on;
   • ttys and ptys: a proper terminal layer (termios, sessions, controlling terminals, /dev/tty, pseudo-terminals), so hvc0 gets job control and a terminal emulator becomes possible;
   • smaller things musl expects: nanosleep, readv/writev, openat with C strings, a bigger stat, getuid and friends.
6. The kernel/OS split and a musl port (next; this is the OS stage and waits for Tyler's go): user space moves into its own tree (libc, toolchain, programs), and musl is ported, so ordinary C programs recompiled against Rustbelt's syscalls (numbered like Linux's already) can run. mmap of /dev/fb0 comes with it, so programs can draw without a syscall per rectangle. One kernel job comes first: saving FPU/SSE state (XSAVE/FXSAVE on every switch and in signal frames). Rustbelt's own programs are built soft-float, so the kernel has never needed it, but musl and every ordinary C compiler output use SSE registers freely; without saving them two threads would silently corrupt each other's floating point. Also: paths as C strings (openat), Linux's full struct stat.
7. The Hyprland-style compositor (after that): see below; tile is its first sketch, and stage 15's ptys are what its terminal windows will run shells on.

Long term: a graphical desktop. The goal is a Hyprland-style tiling compositor: windows tile automatically, with keyboard-driven layouts and smooth animations. It will draw natively on the virtio-gpu framebuffer first, with Rustbelt's own programs as clients, so it needs nothing beyond step 3 plus mmap and an input device. Wayland client support comes later, once step 6 can run ordinary Linux programs: the compositor then speaks the Wayland protocol, so existing Wayland apps can open windows on it.

Out of scope for now: drivers for physical hardware (real disks, NICs, USB). Rustbelt targets QEMU's virtio devices. A Raspberry Pi / ARM port is the planned future hardware target. The split into a separate userland and OS (its own repo layout, libc, toolchain) comes later.

Other ideas, any time:

• Shell extras: Left/Right cursor movement, Tab completion of file names, $?, export and $VAR expansion (the environment itself is there since stage 14).
• More programs: more/less (needs raw mode and screen size), an editor.

Small exercises along the way: a timer-based async sleep future, speed up the PIT (OSDev: PIT), add handlers for divide error / general protection fault, make BootInfoFrameAllocator O(1), add a mem stat (bytes allocated) to the allocator.

Other good references: OSDev wiki main page, Intel SDM, the blog_os repo (one branch per post, e.g. post-04), and for a UEFI/framebuffer setup later, the newer bootloader 0.11 crate.

Notes / gotchas

• The target JSON changes occasionally with new nightlies (e.g. target-pointer-width became an integer, rustc-abi was added). If a nightly update breaks the build, compare with the latest x86_64-blog_os.json in the blog_os repo, or pin a date in rust-toolchain.toml (channel = "nightly-2026-10-29").
• The "packages contain code that will be rejected by a future version of Rust: core" warning comes from building core with build-std and is harmless.
• bootimage prints "CARGO_MANIFEST_DIR env variable not set" when run outside cargo run; also harmless.
• cargo test with build-std breaks ("duplicate lang item in crate core: sized") if Cargo.toml sets panic = "abort" in [profile.*]. The target JSON already sets "panic-strategy": "abort", so the profile settings were removed.
• Stale disk image when switching cargo features: cargo re-uses a previously built kernel ELF with an old mtime, so bootimage thinks its cached image is current and boots the wrong build. scripts/runner.sh and the scripts touch the ELF first to avoid this. If you call cargo bootimage --features ... by hand after switching features, touch target/x86_64-rust_kernel/debug/rustbelt first.
• build.rs reruns when user/, disk/, disk2/, disk3/, disk4/, build/ or build.rs change and rebuilds target/disk.img and the ext2, exFAT and btrfs images. QEMU opens them with snapshot=on, so the kernel's writes (the tests create and delete files) never reach the files on disk (except in scripts/ext2-check.sh and scripts/exfat-check.sh).
• e2fsprogs: sudo apt install e2fsprogs. mke2fs, e2fsck, debugfs and dumpe2fs live in /usr/sbin, which isn't on a normal user's PATH (build.rs and the script look there). Newer mke2fs dropped -r 0; revision 0 is -E revision=0. mke2fs -d stores all-zero blocks as holes, so sparse.bin's data chunks are deliberately non-zero.
• exfatprogs: sudo apt install exfatprogs. mkfs.exfat, fsck.exfat and dump.exfat live in /usr/sbin too. Mounting an exFAT image to fill it needs root and a loop device, and exfat-fuse can't mount here either (fusermount3: unknown option 'user=box'; with sudo, "Block device required"), hence build/exfat_populate.rs. dump.exfat -d PATH in exfatprogs 1.2.9 can't find files inside some directories (e.g. /NewFolder/Note.txt), even on images that only mkfs and the populator touched; dump.exfat -s DIR lists them fine, so exfat-check.sh uses that.
• btrfs-progs: sudo apt install btrfs-progs (6.14 here; mkfs.btrfs, btrfs, btrfs-convert are in /usr/bin). mkfs.btrfs --rootdir writes sparse files as real data (no holes, no prealloc) and numbers -u subvolumes in its own order (-u projects -u ro:archive gives archive 256 and projects 257). Compressed extents are at most 128 KiB, uncompressed ones 1 MiB, and data that doesn't compress is stored plain. btrfs-convert refuses ext2 with 1 KiB blocks and fails silently on a 64 MiB image (256 MiB works). btrfs inspect-internal dump-tree and dump-super on an image show exactly what the driver parses; they were the main debugging tool.
• The VGA console is ASCII-only: Ünïcödé café.txt shows as ■n■c■d■ caf■.txt on screen (serial shows it right). Mapping UTF-8 to code page 437 in vga_buffer.rs would be a small fix.
• docs/long-link points through lots of .. and ., but along a path that also works on Linux (cat it after debugfs -R "rdump /docs /tmp/x"). The VFS resolves .. lexically, so a target like file.txt/.. would work in Rustbelt but not on Linux; the test image avoids that.
• Debug builds use a lot of stack: deep call chains (syscall -> VFS -> fatfs -> ELF loader) needed more than 16 KiB. If you see "kernel stack overflow in thread", that's the guard page catching it: make the stack bigger (STACK_SIZE in src/stack.rs) or the call chain shallower.
• SMP pitfalls hit in stage 11:
  • join returned before the memory was free. The dead thread was removed from the table first and its stack freed a moment later, on another CPU, so the leak test saw 7 extra frames now and then. Now the stack and process reference are dropped first, then the thread is removed.
  • Mixed-up output. With two programs printing on two CPUs, lines came out as count = CPU900000000: println! sent each formatted piece as its own write, and the console printed one write piece by piece. Now ulib's println! formats a whole line into a buffer first, and the console holds a lock for the whole of one write.
  • A lost wake-up through the back door. If a thread blocks but there's nothing else to run, it halts in place; a wake from another CPU then marks it Ready and queues it while it's still running here. Another CPU could then pick it up too. done_waiting takes it back off the queue.
  • A process exiting before its spawner stored its thread id. On one CPU that couldn't happen; on four it can. reap waits for the id.
  • Freeing a kernel stack's page tables would need every CPU to flush its paging-structure caches; keeping them and reusing the slot avoids it.
• Networking pitfalls hit in stage 12:
  • A double fault after httpd exited, now and then. The cause was a spurious interrupt. When an interrupt line drops before the PIC can deliver it (the level-triggered virtio-net line, here), the PIC still raises something: IRQ 7 on the first chip or IRQ 15 on the second. Our IDT had no entry for IRQ 15, and an empty IDT entry doesn't give a friendly error. It gives a "segment not present" fault, and with no handler for that either, a double fault. Now IRQ 7 and 15 have handlers that check nothing and send no end-of-interrupt to the chip that didn't really interrupt (for IRQ 15, only the first chip gets one, because it did see the cascade).
  • [net] DHCP lease lost at boot. smoltcp's DHCP socket starts by reporting "deconfigured". It's only printed now if there was a lease.
  • Slirp's TFTP server answers from port 69, not from a fresh port like RFC 1350 says. tests/net.rs replies to wherever the data came from.
  • QEMU's default network card is an e1000. Adding -netdev/-device virtio-net-pci replaces it, so the PCI device count stayed 10 (stage 13's GPU, rng and console make it 12).
• Pitfalls hit in stage 17:
  • A number with two names. Rustbelt's own sysinfo had taken 502 while Linux's sysinfo is 99 with a different struct. Both exist now, and 502 is traced as rb_sysinfo so a trace never says "sysinfo" for two different calls.
  • Listing a directory while deleting from it. getdents64 skipped entries by count, so rm -r silently missed every other file. Tests that only list (never delete while listing) can't see this bug.
  • nanosleep ends early when a signal arrives. A test that sleeps 500 ms to count a periodic timer's signals woke after the first one and counted 1. Sleep in a loop until the time is really up.
  • A forked child keeps the parent's open files. With flock, the lock belongs to the open file, so the child's inherited copy kept it alive and a waiting child blocked forever (Linux does the same). The child has to close its copies first.
  • Linux's saved-id rule is subtle. setreuid(-1, x) only copies x into the saved id if x isn't the old real id. Read man 2 setreuid before writing the test, not after.
  • Version and count pins again: nbtest wanted "stage 16", the shell test "0.16.0", run-headless.sh 37 programs in bin (now 38, with coretest).
• Pitfalls hit in stage 16:
  • Tests that pin a version or a count. nbtest checked that uname said "stage 15", the shell test looked for "Rustbelt 0.15.1", and run-headless.sh expected dir bin | wc -l to print 36. A new stage and one new program (libctest) broke all three. Each is now updated, but expect it again next stage.
  • Things that quietly lied. F_GETFL always said O_RDWR, every stat time was 0, ramfs had no modes, and Inode::truncate could only cut a file to 0. Nothing failed, so nobody noticed until a C program asked.
  • Who is pid 1? Linux gives orphans to pid 1, but our pid 1 is usually the shell, which never calls wait(-1). Handing it orphans would have turned a memory leak into a zombie leak. See "Orphans" in stage 16.
  • sched_getaffinity returns a size, not 0. The raw call returns how many bytes it wrote; only the libc wrapper returns 0. musl's wrapper uses that size to zero the rest of the caller's buffer.
• Pitfalls hit in stage 15.1:
  • QEMU's TCG never raises #XM. With every SSE exception unmasked, 1.0 / 0.0 set MXCSR's divide-by-zero flag and quietly gave infinity: no exception, no SIGFPE. Real CPUs (and KVM) do raise it. fputest now checks for that case and says it skipped the test, instead of failing.
  • QEMU's default qemu64 CPU has no XSAVE, so the XSAVE/AVX code would never have run. -cpu max is in Cargo.toml's run and test args, and the FXSAVE path is checked by a run with -cpu qemu64.
  • gcc can't link -no-pie code above 4 GiB. Its default code assumes every address fits in 32 bits (mov $sym, %eax). -fpie makes the code RIP-relative, but then a GOTPCREL load of an asm-defined symbol still failed to link ("failed to convert GOTPCREL relocation; relink with --no-relax"). Marking that symbol visibility("hidden") lets gcc address it directly.
  • rawabi failed its own signal check on Linux, before it ever ran on Rustbelt: sqrt(2) * sqrt(8) is not exactly 4.0 in floating point. The handler now computes sqrt(16) * sqrt(0.25), which is exact. Test a portable test on the host first.
  • One more file in /bin broke demo. Its getdents64 buffer (1 KiB) held all of /bin until rawabi made it 36 names, and the test assumed one call reads everything. It now calls until 0, like readdir does.
  • ls -l and the old Stat.mode == 0. FAT and ramfs files used to show no permission bits. With Linux's struct stat, st_mode always has some, so ls -l always prints them (-rw-r--r--).
  • Typing ahead into the screenshot. screenshot.sh waits 0.8 s after each command; fputest takes longer, so the next command's echo was printed in the middle of its output. A session line {SLEEP n} now waits before typing the next command.
• Pitfalls hit in stage 15:
  • write must return what the caller asked for, not what went out. With ONLCR a 6-byte "hello\n" becomes 7 bytes on the wire. The tty returned 7; the shell's write_all then sliced past the end of its buffer, panicked, and its panic handler overflowed the stack. Now the tty returns the caller's count (and maps a partial write back).
  • Sleeping is only as fine as the timer. nanosleep(1 ms) and a 50 ms futex timeout could return early or late by up to one 55 ms PIT tick, depending on where in the tick they started. Deadlines now add a tick (never early), and nanosleep tops up by yielding until the TSC clock says the time has really passed.
  • A closed slave means hang-up. ptytest closed its copy of the slave after forking; for a moment no one had it open, the master reported POLLHUP, and the test read EOF. Keep the slave open in the parent (Linux behaves the same).
  • Read one byte at a time in raw mode if you might exec. The shell's line editor read 64 bytes at once; when commands were pasted into hvc0, the lines after the first were sitting in its buffer and were lost when it ran the first. Reading one byte per call leaves the rest in the tty, where the next reader finds it.
  • Lazily started kernel threads look like leaks. The hvc0 input thread started on first use, so its stack (16 frames) appeared between the headless self-test's two frame counts. It is now started at boot when a virtio console is present.
• Pitfalls hit in stage 14:
  • The kernel wrote straight through copy-on-write. Ring 0 ignores read-only page flags unless CR0.WP is set, so a read() into a forked child's buffer changed the parent's memory too. memory::init now sets CR0.WP, and every syscall runs user buffers through the fault logic first (mm::user_access_ok).
  • sysretq can't return from a signal handler. It loads RIP and RFLAGS from rcx and r11, so those two registers can never be restored. Syscalls now return through iretq like interrupts (src/trap.rs).
  • siginfo field offsets. For SIGCHLD, Linux puts si_pid at offset 16, si_uid at 20 and si_status at 24. ulib's struct had the status one i32 too far, so sigtest saw 0 instead of the exit code.
  • Leaked file descriptors across exec. The shell's script file was open as fd 3 when it forked, so every program inherited it and run --trace showed open(...) = 4 instead of 3. The shell now opens scripts, pipes and redirections with O_CLOEXEC, as real shells do.
  • Tracing execve. The trace line was built after the call, when the path string lived in an address space that no longer existed (<bad pointer>). It's now formatted before the call.
  • kill %1 without job control. In a script (no terminal), jobs aren't in their own process group, so kill -pgid found nobody (ESRCH); it now signals the job's pid there.
  • A fork bomb must leave nothing behind. MAX_PROCESSES (64) makes the 65th fork fail with EAGAIN. The test counts frames and processes before and after killing the bomb, and after a warm-up run, so lazily allocated kernel structures don't look like leaks.
  • AT_EXECFN was just forktest when started by spawn, which only passed the basename along. spawn_image now gets the full path.
• Pitfalls hit in stage 13:
  • memory allocation of 65536 bytes failed in tile. User programs have a fixed 64 KiB heap (user/ulib.rs), so a 64 KiB row buffer can never fit. The wallpaper now goes out 8 rows (32 KiB) at a time; a growable heap needs mmap or brk, next stage.
  • virtio-rng's PCI class is 0x00 ("unclassified") and the console's is 0x07 (communication); pci::class_name didn't know them. With the GPU, rng and console the PCI bus now has 12 devices instead of 10.
  • virtio-gpu has no legacy interface, so the stage-6 style "one I/O port BAR" driver couldn't work: the modern transport (capabilities, memory BARs, a 64-bit BAR for the GPU) had to come first.
  • The log mirror must never deadlock. Every serial_print! also goes to hvc0. serial::_print releases the serial lock first and calls the console with interrupts off, so an interrupt handler that prints can't find the hvc0 lock already held on its own CPU.
  • A test counted mounts: /dev is a new mount, so tests/vfs.rs expects 4, and the root listing has dev in it.
  • -vga none -device virtio-vga keeps the old VGA text memory (virtio-vga is VGA-compatible), so the console tests that read 0xb8000 still pass, and the bootloader's text output still shows until the switch.
  • A boot that sat at A:/ $ for 90 s turned out to be a stale, non-headless image (the headless build had failed to compile), waiting for keyboard input; not a driver bug. Check the build before the boot.
• The \ | key: pc-keyboard 0.7's scancode set 1 decodes it (scancode 0x2B) as KeyCode::Oem7, but its US 104-key layout only maps Oem5 to \ and |. So that key typed nothing, which went unnoticed until the shell needed |. task/keyboard.rs renames Oem7 to Oem5 before decoding.
• User programs: the linker drops empty output sections, and their ALIGN(4K) with them. A program without .data then had its .bss on the read-only .rodata page (see "ELF loading"). user/link.ld uses . = ALIGN(4K); before .data, and the loader now rejects such files.