# Rustbelt **Rustbelt** is Tyler's hobby x86_64 kernel written in Rust. It started from the 2nd edition of Philipp Oppermann's [Writing an OS in Rust](https://os.phil-opp.com/) and keeps going past the end of it. It boots in QEMU straight into **its own shell**: a user-mode program that has a Unix feel with DOS touches. It offers `ls`, `cat`, pipes and redirection, plus `dir`, `type`, `ver`, and drive letters from `A:` to `E:`. Under the shell: - **A framebuffer console and serial output.** The kernel draws its text console with a built-in 8x16 bitmap font on a 1024x768 **virtio-gpu** framebuffer (falling back to the old VGA text screen when there's no virtio-gpu), and mirrors everything to the serial port (COM1) and a second console, **hvc0**. - **CPU exceptions.** It handles breakpoints, page faults, and double faults (on their own stack). - **Several CPUs.** It starts every CPU core QEMU gives it (`-smp 4`) through the local APIC, and its scheduler runs threads and programs on all of them at once. - **Timers and keyboard.** The PIT drives an uptime clock and wakes sleeping threads; each CPU's local APIC timer drives preemption. The PS/2 keyboard feeds a tty with cooked and raw modes. - **Memory.** It walks and edits its own page tables. It has a kernel heap with a fixed-size-block allocator, and frees every frame a process or thread used when it exits. Every stack has an unmapped **guard page** under it, so an overflow is caught and reported instead of silently corrupting memory. - **Multitasking.** It multitasks two ways: an **async/await executor** and **preemptive kernel threads**. Threads can sleep and wait for events (wait queues) without using any CPU while they wait. - **User programs.** Small Rust ELF binaries run in **ring 3** as processes, each with its own address space. Their memory is freed when they exit. They talk to the kernel through 83 `syscall`s, numbered like Linux's: `read`, `write`, `open`, `stat`, `getdents64`, `pipe2`, `uname`, `socket`, `connect`, `getrandom`... A few are our own: `spawn`, `mounts`, `sysinfo`, `resolve`, `netinfo` and `trace_me`. - **The Unix process model.** `fork` with **copy-on-write**, `execve` with Linux's `argv`/`envp`/auxiliary-vector stack layout, `wait4`, **`mmap`** (anonymous, shared, and file-backed private mappings), `mprotect`, `munmap` and `brk` with **demand paging**, and **signals** with real signal frames: handlers, masks, `sigreturn`, SIGSEGV on bad accesses, SIGCHLD, SIGPIPE, and **Ctrl+C** to the foreground process group. The shell runs programs with fork + exec and has job control (`kill`, `fg`, `bg`, Ctrl+Z). - **Files.** Files live behind a Linux-style **VFS**: a ramfs at `/` and `/tmp`, a **FAT32** partition mounted at `/mnt/disk0`, an **ext2** disk at `/mnt/disk1`, an **exFAT** disk at `/mnt/disk2` and a **btrfs** disk at `/mnt/disk3`. The ext2 and exFAT drivers are hand-written and read and write their disks (ext2: permissions, symlinks, indirect blocks; exFAT: long Unicode names, case-insensitive lookup, FAT chains). The real `e2fsck` and `fsck.exfat` check what they write. The btrfs driver, also hand-written, is **read-only**: copy-on-write B-trees, chunk mapping (single and DUP), crc32c checksums on every node and data block, zlib/lzo/zstd compression, holes, symlinks and subvolumes. The disks are **virtio-blk** devices found by **PCI** enumeration, with their **GPT/MBR** partitions parsed. The shell and its programs are loaded from the FAT disk. - **Networking.** A **virtio-net** card (interrupt-driven) carries Ethernet frames; the [smoltcp](https://github.com/smoltcp-rs/smoltcp) crate does ARP, IPv4, ICMP, UDP, TCP, DHCP and DNS on top. User programs get BSD-style sockets (`socket`, `connect`, `bind`, `listen`, `accept`, `sendto`, `recvfrom`) and come with `ifconfig`, `ping`, `host`, `fetch` (a tiny `wget`) and `httpd` (a tiny web server you can reach from the host). - **Virtual devices.** Hand-written drivers for the modern (virtio 1.0) **virtio-rng** (entropy: the `getrandom` syscall, `/dev/random`, smoltcp's seed and random TCP ports), **virtio-console** (`/dev/hvc0`, a second terminal the host reaches through a socket; `getty` runs a shell on it) and **virtio-gpu** in 2D mode (`/dev/fb0` with ioctls; `tile` draws a tiling-window-manager mock-up on it). - **What a C library needs (stage 15).** `poll`, `select` and **`epoll`** over pipes, sockets, terminals and files; **`O_NONBLOCK`** with `EAGAIN`; a **loopback** interface (127.0.0.1) so programs talk TCP/UDP to each other without a network card; **threads**: `clone(CLONE_VM | CLONE_THREAD ...)`, **futexes**, thread-local storage (`arch_prctl`), `gettid`, `tgkill`, `exit` vs `exit_group`, running on all 4 CPUs at once; real **ttys and ptys** with termios, canonical/raw modes, echo, ^C/^Z/^\\ as signals on every terminal, `/dev/tty`, `/dev/ptmx` and `/dev/pts/N`; plus `nanosleep`, `clock_gettime(CLOCK_REALTIME)`, `readv`/`writev`, `pread`/`pwrite` and `readlink /proc/self/exe`. It has a `cargo test` suite that runs inside QEMU, and a headless self-test that drives the shell with a script. Most code follows the blog_os sources (MIT / Apache-2.0) closely, with extra comments. Everything else is new: the serial mirroring, uptime clock, demo features and scripts, and everything past chapter 12. That covers preemptive threads, user mode, syscalls, fork/exec/mmap/signals, ELF loading, the VFS, PCI, virtio-blk, partitions, FAT32, the shell, the ext2, exFAT and btrfs drivers, multi-core (SMP) support, the virtio-net driver and socket layer, the virtio-rng, virtio-console and virtio-gpu drivers with the framebuffer console, and stage 15's poll/epoll, loopback, threads, futexes and ptys. The 8x16 font is [Spleen](https://github.com/fcambus/spleen) by Frederic Cambus (BSD 2-clause, license in `src/font.rs`). ![Rustbelt 0.15.0 (stage 15: libc plumbing) on the 1024x768 virtio-gpu framebuffer: `tile` drew four tiled windows; in the left one the shell interrupts a CPU hog with Ctrl+C ("[exit code 130] (interrupted: SIGINT)"), `threads` sums 1..=40000000 with 4 threads and a futex mutex, `echoserver` echoes over 127.0.0.1 with epoll and poll, and `ptytest` and `nbtest` report "all checks passed"](boot.png) ## New to Rust and C? Start here Rustbelt is meant to be read. You don't need to know Rust or C well to follow it; here is a path through it, and the handful of ideas that make kernel code look strange at first. **A reading order.** Each file starts with a comment explaining what it is for. Read them roughly in the order the kernel uses them: 1. `src/main.rs` -- `kernel_main` is the first Rust code that runs. It calls the `init` functions below in order, then starts the shell. 2. `src/vga_buffer.rs`, `src/serial.rs` -- printing text: the screen is just memory at address `0xb8000`, the serial port is a few I/O ports. 3. `src/interrupts.rs`, `src/gdt.rs` -- what happens when the CPU is interrupted (a key press, a timer, a crash). 4. `src/memory.rs`, `src/allocator.rs` -- page tables (how addresses are translated) and the heap (`Box`, `Vec`). 5. `src/thread.rs` -- threads, the scheduler and sleeping. `src/smp.rs` and `src/apic.rs` -- how the other CPUs get started. 6. `src/process.rs`, `src/syscall.rs`, `src/elf.rs` -- user programs. 7. `user/hello.rs`, then `user/sh.rs` -- what a program on Rustbelt looks like. 8. `src/fs/` -- the file systems, once the rest makes sense. 9. `src/virtio/` -- devices: `mod.rs` and `queue.rs` (how a driver talks to a virtio device), then `rng.rs` (the smallest driver), `console.rs` and `gpu.rs`; `src/fbcon.rs` draws text on the framebuffer. **Rust in kernel code, in five ideas:** - **`#![no_std]`**: there is no operating system underneath, so Rust's standard library (files, threads, `println!`) isn't available. Only `core` (basic types) and, once there's a heap, `alloc` (`Box`, `Vec`, `String`). Rustbelt defines its own `println!`. - **`unsafe`**: Rust normally guarantees memory safety; inside an `unsafe {}` block *you* promise it instead. Kernels need it to touch hardware (raw addresses, port I/O, CPU registers). Rustbelt keeps these blocks small and says in a comment why each one is OK. - **Ownership and `Drop`**: every value has one owner, and when the owner goes away Rust runs its `Drop`. Rustbelt uses that for cleanup: dropping a `KernelStack` unmaps it and frees its memory (`src/stack.rs`); dropping the last `Arc` frees the whole address space. - **`Mutex` and `Arc`**: `spin::Mutex` is a lock: `.lock()` gives you the `T` and unlocks automatically when you're done. `Arc` is a shared pointer with a reference count, so several threads can hold the same process. - **`global_asm!` / `asm!`**: a few things (switching stacks, entering ring 3, starting a CPU) need raw assembly. Each block is commented line by line. **C ideas you'll see** (the hardware docs and OSDev wiki are written in C terms): a *pointer* is just an address; `#[repr(C)]` lays out a struct the way C would, so it matches what the CPU or a disk format expects; *volatile* reads and writes tell the compiler a memory access really has to happen (it's a device, not ordinary memory). **Kernel words, briefly:** a *page* is 4 KiB of memory; a *frame* is the physical memory behind it. An *interrupt* stops the CPU and runs a handler. *Ring 0* is the kernel, *ring 3* user programs. A *syscall* is a user program asking the kernel for something. *SMP* means several CPUs sharing memory; the first one to boot is the *BSP*, the others *APs*. **Try things.** `cargo run`, then in the shell: `help`, `ver`, `sysinfo`, `spin 200 > A:/tmp/spin.txt &` followed by `sysinfo` (watch one CPU go to 100%), `spin 1000` and then Ctrl+C, `sigtest`, `overflow` (a stack overflow, caught), `tile` (the console moves into a window; `tile -r` moves it back), `random -d 6 5`. Change something small -- the message in `user/overflow.rs`, the bar width in `user/sysinfo.rs` -- and run `cargo test` to check nothing broke. When something crashes, the `[kernel]` lines on the serial output tell you which handler caught it and where. ## What's installed (on this box) | Tool | Version | Why | |---|---|---| | rustup | 1.29.2 | toolchain manager (in `~/.cargo/bin`, ahead of Debian's `/usr/bin/cargo`) | | Rust nightly | rustc 1.101.0-nightly (2026-10-29) | custom JSON targets + `build-std` are nightly-only | | `rust-src` component | – | source of `core`, recompiled for our bare-metal target | | `llvm-tools-preview` component | – | `llvm-objcopy` etc., used by `bootimage` | | `bootimage` (cargo subcommand) | 0.10.5 | links kernel + bootloader into a bootable disk image | | QEMU (`qemu-system-x86`, `qemu-system-gui`) | 10.0.13 | the emulator we boot in (`-gui` adds the GTK/SDL window) | | `dosfstools`, `mtools`, `fdisk` | Debian packages | `build.rs` makes the FAT32 test disk with `sfdisk` + `mkfs.vfat` + `mmd`/`mcopy` | | `e2fsprogs`, `exfatprogs` | Debian packages | the ext2 and exFAT images, and `e2fsck` / `fsck.exfat` to check Rustbelt's writes | | `btrfs-progs` | 6.14 | `mkfs.btrfs --rootdir` and `btrfs-convert` make the btrfs images; `btrfs check --readonly` checks them | `rust-toolchain.toml` pins the project to nightly with the two components, so `cargo` inside this directory picks them up automatically. It also installs the built-in `x86_64-unknown-none` target, used to compile the user programs. Crates: `bootloader` 0.9.35 (BIOS bootloader used by the 2nd edition), `volatile` 0.2, `spin` 0.5, `x86_64` 0.15, `uart_16550` 0.3, `lazy_static` 1.5, `pic8259` 0.11 (interrupt controller), `pc-keyboard` 0.7 (scancode decoding), `linked_list_allocator` 0.9 (fallback heap allocator), `fatfs` (git, `no_std` + `alloc`; FAT12/16/32 reading and writing), `miniz_oxide` 0.8 (zlib inflate) and `ruzstd` 0.8 (zstd decoding) for btrfs's compressed extents; LZO is hand-written (`src/fs/lzo.rs`). Fresh setup on another machine: ```sh curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # rustup rustup toolchain install nightly -c rust-src -c llvm-tools-preview -t x86_64-unknown-none cargo install bootimage sudo apt install qemu-system-x86 qemu-system-gui dosfstools mtools fdisk e2fsprogs exfatprogs btrfs-progs ``` ## Build and run ```sh cargo bootimage # build -> target/x86_64-rust_kernel/debug/bootimage-rustbelt.bin cargo run # build + boot in a QEMU window; serial output also prints in the terminal ``` **Using it:** `cargo run` opens a QEMU window. The kernel prints a few boot lines, then you're at the shell prompt `A:/ $`. Click into the window and type, for example `help`, `cd B:`, `ls` or `cat readme.txt | wc`, or `ls -l C:/` for the ext2 disk, `ls -l D:/` for the exFAT one, or `ls -l E:/` for the read-only btrfs one. See [The shell](#the-shell-stage-7) for everything it can do. Everything on the screen is also mirrored to your terminal over serial. Typing `exit` ends the shell, and the kernel starts a fresh one. **Networking in QEMU:** `cargo run` gives the guest QEMU's *user-mode* network (`-netdev user`, also called slirp). It needs no root and no setup on the host: QEMU itself plays a little router with a DHCP server (the guest gets `10.0.2.15`), a gateway at `10.0.2.2` that answers pings and forwards TCP/UDP out through the host's own sockets, and a DNS forwarder at `10.0.2.3`. Two extras are set up in `Cargo.toml`: - `guestfwd`: connections from the guest to `10.0.2.100:80` and `:7` start `scripts/http-reply.sh` (a 20-line HTTP responder) and `cat` (an echo server) on the host, so `fetch http://10.0.2.100/` always works, even with no internet. - `hostfwd=tcp:127.0.0.1:5580-:80`: run `httpd` in the shell, then `curl http://127.0.0.1:5580/` on the host reaches it. Try `ifconfig`, `ping 10.0.2.2`, `host example.com` or `wget -o D:/page.html http://example.com/`. **The second console (hvc0):** `cargo run` connects the guest's virtio-console to a UNIX socket, `target/hvc0.sock`. The kernel log is mirrored there; to get a shell on it, type `getty /dev/hvc0 &` in the QEMU window, then on the host: ```sh socat - UNIX-CONNECT:target/hvc0.sock # or: nc -U target/hvc0.sock ``` Type commands and press Enter (that shell reads whole lines, like a script, and echoes each one after its prompt; `exit` ends it and getty starts a new one). The headless runs use a file instead of the socket (`-chardev file,...,input-path=`), see below. **The display** is virtio-gpu (`-vga none -device virtio-vga`): the guest sets up a 1024x768 framebuffer and QEMU shows it scaled in its window. Leave those two options out and the kernel falls back to the VGA text console. To get the mouse and keyboard back from QEMU, press Ctrl+Alt+G. To quit, close the window or press Ctrl+C in the terminal. Over SSH without X, use `cargo run -- -display curses -vga std` to see the screen right in your terminal (curses can only show text mode, so this switches back to the VGA console), and quit from another terminal with `pkill qemu-system`. Headless (no window, good for SSH / scripts): ```sh ./scripts/run-headless.sh # boot, the shell runs disk/scripts/selftest.sh, QEMU exits; # passes if the shell exits 0 AND the expected lines appear # (and the host's curl got a page from the guest's httpd, # and the shell on hvc0 answered in target/headless-hvc0.txt) ./scripts/run-headless.sh stack-overflow # deliberately overflow the stack; passes if the double-fault handler catches it ./scripts/run-headless.sh page-fault # write to unmapped 0xdeadbeaf; passes if the page-fault handler catches it EXT2_IMG=copy.img ./scripts/run-headless.sh # use copy.img as the ext2 disk, writable (no snapshot) EXFAT_IMG=copy.img ./scripts/run-headless.sh # the same for the exFAT disk BTRFS_IMG=other.img ./scripts/run-headless.sh # another btrfs image as E: (always snapshot=on) ./scripts/ext2-check.sh # Rustbelt writes to copies of the ext2 images, then the # host's e2fsck -fn checks them (see "ext2 (stage 8)") ./scripts/exfat-check.sh # the same for exFAT with fsck.exfat -n (see "exFAT (stage 9)") ./scripts/btrfs-check.sh # btrfs check --readonly on the images, then every file Rustbelt # reads is compared with the host's copy (see "btrfs (stage 10)") cargo run --features headless -- -display none # the self-test via cargo # (QEMU's exit status is 33 = success; cargo just reports it as non-zero) cargo run --features demos # the earlier stages' demos first, then the shell ./scripts/screenshot.sh [out.png] [session.txt] [expected...] # boot, type a shell session via QEMU monitor `sendkey` # (one command per line; {BS} = Backspace, {UP} = up arrow), # check the expected texts appear, save the screen (.png, or .ppm) # (default: the session in boot.png); exits 1 if text is missing ./scripts/gpu-check.sh # type `tile`, dump the virtio-gpu screen, compare 12 pixels # against the colours tile.rs draws (see "Virtual drivers") ``` Cargo features (in `Cargo.toml`): | Feature | Effect | |---|---| | (none) | boot, start the shell on the console (the default) | | `headless` | the shell runs `/mnt/disk0/scripts/selftest.sh` instead of reading the keyboard; its exit code becomes QEMU's (0 -> success); a 60 s timer safety net fails the run if it hangs; panics exit with failure | | `demos` | before the shell: the breakpoint, paging and heap demos, threads A/B/C, the async example, and the `hello`/`evil`/`spin`/`demo` programs of stages 2-6 | | `stack-overflow` | after boot, recurse forever to test the double-fault handler (with `headless`: exit success when caught) | | `page-fault` | after boot, write to unmapped address `0xdeadbeaf` (with `headless`: exit success when caught) | Expected serial output of `./scripts/run-headless.sh`. Boot is quick now that the demos are behind a feature; most of the output is the shell running `disk/scripts/selftest.sh`. Each command is echoed after its prompt. The `[process] forked/exec/exited` lines are the kernel's own log (every program is started with fork + exec since stage 14; the `...` lines are cut here). They go to serial only, so the VGA screen shows just the shell session (and kernel messages such as a killed process). ``` Rustbelt 0.15.0 -- a hobby x86_64 kernel in Rust [smp] 4 CPUs online [pci] 00:00.0 8086:1237 host bridge ... (12 PCI devices: the network card, the GPU, the rng, the console, 4 disks) [pci] 00:03.0 1af4:1050 display controller [pci] 00:04.0 1af4:1005 unclassified [pci] 00:05.0 1af4:1003 communication controller ... [pci] 12 devices on the PCI bus [disk] vda: virtio-blk, 131072 sectors (64 MiB) [disk] vda: GPT partition table, 1 partition(s) [disk] vda1: start 2048 sectors 126976 type Microsoft basic data (FAT/exFAT/NTFS) name "fat32" [disk] vdb: virtio-blk, 65536 sectors (32 MiB) [disk] vdb: no partition table (NoSignature) [disk] vdc: virtio-blk, 65536 sectors (32 MiB) [disk] vdc: no partition table (WholeDiskExfat) [disk] vdd: virtio-blk, 262144 sectors (128 MiB) [disk] vdd: no partition table (NoSignature) [disk] vda1: FAT32 volume "TYLERDISK" mounted at /mnt/disk0 [disk] vdb: ext2 volume "RUSTBELT" (rev 1, 1 KiB blocks, 4 groups) mounted at /mnt/disk1 [disk] vdc: exFAT volume "TYLEREXFAT" (4 KiB clusters, 7680 clusters) mounted at /mnt/disk2 [disk] vdd: btrfs volume "TYLERBTRFS" (16 KiB nodes, metadata DUP, data single, 2 subvolumes, read-only) mounted at /mnt/disk3 [vfs] mounted ramfs at / [vfs] mounted ramfs at /tmp [vfs] mounted devfs at /dev [vfs] mounted fat32 at /mnt/disk0 [vfs] mounted ext2 at /mnt/disk1 [vfs] mounted exfat at /mnt/disk2 [vfs] mounted btrfs at /mnt/disk3 [virtio] rng: virtio-rng at 00:04.0 [virtio] hvc0: virtio-console at 00:05.0, IRQ 10 [virtio] gpu: virtio-gpu 2D at 00:03.0, display 1280x800, framebuffer 1024x768 (3072 KiB) [virtio] fbcon: text console on the framebuffer, 128x48 characters [net] lo: loopback, 127.0.0.1/8 [net] eth0: virtio-net, MAC 52:54:00:12:34:56, IRQ 10 [net] DHCP lease: 10.0.2.15/24, gateway 10.0.2.2, DNS 10.0.2.3 [process] started pid 1 (sh), entry 0x800040ada5 A:/ $ # Rustbelt shell self-test: `cargo run --features headless` runs this with A:/ $ # `sh /mnt/disk0/scripts/selftest.sh` (B:/scripts/selftest.sh) and scripts/run-headless.sh checks the output. A:/ $ ver Rustbelt 0.15.0 (stage 15: libc plumbing) x86_64 A:/ $ drives A: / ramfs B: /mnt/disk0 fat32 C: /mnt/disk1 ext2 D: /mnt/disk2 exfat E: /mnt/disk3 btrfs A:/ $ pwd / A:/ $ cd B: B:/ $ ls [process] pid 1 forked pid 2 [process] pid 2 exec /mnt/disk0/bin/ls bin/ notes/ readme.txt scripts/ [process] pid 2 (ls) exited with code 0 B:/ $ ls -l notes [process] pid 1 forked pid 3 [process] pid 3 exec /mnt/disk0/bin/ls 125 todo.txt [process] pid 3 (ls) exited with code 0 B:/ $ type B:/readme.txt [process] pid 1 forked pid 4 [process] pid 4 exec /mnt/disk0/bin/cat Hello from the FAT32 disk! This file lives on a virtio-blk disk, in a GPT partition, in a FAT32 file system -- read through the kernel's VFS. In the Rustbelt shell this disk is drive B: -- try `cd B:` and `ls`. [process] pid 4 (cat) exited with code 0 B:/ $ cat b:notes/todo.txt | grep -i EXT | wc -l [process] pid 1 forked pid 5 [process] pid 1 forked pid 6 [process] pid 5 exec /mnt/disk0/bin/cat [process] pid 1 forked pid 7 [process] pid 5 (cat) exited with code 0 [process] pid 6 exec /mnt/disk0/bin/grep [process] pid 7 exec /mnt/disk0/bin/wc [process] pid 6 (grep) exited with code 0 1 [process] pid 7 (wc) exited with code 0 B:/ $ dir bin | wc -l [process] pid 1 forked pid 8 [process] pid 1 forked pid 9 [process] pid 8 exec /mnt/disk0/bin/ls [process] pid 9 exec /mnt/disk0/bin/wc [process] pid 8 (ls) exited with code 0 29 [process] pid 9 (wc) exited with code 0 B:/ $ echo first line > A:/tmp/out.txt B:/ $ echo second line >> /tmp/out.txt B:/ $ cat < /tmp/out.txt [process] pid 1 forked pid 10 [process] pid 10 exec /mnt/disk0/bin/cat first line second line [process] pid 10 (cat) exited with code 0 B:/ $ md /tmp/work [process] pid 1 forked pid 11 [process] pid 11 exec /mnt/disk0/bin/mkdir [process] pid 11 (mkdir) exited with code 0 B:/ $ copy B:/notes/todo.txt /tmp/work [process] pid 1 forked pid 12 [process] pid 12 exec /mnt/disk0/bin/cp [process] pid 12 (cp) exited with code 0 B:/ $ touch /tmp/work/empty.txt [process] pid 1 forked pid 13 [process] pid 13 exec /mnt/disk0/bin/touch [process] pid 13 (touch) exited with code 0 B:/ $ ls -l A:/tmp/work [process] pid 1 forked pid 14 [process] pid 14 exec /mnt/disk0/bin/ls 0 empty.txt 125 todo.txt [process] pid 14 (ls) exited with code 0 B:/ $ del /tmp/work/todo.txt /tmp/work/empty.txt /tmp/work [process] pid 1 forked pid 15 [process] pid 15 exec /mnt/disk0/bin/rm [process] pid 15 (rm) exited with code 0 B:/ $ grep "B:" readme.txt [process] pid 1 forked pid 16 [process] pid 16 exec /mnt/disk0/bin/grep In the Rustbelt shell this disk is drive B: -- try `cd B:` and `ls`. [process] pid 16 (grep) exited with code 0 B:/ $ alias ll="ls -l" B:/ $ ll A:/tmp [process] pid 1 forked pid 17 [process] pid 17 exec /mnt/disk0/bin/ls 23 out.txt [process] pid 17 (ls) exited with code 0 B:/ $ run --trace cat B:/notes/todo.txt [process] pid 1 forked pid 18 [process] pid 18 exec /mnt/disk0/bin/cat [trace 18] execve("/mnt/disk0/bin/cat", ...) = 0 [trace 18] open("/mnt/disk0/notes/todo.txt", 0o0) = 3 [trace 18] read(3, "Done: the shell (cd, ls, cat, run, start"..., 512) = 125 Done: the shell (cd, ls, cat, run, start, pipes, redirection, drive letters). Next: more file systems -- ext2, exFAT, btrfs. [trace 18] write(1, "Done: the shell (cd, ls, cat, run, start"..., 125) = 125 [trace 18] read(3, "", 512) = 0 [trace 18] close(3) = 0 [trace 18] exit(0) [process] pid 18 (cat) exited with code 0 B:/ $ start sleep 1 [process] pid 1 forked pid 19 [pid 19] sleep started in the background B:/ $ jobs [1] [pid 19] [process] pid 19 exec /mnt/disk0/bin/sleep running start sleep 1 B:/ $ wait [process] pid 19 (sleep) exited with code 0 [done] pid 19 (start sleep 1), exit code 0 B:/ $ cd C: C:/ $ ls [process] pid 1 forked pid 20 [process] pid 20 exec /mnt/disk0/bin/ls big.txt docs/ hello-link@ hello.txt lost+found/ readme.txt sparse.bin [process] pid 20 (ls) exited with code 0 C:/ $ cat hello-link [process] pid 1 forked pid 21 [process] pid 21 exec /mnt/disk0/bin/cat Hello from the ext2 disk! This file lives on a second virtio disk, in an ext2 file system that Rustbelt reads and writes with its own driver. [process] pid 21 (cat) exited with code 0 C:/ $ grep "line 12345" big.txt [process] pid 1 forked pid 22 [process] pid 22 exec /mnt/disk0/bin/grep big.txt line 12345 [process] pid 22 (grep) exited with code 0 C:/ $ cp B:/notes/todo.txt C:/todo.txt [process] pid 1 forked pid 23 [process] pid 23 exec /mnt/disk0/bin/cp [process] pid 23 (cp) exited with code 0 C:/ $ cat C:/todo.txt | grep -i ext2 [process] pid 1 forked pid 24 [process] pid 1 forked pid 25 [process] pid 24 exec /mnt/disk0/bin/cat [process] pid 24 (cat) exited with code 0 [process] pid 25 exec /mnt/disk0/bin/grep Next: more file systems -- ext2, exFAT, btrfs. [process] pid 25 (grep) exited with code 0 C:/ $ md C:/made-here [process] pid 1 forked pid 26 [process] pid 26 exec /mnt/disk0/bin/mkdir [process] pid 26 (mkdir) exited with code 0 C:/ $ echo written by Rustbelt > C:/made-here/note.txt C:/ $ type C:/made-here/note.txt [process] pid 1 forked pid 27 [process] pid 27 exec /mnt/disk0/bin/cat written by Rustbelt [process] pid 27 (cat) exited with code 0 C:/ $ ls -l C:/ [process] pid 1 forked pid 28 [process] pid 28 exec /mnt/disk0/bin/ls -rw-r--r-- 304000 big.txt drwxr-xr-x docs/ lrwxrwxrwx hello-link -> hello.txt -rw-r--r-- 142 hello.txt drwx------ lost+found/ drwxr-xr-x made-here/ -rw-r--r-- 284 readme.txt -rw-r--r-- 69206016 sparse.bin -rw-r--r-- 125 todo.txt [process] pid 28 (ls) exited with code 0 C:/ $ cd D: D:/ $ ls [process] pid 1 forked pid 29 [process] pid 29 exec /mnt/disk0/bin/ls Long File Name With Spaces.txt MixedCase.TXT Photos/ a-really-long-file-name-that-takes-several-file-name-directory-entries-in-exfat-xxxxxxxxxxxxxxxxxxxx.txt frag-chain.txt numbers.txt readme.txt Ünïcödé café.txt [process] pid 29 (ls) exited with code 0 D:/ $ cat "long file name with spaces.txt" [process] pid 1 forked pid 30 [process] pid 30 exec /mnt/disk0/bin/cat This file has a long name with spaces in it. exFAT stores names of up to 255 UTF-16 characters, 15 per "File Name" directory entry. [process] pid 30 (cat) exited with code 0 D:/ $ cp B:/notes/todo.txt D:/ [process] pid 1 forked pid 31 [process] pid 31 exec /mnt/disk0/bin/cp [process] pid 31 (cp) exited with code 0 D:/ $ cp C:/hello.txt "Copied From Ext2.txt" [process] pid 1 forked pid 32 [process] pid 32 exec /mnt/disk0/bin/cp [process] pid 32 (cp) exited with code 0 D:/ $ cp mixedcase.txt C:/from-exfat.txt [process] pid 1 forked pid 33 [process] pid 33 exec /mnt/disk0/bin/cp [process] pid 33 (cp) exited with code 0 D:/ $ cat C:/from-exfat.txt [process] pid 1 forked pid 34 [process] pid 34 exec /mnt/disk0/bin/cat MixedCase.TXT: exFAT keeps the case you wrote but ignores it when looking names up. [process] pid 34 (cat) exited with code 0 D:/ $ md "New Folder" [process] pid 1 forked pid 35 [process] pid 35 exec /mnt/disk0/bin/mkdir [process] pid 35 (mkdir) exited with code 0 D:/ $ echo hi from exFAT > "New Folder/Note.txt" D:/ $ type "new folder/NOTE.TXT" [process] pid 1 forked pid 36 [process] pid 36 exec /mnt/disk0/bin/cat hi from exFAT [process] pid 36 (cat) exited with code 0 D:/ $ ls -l D:/ [process] pid 1 forked pid 37 [process] pid 37 exec /mnt/disk0/bin/ls 142 Copied From Ext2.txt 132 Long File Name With Spaces.txt 84 MixedCase.TXT New Folder/ Photos/ 88 a-really-long-file-name-that-takes-several-file-name-directory-entries-in-exfat-xxxxxxxxxxxxxxxxxxxx.txt 38000 frag-chain.txt 340000 numbers.txt 363 readme.txt 125 todo.txt 61 Ünïcödé café.txt [process] pid 37 (ls) exited with code 0 D:/ $ cd E: E:/ $ ls -l [process] pid 1 forked pid 38 [process] pid 38 exec /mnt/disk0/bin/ls drwxr-xr-x Music/ drwxr-xr-x archive/ -rw-r--r-- 2160000 big.txt drwxr-xr-x docs/ -rw-r--r-- 0 empty.txt lrwxrwxrwx hello-link -> hello.txt -rw-r--r-- 78 hello.txt drwxr-xr-x projects/ -rw-r--r-- 1500000 random.bin -rw-r--r-- 637 readme.txt -rw-r--r-- 1048576 zeros.bin [process] pid 38 (ls) exited with code 0 E:/ $ cat hello.txt [process] pid 1 forked pid 39 [process] pid 39 exec /mnt/disk0/bin/cat Hello from btrfs! (an inline extent: these bytes live inside the B-tree leaf) [process] pid 39 (cat) exited with code 0 E:/ $ cat hello-link [process] pid 1 forked pid 40 [process] pid 40 exec /mnt/disk0/bin/cat Hello from btrfs! (an inline extent: these bytes live inside the B-tree leaf) [process] pid 40 (cat) exited with code 0 E:/ $ cat docs/btrfs/cow.txt [process] pid 1 forked pid 41 [process] pid 41 exec /mnt/disk0/bin/cat Copy-on-write: btrfs never overwrites a block in place. To change a leaf it writes a new copy somewhere free, then a new copy of its parent that points to it, and so on up to the root. The superblock is the last thing written; until then the old tree is still complete on disk. [process] pid 41 (cat) exited with code 0 E:/ $ wc big.txt [process] pid 1 forked pid 42 [process] pid 42 exec /mnt/disk0/bin/wc 120000 360000 2160000 big.txt [process] pid 42 (wc) exited with code 0 E:/ $ grep "line 119999" big.txt [process] pid 1 forked pid 43 [process] pid 43 exec /mnt/disk0/bin/grep btrfs line 119999 [process] pid 43 (grep) exited with code 0 E:/ $ cat projects/rustbelt/stage10.txt [process] pid 1 forked pid 44 [process] pid 44 exec /mnt/disk0/bin/cat Stage 10: a read-only btrfs driver. Next: core polish (SMP, memory cleanup, guard pages, a proper scheduler), then networking. [process] pid 44 (cat) exited with code 0 E:/ $ cp docs/btrfs/cow.txt D:/cow.txt [process] pid 1 forked pid 45 [process] pid 45 exec /mnt/disk0/bin/cp [process] pid 45 (cp) exited with code 0 E:/ $ type D:/COW.TXT | wc -l [process] pid 1 forked pid 46 [process] pid 1 forked pid 47 [process] pid 46 exec /mnt/disk0/bin/cat [process] pid 46 (cat) exited with code 0 [process] pid 47 exec /mnt/disk0/bin/wc 4 [process] pid 47 (wc) exited with code 0 E:/ $ cp docs/btrfs/chunks.txt C:/chunks.txt [process] pid 1 forked pid 48 [process] pid 48 exec /mnt/disk0/bin/cp [process] pid 48 (cp) exited with code 0 E:/ $ cat C:/chunks.txt | wc -l [process] pid 1 forked pid 49 [process] pid 1 forked pid 50 [process] pid 49 exec /mnt/disk0/bin/cat [process] pid 49 (cat) exited with code 0 [process] pid 50 exec /mnt/disk0/bin/wc 3 [process] pid 50 (wc) exited with code 0 E:/ $ echo hi > new.txt sh: new.txt: Read-only file system E:/ $ rm hello.txt [process] pid 1 forked pid 51 [process] pid 51 exec /mnt/disk0/bin/rm rm: hello.txt: Read-only file system [process] pid 51 (rm) exited with code 1 [exit code 1] E:/ $ mkdir newdir [process] pid 1 forked pid 52 [process] pid 52 exec /mnt/disk0/bin/mkdir mkdir: newdir: Read-only file system [process] pid 52 (mkdir) exited with code 1 [exit code 1] E:/ $ run evil [process] pid 1 forked pid 53 [process] pid 53 exec /mnt/disk0/bin/evil [evil] pid 53: reading kernel memory at 0x200000... [kernel] user page fault: address 0x200000, rip 0x8000400558, PageFaultErrorCode(PROTECTION_VIOLATION | USER_MODE), Unmapped [process] pid 53 (evil) killed: page fault (SIGSEGV) [exit code 139] (killed by SIGSEGV) E:/ $ # Stage 11: a stack overflow hits the guard zone and is killed cleanly; E:/ $ # then two CPU hogs in the background, and sysinfo shows the load per CPU. E:/ $ run overflow [process] pid 1 forked pid 54 [process] pid 54 exec /mnt/disk0/bin/overflow [overflow] pid 54: recursing until the stack runs out... [kernel] user page fault: address 0xffffefed38, rip 0x80004000e1, PageFaultErrorCode(CAUSED_BY_WRITE | USER_MODE), Unmapped [process] pid 54 (overflow) killed: stack overflow (SIGSEGV) [exit code 139] (killed by SIGSEGV) E:/ $ start spin [process] pid 1 forked pid 55 [pid 55] spin started in the background E:/ $ start spin [process] pid 55 exec /mnt/disk0/bin/spin [process] pid 1 forked pid 56 [spin] pid 55: spinning without syscalls... [pid 56] spin started in the background [process] pid 56 exec /mnt/disk0/bin/spin E:/ $ sysinfo [spin] pid 56: spinning without syscalls... [process] pid 1 forked pid 57 [process] pid 57 exec /mnt/disk0/bin/sysinfo [spin] round 1 done (count = 30000000) [spin] round 1 done (count = 30000000) [spin] round 2 done (count = 60000000) [spin] round 2 done (count = 60000000) 4 CPUs online (this program ran on CPU 2), up 15 s 11 threads, 4 processes, 4572 KiB of memory frames in use CPU load over the last quarter second: CPU0 [....................] 0% CPU1 [....................] 0% [spin] round 3 done (count = 90000000) [process] pid 55 (spin) exited with code 0 CPU2 [####################] 100% [spin] round 3 done (count = 90000000) [process] pid 56 (spin) exited with code 0 CPU3 [####################] 100% [process] pid 57 (sysinfo) exited with code 0 [done] pid 55 (start spin), exit code 0 [done] pid 56 (start spin), exit code 0 E:/ $ wait E:/ $ # Stage 12: networking. DHCP gave us an address; ping QEMU's gateway; fetch E:/ $ # from the test web server QEMU runs for 10.0.2.100 (guestfwd), and save a E:/ $ # 108 KB file to D:; look up a name (only works if the host has DNS); then E:/ $ # serve one request with httpd (run-headless.sh curls it through hostfwd). E:/ $ ifconfig [process] pid 1 forked pid 58 [process] pid 58 exec /mnt/disk0/bin/ifconfig eth0: virtio-net MAC 52:54:00:12:34:56 IRQ 10 inet 10.0.2.15/24 gateway 10.0.2.2 DNS 10.0.2.3 (DHCP) RX 15 packets (1 KiB) TX 14 packets (1 KiB) [process] pid 58 (ifconfig) exited with code 0 E:/ $ ping -c 3 -i 200 10.0.2.2 [process] pid 1 forked pid 59 [process] pid 59 exec /mnt/disk0/bin/ping PING 10.0.2.2 (10.0.2.2): 56 data bytes 64 bytes from 10.0.2.2: icmp_seq=1 time=8.97 ms 64 bytes from 10.0.2.2: icmp_seq=2 time=2.15 ms 64 bytes from 10.0.2.2: icmp_seq=3 time=2.19 ms --- 10.0.2.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss rtt min/avg/max = 2.15/4.43/8.97 ms [process] pid 59 (ping) exited with code 0 E:/ $ fetch http://10.0.2.100/ [process] pid 1 forked pid 60 [process] pid 60 exec /mnt/disk0/bin/fetch Hello from the host, via QEMU guestfwd! [process] pid 60 (fetch) exited with code 0 E:/ $ wget -o D:/seq.txt http://10.0.2.100/big [process] pid 1 forked pid 61 [process] pid 61 exec /mnt/disk0/bin/fetch fetch: HTTP/1.0 200 OK, saved 108894 bytes to /mnt/disk2/seq.txt in 126 ms (843 KiB/s) [process] pid 61 (fetch) exited with code 0 E:/ $ cat D:/seq.txt | wc -l [process] pid 1 forked pid 62 [process] pid 1 forked pid 63 [process] pid 62 exec /mnt/disk0/bin/cat [process] pid 63 exec /mnt/disk0/bin/wc [process] pid 62 (cat) exited with code 0 20000 [process] pid 63 (wc) exited with code 0 E:/ $ host example.com [process] pid 1 forked pid 64 [process] pid 64 exec /mnt/disk0/bin/host example.com has address 198.18.0.1 [process] pid 64 (host) exited with code 0 E:/ $ httpd -n 1 -t 20 [process] pid 1 forked pid 65 [process] pid 65 exec /mnt/disk0/bin/httpd [httpd] listening on port 80 [httpd] 10.0.2.2:48206 GET / -> 200 (133 bytes) [httpd] served 1 request, bye [process] pid 65 (httpd) exited with code 0 E:/ $ # Stage 13: virtual drivers. The devices in /dev; random bytes from E:/ $ # virtio-rng; a shell on the second console (getty -1 runs the commands E:/ $ # QEMU feeds into hvc0 from a file; its output goes to another file that E:/ $ # run-headless.sh checks); the tiling demo on the virtio-gpu framebuffer. E:/ $ ls -l /dev [process] pid 1 forked pid 66 [process] pid 66 exec /mnt/disk0/bin/ls crw-rw-rw- fb0 crw-rw-rw- hvc0 crw-rw-rw- null crw-rw-rw- random crw-rw-rw- urandom [process] pid 66 (ls) exited with code 0 E:/ $ random -n 8 [process] pid 1 forked pid 67 [process] pid 67 exec /mnt/disk0/bin/random a83abb283794a4d6 [process] pid 67 (random) exited with code 0 E:/ $ random -d 6 5 [process] pid 1 forked pid 68 [process] pid 68 exec /mnt/disk0/bin/random 3 3 6 6 5 [process] pid 68 (random) exited with code 0 E:/ $ getty -1 /dev/hvc0 [process] pid 1 forked pid 69 [process] pid 69 exec /mnt/disk0/bin/getty [process] started pid 70 (sh), entry 0x800040ada5 [process] pid 70 forked pid 71 [process] pid 71 exec /mnt/disk0/bin/random [process] pid 71 (random) exited with code 0 [process] pid 70 forked pid 72 [process] pid 72 exec /mnt/disk0/bin/ls [process] pid 72 (ls) exited with code 0 [process] pid 70 (sh) exited with code 0 [process] pid 69 (getty) exited with code 0 E:/ $ tile [process] pid 1 forked pid 73 [process] pid 73 exec /mnt/disk0/bin/tile tile: 4 windows on 1024x768, console 61x46 [process] pid 73 (tile) exited with code 0 E:/ $ tile -r [process] pid 1 forked pid 74 [process] pid 74 exec /mnt/disk0/bin/tile console: full screen, 128x48 characters [process] pid 74 (tile) exited with code 0 E:/ $ # Stage 14: processes. Every program above was started with fork + execve; E:/ $ # these check the process model itself (each prints "all checks passed"). E:/ $ # Then a background job ended with the kill builtin (SIGTERM: 128+15). E:/ $ forktest [process] pid 1 forked pid 75 [process] pid 75 exec /mnt/disk0/bin/forktest [process] pid 75 forked pid 76 [process] pid 76 (forktest) exited with code 7 [forktest] ok: fork returns the child's pid in the parent [forktest] ok: child saw getppid() == parent; wait4 gives its exit code 7 ... [forktest] all checks passed [process] pid 75 (forktest) exited with code 0 E:/ $ cowtest [process] pid 1 forked pid 92 [process] pid 92 exec /mnt/disk0/bin/cowtest [process] pid 92 forked pid 93 [process] pid 93 (cowtest) exited with code 0 [cowtest] ok: the child starts with the parent's data and can change its copy [cowtest] ok: the child's writes to global data stay in the child ... [cowtest] all checks passed [process] pid 92 (cowtest) exited with code 0 E:/ $ mmaptest [process] pid 1 forked pid 98 [process] pid 98 exec /mnt/disk0/bin/mmaptest [mmaptest] ok: mmap 3 pages of anonymous memory [mmaptest] ok: it starts zeroed [mmaptest] ok: it keeps what we write [mmaptest] ok: mmap 64 MiB: no memory used yet (pages come on first touch) ... [mmaptest] all checks passed [process] pid 98 (mmaptest) exited with code 0 E:/ $ sigtest [process] pid 1 forked pid 99 [process] pid 99 exec /mnt/disk0/bin/sigtest [sigtest] ok: SIG_IGN: an ignored signal does nothing [sigtest] ok: rt_sigaction installs a handler [sigtest] ok: kill(getpid(), SIGUSR1) runs the handler once, with sig = 10 [sigtest] ok: after rt_sigreturn the interrupted code computes the same result ... [sigtest] all checks passed [process] pid 99 (sigtest) exited with code 0 E:/ $ start spin 1000 [process] pid 1 forked pid 110 [pid 110] spin started in the background E:/ $ kill %1 [process] pid 110 exec /mnt/disk0/bin/spin [process] pid 110 (spin) killed by SIGTERM E:/ $ wait [done] pid 110 (start spin 1000), exit code 143 E:/ $ kill -l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR 31) SIGSYS E:/ $ nosuchprogram sh: nosuchprogram: command not found (try `help`) E:/ $ cd Q: cd: there is no drive Q: (see `drives`) E:/ $ cd A:/tmp A:/tmp $ pwd /tmp A:/tmp $ exit 0 [process] pid 1 (sh) exited with code 0 [ok] shell self-test finished: Exited(0) [memory] 1015 frames in use before the shell, 1015 after it and all its children exited [run-headless] frames in use returned to the baseline [run-headless] host curl got httpd's page: Hello from Rustbelt! Rustbelt 0.15.0 (stage 15: libc plumbing) [run-headless] all 67+12 expected shell and hvc0 lines found [run-headless] kernel exited OK ``` And `target/headless-hvc0.txt`, what the host got from hvc0: the kernel log up to the moment getty opened the port, then the shell running the commands in `scripts/hvc0-session.txt`: ``` ... E:/ $ getty -1 /dev/hvc0 [process] pid 1 forked pid 69 [process] pid 69 exec /mnt/disk0/bin/getty Rustbelt getty: shell on this terminal (type `exit` to end it) E:/ $ ver Rustbelt 0.15.0 (stage 15: libc plumbing) x86_64 E:/ $ echo hi over hvc0 hi over hvc0 E:/ $ random -d 20 3 3 12 8 E:/ $ ls /dev fb0 hvc0 null random urandom E:/ $ exit 0 [process] pid 69 (getty) exited with code 0 E:/ $ tile ... ``` (Which CPU each program lands on, the exact order of lines printed by programs running at the same time, ping times and download speeds change from run to run, and so do the random numbers. `example.com`'s address is whatever the host's DNS says; without DNS that line is `host: example.com: ...` and the run still passes.) The run ends with the same number of frames in use before and after the shell, even though about 110 processes have come and gone (forked, exec'd, sharing pages copy-on-write); `run-headless.sh` fails if they differ. Exited processes give back their page tables, code, user stacks and kernel stacks. (Of the 1015 at boot, 768 are the 3 MiB framebuffer; most of the rest are the virtio queues and buffers of the four disks, the network card, the rng, the console and the GPU, the kernel stack area and the other CPUs' stacks.) The kernel boots from the bootimage on the first (IDE) disk; the scripts and `Cargo.toml` attach a **second** disk, `target/disk.img` (GPT + FAT32), a **third**, `target/ext2.img` (raw ext2), a **fourth**, `target/exfat.img` (raw exFAT), and a **fifth**, `target/btrfs.img` (raw btrfs), as `-drive file=target/disk.img,format=raw,if=virtio,snapshot=on` and so on. `snapshot=on` makes QEMU throw away writes when it exits, so every boot and every test sees the same fresh image even though the kernel writes to it. The tests attach more images *after* those five, so the drive letters B: to E: are the same in `cargo run` and `cargo test`: two more ext2 images (4 KiB blocks, revision 0, `/mnt/disk4` and `disk5`), one more exFAT image (512-byte clusters, `disk6`), and four more btrfs images (lzo, zstd, converted from ext2: `disk7` to `disk9`; and one with sha256 checksums that is refused, so it gets no `/mnt/diskN`). Drive letters follow mount order, so a new user-visible disk always goes right after the last one `cargo run` attaches, never after the test-only ones. ## Tests ```sh cargo test # everything (lib unit tests + tests/*.rs), each booted in QEMU headless cargo test --lib # just the unit tests inside src/ cargo test --test heap_allocation # one integration test ``` Each test binary is its own little kernel: `bootimage runner` boots it with the `test-args` from `Cargo.toml` (no display, serial to stdout, isa-debug-exit) and treats QEMU exit code 33 as a pass. Output looks like: ``` Running 5 tests heap_allocation::large_allocation_uses_fallback... [ok] heap_allocation::large_vec... [ok] heap_allocation::many_boxes... [ok] heap_allocation::many_boxes_long_lived... [ok] heap_allocation::simple_allocation... [ok] ``` | Test | What it checks | |---|---| | lib (`src/*.rs` `#[test_case]`s) | trivial assertion, breakpoint handler returns, VGA `println!` writes the right cells; btrfs: crc32c check value and the `"default"` name hash, a hand-built prealloc extent and an explicit hole, an LZO literal run + overlapping match | | `tests/basic_boot.rs` | printing works straight after boot, before any `init()` | | `tests/heap_allocation.rs` | Box, big Vec, 100 K short-lived boxes (memory reuse), same with one long-lived box, >2 KiB fallback allocation | | `tests/should_panic.rs` | a failing `assert_eq!` reaches the panic handler (`harness = false`) | | `tests/stack_overflow.rs` | stack overflow lands in the double-fault handler on the IST stack (`harness = false`) | | `tests/executor.rs` | `SimpleExecutor` runs an async fn to completion; the waker-based `Executor` interleaves two tasks that `yield_now().await` (a1 b1 a2 b2 a3 b3) | | `tests/shell.rs` | **the shell**: tokenizer (operators without spaces, quotes, comments), parser (pipelines, `<` `>` `>>`, `&`, error cases), drive-letter assignment from a mount table, `B:/x` -> `/mnt/disk0/x` translation and back for the prompt; then the real `/bin/sh` runs a script (cd with a drive letter, ls, a pipe, `>`/`>>`/`<`, `ver`, `exit 3`) with its stdout in a pipe and the output is checked; background jobs (`start`, `jobs`, `wait`); 20 processes started and reaped leave `frames_in_use` unchanged | | `tests/user_mode.rs` | programs loaded from the FAT disk: `hello` exits 0, `demo` passes all its checks (exit 0), `evil` is killed by SIGSEGV (code 139), `spin` is preempted; spawn with arguments + trace + exit code 7, then reaped; a child's stdout into a pipe; spawn errors (`ENOENT`, `ENOEXEC`, `EISDIR`, `ECHILD`); `wait` with `WNOHANG` returns `EAGAIN` while the child runs; ELF loader rejects garbage; user pointers give `EFAULT` (length 0 is fine); `sleep` blocks for the right number of ticks; a `WaitQueue` wakes its waiter | | `tests/vfs.rs` | the root holds `dev`, `etc`, `hello.txt`, `mnt`, `tmp`; read `/hello.txt` to EOF, path normalisation (`.`, `..`, relative to a cwd), `ENOENT`/`ENOTDIR`/`EINVAL`/`EISDIR`/`EEXIST`, `/tmp` is a separate ramfs, longest-prefix mounts (4 with `/dev`), sparse writes zero-fill, listing, remove (`ENOTEMPTY`, `EBUSY` on mount points), seek/stat/truncate, `getdents64` record layout, pipes pass bytes and report EOF / `EPIPE` | | `tests/block.rs` | PCI scan finds the virtio disk; sector 1 of the image is the GPT header; multi-sector reads equal single-sector reads; write + read back; out-of-range and partial-sector requests fail; the GPT partition is registered as `vda1` (with a FAT32 boot sector); MBR parsing on a RAM disk | | `tests/fat.rs` | the FAT32 volume is mounted at `/mnt/disk0` with the expected files (`bin`, `notes`, `readme.txt`, `scripts`); read `readme.txt`; every program in `bin/` is a valid ELF; create/write/read/truncate/remove a 10 KB file; directories with long names, `ENOTEMPTY`; the `BlockIo` adapter handles unaligned reads and writes | | `tests/ext2.rs` | the three ext2 images are mounted (`/mnt/disk1`, `disk4`, `disk5`); permissions and case-sensitive names; `big.txt` line by line and in one read (direct, single and double indirect blocks); `sparse.bin`'s data at 0 / 40 MiB / 65 MiB (triple indirect) and its holes; fast and slow symlinks, `readlink`, removing a link keeps its target; 4 KiB blocks and revision 0; on each image: mkdir, create, a 300 KB write, append, overwrite across a block boundary, a write at 70 MiB (hole + triple indirect), truncate, `ENOTEMPTY`/`EISDIR`/`ENOTDIR`, unlink, rmdir, and the superblock's free counts are back where they started; 200 files grow a directory; copy from FAT to ext2. `scripts/ext2-check.sh` then checks the same kernel's writes with the host's `e2fsck` | | `tests/exfat.rs` | both exFAT images are mounted (`/mnt/disk2`, `/mnt/disk6`), label and cluster size; long, mixed-case and Unicode names listed as written; case-insensitive lookup (`ÜNÏCÖDÉ CAFÉ.TXT`, nested paths); `numbers.txt` at a cluster boundary and at its end; the FAT-chained `frag-chain.txt` on 4 KiB and 512-byte clusters; bad names (`EINVAL`, `ENAMETOOLONG`, `EEXIST` for a name in another case); on each image: mkdir, create, write across clusters, append, truncate, a write past the end (zero gap), `ENOTEMPTY`, unlink, rmdir, and the bitmap and PercentInUse are back where they started; a contiguous file turns into a FAT chain; 200 files grow a directory; the root grows; copies between FAT, ext2 and exFAT; a last test leaves files for `scripts/exfat-check.sh`'s `fsck.exfat` | | `tests/btrfs.rs` | the four btrfs images are mounted (`/mnt/disk3`, `disk7`-`disk9`) and described (label, node size, DUP/single, subvolume count); **every file on every image is read and its size and CRC-32 compared with build.rs's manifest of the host's tree** (zlib, lzo with a two-level tree, zstd with 64 KiB nodes, and the converted image); reads at odd offsets across compressed and plain extent boundaries; holes in the converted `sparse.bin`; `ext2_saved/image` starts like an ext2 disk; hash lookups are exact (`HELLO.TXT` is `ENOENT`); symlinks, including one leaving a subvolume; subvolumes listed with ids and read-only flags; the sha256 image is refused with a reason; create/write/truncate/mkdir/remove give `EROFS`; and, last, damage on purpose: copy 0 of a DUP node (copy 1 is used), a data sector (`EIO` for that block only), then both copies (`EIO`). `scripts/btrfs-check.sh` runs `btrfs check --readonly` and compares the same kernel's output with the host's files | | `tests/threads.rs` | two busy-looping threads that never yield both make progress while the main thread busy-waits too (only possible with preemption); a thread can exit and the others keep running | | `tests/smp.rs` | **multi-core and the scheduler**: all 4 CPUs are online and take timer ticks; 4 busy threads run on 2+ CPUs *at the same time* (per-CPU busy ticks); 3 `spin` processes run on 2+ CPUs; 4 threads on 4 CPUs add 200,000 to a locked counter without losing one; `sleep_ticks(10)` lasts 10-12 ticks; sleepers wake in deadline order (3, 6, 9); while 4 threads sleep the CPUs are idle (<10% busy, so no busy-polling); two threads ping-pong 1,000 times through a `WaitQueue`; **leak checks**: 200 threads and 40 processes spawned and exited leave `frames_in_use`, the thread count, the process count and the number of kernel stacks exactly where they were | | `tests/net.rs` | **networking**, all against things QEMU itself provides, so it's deterministic and works offline: DHCP gives 10.0.2.15/24, gateway 10.0.2.2, DNS 10.0.2.3 and the MAC is QEMU's default; 3 ICMP echoes to the gateway come back with their payload; a UDP **TFTP** read of `readme.txt` from slirp's TFTP server (`tftp=disk` in `test-args`); a 64 KiB TCP round trip through the `cat` echo server (guestfwd 10.0.2.100:7); an HTTP GET of 108,894 bytes through the socket API; `ECONNREFUSED` from a closed host port; a UDP `recv` with a 300 ms timeout returns `EAGAIN` after 250-1000 ms; the `ping` and `fetch` programs (saving to a file, and exit 1 on a 404); 20 TCP + 20 UDP sockets opened and closed leave the socket count and `frames_in_use` unchanged; DNS for `example.com`, **skipped** (not failed) if the host has no DNS | | `tests/virtio.rs` | **the virtual drivers**: 2 x 4 KiB from virtio-rng differ, aren't constant, have 15384-17384 one bits out of 32768 and contain all 256 byte values; the `random` program (hex, dice, a usage error); hvc0 receives exactly the bytes QEMU feeds in from `tests/hvc0-input.txt`, and 8 KiB written to it go out with nothing dropped; virtio-gpu answers `GET_DISPLAY_INFO`, refuses a flush of a resource that doesn't exist with `ERR_INVALID_RESOURCE_ID`, full and clipped flushes work, a pattern written into the framebuffer reads back; the text console is on the framebuffer (128x48) and printing lights up pixels; `/dev` lists `fb0 hvc0 null random urandom`; `tile` draws the expected colours at known pixels, moves the console to 61x46 and `tile -r` back, with no frames leaked. `scripts/gpu-check.sh` checks what QEMU actually displays | | `tests/guard_pages.rs` | the guard page sits right below each kernel stack, and the user stack's 64 KiB guard zone has the right bounds; a kernel thread that recurses forever is killed with "kernel stack overflow" three times in a row (the rest of the kernel carries on, and its stack is freed); the user `overflow` program ends with `Killed(SIGSEGV, "stack overflow")` and leaves no frames behind | | `tests/process.rs` | **the process model** (stage 14), mostly by running the user test programs: `cowtest` (globals, heap and stack stay separate after fork in both directions, three generations, `MAP_SHARED` vs `MAP_PRIVATE`); `forktest` (fork/wait4/getppid, 10 children reaped with `wait4(-1)`, `ECHILD`, execve errors, vfork + exec, `pipe2(O_CLOEXEC)`, fork + dup2 + exec through a pipe, a child dying of SIGSEGV); **exec argv/envp/auxv** (`forktest args "two words" ""` with `FOO=bar EMPTY=`: argc, every argv and envp string, `AT_PAGESZ`, `AT_ENTRY` = `_start`, `AT_PHDR` covering the entry, `AT_RANDOM`, `AT_EXECFN`, stack position); `mmaptest` (zeroed anonymous memory, 64 MiB that costs nothing until touched, SIGSEGV handlers that see `si_addr` and `SEGV_ACCERR`/`SEGV_MAPERR` and fix the fault so the access is retried, `PROT_NONE`, file-backed private mappings, `brk`, `EINVAL`s, memory back to the start); an unhandled fault is `Killed(SIGSEGV)` with status 11 and exit code 139; `sigtest` (handler round trip through sigreturn with registers intact, masks and pending signals, SIGKILL can't be caught, default actions, SIGCHLD with `si_pid`/`si_status`, SIGPIPE, stop/continue seen by `wait4(WUNTRACED/WCONTINUED)`, `EINTR` vs `SA_RESTART`, `kill` errors, signal 0, `kill(-pgid)`); wait-status encoding; a **fork bomb** stops at `EAGAIN`, is killed and reaped, and leaves frames and the process count unchanged; **20 fork/exec/exit loops leak no frames** and leave no shared frames; **Ctrl+C** (`tty::input_char('\x03')`) kills `spin` in the foreground group with SIGINT (exit code 130), and does nothing when no group is in the foreground | | `tests/libc.rs` | **libc plumbing** (stage 15), by running four user test programs twice each (a warm-up, then a run that must leave the thread, process, futex-waiter, pty, socket and frame counts exactly as they were): `threads` (4 threads sum 1..=40,000,000 on several CPUs; a futex mutex keeps a shared counter exact while an unlocked one loses updates; each thread sees its own TLS value through `fs:`; `gettid`; `FUTEX_WAIT` with a stale value is `EAGAIN`, times out, `FUTEX_WAKE` with no waiters; `CLONE_CHILD_CLEARTID` + futex wake for join; `exit` ends one thread, `exit_group` all; `tgkill` and `ESRCH`); `echoserver` (an epoll server and a poll server on 127.0.0.1, three forked clients each, plus UDP over loopback); `ptytest` (canonical mode holds a half line back, backspace editing, CR -> NL, echo, ONLCR; raw mode passes bytes through without echo; `TCGETS`/`TCSETS`; ^C and ^\\ kill the child that made the pty its controlling terminal with SIGINT/SIGQUIT and echo "^C"; `/dev/tty` with and without a controlling terminal; closing the master gives the slave EOF); `nbtest` (`EAGAIN` from pipes, sockets and ptys; 64 KiB pipe capacity; poll bits; `fcntl(F_SETFL, O_NONBLOCK)` and `FIONBIO`; nonblocking `connect` is `EINPROGRESS` then `POLLOUT`; `select`, `ppoll`, `readv`/`writev`, `pread`/`pwrite`, `getcwd`, `uname`, `CLOCK_REALTIME`, `nanosleep`, `readlink /proc/self/exe`) | How it works: `#![feature(custom_test_frameworks)]` collects every `#[test_case]` function and passes them to `rustbelt::test_runner` (in `src/lib.rs`), which prints each name to serial, runs it, prints `[ok]`, then exits QEMU with success. A panic goes to `test_panic_handler`, which prints `[failed]` and exits with failure. Extra args after `--` in `cargo run -- ...` are appended to the QEMU command line, e.g. `cargo run -- -d int -no-reboot` to log every interrupt and stop on triple faults, or `cargo run -- -s -S` to wait for `gdb` on port 1234. ## Project layout ``` .cargo/config.toml build-std (core, alloc) + custom target + scripts/runner.sh as the cargo runner rust-toolchain.toml pin nightly + rust-src + llvm-tools-preview x86_64-rust_kernel.json custom target: no OS, no red zone, soft-float, abort on panic, rust-lld Cargo.toml deps, features, test settings, QEMU run/test args src/lib.rs the kernel library: module list, init(), kprintln!, test framework, exit_qemu src/main.rs the bootable binary: kernel_main (init, then a thread that runs the shell and restarts it when it exits), the `demos` feature's demos, panic handler src/gdt.rs GDT (kernel + user segments) + TSS (double-fault IST stack, rsp0 per thread); one GDT + TSS per CPU src/interrupts.rs IDT; breakpoint / page fault / GPF / invalid opcode / double fault handlers (faults from ring 3 kill the process; guard-page hits kill with "stack overflow"); 8259 PIC; local APIC timer, reschedule IPI; PIT handler (uptime, ticks, wakes sleeping threads), keyboard handler (just queues the scancode) src/memory.rs OffsetPageTable setup from CR3, example mapping, BootInfoFrameAllocator, global frame allocator + free list (free_frame), per-process address spaces (and freeing them), user pointer checks, kernel stack mappings src/allocator.rs heap location/size, init_heap (maps heap pages), #[global_allocator], Locked src/allocator/bump.rs bump allocator (not active, for learning) src/allocator/fixed_size_block.rs fixed-size-block allocator (ACTIVE), falls back to linked_list_allocator src/task/mod.rs async: Task (pinned boxed future + id), yield_now() src/task/simple_executor.rs first executor: polls every task in a loop (no wakers) src/task/executor.rs real executor: task queue + wakers, sleeps with hlt when idle src/task/keyboard.rs lock-free scancode queue, ScancodeStream, async keyboard task (keys -> tty) src/thread.rs threads + the multi-core scheduler: spawn, one shared run queue, per-CPU idle threads, asm context switch, sleep/wake, WaitQueue, join, stack-overflow kills src/stack.rs kernel stacks: 64 KiB each with an unmapped guard page below, in their own region (L4 slot 200), slots reused when threads exit src/smp.rs multi-core bring-up: AP trampoline (real mode -> long mode, in assembly), INIT/SIPI, per-CPU data (swapgs), CPU numbering, waking idle CPUs src/apic.rs local APIC: enable, timer (calibrated against the PIT), EOI, IPIs src/acpi.rs finds the ACPI RSDP/RSDT/MADT to list the CPUs src/syscall.rs syscall MSR setup, asm entry stub, dispatch + all syscalls (table at the top) src/trap.rs TrapFrame (all user registers) + the common iretq return to user mode src/process.rs Process (pid, parent, pgid, cwd, env, fd table + close-on-exec, exit status), spawn, fork, execve (argv/envp/auxv stack), exit/kill, wait4 (reaping), groups src/mm.rs a process's memory map: regions (VMAs), demand paging, copy-on-write faults, mmap/munmap/mprotect/brk, fork of an address space src/signal.rs signals: actions, masks, pending sets, default actions, rt_sigframe setup and sigreturn, SIGCHLD/stop/continue, delivery on the way back to user mode src/pci.rs PCI bus scan through config ports 0xCF8/0xCFC, device names src/block/mod.rs BlockDevice trait, device registry (vda, vda1...), RamDisk (for tests) src/block/virtio_blk.rs legacy virtio-blk driver: one virtqueue, polled, bounce buffer src/block/partition.rs MBR + GPT parsing, Partition = a BlockDevice window onto its disk src/elf.rs minimal ELF64 loader (PT_LOAD segments -> regions; program headers for AT_PHDR) src/errno.rs Linux errno numbers src/tty.rs ttys (stage 15): the console, hvc0 and ptys share one line discipline (termios, canonical/raw, echo, ^C/^Z/^\\ -> signals), sessions and controlling terminals, /dev/tty, /dev/ptmx + /dev/pts/N src/poll.rs poll, ppoll, select, pselect6 and epoll (one global "something changed" counter) src/futex.rs futex WAIT/WAKE (+ bitsets, private flag, timeouts): what libc mutexes and pthread_join sleep on src/fs/mod.rs VFS: Inode + FileSystem traits, mount table, path normalize/lookup/create/remove, symlink following (ELOOP after 8), init (boot files), mount_disks (ext2, exFAT, btrfs or FAT partitions/disks -> /mnt/diskN) src/fs/file.rs File trait (what an fd points to), InodeFile (inode + offset), Stat, getdents64 src/fs/ramfs.rs in-memory filesystem src/fs/fat.rs FAT32 via the fatfs crate: BlockIo adapter + FatFs/FatInode for the VFS src/fs/ext2.rs hand-written ext2: superblock, groups, bitmaps, inodes, indirect blocks, directories, symlinks; read + write (the layout is explained at the top) src/fs/exfat.rs hand-written exFAT: boot checksum, bitmap, FAT chains + NoFatChain, up-case table, entry sets + SetChecksum; read + write (layout explained at the top) src/fs/btrfs.rs hand-written read-only btrfs: superblock, chunk tree (single/DUP), root and FS trees, crc32c on nodes and data, inline/regular/compressed extents, holes, symlinks, subvolumes (the format is explained at the top) src/fs/lzo.rs LZO1X decompressor for btrfs's lzo extents src/fs/pipe.rs pipes: a byte queue with a blocking read end and a write end build.rs compiles user/*.rs to ELF files, then builds target/disk.img (GPT + FAT32) and the ext2 images target/ext2.img, ext2-4k.img, ext2-rev0.img (mke2fs -d), and the exFAT images target/exfat.img, exfat-512.img (mkfs.exfat + populator), and the btrfs images target/btrfs*.img (mkfs.btrfs --rootdir, btrfs-convert) with manifests of their files (target/btrfs-*.manifest) build/exfat_populate.rs build.rs's own little exFAT writer: copies a tree into a fresh mkfs.exfat image disk/ files copied onto the FAT32 disk (readme.txt, notes/todo.txt, scripts/selftest.sh = the headless shell session) disk2/ files copied onto the ext2 disk (hello.txt, readme.txt, docs/...); build.rs adds big.txt, sparse.bin and two symlinks disk3/ files copied onto the exFAT disk (long, mixed-case, Unicode names, Photos/...); build.rs adds numbers.txt and frag-chain.txt disk4/ files copied onto the btrfs disks (hello.txt, docs/, Music/, projects/ and archive/ become subvolumes); build.rs adds big.txt, random.bin, zeros.bin, empty.txt and symlinks user/ulib.rs tiny user-space "libc": syscall wrappers (fork, execve, wait4 + W* macros, mmap, signals with a restorer), println!/eprintln!, a brk-based heap (Vec/String), line reading, strerror, _start reading argv/envp/auxv, panic handler user/link.ld linker script for user programs (base 0x80_0040_0000) user/sh.rs the shell: line editor, builtins, aliases, PATH, pipes, redirection, fork + exec, job control (process groups, Ctrl+C, kill, fg, bg) user/shparse.rs the shell's parser + drive letters (also compiled into tests/shell.rs) user/{ls,cat,wc,grep,mkdir,rm,touch,cp,sleep}.rs the shell's tools user/{hello,evil,spin,demo,child}.rs test/demo programs (all programs go to /mnt/disk0/bin) user/sysinfo.rs CPUs, per-CPU load bars, threads, processes, memory (the sysinfo syscall) user/overflow.rs recurses forever: killed with "stack overflow" by the user stack guard zone src/time.rs microsecond clock: the TSC, calibrated against the PIT at boot src/net/mod.rs networking: the smoltcp Interface, DHCP + DNS, the `netd` thread, IRQ handling src/net/virtio_net.rs legacy virtio-net driver: RX/TX virtqueues, MAC src/net/device.rs smoltcp's Device: frames for our own MAC/IP (127.x) loop back, the rest go to the NIC src/net/socket.rs BSD-style sockets (TCP, UDP, ICMP) on smoltcp, blocking on a wait queue user/{ifconfig,ping,host,fetch,httpd}.rs the network programs user/{random,getty,tile}.rs stage 13: getrandom, a shell on hvc0, the framebuffer tiling demo user/{forktest,cowtest,mmaptest,sigtest}.rs stage 14: the process-model checks (each prints ok/FAILED lines and "all checks passed"; tests/process.rs runs them) user/{threads,echoserver,ptytest,nbtest}.rs stage 15: threads + futexes, poll/epoll over loopback, ptys, nonblocking I/O (tests/libc.rs runs them) scripts/hvc0-session.txt the commands run-headless.sh feeds into hvc0 (for `getty -1 /dev/hvc0`) scripts/gpu-check.sh types `tile`, dumps the virtio-gpu screen (PPM), compares pixels scripts/http-reply.sh the host side of `fetch http://10.0.2.100/`: QEMU runs it per connection src/vga_buffer.rs the console writer, print!/println!, print_at (status line), backspace: VGA text mode, or an FbConsole once virtio-gpu is up (use_framebuffer, set_viewport) src/virtio/mod.rs modern (1.0) virtio over PCI: capabilities, feature negotiation, the Transport (common config, notify, ISR, device config), IRQ fan-out src/virtio/queue.rs split virtqueue: descriptor table, available and used rings src/virtio/rng.rs virtio-rng: fill(buf) for getrandom, /dev/random and seeds (+ fallback) src/virtio/console.rs virtio-console (hvc0): 8 RX + 8 TX buffers, IRQ-driven input, log mirroring src/virtio/gpu.rs virtio-gpu 2D: display info, resource, backing, scanout, transfer + flush src/fbcon.rs text console on the framebuffer: cells, cursor, scrolling, dirty rows, a viewport src/font.rs the 8x16 bitmap font (Spleen, BSD-2), Latin-1 + box drawing src/dev.rs devfs at /dev: null, random, urandom, console, tty, ptmx, pts/, hvc0, fb0 src/serial.rs COM1 UART writer + serial_print!/serial_println! tests/ integration tests, one QEMU-booted kernel each scripts/runner.sh `cargo run`/`cargo test` runner: `bootimage runner` + stale-image fix scripts/run-headless.sh headless boot + shell self-test, timeout, exit-code and output checks scripts/screenshot.sh headless boot + a typed shell session + PNG/PPM screenshot via QEMU monitor scripts/ext2-check.sh Rustbelt writes to copies of the ext2 images, then host e2fsck -fn + debugfs scripts/exfat-check.sh Rustbelt writes to copies of the exFAT images, then host fsck.exfat -n + dump.exfat scripts/btrfs-check.sh btrfs check --readonly, then tests/btrfs.rs's file CRCs vs. the host's files boot.png screenshot of the virtio-gpu screen (tile + a shell session) ``` ## What the interrupt code does (chapters 5-7) - **CPU exceptions / IDT** (`interrupts.rs`): the IDT maps vector numbers to handler functions written with `extern "x86-interrupt"` (nightly `abi_x86_interrupt`), so the compiler saves all registers and returns with `iretq`. `_start` fires an `int3` on purpose; the breakpoint handler prints the stack frame and execution continues. A page-fault handler prints the faulting address from CR2. - **Double faults** (`gdt.rs`): a stack overflow hits the guard page -> page fault -> the CPU can't push the exception frame on the broken stack -> double fault -> same again -> *triple fault* = machine reset. We load our own GDT containing a TSS whose Interrupt Stack Table entry 0 points at a separate 20 KiB static stack, and mark the double-fault IDT entry to use it. Try `./scripts/run-headless.sh stack-overflow`. - **Hardware interrupts**: the two 8259 PICs are remapped to vectors 32-47 (0-31 are CPU exceptions) and then `sti` enables interrupts. The timer (IRQ 0, ~18.2 Hz) bumps an atomic tick counter and redraws the uptime clock; the keyboard (IRQ 1) reads the scancode from port 0x60 and queues it; an async task decodes it with `pc-keyboard` (US layout, scancode set 1) and echoes it (see Multitasking below). Each handler must send End-Of-Interrupt to the PIC or no further interrupts arrive. `print!`/`serial_print!` disable interrupts while holding their lock, otherwise an interrupt handler printing while `_start` holds the lock would deadlock. The idle loop uses `hlt`, so QEMU doesn't burn a host CPU core. ## Memory (chapters 8-11) **Paging.** Every address the kernel uses is *virtual*. The CPU translates it with a 4-level page table tree (level 4 -> 3 -> 2 -> 1 -> 4 KiB page); 9 bits of the address index each level, the low 12 bits are the offset in the page. CR3 holds the *physical* address of the level-4 table (here `0x1000`). The `page-fault` feature writes to `0xdeadbeaf`, which no table maps, and the page-fault handler reports the address (from CR2) and the reason (`CAUSED_BY_WRITE`, not `PROTECTION_VIOLATION` -> the page simply isn't present). **Reaching page tables.** Page tables contain physical addresses, but we can only dereference virtual ones. With the bootloader's `map_physical_memory` feature, all of physical RAM is also mapped at `physical_memory_offset` (`0x18000000000` here), so `virt = offset + phys`. `memory::init` wraps that in the x86_64 crate's `OffsetPageTable`, which can `translate_addr` (walk the tables) and `map_to` (add entries). **Virtual memory layout** (as set up by bootloader 0.9 in this build): | Virtual address | What | |---|---| | `0xb8000` | VGA text buffer (identity-mapped: virt = phys) | | `0x200000`... | kernel code + data (the ELF; maps to phys around `0x400000`) | | `0x0100_0000_0000`... | kernel stack (80 pages, guard page below it) | | `0x0180_0000_0000`... | all physical memory (`physical_memory_offset`) | | `0x4444_4444_0000` | kernel heap, 4 MiB (also holds thread stacks) (we map it in `allocator::init_heap`) | | `0xdead_beaf_000` | demo page we map to the VGA frame (writes "New!" on screen) | The "New!" demo writes to screen row 20 through the new mapping; by the time you look, later lines have scrolled it up, so in `boot.png` it overwrote the start of the first line. **Physical frames.** `BootInfoFrameAllocator` hands out 4 KiB frames from the regions the BIOS memory map marks `Usable` (~121 MiB of QEMU's default 128 MiB). It never frees; fine for now. **Heap.** `allocator::init_heap` maps the 1024 pages (4 MiB) of `HEAP_START..+HEAP_SIZE` to fresh frames, then gives the range to the `#[global_allocator]`. From then on the `alloc` crate (`Box`, `Vec`, `Rc`, `String`, `BTreeMap`...) works. `alloc` is compiled for our target by `build-std` in `.cargo/config.toml`. **Allocator designs** (`src/allocator/`): the *bump* allocator just moves a pointer forward and can only reclaim memory when every allocation is freed (it fails the `many_boxes_long_lived` test). The `linked_list_allocator` crate keeps a list of free regions (first fit, slower, but reuses any freed memory). The active *fixed-size-block* allocator rounds each request up to 8/16/.../2048 bytes and keeps one free list per size, so alloc/free are O(1) pops/pushes; bigger requests and empty lists fall back to the linked-list allocator. Switch allocators by editing one line in `src/allocator.rs`. ## Multitasking (async/await + threads) ### Cooperative: async/await (`src/task/`, phil-opp chapter 12) An `async fn` compiles into a state machine (a `Future`). An *executor* calls `poll` on it; the future runs until it has to wait, then returns `Pending` and remembers a `Waker`. Whoever makes progress possible later (here: the keyboard interrupt) calls `waker.wake()`, which puts the task's id back into the executor's queue. When the queue is empty, the executor `hlt`s -- carefully: it disables interrupts, checks the queue, and then uses `sti; hlt` as one step (`enable_and_hlt`), so a wakeup arriving in between can't be missed. The keyboard interrupt handler now does almost nothing: read the scancode from port 0x60 and push it into a fixed-size lock-free queue (`crossbeam_queue::ArrayQueue`), then wake the keyboard task (`futures_util::task::AtomicWaker`). No locks in the interrupt handler means no deadlock if the interrupt arrives while the task holds one. The task (`print_keypresses`) is an `async` loop over a `ScancodeStream` that decodes and echoes characters. "Cooperative" means a task only gives up the CPU at an `.await`. A task that loops forever without awaiting freezes every other task. Upside: switching is cheap (just return from `poll`), and all tasks share one stack. ### Preemptive: kernel threads (`src/thread.rs`, beyond the blog series) Each thread has its own 64 KiB stack (a `KernelStack`, with a guard page under it). On every local APIC timer interrupt (~100 times a second, on each CPU) the handler sends the EOI and then calls `thread::schedule()`, which picks the next ready thread round-robin and calls `switch_context(&mut old.rsp, new.rsp)` (a dozen lines of `global_asm!`): ``` push rbp, rbx, r12, r13, r14, r15 ; callee-saved regs of the OLD thread, onto its stack mov [rdi], rsp ; remember where the old stack ends mov rsp, rsi ; ---- now on the NEW thread's stack ---- pop r15, r14, r13, r12, rbx, rbp ; its callee-saved regs ret ; return to wherever IT called switch_context ``` Why only those six registers? `switch_context` is called like a normal function, so the compiler already saved the caller-saved registers, and the `x86-interrupt` handler around it saved everything the interrupted code was using (plus the CPU pushed RIP/RSP/RFLAGS). So a paused thread's complete state sits on its own stack; the scheduler only stores its `rsp`. The new thread "returns" from `schedule()`, from its timer handler, and `iretq`s back to where it was interrupted. A brand-new thread was never interrupted, so `spawn` hand-builds a stack that looks as if `switch_context` had saved it: six register slots (r12 = the thread's function) and a return address pointing to `thread_trampoline`, which moves r12 into rdi and calls `thread_start(entry)`. That re-enables interrupts (we arrive from inside the timer interrupt, where they're off), runs the function, and marks the thread finished when it returns. The code that booted the kernel becomes thread 0, "main", which runs the async executor. Preemption means even `loop {}` can't hog the CPU (thread C in the demo is a pure busy loop). The price: any code can be interrupted anywhere, so shared data needs locks, and a lock that the timer handler might also want must be taken with interrupts disabled. Here: the print macros and `print_at` already hold their spinlocks with interrupts off (`without_interrupts`), so a thread can't be switched away while holding them; the scheduler lock is only taken with interrupts off and is released *before* switching (the next thread would otherwise wait forever for a lock held by a paused thread). **The locking rule:** a lock that is *ever* taken with interrupts off must *always* be taken with interrupts off. Otherwise: thread T takes the lock with interrupts on, gets preempted, and some code running with interrupts off (so it can never be preempted back to T) spins on that lock forever. Stage 6 hit exactly this with the heap allocator: wait-queue code runs with interrupts off and allocates, so `GlobalAlloc::alloc`/`dealloc` now take the heap lock inside `without_interrupts`. **Blocking (stage 6).** A thread can be *blocked*: the scheduler skips it until someone wakes it. Two ways to be woken: - `sleep_ticks(n)` marks the thread sleeping until tick `now + n`; the PIT handler wakes it once that tick has passed (`nanosleep` sleeps whole 55 ms ticks this way, then yields until the TSC clock says the time is really up). - A `WaitQueue` (a list of waiting thread ids). `wait_until(|| condition)` turns interrupts off, checks the condition, and if it's false puts the thread on the queue, marks it blocked and switches away; when woken it checks again. `wake_all()` unblocks everyone on the queue. Checking with interrupts off is what prevents a *lost wakeup* (the event firing between "check" and "sleep"). Users: the console (`read(0)` waits for Enter; the keyboard path wakes it), pipes (reader waits for data or EOF) and `wait(pid)` (parent waits on the child's `exited` queue). If every thread is blocked, the scheduler idles with `sti; hlt` until an interrupt makes something runnable. **Stack size.** Stage 6 first used 16 KiB stacks and crashed with a ring-0 jump to address 0: the syscall -> VFS -> FAT -> ELF loader path in an unoptimized debug build used more than that, overflowed into the heap and smashed a neighbour. Stacks are 64 KiB now, and since stage 11 each has a real guard page under it (see "Multiple CPUs" below), which replaced the old canary check. Stage 11 rewrote the scheduler for several CPUs; the next section explains what changed. ## Multiple CPUs, sleep/wake and guard pages (stage 11) Until stage 10, Rustbelt used one CPU. QEMU now runs with `-smp 4`, and the kernel starts all four. ### Starting the other CPUs (`src/smp.rs`, `src/apic.rs`, `src/acpi.rs`) At power-on only one CPU runs: the *bootstrap processor* (BSP, CPU 0). The others (*application processors*, APs) wait, halted, until the BSP sends them messages through its **local APIC**, a small interrupt controller built into every CPU core: 1. **Which CPUs exist?** The firmware describes them in ACPI tables. `acpi.rs` finds the RSDP (a signature in the BIOS area), follows it to the RSDT, then to the MADT, which lists one entry per CPU with its APIC id. 2. **Trampoline.** An AP starts in 16-bit *real mode*, like an 8086, at an address below 1 MiB. So `smp.rs` copies a small assembly trampoline to physical address `0x8000`, plus parameters: the page table (CR3), the control registers, a fresh kernel stack, the CPU number and the Rust function to call. The trampoline jumps straight from real mode into 64-bit long mode using the BSP's settings. 3. **INIT, then STARTUP.** The BSP sends an *INIT* inter-processor interrupt (IPI), waits ~110 ms (two PIT ticks), then a *STARTUP* IPI whose vector (`0x08`) means "start at `0x8000`"; a second STARTUP if the AP didn't answer. Then it waits for the AP to say it's ready, one CPU at a time. 4. **AP setup** (`ap_entry`): its own GDT and TSS (with its own double-fault stack), the shared IDT, the syscall MSRs, its local APIC and timer, and then it becomes an idle thread in the scheduler. **Per-CPU data.** Some things must exist once per CPU: the TSS (it holds the stack used when an interrupt arrives from ring 3), the "current thread", the syscall entry's scratch slots. `cpu_id()` reads the local APIC id and maps it to 0..3. The syscall entry can't call Rust before it has a stack, so it uses `swapgs`: the `KERNEL_GS_BASE` MSR of each CPU points at that CPU's `PerCpu { syscall_kernel_rsp, syscall_user_rsp }`, and `swapgs` makes `gs:[0]`/`gs:[8]` reach them. **Timers.** The old PIT (on the 8259 PIC) still runs, on CPU 0 only: it keeps the uptime and wakes sleeping threads. Each CPU's local APIC timer, calibrated against the PIT to ~10 ms, preempts whatever that CPU runs. **No I/O APIC**: it routes device interrupts to chosen CPUs, but Rustbelt's devices are polled (virtio-blk) or happy on CPU 0 (keyboard, PIT), so the 8259 is kept and wired to CPU 0's local APIC (LINT0 in "ExtINT" mode). Moving to the I/O APIC is a good exercise once interrupt-driven virtio needs it. ### The scheduler (`src/thread.rs`) - **One shared ready queue** (a `VecDeque`) under one lock. Each CPU has its own *current* thread and its own *idle* thread, which runs `sti; hlt` when there's nothing to do. Simple and fair; per-CPU queues with work stealing would scale better, but four CPUs don't need them. - **Thread states:** `Ready`, `Running`, `Blocked` (on a wait queue), `Sleeping(until tick)`, `Finished`. - **Handing a thread to another CPU safely.** When CPU 0 switches away from thread T, T stays marked `on_cpu` until CPU 0 has fully left T's stack (`finish_switch`, right after the switch). Until then no other CPU may pick T, or two CPUs would run on the same stack. - **Sleeping without polling.** `sleep_ticks(n)` marks the thread `Sleeping(now + n)` and switches away; it's not on the ready queue, so it costs nothing. The PIT handler wakes threads whose time has come (`wake_sleepers`) and, if some CPU is idle, sends it a *reschedule IPI* (vector `0x41`) so it picks the thread up at once instead of at its next timer tick. - **Wait queues.** `queue.wait_until(|| condition)` blocks until the closure returns `Some`; `queue.wake_all()` wakes the waiters. The classic *lost wake-up* (the event happens between "check" and "block", possibly on another CPU) is prevented by joining the queue *before* checking, plus a `woken` flag: a wake that arrives early makes the block return immediately. Console input, pipes, `wait`, `join` and the async executor (when it has nothing to do) all use wait queues. - **Statistics.** Each local APIC tick counts as *busy* or *idle* for that CPU; the `sysinfo` syscall (502) returns these, and `user/sysinfo.rs` turns two samples into load bars. **The locking rule now matters more:** a lock that is ever taken with interrupts off must always be taken with interrupts off. On one CPU breaking it deadlocks only sometimes; on four, a thread spinning with interrupts off on CPU 1 can wait forever for a lock held by a preempted thread. The scheduler, frame allocator, heap, process table and kernel page-table locks all follow it. ### Memory cleanup Everything a thread or process had is given back when it ends: - A finished thread can't free the stack it's running on, so the *next* thread on that CPU frees it (`finish_switch`), before removing the thread from the table. So when `join` returns, the memory really is free. - A process's address space (its page tables at every level, code, data, heap, user stack) is freed in `Drop for Process`, when the last `Arc` to it goes away. `wait` joins the process's thread before reaping, so the parent holds the last reference. - Children whose parent exits first are *orphaned* and clean themselves up when they exit. `memory::frames_in_use()` counts frames; the headless run and `tests/smp.rs` check it returns to exactly the same number after hundreds of threads and processes. ### Guard pages (`src/stack.rs`, `src/process.rs`) A stack grows down. If it grows past its end it would silently overwrite whatever lies below. A **guard page** is an address range under the stack that is deliberately *not mapped*: touching it causes a page fault, so the overflow is caught at the first byte. - **Kernel stacks** live in their own region of the address space (L4 slot 200, `0x0000_6400_0000_0000`): 4 KiB guard page, 64 KiB stack, guard page, stack, ... When a kernel stack overflows, the CPU can't even push the page-fault frame (it would land in the guard page too), so it raises a *double fault*, which runs on its own IST stack. That handler sees the faulting address is a guard page and prints `[kernel] kernel stack overflow in thread 'NAME' (hit guard page 0x...)`, then kills that thread (or its process) and keeps going. - **Stack probes.** A function with a big local array could jump *over* a 4 KiB guard page. `"stack-probes": {"kind": "inline"}` in the target JSON makes the compiler touch each page of a big frame in order, so it can't. - **User stacks** (1 MiB since stage 14, pages appearing as it grows) get a 64 KiB never-mapped zone underneath, and the ELF loader refuses programs whose segments reach into it. A fault there is a SIGSEGV, reported as "stack overflow" (exit code 139). Try `overflow` in the shell. Kernel stacks are reused: a freed stack's slot goes on a free list and its page tables stay, so they never need freeing. **Why no TLB shootdown?** When a stack is freed, other CPUs may still have its old addresses cached in their TLB (the CPU's cache of page translations). Normally the freeing CPU would send them all an IPI to flush it. Here nothing can use a freed slot until a *new* thread gets it, so each thread instead flushes its stack's pages (`invlpg`) on each CPU the first time it runs there. Cheaper and simpler, and only possible because nobody else ever touches a thread's stack. Known simplifications (good exercises): no priorities; one shared run queue; no CPU affinity; device interrupts all go to CPU 0. ## User mode, syscalls and processes The user programs in `user/` run in **ring 3** as separate processes, each with its own address space and file-descriptor table. The shell and its tools are described in [The shell](#the-shell-stage-7). These are the test and demo programs from stages 5 and 6 (run them from the shell: `run hello`, `run evil`...): | Program | What it does | |---|---| | `hello` | prints "Hello from user mode!", `getpid`, reads `/hello.txt` from the ramfs, creates and reads back `/tmp/note.txt`, checks error cases (`ENOENT`, `EBADF`, `EFAULT`, `ENOSYS`), exits with the number of failed checks (0) | | `evil` | reads kernel memory at `0x200000` -> page fault in ring 3 -> the kernel kills it and keeps going | | `spin` | busy-loops ~2 s with no syscalls; it still gets preempted (threads A/B/C keep printing) | | `demo` | stage 6 checks: cwd, reads `/mnt/disk0/readme.txt`, `stat`/`fstat`/`lseek`, lists `bin/` with `getdents64`, `chdir` errors, `mkdir`/`unlink`/`rmdir` on ramfs and FAT, a pipe from a child, arguments, `wait` exit codes (42, 139 for killed `evil`), `ECHILD`/`ENOENT`/`ENOEXEC`, `sleep_ms`; exits with the number of failed checks | | `child` | helper for `demo` and tests: `child pipe` writes a line to stdout, `child cat` copies stdin to stdout, `child 42` exits with 42, otherwise prints its arguments | All of them are read from `/mnt/disk0/bin/`, on the FAT32 partition of the virtio disk; they're not compiled into the kernel any more. ### Rings in one paragraph The CPU's *current privilege level* (CPL) is the low 2 bits of the CS register: 0 = kernel, 3 = user. In ring 3 the CPU refuses privileged instructions (`hlt`, `cli`, writing CR3, `in`/`out`...) and refuses to touch any page whose page-table entries lack the `USER_ACCESSIBLE` bit. User code can only get back into ring 0 through doors the kernel set up: interrupts/exceptions (via the IDT) and the `syscall` instruction (via the LSTAR MSR). Every such door switches to a kernel stack, so a user program can't corrupt kernel state through its own stack pointer. ### GDT and TSS (`src/gdt.rs`) ``` 0x00 null 0x08 kernel code (ring 0) 0x10 kernel data (ring 0) 0x18 user data (ring 3) |3 0x20 user code (ring 3) |3 0x28 TSS (16 bytes: two slots) ``` The odd order (user *data* before user *code*) is forced by `sysret`: it loads SS = STAR[63:48] + 8 and CS = STAR[63:48] + 16, so the two user segments must be adjacent in that order. The TSS has two jobs: IST[0] is the double-fault stack (stage 2), and **`rsp0`** is the stack the CPU switches to when an interrupt arrives while in ring 3. Each thread has its own kernel stack, so the scheduler writes the next thread's stack top into `TSS.rsp0` on every switch (`gdt::set_kernel_stack`). Since stage 11 every CPU has its own GDT and TSS (the TSS is per CPU by nature: it says where *this* CPU's interrupts land). ### The syscall path (`src/syscall.rs`) `syscall` was chosen over `int 0x80`: it's what Linux uses on x86_64, it's faster, and the setup is only four MSRs. The catch is that, unlike an interrupt, `syscall` does **not** switch stacks -- the entry code must. Setup (`syscall::init`): `EFER.SCE = 1` (enable), `STAR` = the selectors above, `LSTAR` = address of `syscall_entry`, `SFMASK = IF|DF|TF` (so we enter with interrupts off until we're on a kernel stack). One call, step by step: 1. User code puts the number in `rax`, args in `rdi, rsi, rdx` (Linux ABI; we also save `r10, r8, r9`) and executes `syscall`. 2. CPU: `rcx` = user RIP, `r11` = user RFLAGS, CS/SS = kernel, RFLAGS &= ~SFMASK, RIP = LSTAR. RSP is still the *user* stack. 3. `syscall_entry` (asm): `swapgs` to reach this CPU's `PerCpu` data, stash user RSP there, load this thread's kernel stack top (set by the scheduler), push user RSP, `swapgs` back, push `rcx`, `r11` and the args. 4. Shuffle into the C ABI and call `dispatch(nr, a1..a5)` in Rust with interrupts re-enabled (so a long syscall, e.g. a blocking `read`, can be preempted). 5. `dispatch` validates user pointers (`user_slice`: the range must be inside the user half *and* inside the process's memory regions; pages not there yet are faulted in, copy-on-write pages copied before a write), runs the call, and returns a value or `-errno` in `rax`. 6. Since stage 14 the entry saves *all* registers as a `TrapFrame` (`src/trap.rs`) and returns through the same exit as interrupts: deliver any pending signal (which may rewrite the frame to enter a handler), pop everything back and `iretq`. Stage 13 used `cli` + `sysretq` (RIP = rcx, RFLAGS = r11), which is faster but can't restore `rcx` and `r11`, and a return from a signal handler needs every register exactly as it was. With several CPUs, the "current kernel stack" and "saved user RSP" can't be globals (two CPUs can be in a syscall at once), so they're per-CPU, reached with `swapgs` (see "Multiple CPUs" below). Numbers are Linux's wherever Linux has the same call, so man pages and `strace` habits mostly apply. Args 4 and 5 go in `r10` and `r8` like Linux. | nr | name | args | notes | |---|---|---|---| | 0 | `read` | fd, buf, len | console: blocks until Enter; pipe: blocks until data or EOF; socket: like `recv` | | 1 | `write` | fd, buf, len | console = screen + serial; pipe: `EPIPE` **and SIGPIPE** once the reader is gone; socket: like `send` | | 2 | `open` | path, path_len, flags | `O_CREAT` 0o100, `O_TRUNC` 0o1000, `O_APPEND` 0o2000 (every write goes to the end: `>>`); `O_CLOEXEC` 0o2000000 (closed by `execve`); directories can be opened (for `getdents64`) | | 3 | `close` | fd | | | 4 | `stat` | path, path_len, *Stat | follows symlinks | | 5 | `fstat` | fd, *Stat | | | 7 | `poll` | *pollfd, n, timeout_ms | **stage 15**: -1 = forever; pipes, sockets, ttys, files, epoll fds | | 8 | `lseek` | fd, offset, whence | `SEEK_SET/CUR/END` = 0/1/2; `ESPIPE` on console/pipes | | 9 | `mmap` | addr, len, prot, flags, fd, offset | MAP_PRIVATE/SHARED, MAP_ANONYMOUS, MAP_FIXED; files: private only | | 10 | `mprotect` | addr, len, prot | PROT_READ 1, PROT_WRITE 2, PROT_EXEC 4 | | 11 | `munmap` | addr, len | | | 12 | `brk` | addr | returns the (new) end of the heap; brk(0) = current | | 13 | `rt_sigaction` | sig, *new, *old, 8 | kernel `struct sigaction`; handlers need SA_RESTORER | | 14 | `rt_sigprocmask` | how, *set, *old, 8 | SIG_BLOCK 0 / SIG_UNBLOCK 1 / SIG_SETMASK 2 | | 15 | `rt_sigreturn` | | only from a signal handler's restorer | | 16 | `ioctl` | fd, request, arg | any fd, **stage 15**: `FIONBIO` (0x5421). Ttys (console, hvc0, ptys), **stage 15**: `TCGETS`/`TCSETS`/`TCSETSW`/`TCSETSF` (0x5401-0x5404, the 36-byte kernel `struct termios`), `TIOCSCTTY`, `TIOCNOTTY`, `TIOCSWINSZ`, `TIOCGPTN`/`TIOCSPTLCK` on `/dev/ptmx`. Console: `TTY_SET_RAW` (1): raw mode on (arg 1) / off (0); returns the old mode. `/dev/fb0`: `FB_GET_INFO` (0x4600) writes `[width, height, bytes per row, 32]` to `*arg`; `FB_FLUSH` (0x46f0) shows `*[x, y, w, h]` (arg 0: the rows this fd wrote); `FB_FILL` (0x46f1) fills `*[x, y, w, h, 0xRRGGBB]`; `FB_SET_CONSOLE` (0x46f2) moves the text console into `*[x, y, w, h]` (arg 0: full screen) and returns `cols << 16 \| rows`. Console, stage 14: `TIOCGPGRP`/`TIOCSPGRP` (0x540f/0x5410) get/set the foreground process group (Ctrl+C goes to it), `TIOCGWINSZ` (0x5413) the size in characters. Others: `ENOTTY` | | 17 | `pread64` | fd, buf, len, offset | **stage 15**: regular files; leaves the file position alone | | 18 | `pwrite64` | fd, buf, len, offset | **stage 15** | | 19 | `readv` | fd, *iovec, n | **stage 15**: `struct iovec {base, len}`, up to 1024 | | 20 | `writev` | fd, *iovec, n | **stage 15** | | 22 | `pipe` | *[u32; 2] | [read end, write end]; `pipe2` (293) also takes `O_CLOEXEC` | | 23 | `select` | n, *rd, *wr, *ex, *timeval | **stage 15**: fd bitmaps (n <= 1024) | | 24 | `sched_yield` | | | | 32 | `dup` | fd | lowest free fd | | 33 | `dup2` | old, new | closes `new` first if open | | 34 | `pause` | | sleeps until a signal; returns EINTR | | 35 | `nanosleep` | *timespec, *remaining | **Linux's `nanosleep` since stage 15** (stages 2-14 had our own `sleep_ms(ms)` here). Sleeps whole PIT ticks, then yields until the TSC clock says the time is up; a signal ends it early with `EINTR` and the time left in `*remaining` | | 39 | `getpid` | | | | 41 | `socket` | domain, type, protocol | `AF_INET` (2) only; `SOCK_STREAM` (1) = TCP, `SOCK_DGRAM` (2) = UDP, or `SOCK_DGRAM` + `IPPROTO_ICMP` (1) = ping socket (like Linux's unprivileged ping sockets). Returns an fd; `close` closes it | | 42 | `connect` | fd, *sockaddr_in, len | TCP: blocks for the handshake (10 s, `ETIMEDOUT`; `ECONNREFUSED` on a reset). UDP: sets the default peer | | 43 | `accept` | fd, *sockaddr_in, *len | blocks for a connection on a listening socket; returns a new fd | | 44 | `sendto` | fd, buf, len, flags, *sockaddr_in, len | address 0 = the connected peer. ICMP: you write the echo header, the kernel fills in the id and checksum | | 45 | `recvfrom` | fd, buf, len, flags, *sockaddr_in, *len | blocks (wait queue) until data; 0 = the peer closed (TCP); `EAGAIN` after `SO_RCVTIMEO` | | 49 | `bind` | fd, *sockaddr_in, len | local port (address must be 0 or ours) | | 50 | `listen` | fd, backlog | TCP; up to 4 connections can be waiting | | 54 | `setsockopt` | fd, level, name, *val, len | only `SOL_SOCKET`/`SO_RCVTIMEO` (a `timeval`) | | 56 | `clone` | flags, stack, *ptid, *ctid, tls | fork-like (SIGCHLD, optional CLONE_VFORK), or since **stage 15** a thread: `CLONE_VM|FS|FILES|SIGHAND|THREAD|SYSVSEM` plus optional `SETTLS`, `PARENT_SETTID`, `CHILD_SETTID`, `CHILD_CLEARTID`. The child starts on `stack` with rax = 0 | | 57 | `fork` | | copy-on-write; parent gets the pid, child 0 | | 58 | `vfork` | | fork + the parent waits until the child execs or exits | | 59 | `execve` | path, argv, envp (C strings) | Linux's stack layout: argc, argv, envp, auxv | | 60 | `exit` | code | since **stage 15** ends only the calling *thread*; `exit_group` (231) ends the whole process | | 61 | `wait4` | pid, *status, options, *rusage | **Linux's `wait4` since stage 14** (stage 13 had a simpler `wait`): pid > 0 one child, -1 any, 0 / < -1 a process group; `*status` gets the packed status (`WIFEXITED`, `WEXITSTATUS`, `WIFSIGNALED`, `WTERMSIG`, `WIFSTOPPED`...); options `WNOHANG` 1 (returns 0 if nobody is done), `WUNTRACED` 2, `WCONTINUED` 8; reaps the child; `ECHILD` if none; rusage must be 0 | | 62 | `kill` | pid, sig | pid 0 = my group, -1 = all, < -1 = group -pid | | 63 | `uname` | *utsname | Linux's struct: 6 NUL-padded 65-byte fields: "Rustbelt", "qemu", "0.15.0", "stage 15: libc plumbing", "x86_64", "(none)" | | 72 | `fcntl` | fd, cmd, arg | F_DUPFD 0, F_GETFD 1, F_SETFD 2, F_GETFL 3, F_SETFL 4 (`O_NONBLOCK` on pipes, sockets, ttys), F_DUPFD_CLOEXEC 1030 | | 79 | `getcwd` | buf, len | like Linux since stage 15: writes a NUL and returns the length including it; `ERANGE` if too small | | 80 | `chdir` | path, path_len | per-process cwd, inherited by forked and spawned children | | 83 | `mkdir` | path, path_len | | | 84 | `rmdir` | path, path_len | empty directories only (`ENOTEMPTY`); `EBUSY` on a mount point | | 87 | `unlink` | path, path_len | files and symlinks (the link, not its target); directories give `EISDIR` | | 89 | `readlink` | path, path_len, buf, len | a symlink's target (no NUL); returns its length, cut to `len` like Linux; `EINVAL` if not a symlink. Stage 15: `/proc/self/exe` is the running program's path | | 109 | `setpgid` | pid, pgid | 0 = myself / a new group | | 110 | `getppid` | | 0 if started by the kernel or orphaned | | 111 | `getpgrp` | | | | 112 | `setsid` | | **stage 15**: new session + group, no controlling tty | | 121 | `getpgid` | pid | | | 124 | `getsid` | pid | **stage 15** | | 158 | `arch_prctl` | code, addr | **stage 15**: ARCH_SET_FS 0x1002 / ARCH_GET_FS 0x1003: the TLS pointer | | 186 | `gettid` | | **stage 15**: thread id (= pid for the first thread) | | 200 | `tkill` | tid, sig | **stage 15** | | 202 | `futex` | *u32, op, val, *timespec, *u32, val3 | **stage 15**: FUTEX_WAIT 0, WAKE 1, WAIT_BITSET 9, WAKE_BITSET 10 (+ PRIVATE 128) | | 213 | `epoll_create` | size | **stage 15**: = epoll_create1(0) | | 217 | `getdents64` | fd, buf, len | real Linux `struct linux_dirent64` records (ino, off, reclen, type, name\0) | | 218 | `set_tid_address` | *u32 | **stage 15**: cleared + futex-woken when this thread exits; returns the tid | | 228 | `clock_gettime` | clock, *timespec | microsecond resolution (TSC calibrated against the PIT). `CLOCK_MONOTONIC` (1) counts from boot; since stage 15 `CLOCK_REALTIME` (0) is real Unix time (the CMOS clock read once at boot, plus uptime) | | 231 | `exit_group` | code | ends the process: every thread (stage 15; `exit`, 60, ends one thread) | | 232 | `epoll_wait` | epfd, *epoll_event, max, timeout_ms | **stage 15**: level-triggered; EPOLLONESHOT supported | | 233 | `epoll_ctl` | epfd, op, fd, *epoll_event | **stage 15**: ADD 1, DEL 2, MOD 3 | | 234 | `tgkill` | tgid, tid, sig | **stage 15**: a signal for one thread | | 270 | `pselect6` | n, *rd, *wr, *ex, *timespec, *{*mask, 8} | **stage 15** | | 271 | `ppoll` | *pollfd, n, *timespec, *mask, 8 | **stage 15**: sets the mask while waiting | | 281 | `epoll_pwait` | epfd, *events, max, timeout, *mask, 8 | **stage 15** | | 288 | `accept4` | fd, *addr, *len, flags | **stage 15**: flags: SOCK_NONBLOCK, SOCK_CLOEXEC | | 291 | `epoll_create1` | flags | **stage 15**: EPOLL_CLOEXEC | | 292 | `dup3` | old, new, flags | flags: O_CLOEXEC | | 293 | `pipe2` | *[u32; 2], flags | flags: O_CLOEXEC | | 318 | `getrandom` | buf, len, flags | random bytes from virtio-rng (the host's `/dev/urandom`), up to 64 KiB per call; never blocks, flags ignored. Without virtio-rng: a TSC-seeded xorshift (not secure) | | 500 | `spawn` | path, path_len, argv, argv_len, *SpawnOptions | **ours**: start a program, returns its pid (fork + exec in one step, like `posix_spawn`; the kernel's first shell and `getty` use it, the shell uses fork + exec) | | 501 | `mounts` | buf, len | **ours** (Linux reads `/proc/mounts`): the mount table as text, one `path fstype` line per mount, in mount order; returns the length, `ERANGE` if too small. The shell assigns drive letters from it | | 502 | `sysinfo` | *SysInfo | **ours**: CPU count, the CPU you're on, thread/process/frame counts, uptime ticks, and per-CPU timer ticks and busy ticks (sample twice for the load) | | 503 | `resolve` | name, name_len, *addrs, max | **ours** (Linux does DNS in libc, in user space): IPv4 addresses for a name via the DHCP-given DNS server; returns how many; `ENOENT` / `ETIMEDOUT` | | 504 | `netinfo` | *NetInfo | **ours** (Linux has `ioctl`s and netlink): MAC, IRQ, address/prefix, gateway, DNS, packet and byte counters | | 505 | `trace_me` | on | ours: print my syscalls from now on (survives execve; `run --trace`) | Deviations from Linux worth knowing: - **Paths are pointer + length**, not NUL-terminated C strings (easier to validate, and Rust `&str`s aren't NUL-terminated). Relative paths start at the process's cwd; `.` and `..` are resolved in the kernel (`fs::normalize`). - **`Stat` is tiny**: `#[repr(C)] struct Stat { kind: u64, size: u64, mode: u64 }`, kind 1 = file, 2 = directory, 3 = console, 4 = pipe, 5 = symlink, 6 = socket, 7 = device (`/dev/*`; `getdents64` says `DT_CHR`); `mode` is the permission bits (`0o644`), or 0 on file systems without them (ramfs, FAT). No inode numbers, owners or times yet. - **Stage 14's calls are exactly Linux's.** `execve` takes NUL-terminated C strings and `char **` arrays, `rt_sigaction` takes the kernel's `struct sigaction`, `wait4` writes Linux's status word, `siginfo` and the signal frame have Linux's layout: a C library can use them unchanged (see "Processes" below for how they work). - **spawn as well as fork + exec.** `spawn` (500) does both steps at once, like Windows `CreateProcess` or POSIX `posix_spawn`. What Unix shells do between fork and exec -- pointing fds 0/1/2 at a pipe or a file -- is spawn's `SpawnOptions`: `{ stdin, stdout, stderr: i64 (one of *my* fds, -1 = same as mine), flags }`. `flags = SPAWN_TRACE` (1) prints every syscall the child makes, as `[trace pid] open("notes/todo.txt", 0o0) = 3` in strace style (strings shown, errors as `-2 ENOENT`). The shell now uses fork + exec, and `trace_me` (505) to get the same tracing for `run --trace`. spawn's argv is one buffer with NUL-separated arguments (`"child\0with\0three args"`); either way the program starts with Linux's `argc`/`argv`/`envp`/auxv stack, which `ulib` reads as `arg(i)`/`getenv`/`getauxval`. Errors are negative errno values like Linux (`-2` ENOENT, `-9` EBADF, `-14` EFAULT, `-38` ENOSYS...). Unknown numbers return `-ENOSYS`. ### Address spaces (`src/memory.rs`, `src/process.rs`) Each process gets a fresh level-4 page table. The kernel's L4 entries are **copied** into it (so the kernel, heap, physical-memory map and thread stacks are mapped in every address space and the kernel keeps working after a CR3 switch), and L4 slot 1 is the process's own: ``` 0x0000_0000_0020_0000 kernel code (bootloader 0.9 puts it here) shared, ring 0 only 0x0000_0080_0000_0000 USER_START (L4 slot 1) per process 0x0000_0080_0040_0000 program (.text R-X, .rodata R--, .data/.bss RW-), then the brk heap ... mmap regions, handed out downwards (stage 14, see below) 0x0000_00ff_ffef_f000 user stack (1 MiB, RW-, grows down; a 64 KiB guard zone under it) 0x0000_0100_0000_0000 USER_END 0x0000_0100_0020_0000 boot stack (L4 slot 2) shared 0x0000_0180_0000_0000 all physical memory (phys_mem_offset, slot 3) shared 0x0000_4444_4444_0000 kernel heap (thread kernel stacks live here) shared ``` Real kernels put the kernel in the upper half (`0xffff_8000_...`) and give user space the whole lower half. With bootloader 0.9 the kernel sits low, so here user space is one 512 GiB slot that the kernel doesn't use. Only user pages have `USER_ACCESSIBLE`, which is why `evil` faults: the page at `0x200000` *is* mapped, just not for ring 3 (`PROTECTION_VIOLATION | USER_MODE`). `.text` is not writable and everything else is `NO_EXECUTE` (when EFER.NXE is on). A **process** = a kernel thread (`thread.rs`, it already has a kernel stack and gets preempted) + a CR3 value + a `Process` struct (pid, name, fd table, exit status). The scheduler, on each switch, sets `TSS.rsp0` and the syscall kernel stack to the next thread's stack and writes CR3 if it differs. A new process's thread starts in the kernel, then `iretq`s into ring 3 with a hand-built frame (SS=user data, RSP=user stack, RFLAGS=IF, CS=user code, RIP=ELF entry). When ring-3 code faults (page fault with `USER_MODE`, general protection fault or invalid opcode with CS RPL 3), the kernel doesn't panic. A page fault may just be demand paging or copy-on-write (see stage 14); otherwise the process gets a signal (SIGSEGV, SIGILL, SIGFPE...). If it has no handler for it, it is killed (`Killed(SIGSEGV, ...)`, exit code 139), its thread exits and the scheduler picks someone else. A fault in ring 0 still panics -- that's a kernel bug. ### ELF loading and building user programs (`src/elf.rs`, `build.rs`, `user/`) `build.rs` compiles each `user/*.rs` with plain `rustc` for the built-in `x86_64-unknown-none` target (no build-std needed: it ships a precompiled `core` and uses soft-float, so no SSE state to save). Flags worth knowing: `-C relocation-model=static` (no dynamic loader), `-C code-model=large` (the program lives above 4 GiB; the default model assumes 32-bit addresses and fails with "relocation overflow"), `-T user/link.ld` (one PT_LOAD segment per permission, page-aligned). The large code model names its sections `.ltext`/`.lrodata`/`.ldata`/`.lbss`, so the linker script lists those too -- forgetting that put the code into a non-executable segment and every program died at its first instruction. The ELFs (~15-20 KB) used to be embedded in the kernel with `include_bytes!`; since stage 6 `build.rs` copies them onto the FAT32 disk image instead and `process::spawn_path` reads them through the VFS (see "Disks" below). `user/ulib.rs` is a tiny libc-equivalent included by each program: the `syscall` inline asm, wrappers, `print!/println!/eprintln!`, `_start` (calls `main`, then `exit` with its return value) and a panic handler (`exit(101)`). Since stage 7 it also has a **heap** with a power-of-two size-class allocator (free list per size) as the `#[global_allocator]`; since stage 14 its memory comes from `brk` (up to 1 MiB blocks) instead of a fixed 64 KiB array. A program that says `extern crate alloc;` can use `Vec`, `String` and `format!`; the `x86_64-unknown-none` target ships a precompiled `alloc` too. The loader (~160 lines, no crate) checks the header (magic, 64-bit, little endian, x86_64, `ET_EXEC`), and for every `PT_LOAD` segment: checks it's inside the user range, maps zeroed frames with flags from the segment (`W` -> writable, no `X` -> NX), copies `filesz` bytes through the physical-memory map and leaves the rest (`.bss`) zero. It refuses (`ENOEXEC`) a file where two segments with different permissions share a page. A page has one set of permissions, so the second segment would silently get the first one's. This happened for real in stage 7: `sleep` has no `.data`, so the linker put its `.bss` on the read-only `.rodata` page, and `sleep` died on its first write to a static. `user/link.ld` now aligns the location counter before `.data`, not just the (empty, hence dropped) section. ### VFS (`src/fs/`) and file descriptors Designed like Linux's so real filesystems can plug in later: - `trait Inode` (`fs/mod.rs`): a file, directory or symlink: `kind`, `size`, `mode`, `readlink`, `read_at(offset, buf)`, `write_at`, `truncate`, `lookup(name)`, `create(name, kind)`, `list` (names + kinds), `remove(name)`. Methods have default impls returning an error, so a read-only fs implements only what it supports. - `trait FileSystem`: `name()` and `root() -> Arc`. A disk filesystem will hold its block device and parsed superblock. - **Mount table**: `mount("/tmp", fs)`; `lookup(path)` picks the longest mount point that prefixes the path, then walks the rest with `Inode::lookup`. Boot mounts a ramfs on `/` (`/hello.txt`, `/etc/motd`, `/tmp`, `/mnt`), a second, separate ramfs on `/tmp`, and each FAT partition at `/mnt/diskN`. `fs::normalize(cwd, path)` turns any path into a clean absolute one first. - `trait File` (`fs/file.rs`): what a file descriptor points to: `read(buf)`, `write(buf)`, `seek`, `stat`, `getdents`. `InodeFile` = an inode + its own offset (two opens of the same file have separate offsets, like Linux's `struct file`). `tty::Console` is another `File`: writes go to VGA+serial, reads come from the keyboard line buffer. `fs/pipe.rs` gives two more: a pipe's read and write ends. - **fd table**: `Vec>>` per process (max 16), fds 0-2 = Console. `open` = `fs::lookup` (or `fs::create` with `O_CREAT`) + `InodeFile::open` + first free slot. Keyboard input for `read(0)` goes through a small tty line discipline (`src/tty.rs`), like Linux's termios: - **Cooked mode** (the default): the kernel echoes the characters and Backspace edits the current line. Enter makes the line, with its `\n`, available to readers. Programs like `cat` with no arguments see this. - **Raw mode** (`ioctl(0, TTY_SET_RAW, 1)`): every key goes to the reader at once and nothing is echoed. Backspace is `\x08`, and the arrows, Home, End and Delete arrive as ANSI escape sequences (`ESC [ A` = up...). The shell uses this while you edit a line, so it can do the echo and history itself. It switches back to cooked before running a program. Raw mode is also switched off automatically if its owner process exits. The console understands two control characters on output. `\x08` moves back one column and blanks it, and `\x0c` (form feed) clears the screen, which is what `cls` writes. **Process cleanup.** When `wait` reaps a process, `memory::free_address_space` walks L4 slot 1 (the user half) and frees every page, page table and the L4 table. Freed frames go on a free list that `alloc_frame` uses first. The list is an intrusive linked list: each free frame stores the address of the next one, written through the physical-memory map, so it needs no heap. The exiting process disables interrupts before publishing its exit status. That way the parent can't free the address space while the dying thread might still run on it. Known simplifications (exercises): a process nobody waits for keeps its memory; orphans aren't reparented; no signals; user pointer checks walk the page table but don't guard against another thread unmapping the page mid-syscall (there's no `munmap` yet, so that's fine for now). ## Disks: PCI, virtio-blk, partitions, FAT32 (stage 6) ``` /mnt/disk0/readme.txt VFS (fs/mod.rs) longest mount prefix -> FatFs, rest "readme.txt" FatInode (fs/fat.rs) fatfs crate: clusters, FAT chains, directory entries BlockIo adapter byte offsets -> whole 512-byte sectors Partition vda1 sector + 2048, bounds-checked VirtioBlk vda virtqueue request to QEMU PCI found the device and its I/O port at boot ``` **PCI** (`src/pci.rs`). Every PCI device has a 256-byte configuration space (vendor/device id, class code, BARs = where its registers live). On a PC you read it by writing `bus/device/function/offset` to port `0xCF8` and reading port `0xCFC`. `pci::scan()` tries every bus/device/function, keeps the ones whose vendor id isn't `0xFFFF`, and prints them with a class name. QEMU's default machine has 7: host bridge, ISA/IDE/ACPI parts of the PIIX chipset, VGA, the e1000 network card, and our disk, `1af4:1001`. **Why virtio-blk and not ATA PIO?** ATA PIO is simpler (a few port writes, then 256 `in` instructions per sector) but it's a 1980s interface that only emulators and old hardware have. virtio is what QEMU/KVM and cloud VMs actually use, and it teaches the pattern every modern device uses: the driver and device share **queues of descriptors in RAM**. The legacy (virtio 0.9.5) interface is used, through I/O port BAR0, because it needs no MMIO mapping or PCI capability parsing; QEMU's device is "transitional" and speaks both. **virtio-blk in one paragraph** (`src/block/virtio_blk.rs`, ~280 lines). Reset the device, set ACKNOWLEDGE + DRIVER, accept no optional features, read the queue size, allocate the virtqueue (descriptor table + available ring + used ring, physically contiguous: `memory::alloc_contiguous_frames`) and give its page number to the device, set DRIVER_OK. A request is a chain of three descriptors: a 16-byte header (`IN` = read or `OUT` = write, sector number), the data buffer, and a 1-byte status the device fills in. Put the chain's head in the available ring, write the queue number to the notify port, then **poll** the used ring until the device hands it back. Data goes through a one-page **bounce buffer** whose physical address we know (heap buffers are virtually contiguous but not physically). Interrupts are switched off for the device (`VRING_AVAIL_F_NO_INTERRUPT`); polling is fine because QEMU completes requests almost instantly. Exercises: IRQ-driven completion (block the thread on a wait queue), several requests in flight, virtio 1.0 "modern" MMIO. **`BlockDevice`** (`src/block/mod.rs`): `sector_count()`, `read_sectors(lba, buf)`, `write_sectors(lba, buf)`, buffers a whole number of 512-byte sectors. Devices get Linux-style names in a registry (`block::devices()`, `block::get("vda1")`). `RamDisk` is a `Vec` disk used by tests. **Partitions** (`src/block/partition.rs`). Sector 0 is the MBR: four 16-byte entries at offset 446 and the `55 AA` signature. A single entry of type `0xEE` is a "protective MBR" meaning *look for a GPT*: sector 1 has the `EFI PART` header, which says where the partition entry array is (128 bytes each: type GUID, unique GUID, first/last LBA, UTF-16 name). Each partition becomes a `Partition` -- itself a `BlockDevice` that adds its start sector and refuses to go past its end -- registered as `vda1`, `vda2`... CRCs aren't checked yet. **FAT32 via the `fatfs` crate** (`src/fs/fat.rs`). `fatfs` does the filesystem logic; it wants something implementing its `Read + Write + Seek` traits on *bytes*. `BlockIo` is that adapter: it keeps a byte position, sends whole aligned sectors straight to the device, and handles partial sectors through a one-sector buffer (read the sector, change the bytes, write it back). Writes go straight through, so `flush` has nothing to do. (`fatfs` 0.3 from crates.io depends on the abandoned `core_io` crate, which doesn't build on current nightlies, so `Cargo.toml` pins a git revision of the 0.4 branch with its own `no_std` IO traits.) For the VFS, `FatFs` holds the `fatfs::FileSystem` in an `Arc>`, and a `FatInode` is just *(that filesystem, a path, a kind)*: each call reopens its path from the root. Slower than caching directory entries, but simple and never stale. Lookups are case-insensitive, like FAT itself. No timestamps (`NullTimeProvider`) and long file names on. **The test disk** (`build.rs`). After compiling the user programs, `build.rs` makes `target/disk.img` (64 MiB): `sfdisk` writes a GPT with one "Microsoft basic data" partition at sector 2048, `mkfs.vfat -F 32 -s 1 -n TYLERDISK` formats a separate partition image (1 sector per cluster so 62 MiB is enough clusters for FAT32), `mmd`/`mcopy` (mtools, which edit FAT images without mounting, so no root) copy in `disk/` and `bin/`, and the partition image is spliced into the disk at sector 2048. If a tool is missing the build stops with `sudo apt install dosfstools mtools fdisk` (`mkfs.vfat` lives in `/usr/sbin`, which isn't in a normal user's `PATH` on Debian, so `build.rs` looks there too). Look inside it from Linux: ```sh /usr/sbin/sfdisk -l target/disk.img # not in a normal user's PATH on Debian export MTOOLS_SKIP_CHECK=1 # mtools: don't complain about the small-cluster FAT32 mdir -i target/disk.img@@1M ::/bin # @@1M = partition offset (2048 * 512) mtype -i target/disk.img@@1M ::/readme.txt ``` ## The shell (stage 7) `/bin/sh` on the disk (`user/sh.rs`, about 550 lines) is an ordinary ring-3 program. It uses nothing but the syscalls above. Its design is a **hybrid**: Unix navigation and pipes, with DOS-style program running and drive letters on top. At boot a kernel thread starts it on the console and waits for it. When it exits, the thread starts a new one, the way `init` respawns a login shell. ``` Rustbelt shell. Type `help` for commands. Drives: A: = /, B: = /mnt/disk0, C: = /mnt/disk1 A:/ $ cd b: B:/ $ ls bin/ notes/ readme.txt scripts/ B:/ $ type readme.txt | grep -i rustbelt In the Rustbelt shell this disk is drive B: -- try `cd B:` and `ls`. B:/ $ ls bin | wc -l 15 B:/ $ echo first > A:/tmp/a.txt B:/ $ echo second >> A:/tmp/a.txt B:/ $ wc < A:/tmp/a.txt 2 2 13 B:/ $ run --trace cat notes/todo.txt [trace 8] open("notes/todo.txt", 0o0) = 3 [trace 8] read(3, "Done: the shell (cd, ls, cat, run, start"..., 512) = 125 Done: the shell (cd, ls, cat, run, start, pipes, redirection, drive letters). Next: more file systems -- ext2, exFAT, btrfs. [trace 8] write(1, "Done: the shell (cd, ls, cat, run, start"..., 125) = 125 [trace 8] read(3, "", 512) = 0 [trace 8] close(3) = 0 [trace 8] exit(0) B:/ $ start sleep 5 [pid 9] sleep started in the background B:/ $ jobs [pid 9] running start sleep 5 B:/ $ run evil [evil] pid 10: reading kernel memory at 0x200000... [process] pid 10 (evil) killed: page fault [exit code 139] B:/ $ ver Rustbelt 0.8.0 (stage 8: ext2) x86_64 B:/ $ <- Enter on an empty line, a few seconds later [done] pid 9 (start sleep 5), exit code 0 B:/ $ ``` ### Commands | Command | What it does | |---|---| | `cd [dir]` | change directory (`cd` alone = `A:/`); `cd B:` goes to the root of drive B | | `pwd` | print the current directory as a Unix path (the prompt shows the drive form) | | `echo text...` | print the arguments | | `ver` | kernel name and version, from the `uname` syscall | | `drives` | the drive letters and what they map to | | `alias [name=value]` | list aliases or define one: `alias ll="ls -l"` | | `history` | the commands typed so far | | `jobs` / `wait [pid]` | list background jobs (`[1] [pid 5] running ...`) / wait for them (all, or one pid) | | `kill [-SIG] pid\|%job...` / `kill -l` | send a signal (default SIGTERM; `-9`, `-KILL`, `-INT`...); `%1` = job 1, a whole process group; `-l` lists the signals | | `fg [%job]` / `bg [%job]` | continue a stopped job (Ctrl+Z) in the foreground / background | | `cls`, `clear` | clear the screen | | `help` | the command list | | `exit [code]` | leave the shell (with the last command's exit code, or `code`) | | `run [--trace] prog args` | run a program in the foreground; `--trace` prints every syscall it makes | | `start prog args` | run it in the background and print its pid (same as `prog args &`) | | `prog args` | a bare name is looked up in each drive's `bin` directory, in order: `A:/bin`, `B:/bin`... | Programs in `B:/bin` (`user/*.rs`, each 30-150 lines): | Program | What it does | |---|---| | `ls [-l] [path...]` | list a directory, sorted; in columns on the console, one name per line into a pipe or file (like real `ls`); symlinks get an `@`; `-l` shows sizes and ``, plus permissions (`-rw-r--r--`) and `link -> target` on ext2 | | `cat [file...]` | print files, or stdin (also for `-`) | | `wc [-l] [file...]` | lines, words, bytes | | `grep [-i] [-v] [-n] text [file...]` | lines containing `text` (plain text, no regexes); exit code 0 = found, 1 = not found | | `mkdir dir...` | create directories | | `rm path...` | delete files or empty directories | | `touch file...` | create empty files | | `cp src dest` | copy a file (into `dest` if it's a directory) | | `sleep N` / `sleep 250ms` | wait (blocks in the kernel, no busy loop) | | `ifconfig` (`ip`) | the network card, its address from DHCP, gateway, DNS, packet counters | | `ping [-c N] [-i ms] host` | ICMP echo, with times and a summary like the real one | | `host name` (`nslookup`) | look up a name's IPv4 addresses | | `fetch [-o file] http://host[:port]/path` (`wget`) | HTTP/1.0 GET; prints the body, or saves it (`-o D:/page.html`) with a size and speed report; exit 1 on non-200 | | `httpd [-p port] [-n count] [-t secs]` | a tiny web server: `/` is a greeting with a visitor count, other paths serve files from the current directory | | `random [-n bytes]` / `random -d sides [count]` | random bytes as hex (default 16), or fair dice rolls; uses `getrandom` | | `getty [-1] device` | run a shell on another terminal, e.g. `getty /dev/hvc0 &`; restarts it when it exits (`-1`: once) | | `tile [1-6]` / `tile -r` | draw a tiling-window-manager mock-up on `/dev/fb0` and move the console into the first window; `-r` gives the console the whole screen back | | `hello`, `demo`, `evil`, `spin`, `child` | the stage 5/6 test programs: `run demo` runs all the stage 6 syscall checks | | `forktest`, `cowtest`, `mmaptest`, `sigtest` | the stage 14 checks of fork/exec/wait4, copy-on-write, mmap and signals; each ends with "all checks passed" (`forktest bomb` is a bounded fork bomb, `mmaptest crash` dies of SIGSEGV) | | `threads [n]`, `echoserver [port]`, `ptytest`, `nbtest` | the stage 15 checks: a parallel sum on `n` threads with a futex mutex; an epoll + poll echo server over 127.0.0.1 (`echoserver 7070` just serves forever); ptys and the line discipline; nonblocking I/O and the libc extras. Each ends with "all checks passed" | Built-in aliases are the DOS names: `dir` = `ls`, `type` = `cat`, `del` = `rm`, `md` = `mkdir`, `copy` = `cp`; and for networking, `ip` = `ifconfig`, `nslookup` = `host`, `wget` = `fetch`. **Ctrl+C** interrupts the program in the foreground (SIGINT to its process group; the shell shows `[exit code 130] (interrupted: SIGINT)`), and on an empty or half-typed line it just starts a new one. **Ctrl+Z** stops it (`fg` continues it). Since stage 14 the shell starts every program with `fork` + `execve`, in its own process group (see "Processes" below). **Line editing**: Backspace, and Up/Down to browse the history. Left/Right and Tab aren't supported yet. **Scripts**: `sh file` runs the commands in a file and echoes each one after its prompt. If stdin isn't the console (`sh < file`, `cat file | sh`), the shell reads commands from it the same way. `#` starts a comment. ### Pipes and redirection `a | b | c`, `> file` (create or truncate), `>> file` (append) and `< file` can be combined: `cat < in.txt | grep x | wc -l > count.txt`. The operators don't need spaces around them (`ls|wc`). Quotes (`'...'` or `"..."`) group words and hide operators. There are no variables or escapes. How a pipeline runs (`Shell::run_pipeline`): for each stage, the shell picks its stdin and stdout and spawns it. - **stdin**: the `< file`, else the read end of the previous stage's pipe, else the shell's own stdin. - **stdout**: a new `pipe()` to the next stage, overridden by `> file`, else the shell's own stdout. - **Spawning**: the stage starts with `spawn(path, argv, SpawnOptions { stdin, stdout, ... })`. Then the shell **closes its own copies** of all pipe ends and files. A reader only sees end-of-file once every write end is closed, and the shell holds one. Last, it waits for every stage; the last stage's exit code is the pipeline's. A builtin can be a stage too (`help | grep run`): it just writes to the pipe. Pipes have no size limit here, so that can't block. ### Drive letters Drive letters are only shell syntax. The kernel keeps one Unix tree: - **Why start at A:?** In DOS and Windows the system drive is `C:` only for historical reasons. `A:` and `B:` were reserved for floppy drives, so the first hard disk got `C:`, and Windows still keeps that habit. Rustbelt has no floppies, so its letters simply start at the front of the alphabet. (The first version of the shell used `C:` for `/`.) - **Mapping**: `A:` is `/`. `B:`, `C:`... go to the mounts under `/mnt`, in the order they were mounted. The shell reads them from the `mounts` syscall before every prompt, so a newly mounted disk would get a letter right away. Other mounts, such as `/tmp`, stay under A: (`A:/tmp`). - **Translation**: every unquoted word that starts with a known letter is translated before any syscall sees it (`shparse::translate`): - `B:/notes/todo.txt` becomes `/mnt/disk0/notes/todo.txt`. - `b:` becomes `/mnt/disk0` (any case). - `B:notes` becomes `/mnt/disk0/notes`, since drives don't have their own current directory. Quote a word to keep it literal: `grep "B:" readme.txt`. - **Prompt**: the prompt goes the other way (`shparse::display`). The longest matching drive path wins, so `/mnt/disk0/bin` shows as `B:/bin`. - **Unknown letters**: these are left alone, so `cd Q:` says `there is no drive Q:`. - **Stable letters**: since letters follow mount order, `Cargo.toml` and the scripts attach the user-visible disks first (B: FAT32, C: ext2, D: exFAT, E: btrfs) and the test-only images after them. A disk that fails to mount (like the sha256 btrfs test image) gets no `/mnt/diskN` and no letter, so it can't shift the others either. ### How the pieces fit - `user/shparse.rs`: the tokenizer, the parser (`Pipeline` of `Command`s, each with words and redirections) and the drive-letter functions. All of them are pure functions with no syscalls. The same file is compiled into the shell *and* into `tests/shell.rs` (`#[path = "../user/shparse.rs"]`), so the parser is unit-tested inside the kernel test suite. - `user/sh.rs`: the main loop runs these steps: refresh the drives, report finished background jobs (`wait` with `WNOHANG`), print the prompt, read a line, then run it. Before running, each command goes through `prepare()`: it translates drive letters, expands aliases, and strips `run` / `run --trace` / `start` prefixes. The command then runs as a builtin or is spawned with the right fds. - **Kernel support added for the shell**: - `ioctl` (raw mode), `uname`, `mounts`, `wait` flags and `O_APPEND`. - Freeing process memory, so a long session doesn't run out of frames. - strace-style `--trace` output. - The `\x08` and `\x0c` console controls. - A fix for the `|` key (see Notes). ## ext2 (stage 8) The third disk, `target/ext2.img`, holds an **ext2** file system that Rustbelt reads and writes with its own driver, `src/fs/ext2.rs` (about 1100 lines including a long explanation at the top, no crates). It is mounted at `/mnt/disk1`, which is drive **`C:`** in the shell: ``` A:/ $ drives A: / ramfs B: /mnt/disk0 fat32 C: /mnt/disk1 ext2 A:/ $ cd c: C:/ $ ls -l -rw-r--r-- 304000 big.txt drwxr-xr-x docs/ lrwxrwxrwx hello-link -> hello.txt -rw-r--r-- 142 hello.txt drwx------ lost+found/ -rw-r--r-- 284 readme.txt -rw-r--r-- 69206016 sparse.bin C:/ $ cat hello-link Hello from the ext2 disk! This file lives on a second virtio disk, in an ext2 file system that Rustbelt reads and writes with its own driver. C:/ $ cp B:/notes/todo.txt C:/ C:/ $ cat todo.txt | grep -i ext2 Next: more file systems -- ext2, exFAT, btrfs. ``` ### The on-disk layout, for a learner ext2 (1993) was Linux's main file system through the 1990s, and ext3 and ext4 grew out of it. It is the classic file system to learn from, because everything on disk is an **array of fixed-size records** that you find with plain arithmetic. You can follow along with `dumpe2fs target/ext2.img` and `debugfs target/ext2.img` (both in `/usr/sbin`). - **Blocks and groups.** The disk is cut into blocks of 1, 2 or 4 KiB, and the blocks into **block groups** (8192 blocks each with 1 KiB blocks). Our 32 MiB disk has 1 KiB blocks, so it has 4 groups. Each group keeps its own bitmaps and inode table, so a file's inode and its data usually sit close together. ``` byte 0 1024 2048 (block 2) | boot block | superblock | group descriptors | reserved | block bitmap | inode bitmap | inode table | data... <- group 0 | superblock + descriptor backups | block bitmap | inode bitmap | inode table | data... <- group 1 ``` - **Superblock** (always at byte 1024, whatever the block size). It holds the totals: block size (`1024 << s_log_block_size`), blocks and inodes per group, the free counts, the magic number `0xEF53`, the revision and three sets of **feature flags**. `probe()` reads just that magic, and `mount_disks` tries ext2 first, then FAT. - **Feature flags.** There are three sets, and each set tells an old driver what to do with a flag it doesn't know: - **compat**: ignore it. - **ro_compat**: mount read-only (Rustbelt then says `read-only` in the boot log, and writes give `EROFS`). - **incompat**: don't mount at all. ext4's extents are one, so Rustbelt refuses ext4 instead of misreading it. - **Group descriptors** (32 bytes per group, in the block after the superblock). Each says where that group's block bitmap, inode bitmap and inode table are, and how many blocks and inodes are free there. - **Bitmaps.** One bit per block or per inode, 1 = in use. Allocating is "find a 0 bit, set it, decrement the free count in the group descriptor *and* the superblock". Rustbelt allocates a file's blocks in the group of its inode first. - **Inodes.** Inode N is at index `(N-1) % inodes_per_group` of group `(N-1) / inodes_per_group`'s table. Inode 2 is the root directory, and 11 is `lost+found`. An inode has the mode (type + permissions), owner, size, times, link count and **15 block pointers**: ``` i_block[0..12] data blocks 0-11 1 KiB blocks: the first 12 KiB i_block[12] -> 256 block numbers single indirect, up to 268 KiB i_block[13] -> 256 single-indirect blocks double indirect, up to ~64 MiB i_block[14] -> 256 double-indirect blocks triple indirect, up to ~16 GiB ``` `Volume::bmap` turns "block 40 960 of this file" into a disk block by working out which level it falls in and walking down the pointers. A 0 pointer is a **hole**, which reads as zeros. That is how `sparse.bin` is 66 MiB while using only a few blocks. With 4 KiB blocks, each indirect block holds 1024 numbers. - **Directories** are files full of variable-length entries: `inode (u32), rec_len (u16), name_len (u8), file_type (u8), name`. `rec_len` is the distance to the next entry. To delete an entry, Rustbelt adds its length to the previous entry's `rec_len`. To add one, it looks for an entry with spare room behind it, and if there is none, it gives the directory a new block. Revision 0 has no `file_type` byte, so the driver reads the inode to tell files from directories. - **Symlinks.** A target shorter than 60 bytes is stored right inside `i_block` (a *fast* symlink, like `hello-link`). Longer targets get a data block (a *slow* symlink, like `docs/long-link`, 74 bytes). The VFS follows up to 8 links in a path, then gives `ELOOP`. `stat` follows them, `rm` doesn't, and the new `readlink` syscall (89) returns the target, which `ls -l` prints as `name -> target`. ### What the driver does and doesn't do - **Reads:** - The revision 0 and revision 1 formats, with 1 KiB and 4 KiB blocks (2 KiB should work too, but isn't tested). - Direct, single, double and triple indirect blocks, and holes. - Fast and slow symlinks. - Directories with `dir_index` hash trees. The index hides inside entries that plain ext2 code skips. - **Writes:** - Creating files and directories (mode 0644 and 0755), writing, appending, writing past the end (which leaves a hole), truncating to 0, `unlink` and `rmdir`. - The bitmaps, the free counts and the parent's link count stay consistent, so `e2fsck` is happy. Changing a hash-tree directory clears its index flag, like Linux's ext2 does. - **Design:** - No cache: every operation re-reads what it needs, and every write goes straight to the disk. - One `Mutex` per volume. - An `Ext2Inode` is just *(volume, inode number, kind)*. - **Not yet:** - Creating symlinks, hard links, rename, and `chmod`/`chown`. - Deleting a file that has an extended-attribute block (gives `EOPNOTSUPP`). - Writes past 2 GiB. - Real timestamps: there is no clock yet, so new files get the time the image was made. - The backup superblocks are never updated (Linux doesn't either), and there is no journal: a crash in the middle of an operation leaves work for `e2fsck`, just like real ext2. ### The test images (`build.rs`) `build.rs` copies `disk2/` (text files, nested directories) to `target/ext2-tree/` and adds a few generated files: - `big.txt`: 16 000 numbered lines, 304 000 bytes, which reaches the double indirect blocks. - `sparse.bin`: 66 MiB with data only at 0, 40 MiB and 65 MiB, so the last one sits in a **triple** indirect block. - The two symlinks. Then `mke2fs -t ext2 -d ` builds the images without root or mounting: | Image | Size | Block size | Notes | Attached as | |---|---|---|---|---| | `target/ext2.img` | 32 MiB | 1 KiB | revision 1, label RUSTBELT, the full tree | `vdb` -> `/mnt/disk1` = `C:` (run and tests) | | `target/ext2-4k.img` | 8 MiB | 4 KiB | label FOURK, small tree | `vdd` -> `/mnt/disk3`, tests only | | `target/ext2-rev0.img` | 4 MiB | 1 KiB | **revision 0** (`-E revision=0`), small tree | `vde` -> `/mnt/disk4`, tests only | **Raw, not partitioned.** The FAT disk has a GPT because that's what real disks have. The ext2 image is the bare file system, with no partition table, so the host's `e2fsck`, `debugfs` and `dumpe2fs` work on the file directly. `mount_disks` mounts a whole disk if it has no partitions, and its partitions if it does. The disks are numbered in device order: `vda1` = disk0, `vdb` = disk1, `vdc` (exFAT) = disk2. ### Checking Rustbelt's writes with the real e2fsck All QEMU disks normally use `snapshot=on`, so nothing reaches the image files. `scripts/ext2-check.sh` does the opposite. It copies the images to `target/fsck/` and boots Rustbelt on the **writable** copies twice: - once for the shell self-test (`EXT2_IMG=... ./scripts/run-headless.sh`), which does `cp B: -> C:`, `md` and `echo >` on C:. - once for the `tests/ext2.rs` kernel, which creates, writes, truncates and deletes on all three images. Then it runs `e2fsck -fn` on every copy (`-f` = check even if marked clean, `-n` = read-only, change nothing) and shows the root directory with `debugfs`: ```sh cargo test --test ext2 && ./scripts/ext2-check.sh ``` ``` == e2fsck -fn target/fsck/selftest.img (after Rustbelt wrote to it) e2fsck 1.47.2 (1-Jan-2025) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information RUSTBELT: 25/8192 files (0.0% non-contiguous), 2777/32768 blocks ... == debugfs: ls -l / on the self-test copy 23 100644 (1) 0 0 125 30-Oct-2026 01:45 todo.txt 24 40755 (2) 0 0 1024 30-Oct-2026 01:45 made-here == debugfs: cat /made-here/note.txt written by Rustbelt == all images are clean ``` Pass 5 is the one that catches allocation bugs: it recounts every bitmap and compares the result with the free counts in the group descriptors and the superblock. ## exFAT (stage 9) The fourth disk, `target/exfat.img`, holds an **exFAT** file system: the format on SD cards over 32 GB and most USB sticks, which Windows, macOS, Linux, cameras and phones all understand. Rustbelt reads and writes it with its own driver, `src/fs/exfat.rs`: about 1000 lines including a long explanation at the top, written from [Microsoft's exFAT specification](https://learn.microsoft.com/en-us/windows/win32/fileio/exfat-specification), with no crates. It is mounted at `/mnt/disk2`, which is drive **`D:`**: ``` A:/ $ cd d: D:/ $ ls -l 132 Long File Name With Spaces.txt 84 MixedCase.TXT Photos/ 88 a-really-long-file-name-that-takes-several-file-name-directory-entries-in-exfat-xxxxxxxxxxxxxxxxxxxx.txt 38000 frag-chain.txt 340000 numbers.txt 363 readme.txt 61 Ünïcödé café.txt D:/ $ cat "long file name with spaces.txt" This file has a long name with spaces in it. exFAT stores names of up to 255 UTF-16 characters, 15 per "File Name" directory entry. D:/ $ cp C:/hello.txt "From Ext2.txt" D:/ $ type "FROM EXT2.TXT" Hello from the ext2 disk! This file lives on a second virtio disk, in an ext2 file system that Rustbelt reads and writes with its own driver. ``` Names keep the case you typed but are found in any case, like on Windows. Quote names with spaces. Inside quotes the shell doesn't translate drive letters, so `cd D:` first (or write `/mnt/disk2/...`). exFAT has no permissions, so `ls -l` shows sizes only, like on the FAT disk. ### The on-disk layout, for a learner exFAT is FAT's successor, redesigned for big flash cards. If you know FAT32, three things are new: a **bitmap** says which clusters are free, contiguous files can skip the FAT entirely, and a file's directory entry is a **set** of 32-byte entries with a checksum. ```text sector 0 1-8 9-10 11 12-23 FatOffset ClusterHeapOffset | boot sector | ext. boot | OEM | checksum | backup of 0-11 | ... | FAT | ... | cluster 2 | cluster 3 | ... ``` - **Boot sector.** Says `EXFAT ` at byte 3; that's how `mount_disks` recognizes it. Then the geometry, all in sectors: where the FAT and the **cluster heap** start, how many clusters there are, the root directory's first cluster, and the sector and cluster sizes as powers of two (`9` = 512 bytes). Byte 112 is **PercentInUse**, which we keep up to date. - **Boot checksum.** Sector 11 is the 32-bit "rotate right, add the next byte" checksum of sectors 0-10, repeated to fill the sector. It skips the three bytes that change while the volume is in use (VolumeFlags and PercentInUse), so updating those doesn't mean rewriting the checksum. We check it at mount. - **FAT.** One 32-bit entry per cluster: the next cluster of the file, or `0xFFFFFFFF` for the end. Clusters are numbered from 2, and cluster N starts at `ClusterHeapOffset + (N - 2) * cluster size`. - **Everything else lives in clusters**, even the "system files": - **Allocation bitmap**: 1 bit per cluster. This, not the FAT, says what's free. We keep it in memory (960 bytes for our disk) and write each changed byte straight through. - **Up-case table**: the upper-case form of every UTF-16 character. Name lookups compare up-cased names. Because the table is on the disk, every implementation agrees on what "the same name" means, even for `é`/`É`. It's stored compressed (`0xFFFF, n` = "the next n characters map to themselves") and has its own checksum, which we verify. We keep only the characters that change, in a sorted `Vec`, and binary-search it. - **Root directory**: like any directory, except that it has no entry of its own, so its size is the length of its FAT chain. **Directories** are files of 32-byte entries. Byte 0 is the type, and bit 7 of it means "in use". Deleting just clears that bit, and type `0x00` marks the end. The root also holds a volume label entry (`0x83`), the bitmap's (`0x81`) and the up-case table's (`0x82`). A file or directory is an **entry set**: ```text 0x85 File SecondaryCount, SetChecksum, attributes (0x10 = directory), 3 timestamps 0xC0 Stream Extension flags, NameLength, NameHash, ValidDataLength, FirstCluster, DataLength 0xC1 File Name 15 UTF-16 characters <- repeated: a 104-character name needs 7 ``` - **SetChecksum** is a 16-bit "rotate right, add" over every byte of the set except itself. Change a file's size and you must recompute it, or other systems treat the file as corrupt. - **NameHash** is the same checksum over the up-cased name, so a lookup can skip most entries without comparing names. - **NoFatChain** (stream flag `0x02`): "my clusters are consecutive; don't read the FAT". Most files are contiguous, so most files never touch the FAT. When a file must grow and the next cluster is taken, the driver writes FAT links for the clusters it already has, clears the flag, and carries on wherever there's room (`Volume::grow`). Directories work the same way. - **ValidDataLength** <= **DataLength**: bytes past it read as zeros (Windows uses this to create big files quickly). We read it that way and always write the two equal. - Directories have no `.` or `..` entries, and their size is always whole clusters. ### What the driver does and doesn't do - **Reading:** contiguous and FAT-chained files, long UTF-16 names, case-insensitive lookup through the volume's own up-case table, nested directories, the volume label, and both cluster sizes we test (4 KiB and 512 bytes). Damaged entry sets (bad SetChecksum) are skipped. - **Writing:** create, write, append, truncate to 0, mkdir, unlink and rmdir (`ENOTEMPTY` if it isn't empty). That includes: - allocating from the bitmap, preferring the cluster right after the file's last one; - growing directories when a new entry set doesn't fit, the root too; - converting NoFatChain files to FAT chains, freeing chains, and recomputing SetChecksums, NameHashes and PercentInUse. - Names are checked: 1-255 UTF-16 characters, no control characters or `" * / : < > ? \ |`. A name that differs only in case is `EEXIST`. - **Not supported:** - rename and symlinks (exFAT has no symlinks); - real timestamps: there's no clock driver, so everything is 1 Jan 2026; - a second FAT (TexFAT): such a volume mounts read-only; - the VolumeDirty flag (writes go straight through); - shrinking directories (nobody does). - **Like the ext2 driver**, there's no block cache: only the bitmap and one FAT sector are cached. An inode is just the path from the root, resolved again on every call, because exFAT has no inode numbers (a file *is* its directory entry). ### The test images, and why there's a populator (`build.rs`) `mkfs.exfat` (exfatprogs) formats an image, but unlike `mke2fs -d` it can't copy files in, and mtools doesn't speak exFAT. The usual way is to mount the image, but that needs root and a loop device (or FUSE, which doesn't work on this box either). So `build/exfat_populate.rs` is a **small exFAT writer of its own** (~230 lines) that `build.rs` runs after `mkfs.exfat`. It only handles a freshly formatted volume, which makes it simple: clusters are handed out in order, each file is one contiguous NoFatChain run, and files named `frag-*` get every other cluster, linked through the FAT, so the kernel has a fragmented file to read. It deliberately shares no code with the kernel driver, so a misreading of the spec in one doesn't hide in the other, and the host's `fsck.exfat` checks its output on every run of `scripts/exfat-check.sh`. The tree is `disk3/` (long names with spaces, mixed case, non-ASCII names, a 104-character name, `Photos/2026 Trip/trip notes.txt`) plus two generated files: `numbers.txt` (20 000 lines, 340 000 bytes, contiguous) and `frag-chain.txt` (38 000 bytes, fragmented). | Image | Size | Clusters | Notes | Attached as | |---|---|---|---|---| | `target/exfat.img` | 32 MiB | 4 KiB | label TYLEREXFAT, the full tree | `vdc` -> `/mnt/disk2` = `D:` (run and tests) | | `target/exfat-512.img` | 8 MiB | 512 bytes | label SMALLCLUST, small tree: every file spans many clusters | `vdf` -> `/mnt/disk5`, tests only | Like the ext2 disk, it's a bare file system with no partition table, so `fsck.exfat` and `dump.exfat` work on the file directly. An exFAT boot sector also ends in `55 AA`, like an MBR, so the partition scanner checks for `EXFAT ` and reports `no partition table (WholeDiskExfat)` instead of "MBR, 0 partitions". ### Checking Rustbelt's writes with the real fsck.exfat `scripts/exfat-check.sh` works like `ext2-check.sh`. It copies the images to `target/fsck/` and boots Rustbelt on the **writable** copies: once for the shell self-test (`EXFAT_IMG=... ./scripts/run-headless.sh`), and once for the `tests/exfat.rs` kernel. Its last test, `written_files_stay_for_fsck`, deletes nothing, so fsck sees the tricky cases on disk: 300 files in a grown directory that became a FAT chain, deleted entries between live ones, and a file converted from NoFatChain to a FAT chain. Then it runs `fsck.exfat -n` (report, change nothing) on every copy and reads a new file back with `dump.exfat` and `dd`: ```sh cargo test --test exfat && ./scripts/exfat-check.sh ``` ``` == fsck.exfat -n target/exfat.img (pristine, as built) target/exfat.img: clean. directories 3, files 8 == fsck.exfat -n target/exfat-512.img (pristine, as built) target/exfat-512.img: clean. directories 3, files 4 13 exFAT tests ok == fsck.exfat -n target/fsck/selftest-exfat.img (after Rustbelt wrote to it) target/fsck/selftest-exfat.img: clean. directories 4, files 11 == fsck.exfat -n target/fsck/tests-exfat.img (after Rustbelt wrote to it) target/fsck/tests-exfat.img: clean. directories 4, files 212 == fsck.exfat -n target/fsck/tests-exfat-512.img (after Rustbelt wrote to it) target/fsck/tests-exfat-512.img: clean. directories 4, files 32 == dump.exfat -s "/New Folder": the entry set Rustbelt wrote for Note.txt Path: /New Folder/Note.txt Entrytype: 0x85 SetChecksum: 0xAF76 Entrytype: 0xC0 GeneralSecondaryFlags: 0x03 NameLength: 8 NameHash: 0x2C4C ValidDataLength: 14 FirstCluster: 19 DataLength: 14 ... == its 14 bytes at cluster 19, read with dd: hi from exFAT == all images are clean ``` `fsck.exfat` recomputes the bitmap from every file's clusters, and checks each SetChecksum and the boot checksum. A flipped bitmap bit or one changed byte in an entry set gives "files corrupted 1". It doesn't check PercentInUse, so the tests check that themselves: their `usage()` helper reads the bitmap and byte 112 straight off the disk with a few lines of independent parsing. ## btrfs (stage 10) `E:` is a **btrfs** volume, read by a hand-written, **read-only** driver (`src/fs/btrfs.rs`, about 1 200 lines with comments; `src/fs/lzo.rs` for LZO). btrfs is what most of Tyler's Linux machines will use, so this is the file system Rustbelt most needs to *read*. Writing comes much later (see below). ``` E:/ $ ls -l drwxr-xr-x Music/ drwxr-xr-x archive/ <- a read-only subvolume (id 256) -rw-r--r-- 2160000 big.txt <- 17 zlib-compressed extents drwxr-xr-x docs/ -rw-r--r-- 0 empty.txt lrwxrwxrwx hello-link -> hello.txt -rw-r--r-- 78 hello.txt <- an inline extent, inside a leaf drwxr-xr-x projects/ <- a subvolume (id 257) -rw-r--r-- 1500000 random.bin <- incompressible: stored as is -rw-r--r-- 637 readme.txt -rw-r--r-- 1048576 zeros.bin E:/ $ echo hi > new.txt sh: new.txt: Read-only file system ``` ### The ideas, for a learner **Everything is a B-tree item.** ext2 and exFAT have fixed tables (inode tables, a FAT, bitmaps) at fixed places. btrfs has almost none: just the superblock at 64 KiB (magic `_BHRfS_M`). Everything else is an *item* in a B-tree, sorted by a 17-byte key `(objectid, type, offset)`. A file is a handful of items that sit next to each other because they share an objectid (the inode number): ``` (257 INODE_ITEM 0) size, mode, link count, times, flags (257 INODE_REF 256) "my name in directory 256 is docs" (258 DIR_ITEM ) a directory entry, found by crc32c(name) (258 DIR_INDEX ) the same entry again, in creation order (readdir) (259 EXTENT_DATA 0) file bytes 0.. : inline data, or "see disk extent X" (259 EXTENT_DATA 131072) file bytes 131072.. : the next extent ``` Leaves hold items; internal nodes hold `(key, child pointer)` pairs. Looking something up is "walk down from the root to the last key <= the one you want" (`Volume::search_le`); listing a directory or a file's extents is a range walk (`Volume::range`). `lookup` uses DIR_ITEM (one search by name hash), and `ls` uses DIR_INDEX (a range, in creation order). **Copy-on-write B-trees.** btrfs never overwrites a live block. To change a leaf it writes a *new* copy elsewhere, then a new copy of the parent pointing to it, and so on up to the root; the superblock is written last and switches everything over at once. Until then the old tree is still complete, so a crash just leaves the old version. The same trick makes snapshots nearly free: a snapshot is a second root pointing at the same nodes. For a *reader* it means every node is immutable once written, and each one records the transaction (generation) that wrote it. **Chunks: logical addresses.** Tree pointers and extent addresses are *logical* byte numbers, not disk offsets. The **chunk tree** maps logical ranges to (device, physical offset) "stripes". With **single** there is one stripe; with **DUP** (the default for metadata on one disk) there are two copies on the same disk, and the driver reads the second when the first fails its checksum. RAID profiles would spread stripes over several devices; the driver refuses those with a message. But the chunk tree is itself stored at logical addresses, so how do you read it? The superblock carries a small bootstrap copy of the SYSTEM chunks (`sys_chunk_array`), just enough to find the chunk tree. Open order: superblock, then sys_chunk_array, then chunk tree, then root tree, then FS tree. **Trees of trees.** The **root tree** lists every other tree with a ROOT_ITEM (where its root node is, its level). Tree 5 is the top-level **FS tree** (files and directories); tree 7 is the **csum tree** (checksums of data blocks); others (extent tree, device tree...) only matter for writing. The "default" entry in the root tree's directory 6 says which subvolume to show; the driver honours it. **Subvolumes** are separate FS trees (ids 256 and up) with their own inode numbers, so `(subvolume, inode)` names a file (`Loc` in the driver). A directory entry that points at a subvolume has a ROOT_ITEM key instead of an INODE_ITEM key; following it moves to that tree's top directory. The driver lists them from the root tree's ROOT_REF items with their read-only flag (`BtrfsFs::subvolumes`; on E: `archive` is 256, read-only, and `projects` is 257), and `cd E:/projects` just walks into one. Snapshots are subvolumes too, so they would work the same way. **Checksums everywhere.** Every node starts with a checksum of the rest of the node, plus its own logical address and the file system's id, so a misplaced or stale block is caught as well as a corrupt one. Data blocks are checksummed per sector in the csum tree, keyed by logical address. The driver checks both (crc32c: the Castagnoli CRC, a few lines in software); a bad block is `EIO` with a log line, and with DUP it tries the other copy first. Newer btrfs can use xxhash64, sha256 or blake2b instead; those are refused at mount time: `btrfs, but can't mount it: checksum type 2 (sha256) is not supported, only crc32c`. Directory names are found by `crc32c(name)` too, with seed `~1` and no final inversion (the "default" hash 2378154706 is a unit test). **Extents.** A file's data comes as EXTENT_DATA items, each covering a range of the file: | Kind | What it means | Driver | |---|---|---| | inline | the bytes are inside the item, in the leaf (small files, symlink targets) | copied, or decompressed | | regular | `disk_bytenr` + `offset` + `num_bytes`: a slice of an on-disk extent, which may be shared with other files or snapshots | read + checksummed | | compressed | a regular or inline extent with zlib, lzo or zstd: the whole extent is decompressed, then sliced | `miniz_oxide`, `src/fs/lzo.rs`, `ruzstd` | | prealloc | reserved by `fallocate`, never written | zeros | | hole | `disk_bytenr` 0, or simply no item (the `no_holes` feature) | zeros | btrfs's lzo is not plain LZO: it's a 4-byte total length, then per segment a 4-byte length plus an LZO1X block, and a segment header never crosses a 4 KiB sector boundary (padding instead). zstd extents can be followed by zero padding, so the driver decodes frame by frame instead of "all input". ### Why read-only first, and what writing would take Reading btrfs only needs the trees as they are on disk. Writing needs everything copy-on-write implies: - **Allocation**: the extent tree and free-space tree say which logical ranges are used, with *reference counts* (an extent can be shared by files, snapshots and reflinks). Every new node or data block needs a free range, and every freed one a decrement. - **COW updates**: changing one item means new copies of the leaf and of every node up to the root, in each affected tree (FS tree, extent tree, csum tree, root tree...), plus splitting and merging nodes to stay balanced. - **Transactions**: batch all of that, write it, flush, then write the superblock (all copies) with the new generation, in the right order for crash safety. Also the log tree for `fsync`. - **Checksums and compression on the way out**, inode/dir item updates, back-references (INODE_REF, ROOT_BACKREF...). That's a few thousand careful lines where a bug corrupts the volume, versus about a thousand for reading. Read-only first gets Tyler's disks readable, and it builds the parts writing needs anyway (key order, tree walking, chunk mapping, checksums). A first write step could be overwriting existing data blocks in place on a NODATACOW file. ### What the driver does and doesn't do - Does: superblock (crc32c, feature flags, single device), sys_chunk_array + chunk tree with single and DUP, root tree, the default subvolume, FS and csum trees; node checks (checksum, bytenr, fsid, level); inode items, INODE_REF, DIR_ITEM (hash lookup), DIR_INDEX (readdir); inline, regular, prealloc and hole extents, zlib/lzo/zstd, shared extents (slices into a bigger extent, as btrfs-convert makes them); data checksums; symlinks; subvolumes (listed, entered). A 256 KiB node cache and a one-extent decompression cache keep `cat big.txt` from re-reading and re-inflating. - Doesn't: write anything (`EROFS` for write, truncate, create, mkdir, rm); RAID profiles or multiple devices; checksum types other than crc32c; encryption; extended attributes, hard-link counts and timestamps aren't shown; the extent tree isn't read (not needed to read files). ### The test images (`build.rs`) `build.rs` copies `disk4/` to a staging tree, adds `big.txt` (120 000 lines, 2.16 MB), `random.bin` (1.5 MB of noise), `zeros.bin`, `empty.txt` and three symlinks, and makes 128 MiB images with `mkfs.btrfs --rootdir`: | Image | Options | Why | |---|---|---| | `btrfs.img` (E:) | `--compress zlib -u projects -u ro:archive`, defaults: 16 KiB nodes, metadata DUP | the main one | | `btrfs-lzo.img` | `--compress lzo -n 4096 -m single` | lzo; small nodes make a 2-level FS tree | | `btrfs-zstd.img` | `--compress zstd -n 65536` | zstd; 64 KiB nodes | | `btrfs-convert.img` | `mke2fs -d` (4 KiB blocks), then `btrfs-convert` | real holes and shared extents, an `ext2_saved` subvolume | | `btrfs-sha256.img` | `--csum sha256` | must be refused | `mkfs.btrfs --rootdir` never makes holes or prealloc extents (sparse files are stored as data), so holes come from the converted image and prealloc from a hand-made item in a unit test. Each tree gets a manifest (`target/btrfs-tree.manifest`, `btrfs-convert.manifest`): one line per directory, file (size + CRC-32) and symlink (target). ### Checking against the host `./scripts/btrfs-check.sh` (after `cargo test --test btrfs`): ``` == btrfs check --readonly target/btrfs.img found 2056192 bytes used, no error found ... (all five images) == run the tests/btrfs.rs kernel 11 btrfs tests ok == compare what Rustbelt read with the host's files (zlib.crc32) /mnt/disk3: 22 entries (12 files, 4710275 bytes) match /mnt/disk7: 22 entries (12 files, 4710275 bytes) match /mnt/disk8: 22 entries (12 files, 4710275 bytes) match /mnt/disk9: 7 entries (4 files, 5442941 bytes) match == all good ``` The test kernel itself already compares every file with the manifests; the script checks again independently, from the host's files with Python's `zlib.crc32`, using the `[btrfs-file]` lines the kernel prints. ## Networking (stage 12) Rustbelt can now talk to the world: get an address by DHCP, ping, look up names, download web pages and serve them. The card is QEMU's **virtio-net**, the protocols come from the **smoltcp** crate, and user programs see **BSD sockets** as file descriptors. ### Packets and layers, for a learner A network moves **packets**: chunks of at most about 1500 bytes. Every bit of data, from a ping to a 100 MB download, gets cut into packets, and each one travels on its own. A packet is built like a set of envelopes inside each other. Each **layer** wraps the one inside it with its own header, and each header answers one question: ``` +-------------------+-------------+--------------+---------------------+ | Ethernet header | IPv4 header | TCP header | data: "GET / HTTP.."| | to which card? | to which | to which | | | (MAC 52:54:00:..) | computer? | program? | | | | (10.0.2.2) | (port 80) | | +-------------------+-------------+--------------+---------------------+ 14 bytes 20 bytes 20 bytes ``` - **Ethernet** (link layer) delivers a *frame* to a network card on the same wire, by its 6-byte **MAC address**. That's all the card itself understands. Our virtio-net driver just moves whole frames in and out. - **ARP** answers "which MAC has IP 10.0.2.2?" To send to an IP address on the local network, the kernel broadcasts an ARP question to every card and caches the answer. - **IPv4** (network layer) delivers a packet to a *computer* by its 4-byte **IP address**, across many networks. Each hop is a router. If the address isn't on our own network (`10.0.2.0/24` = the first 24 bits match), the packet goes to the **gateway** (10.0.2.2), which passes it on. - **ICMP** is IP's own control messages. **ping** sends an *echo request* and times the *echo reply*. - **UDP** delivers a *datagram* to a **port**, i.e. a program, on that computer. It's one packet, which may get lost, with no connection. **DNS** (names to addresses), **DHCP** (getting an address at all) and **TFTP** run over UDP. - **TCP** turns lossy packets into a reliable **byte stream** between two programs. It numbers every byte, resends what isn't acknowledged, keeps things in order, and slows down when the network is busy. A connection starts with a three-way handshake (SYN, SYN-ACK, ACK) and ends with FINs. **HTTP** runs over TCP. A web request is literally the text `GET /path HTTP/1.0` + headers + an empty line, and the answer is a status line, headers, an empty line, then the file. How we get an address: at boot nobody knows us, so the kernel broadcasts a **DHCP** "discover". QEMU's built-in DHCP server offers `10.0.2.15/24` with gateway `10.0.2.2` and DNS server `10.0.2.3`. The kernel requests it, the server acknowledges, and the lease is ours (`[net] DHCP lease: ...`). ### Why smoltcp Writing ARP, IP and ICMP by hand is a fun weekend. **TCP** is not: getting retransmission timers, windows, the state machine (11 states), out-of-order segments and congestion control right takes months, and a subtly wrong TCP mostly *seems* to work. [smoltcp](https://github.com/smoltcp-rs/smoltcp) is a TCP/IP stack written for exactly this situation: - `no_std`: it needs only `alloc`, no OS. It never allocates on its own and never spawns threads; it doesn't even read a clock. We hand it `Instant`s. - It's **driven by us**. We call `iface.poll(now, device, sockets)`. It reads the frames that arrived, updates every socket, and sends whatever is due (data, ACKs, retransmissions, ARP, DHCP). `poll_delay` says when it next needs a poll (e.g. a retransmit timer). - It talks to the card through a two-method `Device` trait (`receive`, `transmit`), which `src/net/virtio_net.rs` implements in about 40 lines. - It's small and readable, and used in real embedded and OS projects (Redox, Hermit, ESP32 firmware). The same thinking as using `fatfs` for FAT32: write by hand what teaches something (the virtio driver, the socket layer, ext2/exFAT/btrfs), and use a good crate where the hard part is decades of edge cases. ### The virtio-net driver (`src/net/virtio_net.rs`) It uses the same legacy virtio interface as virtio-blk. PCI finds vendor `1af4` device `1000`; we reset it, accept only the MAC feature, and set up two **virtqueues**: - **RX (queue 0)**: we give the card 32 empty 2 KiB buffers up front. When a frame arrives the card fills one, puts it on the *used* ring, and raises an interrupt. We copy the frame out and hand the buffer straight back. - **TX (queue 1)**: to send, we copy the frame into a free buffer behind a 10-byte virtio-net header (all zeros: no checksum offload) and notify the card. Finished buffers are collected the next time we send; this queue has interrupts turned off because nobody needs to wait for them. **Interrupts.** The card's PCI interrupt line (config byte 0x3C) is 11 on QEMU's PC. It arrives through the old 8259 PIC, which still feeds CPU 0's local APIC (LINT0, "ExtINT"). The handler reads the card's ISR register (which also acknowledges it) and wakes the **`netd`** kernel thread. `netd` calls smoltcp's `poll`, then sleeps on a wait queue until the next interrupt or until `poll_delay` says a timer is due. No busy polling: an idle network costs no CPU. ### Sockets (`src/net/socket.rs`, syscalls 41-54) User programs use the **BSD socket API** that every OS copied from 4.2BSD in 1983: ```rust // user/fetch.rs, roughly (ulib returns a negative errno on failure) let fd = socket(SOCK_STREAM, 0); // a TCP socket, as an fd (AF_INET is implied) connect(fd, ip, 80); // handshake (blocks) write(fd, b"GET / HTTP/1.0\r\n\r\n"); // send bytes let n = read(fd, &mut buf); // receive (blocks); 0 = the server closed close(fd); // FIN; the kernel finishes the goodbye // a server: socket, bind(port), listen(backlog), then accept() per client ``` Each socket is a `File` in the fd table, so `read`/`write`/`close` and the shell's redirections work on it like on a pipe. Inside, it holds a smoltcp socket handle. Everything shared (interface, card, all sockets) sits in one `NET` lock, taken with interrupts off. **Blocking** reuses stage 11's wait queues. A `recv` with nothing to read sleeps on the `CHANGED` queue, which `poll` wakes whenever packets have been processed. Then it looks again. `SO_RCVTIMEO` turns into a deadline on the same wait (`wait_until_deadline`), and `EAGAIN` when it passes, so `ping` can give up on a lost reply. A TCP `connect` sleeps until the handshake is done (or `ECONNREFUSED` / `ETIMEDOUT` after 10 s). **Closing TCP** is a conversation, not an instant (FIN, ACK, maybe a last bit of data). `close` starts it, and the socket stays in a "closing" list that `poll` reaps once smoltcp says it's done. Two calls are Rustbelt's own: **`resolve`** (Linux does DNS inside libc, by sending UDP packets to the server in `/etc/resolv.conf`; we have smoltcp's DNS socket, so the kernel does it) and **`netinfo`** (instead of Linux's `ioctl`s and netlink). **`clock_gettime`** came along for `ping`'s millisecond times: it reads the TSC, calibrated against the PIT at boot (`src/time.rs`). ### QEMU's user-mode network, and the test servers `-netdev user` (slirp) is a whole tiny network inside QEMU: DHCP at 10.0.2.2, DNS at 10.0.2.3, and NAT through the host's ordinary sockets, so nothing on the host needs root, a bridge or a TAP device. That makes it perfect for a headless box and for tests. The test setup uses three of its extras: | What | Where | Used by | |---|---|---| | `guestfwd=tcp:10.0.2.100:80-cmd:sh scripts/http-reply.sh` | a fake web server: each connection runs the script, with the socket as its stdin/stdout. `/` = a greeting, `/big` = `seq 1 20000` (108,894 bytes) | `fetch`, the self-test, `tests/net.rs` | | `guestfwd=tcp:10.0.2.100:7-cmd:cat` | an echo server (`cat` copies its stdin to its stdout) | `tests/net.rs` | | `tftp=disk` (tests only) | slirp's TFTP server at 10.0.2.2:69, serving `disk/` | the UDP test | | `hostfwd=tcp:127.0.0.1:5580-:80` (`cargo run`; run-headless picks a free port) | host port -> guest port 80 | `httpd`, reached with `curl` | Because those servers live inside QEMU and on the box itself, the network tests pass with no internet at all. Only `host example.com` needs the host's DNS, and the tests skip it when that's missing. ### What it does and doesn't do Does: DHCP, ARP, IPv4, ICMP echo, UDP, TCP client and server (up to 4 waiting connections per listener), DNS A records, blocking with timeouts, and an interrupt-driven RX path. Doesn't (yet): IPv6, non-blocking sockets / `poll`/`select`, `shutdown`, `getsockname`, checksum offload or multi-queue, more than one card, a loopback interface (127.0.0.1), or TLS (so `fetch` is plain `http://` only). ## Virtual drivers (stage 13) Stage 13 adds the last three virtual devices Rustbelt wants from QEMU, with no physical hardware involved: - **virtio-rng** (`src/virtio/rng.rs`): random bytes from the host. They feed the `getrandom` syscall (Linux number 318), `/dev/random` and `/dev/urandom`, the `random` program, smoltcp's random seed (which it uses for things like TCP initial sequence numbers) and the first local TCP/UDP port, which used to be a constant 49152. - **virtio-console** (`src/virtio/console.rs`): a second terminal, `hvc0`, like Linux's `/dev/hvc0` in a VM. The kernel log is mirrored to it, and `getty /dev/hvc0` runs a shell on it. On the host it's a QEMU *chardev*: a UNIX socket for `cargo run`, files for the tests. - **virtio-gpu, 2D** (`src/virtio/gpu.rs`): a 1024x768 framebuffer. The text console moves onto it (`src/fbcon.rs`, with a built-in bitmap font, `src/font.rs`), and user programs get `/dev/fb0` (`src/dev.rs`). Without a virtio-gpu, the VGA text console carries on as before. ``` A:/ $ ls -l /dev crw-rw-rw- fb0 crw-rw-rw- hvc0 crw-rw-rw- null crw-rw-rw- random crw-rw-rw- urandom A:/ $ random 82461427e1b4012e64cd289b0db180fe A:/ $ random -d 6 5 6 1 2 6 1 A:/ $ tile tile: 4 windows on 1024x768, console 61x46 ``` ### What virtio is, for a learner Real hardware is complicated because it's real: a network chip has registers, timing rules and errata. An emulator like QEMU *can* copy such a chip register by register (that's how it emulates the e1000 network card or the old VGA card), but it's slow and both sides get complicated. **Virtio** is the alternative: device types designed for virtual machines, where guest and host just agree on a simple protocol in shared memory. The same drivers work in QEMU, KVM, cloud VMs and Firecracker. Every virtio device has the same three parts: 1. **Feature bits.** The device offers a set of features and the driver accepts the ones it understands. Rustbelt only ever accepts `VERSION_1` (bit 32: "I speak virtio 1.0"), which the modern interface requires. 2. **Virtqueues.** Ring buffers in *guest* memory (`src/virtio/queue.rs`). A *descriptor table* lists buffers (physical address, length, and whether the device writes it). The driver puts the index of a request on the *available ring* and writes the queue number to a *notify* register (the "doorbell"). The device does the work, then puts the same index on the *used ring* together with how many bytes it wrote, and maybe raises an interrupt. The driver either sleeps until that interrupt or polls the used ring. That's the whole protocol: what a request *means* is up to the device type. 3. **Config space.** A few device-specific registers: the GPU's number of displays, the console's size, a network card's MAC address. The three drivers show three ways of using queues: | Device | Queues | A request is | Waiting | |---|---|---|---| | rng | 1 | one empty buffer; the device fills it with random bytes | polled (it answers at once) | | console | 2 (receive, transmit) | receive: 8 empty 512-byte buffers always posted; transmit: bytes to print | receive: **interrupt** (input can come at any time); transmit: polled | | gpu | 2 (control, cursor; only control used) | a command (24-byte header + arguments) plus a buffer for the response | polled | **Legacy and modern.** The virtio-blk and virtio-net drivers from stages 6 and 12 speak the *legacy* (pre-1.0) interface: every register at a fixed offset in one I/O-port BAR. That's simpler, but virtio-gpu doesn't support it. The modern interface (`src/virtio/mod.rs`) puts register blocks (common config, notify, interrupt status, device config) in memory BARs. **PCI capabilities** (a linked list in PCI config space, vendor id 0x09) say which BAR and offset each block is at. `Transport::new` walks that list, maps the BARs uncached (`memory::map_mmio`) and does the start-up dance: reset, ACKNOWLEDGE, DRIVER, features, FEATURES_OK, set up the queues, DRIVER_OK. All three new drivers share it. The rng and console cards are "transitional" (they'd speak both), so the modern code could replace the legacy one in virtio-blk and virtio-net later. **Interrupts.** The console's receive queue uses the card's PCI interrupt line, which it shares with the network card (IRQ 10 here). PCI interrupts are level-triggered and can be shared, so `interrupts::pci_irq` asks every driver "was it you?" (`net::handle_irq`, then `virtio::handle_irq`). Reading a device's interrupt-status register says yes or no *and* lowers its line. A driver that forgot that read would leave the line high, and the CPU would take the same interrupt forever. ### Framebuffers and scanout A **framebuffer** is memory with one number per pixel: here a 32-bit `0x00RRGGBB` for each of 1024 x 768 pixels, row after row (3 MiB). Drawing is just writing to memory. **Scanout** is the display hardware reading that memory 60 times a second and sending it to the monitor. A real GPU scans out straight from video memory. virtio-gpu adds a step, because the image the host shows lives on the host: ``` guest RAM (our framebuffer) host (QEMU) window +-------------------------+ TRANSFER_TO_HOST_2D +-----------+ RESOURCE_FLUSH | 0x00RRGGBB pixels | -------------------> | resource 1 | --------------> screen +-------------------------+ (copy a rectangle) +-----------+ (show it) ^ ATTACH_BACKING ties these pages to resource 1; SET_SCANOUT shows resource 1 ``` So setting up (`gpu::init`) is: `GET_DISPLAY_INFO` (QEMU prefers 1280x800; we ask for 1024x768 and QEMU scales the window), `RESOURCE_CREATE_2D` (an image with id 1, format B8G8R8X8, which makes a `u32` read as `0x00RRGGBB`), `RESOURCE_ATTACH_BACKING` (its pixels are these 768 contiguous guest pages) and `SET_SCANOUT` (show resource 1 on display 0). After drawing, `gpu::flush(x, y, w, h)` sends `TRANSFER_TO_HOST_2D` and `RESOURCE_FLUSH` together. Nothing reaches the screen without a flush, so the console keeps a **dirty range**: the pixel rows it changed since the last flush. It flushes once per `print!`, not per character. **The framebuffer console** (`src/fbcon.rs`) keeps the old console's model: a grid of cells, each a character and a VGA colour attribute, and the same 16-colour palette. Drawing a cell copies a glyph from `src/font.rs` (8x16 pixels, one byte per row, one bit per pixel) into the framebuffer. 1024x768 gives 128x48 characters instead of VGA's 80x25. Scrolling moves the pixel rows up with one `memmove` (row by row when the console is in a viewport narrower than the screen). The font is Spleen (BSD-2): ASCII and Latin-1, so `é` now shows up instead of a `■`, plus box-drawing characters. The console can be confined to a **viewport** (`FB_SET_CONSOLE`): that's how `tile` puts it in a window. **/dev/fb0 for programs.** Linux programs `mmap` `/dev/fb0` and draw into it directly. Rustbelt has no `mmap` yet (the next stage), so `/dev/fb0` is a file you `lseek` + `write` pixels into, plus ioctls for what doesn't fit in a write: | ioctl | arg | does | |---|---|---| | `FB_GET_INFO` (0x4600) | `*[u32; 4]` | width, height, bytes per row, bits per pixel | | `FB_FLUSH` (0x46f0) | `*[x, y, w, h]`, or 0 | show a rectangle (0: the rows this fd wrote since the last flush) | | `FB_FILL` (0x46f1) | `*[x, y, w, h, color]` | fill a rectangle in the kernel (much faster than writing it) | | `FB_SET_CONSOLE` (0x46f2) | `*[x, y, w, h]`, or 0 | move the text console into the rectangle (0: the whole screen); returns `cols << 16 \| rows` | `user/tile.rs` uses all of them. It writes a gradient wallpaper 8 rows at a time, splits the screen "dwindle"-style (each window takes half of what's left, alternating left/right and top/bottom), fills borders and windows with `FB_FILL`, flushes, then moves the console into the first window. That's a first sketch of the tiling compositor on the roadmap; the compositor itself will need `mmap` (to draw without a syscall per rectangle), input events and windows owned by other processes. ### Why 2D first, and what 3D would take 2D is all a console, a tiling compositor and a software renderer need: the CPU draws the pixels, the GPU just shows them. It's six commands, and every later step builds on it, because even 3D output ends with `SET_SCANOUT` and `RESOURCE_FLUSH`. 3D through **virgl** is a different scale of project. The guest doesn't send pixels but *rendering commands*. virtio-gpu adds 3D contexts, 3D resources (textures, buffers) and `SUBMIT_3D`, which carries a command stream in Gallium's format. On the host, **virglrenderer** turns that stream into OpenGL calls on the real GPU. In the guest, the stream comes from **Mesa**, the open-source OpenGL/Vulkan implementation, through its virgl driver. So Rustbelt would need to run Mesa: that means a C library (the musl port), `mmap`, threads in user space, a way to load shared libraries or a big static build, and kernel support for 3D resources and fences. Newer QEMU versions also offer **Venus** (Vulkan over virtio-gpu) and **virtio-gpu rutabaga/gfxstream**; they need the same user-space groundwork. For now, 2D and software rendering are plenty for a tiling compositor. ### The second console, and testing it headless `virtconsole` is a port on a `virtio-serial-pci` card. Rustbelt doesn't negotiate the multi-port feature, so there's exactly one port, port 0, which needs no control messages. Bytes the host sends land in the receive buffers; the interrupt handler copies them into a 4 KiB queue and wakes any reader. `read` on `/dev/hvc0` blocks until there's input. Writes go straight to the transmit queue. While a program has `/dev/hvc0` open, the kernel stops mirroring its log there, so a shell on hvc0 isn't interrupted by `[process]` lines. QEMU decides where the other end goes: `-chardev socket,...,server=on` for `cargo run` (connect with `socat`), and `-chardev file,path=OUT,input-path=IN` for tests. The file chardev sends all of `IN` once at start-up and writes everything the guest sends into `OUT`, so a headless run can feed hvc0 a script and check the reply afterwards. `run-headless.sh` feeds `scripts/hvc0-session.txt` to `getty -1 /dev/hvc0` and checks `target/headless-hvc0.txt`; `tests/virtio.rs` checks the exact bytes of `tests/hvc0-input.txt` arrive. ### Testing the GPU Inside the guest, `tests/virtio.rs` checks the protocol (an error comes back for a bad resource id) and reads pixels back from the framebuffer. That only proves the guest's memory is right, though. To check what QEMU actually *displays*, `scripts/gpu-check.sh` boots the kernel, types `tile` through the QEMU monitor, saves the screen with the monitor's `screendump` command as a PPM file (a tiny header, then raw RGB bytes, so a few lines of Python can read it) and compares 12 pixels with the colours `tile.rs` draws: the wallpaper gradient at three heights, the gap between windows, the cyan and gray borders, three window fills, a "text" bar, and the console window's black background with text on it. ### What it does and doesn't do - virtio-rng: 4 KiB per request, a 1 s timeout; if the device is missing or silent, a TSC-seeded xorshift64* takes over (fine for port numbers, not for cryptography). - virtio-console: one port, no resize events, no multiport; TX waits at most 10 ms for a free buffer and counts what it drops. The shell on it isn't interactive (no line editing); the host terminal does the echo. - virtio-gpu: one display, one resource, a fixed 1024x768 (the host's preferred size is read and logged, not used), no hardware cursor, no resize, no 3D. Flushes are synchronous (polling for each answer). - `/dev/fb0`: `write`/`lseek`/ioctls; `mmap` comes with the next stage. - Every new user program's heap is still 64 KiB, so `tile` writes the wallpaper in 32 KiB pieces. ## Processes: fork, exec, memory maps and signals (stage 14) Until stage 13 the kernel had one way to start a program, `spawn`, and one way to stop it: exit, or be killed for a bad memory access. Stage 14 adds the classic Unix process model, with Linux's own syscall numbers and structs, so a C library such as [musl](https://musl.libc.org/) could later run on it unchanged: - **`fork`** (57), **`vfork`** (58) and fork-like **`clone`** (56): copy a running process, using copy-on-write. - **`execve`** (59): replace a process's program with another. It sets up `argv`, `envp` and the auxiliary vector exactly like Linux. - **`mmap`/`munmap`/`mprotect`** (9/11/10) and **`brk`** (12): memory regions created on request and filled lazily (demand paging). - **Signals**: `rt_sigaction`, `rt_sigprocmask`, `rt_sigreturn`, `kill`, `pause`, with real signal frames on the user stack and default actions. A bad memory access is now a SIGSEGV that a program can catch. Children that exit send SIGCHLD, a broken pipe sends SIGPIPE, and Ctrl+C sends SIGINT. - **`wait4`** (61) with Linux's status encoding, `getppid`, `exit_group`, `dup`/`dup2`/`dup3`, `pipe2`, `fcntl`, close-on-exec, and process groups plus a terminal "foreground group" for job control. The shell now starts every program with fork + exec. Ctrl+C interrupts the foreground program. `kill`, `fg` and `bg` are builtins, and Ctrl+Z stops a program. ``` A:/ $ spin 1000 > /tmp/hog.txt ^C [exit code 130] (interrupted: SIGINT) A:/ $ start spin 1000 > /tmp/bg.txt [pid 5] spin started in the background A:/ $ jobs [1] [pid 5] running start spin 1000 > /tmp/bg.txt A:/ $ kill %1 A:/ $ wait [done] pid 5 (start spin 1000 > /tmp/bg.txt), exit code 143 A:/ $ run overflow [overflow] pid 6: recursing until the stack runs out... [process] pid 6 (overflow) killed: stack overflow (SIGSEGV) [exit code 139] (killed by SIGSEGV) A:/ $ kill -l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP ... ``` (130 = 128 + 2, 143 = 128 + 15 (SIGTERM), 139 = 128 + 11 (SIGSEGV): a shell reports "killed by signal N" as exit code 128 + N.) ### fork and exec, and why Unix splits them A **process** is a running program: its memory (the *address space*), its registers, its open files, its current directory. Unix creates processes in two steps: 1. **`fork()`** makes an almost exact copy of the calling process. Both copies return from the same `fork()` call: the parent gets the child's pid, the child gets 0. That return value is the only difference. 2. **`execve(path, argv, envp)`** throws away the calling process's program and memory and loads a new program in their place. The pid, the open files, the current directory and the parent stay the same. A shell running `ls -l > out.txt` does: ``` pid = fork() if pid == 0: # the child (still running the shell's code) fd = open("out.txt", O_CREAT|O_TRUNC|O_WRONLY) dup2(fd, 1) # make stdout point at the file close(fd) execve("/bin/ls", ["ls", "-l"], environ) # becomes ls; never returns on success exit(127) # only reached if execve failed else: # the parent: the shell wait4(pid, &status, 0, 0) # sleep until ls is done ``` **Why two steps?** The gap between `fork` and `exec` is the point. The child is still running *the shell's code*, so it can set up the new program's world with the ordinary system calls: redirect fds with `dup2`, change directory, join a process group, reset signal handlers. `execve` then keeps all of that. Without the split, every one of those options would need a parameter of one big "create process" call. Windows' `CreateProcess` has ten parameters, and our old `spawn` had its `SpawnOptions` struct for exactly this reason. The cost is that `fork` seems to copy a whole address space only for `exec` to throw it away a moment later, which is why the next section matters. (`posix_spawn` and `vfork` exist for the same reason. Rustbelt keeps `spawn` (500) as a convenience. The kernel itself uses it to start the first shell, and `getty` uses it.) **What `execve` hands the new program.** Linux starts a program with everything on its stack, and Rustbelt now does exactly the same: ``` (higher addresses) "PATH=/bin:..." "ls" "-l" 16 random bytes ... the strings themselves AT_NULL 0 auxiliary vector: AT_RANDOM -> the 16 bytes (key, value) pairs that tell AT_ENTRY, AT_PHDR, AT_PHNUM, AT_PAGESZ, ... the C library about the program NULL envp[0] -> "PATH=/bin:..." environment pointers NULL argv[1] -> "-l" argv[0] -> "ls" rsp -> argc = 2 ``` The **auxiliary vector** is how the kernel tells the C library things only the kernel knows: where the program's headers are (`AT_PHDR`, so musl can find its thread-local storage template), the page size (`AT_PAGESZ`), the entry point (`AT_ENTRY`), 16 random bytes for stack canaries and hash seeds (`AT_RANDOM`, from virtio-rng), the path it was run as (`AT_EXECFN`), and the user and group ids. `user/ulib.rs`'s `_start` now reads this layout (`arg(i)`, `getenv`, `getauxval`), and `forktest args` checks every field. ### Copy-on-write Copying all of a process's memory on `fork` would be slow and usually wasted (an `exec` follows). Instead, `fork` copies only the **page tables**. The child's table points at the *same* physical frames as the parent's, and every writable private page is made **read-only in both**: ``` parent page table physical frames child page table 0x...1000 R -----------> [frame 42, refs 2] <----------- 0x...1000 R 0x...2000 R -----------> [frame 77, refs 2] <----------- 0x...2000 R ``` Reading works as before. The first **write** to such a page causes a page fault. The fault handler (`mm::Mm::handle_fault`) looks at the region the address belongs to: it is writable, so the fault is "copy-on-write", not a bug. If the frame's **reference count** is above 1, the handler copies the page into a fresh frame, maps the copy writable for the writer and drops one reference. If the count is 1, the other process has already made its own copy (or exited), so the writer gets write access back with no copy at all. Code pages are never written, so they stay shared forever. The reference counts live in `memory.rs` (a map from frame to count, kept only for frames that are shared). A frame goes back to the allocator when its last reference goes away. The leak tests fork, exec and exit hundreds of times and check that the number of frames in use and the number of shared frames return to where they started. One subtle consequence: the *kernel* also writes into user pages (a `read()` copying file data into a buffer, for example). If that buffer is a copy-on-write page, the kernel would write into the frame the parent still uses. Since stage 14 the CPU's `CR0.WP` bit is on, so even ring 0 faults on read-only pages, and every syscall first runs the buffer through the same fault logic (`mm::user_access_ok`). The pages are copied before the kernel touches them. ### Virtual memory maps: mmap, brk and demand paging Every process sees the same addresses (0x80_0000_0000 and up), but each has its own page tables, so the same address leads to different physical frames in different processes. That is **virtual memory**. Stage 14 describes a process's memory as a list of **regions** (Linux calls them VMAs) instead of a list of mapped pages: ``` 0x0000_0080_0040_0000 program: .text R-X, .rodata R--, .data/.bss RW- (loaded at exec) heap: from the end of the program up to brk (grows with brk) ... free ... mmap regions, handed out downwards from MMAP_TOP (mmap) 0x0000_00ff_ffee_f000 64 KiB guard zone (never mapped: overflow -> SIGSEGV) 0x0000_00ff_ffef_f000 user stack, 1 MiB, pages appear as it grows 0x0000_00ff_ffff_f000 one unmapped page, then USER_END ``` A region is a **promise**, not memory. `mmap(64 MiB)` just records the range. The first touch of a page in it causes a page fault, and the handler finds the region, allocates a zeroed frame (or reads the file's bytes into it), maps it, and lets the instruction run again. This is **demand paging**, and `mmaptest` checks it: a 64 MiB mapping costs no memory until pages are touched, then about one frame per touched page. Programs never notice any of this. What's supported: - `MAP_PRIVATE | MAP_ANONYMOUS`: plain memory (what `malloc` uses for big blocks). It is copy-on-write across fork. - `MAP_SHARED | MAP_ANONYMOUS`: memory that **stays shared** after fork. Parent and child see each other's writes (`cowtest` checks this). Its frames are allocated at mmap time. - `MAP_PRIVATE` of a file: the file's bytes are read in page by page on first touch, with zeros past the end of the file. Writes (after `mprotect`) go to a private copy and never reach the file. `MAP_SHARED` of a file returns `ENODEV`, an honest error rather than silently lost writes. - `MAP_FIXED`, `munmap` of any page range (it can split a region), `mprotect` to `PROT_NONE`/`READ`/`WRITE`/`EXEC`, and `brk` to grow and shrink the heap. `user/ulib.rs`'s allocator now gets its memory from `brk`, so programs aren't stuck with stage 13's fixed 64 KiB heap. An access outside every region, or one the region forbids (writing a read-only page, reading `PROT_NONE`), is a real error. The process gets **SIGSEGV**, with `si_addr` set to the faulting address and `si_code` `SEGV_MAPERR` (nothing mapped there) or `SEGV_ACCERR` (not allowed). ### Signals A **signal** is a small number delivered to a process to say "something happened": SIGINT (2) when you press Ctrl+C, SIGSEGV (11) for a bad memory access, SIGCHLD (17) when a child exits, SIGPIPE (13) when it writes to a pipe nobody reads, SIGTERM (15) for a polite "please exit", SIGKILL (9) for a final one. For each signal a process chooses (`rt_sigaction`) between: - the **default action**: terminate (most), terminate "with a core dump" (SIGQUIT, SIGSEGV... We write no core files, so `WCOREDUMP` stays 0, as on Linux with core dumps turned off), ignore (SIGCHLD, SIGURG, SIGWINCH), stop (SIGSTOP, SIGTSTP from Ctrl+Z) or continue (SIGCONT); - **ignore** it (`SIG_IGN`); - a **handler**: a function in the program that runs when the signal arrives, interrupting whatever the program was doing. A process can also **block** signals for a while (`rt_sigprocmask`). They then stay pending until it unblocks them. SIGKILL and SIGSTOP can't be caught, blocked or ignored, so there is always a way to stop any process. **How a handler runs.** Sending a signal only sets a bit in the target's `pending` set and wakes it if it is sleeping. The target acts on the signal on its way back to user mode: after a syscall, a page fault or a timer interrupt. To run a handler, the kernel pushes a **signal frame** onto the user stack: all the user registers at the moment of interruption, the old signal mask, and a `siginfo` (which signal, who sent it, which address faulted). Then it edits the saved registers so that the return to user mode lands in the handler, with a return address pointing at a tiny "restorer" function (`mov rax, 15; syscall`). When the handler returns, the restorer calls **`rt_sigreturn`**, which copies the saved registers back from the frame, and the program carries on exactly where it was. `sigtest` checks this by interrupting a calculation and comparing the result. The frame is Linux's `struct rt_sigframe`, so musl's handlers will work unchanged. This is also why syscalls now return to user mode with `iretq` through a common `TrapFrame` (`src/trap.rs`) rather than `sysretq`. Returning into a handler, or back from one, has to restore *every* register, including the `rcx` and `r11` that `sysretq` overwrites. **Interrupted syscalls.** A process sleeping in the kernel (reading the keyboard or a pipe, `sleep`, `wait4`, `pause`) is woken when a signal arrives. The call then fails with `EINTR`, or is restarted transparently if the handler was installed with `SA_RESTART`. `sigtest` checks both. **Process groups and Ctrl+C.** A pipeline like `cat x | grep y` is two processes but one **job**. The shell puts them in their own **process group** (`setpgid`) and tells the terminal that this group is in the **foreground** (`ioctl(TIOCSPGRP)`). When you press Ctrl+C, the console (`src/tty.rs`) sends SIGINT to the whole foreground group. The shell itself ignores SIGINT, so only the job dies. `kill -TERM %1` sends a signal to a background job's group, Ctrl+Z (SIGTSTP) stops the foreground job, and `fg` and `bg` continue it with SIGCONT. **wait4 and status codes.** A parent learns how a child ended from `wait4`, which returns a packed status word decoded with Linux's macros: `WIFEXITED`/`WEXITSTATUS` (exited with a code), `WIFSIGNALED`/`WTERMSIG`/ `WCOREDUMP` (killed by a signal), `WIFSTOPPED`/`WSTOPSIG` (with `WUNTRACED`) and `WIFCONTINUED` (with `WCONTINUED`). The parent also gets SIGCHLD each time a child exits, stops or continues. ### What it does and doesn't do - One thread per process: no `CLONE_VM`/`CLONE_THREAD`, no futexes yet (they are next on the roadmap, because libc threads need them). *Stage 15 added both; see the next chapter.* `clone` accepts only fork-style flags (`SIGCHLD`, optionally `CLONE_VFORK`). - `vfork` is fork + "the parent waits until the child execs or exits". It is correct for every legal use of vfork, just not faster than fork. - At most 64 processes (`MAX_PROCESSES`). A fork bomb gets `EAGAIN` at the limit, and the test kills it and checks that nothing leaked. - No `MAP_SHARED` file mappings (no page cache), no swap, no `sigaltstack`, no real-time signal queueing (a pending signal is one bit, as for Linux's standard signals), no users or permissions (uid 0). - Ctrl+C and the foreground group work on the main console. hvc0's `getty` shell runs programs without a foreground group, so there Ctrl+C is just a character. Proper ttys and ptys are on the roadmap. *Stage 15: hvc0 is a real tty now, and `getty` makes it the shell's controlling terminal, so Ctrl+C works there too.* ## Libc plumbing (stage 15) Stage 14 gave Rustbelt processes. A real C library (musl, glibc) expects quite a lot more from a kernel before ordinary programs work: a way to wait on many files at once, files that say "not now" instead of blocking, threads, a way for threads to sleep on a lock, per-thread variables, and terminals that behave like terminals. Stage 15 adds all of that, still with Rustbelt's own Rust programs as the users. This chapter explains each idea from scratch. ### Event loops: poll, select and epoll Imagine a chat server with three clients. If it calls `read` on client 1, it blocks until client 1 says something, while clients 2 and 3 wait. One fix is a thread per client (below). The other, used by nearly every server, GUI toolkit and terminal emulator, is an **event loop**: ```text loop { ready = wait_until_any_is_ready([listener, client1, client2, ...]) for fd in ready { accept / read / write -- none of these block now } } ``` The kernel offers three versions of `wait_until_any_is_ready`: - **`poll(fds, n, timeout_ms)`**: an array of `{fd, events, revents}`. You set `events` (`POLLIN` = "tell me when I can read", `POLLOUT` = "...write"), the kernel fills in `revents`, including `POLLHUP` ("the other end hung up") and `POLLERR`. `ppoll` is the same with a nanosecond timeout and a signal mask. - **`select(n, readfds, writefds, exceptfds, timeout)`**: the 1983 BSD original, with bitmaps instead of an array (`fd_set`: bit `fd` set means "watch this fd"). `pselect6` is its `ppoll`. - **`epoll`**: `epoll_create1` makes an epoll instance (itself an fd); `epoll_ctl(ADD/MOD/DEL, fd, events, data)` registers fds *once*; `epoll_wait` returns only the ready ones, each with your `data` (usually the fd or a pointer to your client struct). With 10 000 connections, poll copies and checks 10 000 entries per call; epoll doesn't. How Rustbelt does it (`src/poll.rs`): every open file can answer "what's ready right now?" (`File::poll`). Whatever might change that (bytes into a pipe, a key, a packet, a pty write, a close) calls `poll::notify()`, which bumps one global counter and wakes every sleeping poller. A poller checks its fds; if none is ready it sleeps until the counter moves or the timeout ends. Simple and it can't miss an event, at the cost of waking pollers that didn't need waking. epoll here is *level-triggered*: a fd is reported as long as it stays ready (Linux's default; `EPOLLET`, "only report changes", is accepted but behaves the same). Try `echoserver` (it runs the same echo test with an epoll server and a poll server) or `echoserver 7070` to keep one running. ### Nonblocking I/O and EAGAIN An event loop must never block, so its fds are put in **nonblocking** mode: `fcntl(fd, F_SETFL, O_NONBLOCK)`, or `ioctl(fd, FIONBIO, &1)`, or `O_NONBLOCK` at `open`/`pipe2`/`socket` time. A read that would have to wait now returns at once with **`EAGAIN`** ("try again later"; also called `EWOULDBLOCK`), and so does a write to a full pipe (Rustbelt's pipes hold 64 KiB, like Linux's). A nonblocking `connect` returns `EINPROGRESS`; poll for `POLLOUT` to learn when it's connected. `nbtest` checks all of these. ### Loopback `127.0.0.1` is "this machine". Programs use it to talk TCP/UDP to each other (a database and its client, a compositor and its apps, X11) with no network card involved. Rustbelt now has an `lo` interface: frames sent to 127.x (or to our own eth0 address) are handed straight back to smoltcp's receive side instead of the virtio-net card (`src/net/device.rs`). It is there even with no network card at all, so `echoserver` works offline. ### Threads vs processes A **process** is an address space plus open files, a cwd, signal handlers and so on. A **thread** is "a place where the CPU is executing": registers, a stack, a signal mask. Stage 14 had exactly one thread per process. Linux has one call for both, `clone(flags, ...)`, and the flags say what to *share* with the parent: | Flag | Shares | |---|---| | `CLONE_VM` | the address space (same memory) | | `CLONE_FILES`, `CLONE_FS` | the fd table; the cwd | | `CLONE_SIGHAND` | signal handlers | | `CLONE_THREAD` | the process itself: same pid, the new thread gets a new *tid* | | `CLONE_SETTLS` | start the new thread with this TLS pointer (below) | | `CLONE_PARENT_SETTID` / `CLONE_CHILD_SETTID` | write the new tid here | | `CLONE_CHILD_CLEARTID` | when the thread exits, write 0 here and futex-wake it (that's `pthread_join`) | `fork` is just `clone` with none of them. `pthread_create` is all of them. `getpid` gives the process id, `gettid` the thread id (the first thread's tid equals the pid). `exit` (60) ends *one thread*; `exit_group` (231, what libc's `exit()` calls) ends them all. `tgkill` sends a signal to one thread. Threads of one process run on different CPUs at the same time: `threads` sums 1..=40,000,000 on 4 threads and prints how many CPUs took part. Sharing memory means two threads can update the same variable at once. `counter += 1` is a load, an add and a store; two threads interleaving them lose updates. `threads` shows exactly that, then fixes it with a lock. ### Futexes: how locks sleep A lock is just an integer. Taking a free lock is one atomic compare-and-swap in user space, no kernel at all. The kernel is only needed when a thread must *wait*, and that's `futex` ("fast userspace mutex"): - `futex(&word, FUTEX_WAIT, expected)`: "sleep, but only if `word` still equals `expected`" - `futex(&word, FUTEX_WAKE, n)`: "wake up to n threads sleeping on `word`" The "only if still equal" check, done under the kernel's lock, closes the race where the holder unlocks between our check and our sleep: then WAIT sees the new value and returns `EAGAIN` immediately. The mutex in `user/threads.rs` is the classic one from Ulrich Drepper's paper *Futexes Are Tricky*: the word is 0 (free), 1 (locked) or 2 (locked and someone may be waiting). Lock: CAS 0 -> 1; if that fails, swap in 2 and `FUTEX_WAIT` while it isn't 0. Unlock: swap in 0; only if the old value was 2 call `FUTEX_WAKE`. With no contention, neither side ever enters the kernel. The kernel keeps no per-lock state, just a list of sleepers, each tagged with the word it waits on: (address space, virtual address), or the physical address for `MAP_SHARED` memory (`src/futex.rs`). Timeouts, `FUTEX_WAIT_BITSET`/`WAKE_BITSET`, `REQUEUE` and the private flag are there too, because libc uses them. ### Thread-local storage `errno` is a global variable... but each thread needs its own, or one thread's failing `read` would overwrite another's. Such per-thread variables are **thread-local storage** (TLS). On x86_64 the trick is a segment register: every thread has its own **FS base** address, and code reads its TLS at `fs:offset`. libc allocates a TLS block per thread and sets FS with `arch_prctl(ARCH_SET_FS, addr)` (or `CLONE_SETTLS`). The kernel's whole job is to save and restore that address on each thread switch (it lives in the `IA32_FS_BASE` MSR, `src/thread.rs`). `set_tid_address` registers the `CLONE_CHILD_CLEARTID` word for the first thread. ### ttys and ptys, and why terminals are weird Terminals are strange because they're a 1970s design that never went away. A terminal was a keyboard and screen at the end of a serial line; the kernel sat between the line and the programs, and that middle layer, the **line discipline**, is still in every Unix: - **Canonical mode** (the default): what you type collects in a line buffer you can edit (Backspace, ^U erases the line) and `read` gets nothing until Enter. ^D at the start of a line is end-of-file. Programs that want every key (editors, Rustbelt's shell line editor) switch to **raw** (non-canonical) mode. - **Echo**: the *kernel* prints what you type, not the program. That's why `cat` shows your typing before it prints the line back, and why a password prompt turns `ECHO` off. - **Signals**: ^C sends `SIGINT`, ^\ `SIGQUIT`, ^Z `SIGTSTP`, not to "the program" but to the terminal's **foreground process group**, the job the shell put in front (`TIOCSPGRP`). That's why ^C kills `cat` and not the shell waiting for it. - **Output processing**: `ONLCR` turns `\n` into `\r\n`, because a teletype needed a carriage return to move back to column 0. All of it is configured by one struct, **termios**, read and written with the `TCGETS`/`TCSETS` ioctls (libc: `tcgetattr`/`tcsetattr`). **Sessions.** `setsid()` starts a session (roughly, a login). The first terminal it opens becomes its **controlling terminal**; `/dev/tty` always means "my controlling terminal", whatever its real name. When the session leader exits, the foreground job gets `SIGHUP` ("hang up", from the modem days). `getty` on hvc0 is now a classic one: it forks, the child calls `setsid`, opens `/dev/hvc0` (which becomes its controlling terminal) and `exec`s the shell, so ^C on hvc0 interrupts programs there just like on the console. **Pseudo-terminals.** A terminal window has no serial line; a program plays the terminal instead. Opening **`/dev/ptmx`** gives that program the **master** end and creates **`/dev/pts/N`**, the **slave** end, which the shell gets as its stdin/out/err. Bytes written to the master are "typed" into the slave, through the line discipline (so echo and ^C work); what the shell writes to the slave comes out of the master to be drawn. xterm, ssh, tmux and the future compositor's terminal all work this way. In Rustbelt the console, hvc0 and every pty share one `Tty` (`src/tty.rs`). `ptytest` drives one from both ends. ### What stage 15 does and doesn't do Does: everything above, at `-smp 4`, checked by `tests/libc.rs`, which runs each test program twice and requires the thread, process, futex, pty, socket and memory-frame counts to come back exactly where they were (no leaks). Also `nanosleep` (now exact to the microsecond on top of the 55 ms PIT tick), `clock_gettime(CLOCK_REALTIME)` (the CMOS clock), `uname`, `getcwd`, `readlink /proc/self/exe`, `readv`/`writev`, `pread`/`pwrite`. Doesn't (yet): - No `SIGTTIN`/`SIGTTOU`: a background job may read and write the terminal. - epoll is level-triggered only; no `EPOLLONESHOT`, no `signalfd`, `eventfd` or `timerfd`. - One global "something changed" counter wakes every poller. - Timeouts are rounded up to 55 ms PIT ticks (never early). - No FPU/SSE state saving (Rustbelt's programs are soft-float). musl needs it, so it comes first in the next stage. - Paths are still passed as pointer + length, not C strings. ## Roadmap Done: all 12 chapters of the 2nd edition, plus preemptive kernel threads, user mode + syscalls, ELF loading, a VFS with a ramfs, PCI, virtio-blk, MBR/GPT partitions, FAT32, blocking threads, pipes, spawn/wait, process memory cleanup, the shell, hand-written ext2, exFAT and (read-only) btrfs drivers, multi-core support with a sleep/wake scheduler and guard pages, networking (virtio-net + smoltcp + sockets), and the virtual drivers (virtio-rng, virtio-console, virtio-gpu 2D with a framebuffer console), the process model (fork/exec/mmap/signals), and the kernel side of a libc (poll/epoll, nonblocking I/O, loopback, threads and futexes, ttys and ptys). - [x] Freestanding binary — [phil-opp](https://os.phil-opp.com/freestanding-rust-binary/) - [x] Minimal kernel booting in QEMU — [phil-opp](https://os.phil-opp.com/minimal-rust-kernel/), [OSDev: Bare Bones](https://wiki.osdev.org/Bare_Bones) - [x] VGA text mode printing — [phil-opp](https://os.phil-opp.com/vga-text-mode/), [OSDev: Printing to Screen](https://wiki.osdev.org/Printing_To_Screen) - [x] Serial output + QEMU exit — [phil-opp: Testing](https://os.phil-opp.com/testing/), [OSDev: Serial Ports](https://wiki.osdev.org/Serial_Ports) - [x] CPU exceptions / IDT — breakpoint & page fault handlers. [phil-opp: CPU Exceptions](https://os.phil-opp.com/cpu-exceptions/), [OSDev: IDT](https://wiki.osdev.org/Interrupt_Descriptor_Table), [OSDev: Exceptions](https://wiki.osdev.org/Exceptions) - [x] GDT / TSS / double faults — separate IST stack so stack overflows don't triple-fault. [phil-opp: Double Faults](https://os.phil-opp.com/double-fault-exceptions/), [OSDev: GDT](https://wiki.osdev.org/Global_Descriptor_Table), [OSDev: TSS](https://wiki.osdev.org/Task_State_Segment) - [x] Testing — `cargo test` in QEMU, custom test framework, integration tests. [phil-opp: Testing](https://os.phil-opp.com/testing/) - [x] Hardware interrupts — 8259 PIC, timer, PS/2 keyboard echo, `hlt` idle loop. [phil-opp: Hardware Interrupts](https://os.phil-opp.com/hardware-interrupts/), [OSDev: 8259 PIC](https://wiki.osdev.org/8259_PIC), [OSDev: PIT](https://wiki.osdev.org/Programmable_Interval_Timer), [OSDev: PS/2 Keyboard](https://wiki.osdev.org/PS/2_Keyboard) (later: [APIC](https://wiki.osdev.org/APIC)) - [x] Introduction to paging — CR3, page fault demo. [phil-opp](https://os.phil-opp.com/paging-introduction/), [OSDev: Paging](https://wiki.osdev.org/Paging) - [x] Paging implementation — OffsetPageTable, translation, frame allocator, example mapping. [phil-opp](https://os.phil-opp.com/paging-implementation/), [OSDev: Page Frame Allocation](https://wiki.osdev.org/Page_Frame_Allocation) - [x] Heap allocation — `alloc`, heap mapping, global allocator. [phil-opp](https://os.phil-opp.com/heap-allocation/), [OSDev: Memory Allocation](https://wiki.osdev.org/Memory_Allocation) - [x] Allocator designs — bump, linked list (crate), fixed-size block (active). [phil-opp](https://os.phil-opp.com/allocator-designs/) - [x] Async/await — Task, simple executor, waker-based executor, lock-free async keyboard task (with Backspace). [phil-opp: Async/Await](https://os.phil-opp.com/async-await/), [OSDev: Multitasking Systems](https://wiki.osdev.org/Multitasking_Systems) - [x] Preemptive kernel threads — own stacks, timer-driven round-robin, asm context switch. [OSDev: Brendan's Multi-tasking Tutorial](https://wiki.osdev.org/Brendan%27s_Multi-tasking_Tutorial), [OSDev: Context Switching](https://wiki.osdev.org/Context_Switching) - [x] User mode (ring 3) & syscalls — GDT user segments, TSS.rsp0 per thread, `syscall`/`sysret`, per-process page tables, user faults kill the process. [OSDev: Getting to Ring 3](https://wiki.osdev.org/Getting_to_Ring_3), [OSDev: System Calls](https://wiki.osdev.org/System_Calls), [OSDev: SYSENTER/SYSCALL](https://wiki.osdev.org/SYSENTER) - [x] ELF loading — hand-written PT_LOAD loader, user programs built by `build.rs`. [OSDev: ELF](https://wiki.osdev.org/ELF), [OSDev: ELF Tutorial](https://wiki.osdev.org/ELF_Tutorial) - [x] VFS skeleton + ramfs — Inode/FileSystem/File traits, mount table, fd table, `open/read/write/close`. [OSDev: VFS](https://wiki.osdev.org/VFS), [Linux VFS overview](https://www.kernel.org/doc/html/latest/filesystems/vfs.html) - [x] PCI enumeration + virtio-blk driver + `BlockDevice` trait — [OSDev: PCI](https://wiki.osdev.org/PCI), [OSDev: Virtio](https://wiki.osdev.org/Virtio), [virtio spec (legacy interface)](https://docs.oasis-open.org/virtio/virtio/v1.2/virtio-v1.2.html) - [x] MBR + GPT partitions — [OSDev: MBR](https://wiki.osdev.org/MBR_(x86)), [OSDev: GPT](https://wiki.osdev.org/GPT) - [x] FAT32 through `fatfs`, mounted at `/mnt/disk0`; programs loaded from disk — [OSDev: FAT](https://wiki.osdev.org/FAT) - [x] Blocking threads + wait queues, pipes, `spawn`/`wait`, cwd, `stat`/`lseek`/`getdents64` — [OSDev: Blocking Process](https://wiki.osdev.org/Blocking_Process) - [x] Shell as a user program: line editing, builtins, PATH, pipes, redirection, background jobs, drive letters, `run --trace` — [OSDev: Shell](https://wiki.osdev.org/Shell) - [x] Process cleanup: frame free list, address spaces freed on reap - [x] ext2, hand-written: read + write, indirect blocks, symlinks, 1/4 KiB blocks, revision 0/1, checked with `e2fsck` — [OSDev: Ext2](https://wiki.osdev.org/Ext2), [The Second Extended File System (Dave Poirier)](https://www.nongnu.org/ext2-doc/ext2.html) - [x] exFAT, hand-written: read + write, entry sets + checksums, up-case table, NoFatChain and FAT chains, checked with `fsck.exfat` — [Microsoft exFAT specification](https://learn.microsoft.com/en-us/windows/win32/fileio/exfat-specification) - [x] SMP (stage 11): ACPI MADT, local APIC + timer, INIT/SIPI trampoline, per-CPU GDT/TSS/`swapgs` data, a scheduler for 4 CPUs with sleep/wake and wait queues, frames freed on every exit, guard pages under kernel and user stacks — [OSDev: SMP](https://wiki.osdev.org/Symmetric_Multiprocessing), [OSDev: APIC](https://wiki.osdev.org/APIC), [OSDev: MADT](https://wiki.osdev.org/MADT), [OSDev: APIC timer](https://wiki.osdev.org/APIC_Timer) - [x] btrfs, hand-written, read-only: chunk tree (single/DUP), root/FS/csum trees, crc32c everywhere, zlib/lzo/zstd, holes, subvolumes, checked against `btrfs check` and the host's files — [btrfs on-disk format](https://btrfs.readthedocs.io/en/latest/dev/On-disk-format.html) - [x] Networking (stage 12): virtio-net (legacy, interrupt-driven), smoltcp for ARP/IPv4/ICMP/UDP/TCP/DHCP/DNS, BSD-style socket syscalls with blocking on wait queues, `ifconfig`/`ping`/`host`/`fetch`/`httpd` — [OSDev: Virtio](https://wiki.osdev.org/Virtio), [smoltcp](https://docs.rs/smoltcp), [OSDev: Networking](https://wiki.osdev.org/Category:Networking), [Beej's Guide to Network Programming](https://beej.us/guide/bgnet/) - [x] Virtual drivers (stage 13): a modern (1.0) virtio transport shared by virtio-rng (`getrandom`, `/dev/random`, smoltcp's seed), virtio-console (`/dev/hvc0`, `getty`) and virtio-gpu 2D (a 1024x768 framebuffer, a bitmap-font console, `/dev/fb0`, `tile`) — [virtio 1.2 spec](https://docs.oasis-open.org/virtio/virtio/v1.2/virtio-v1.2.html), [OSDev: Virtio](https://wiki.osdev.org/Virtio), [Linux virtio-gpu driver](https://github.com/torvalds/linux/tree/master/drivers/gpu/drm/virtio) - [x] Processes (stage 14): `fork` with copy-on-write (refcounted frames), `execve` with Linux's argv/envp/auxv stack, `mmap`/`munmap`/`mprotect`/`brk` with demand paging, signals with Linux signal frames and `sigreturn`, `wait4`, `dup2`/`pipe2`/`fcntl`, process groups, Ctrl+C and job control in the shell — [man 2 fork](https://man7.org/linux/man-pages/man2/fork.2.html), [man 2 execve](https://man7.org/linux/man-pages/man2/execve.2.html), [man 7 signal](https://man7.org/linux/man-pages/man7/signal.7.html), [LWN: getauxval() and the auxiliary vector](https://lwn.net/Articles/519085/), [OSDev: Paging](https://wiki.osdev.org/Paging) - [x] Libc plumbing (stage 15): `poll`/`ppoll`/`select`/`pselect6`/`epoll`, `O_NONBLOCK` + `EAGAIN`, a loopback interface, `futex`, `clone` threads with TLS (`arch_prctl`), `set_tid_address`, `gettid`/`tgkill`, `exit` vs `exit_group`, ttys and ptys with termios, `/dev/tty`, `/dev/ptmx` + `/dev/pts/N`, a fork+exec `getty`, `nanosleep`, `CLOCK_REALTIME`, `readv`/`writev`, `pread`/`pwrite` — [man 7 epoll](https://man7.org/linux/man-pages/man7/epoll.7.html), [man 2 futex](https://man7.org/linux/man-pages/man2/futex.2.html), [Drepper: Futexes Are Tricky](https://www.akkadia.org/drepper/futex.pdf), [The TTY demystified](https://www.linusakesson.net/programming/tty/), [man 7 pty](https://man7.org/linux/man-pages/man7/pty.7.html) - [ ] The kernel/OS split and a musl port (next; waits for Tyler's go) - [ ] A Hyprland-style tiling compositor on the framebuffer (later) ### Filesystem roadmap (Tyler's plan) Everything below plugs into the VFS above: a filesystem is a `FileSystem` whose `root()` returns `Inode`s; mounting it is one `fs::mount` call. 1. **ramfs / tmpfs** — done (`src/fs/ramfs.rs`). 2. **Block device driver** — done: `BlockDevice` trait + legacy virtio-blk (stage 6). Still to do: a block cache, IRQ-driven completion, maybe an ATA PIO driver for comparison ([OSDev: ATA PIO Mode](https://wiki.osdev.org/ATA_PIO_Mode)). 3. **Partition tables** — done: MBR + GPT (no CRC checks yet). 4. **FAT32** — done via the `fatfs` crate (`src/fs/fat.rs`). A good exercise later: a hand-written read-only FAT32 to see what the crate does. 5. **ext2** — done (stage 8, `src/fs/ext2.rs`): written by hand, read and write: superblock, block groups, bitmaps, inode table, direct/indirect blocks, directory entries, symlinks. Still to do: symlink creation, rename, hard links, a block cache, real timestamps once there's an RTC driver. Later, ext4's extents would be a nice follow-up. [OSDev: Ext2](https://wiki.osdev.org/Ext2), [The Second Extended File System (Dave Poirier)](https://www.nongnu.org/ext2-doc/ext2.html) 6. **exFAT** — done (stage 9, `src/fs/exfat.rs`): written by hand from Microsoft's spec, read and write: boot checksum, allocation bitmap, FAT chains and NoFatChain runs, up-case table, entry sets with SetChecksum and NameHash. Chosen because it works on literally anything (cameras, phones, Windows, macOS, Linux), so Rustbelt can share USB sticks and SD cards with every other machine. Still to do: rename, timestamps, TexFAT. [Microsoft exFAT specification](https://learn.microsoft.com/en-us/windows/win32/fileio/exfat-specification) 7. **btrfs, read-only** — done (stage 10, `src/fs/btrfs.rs`): because most of Tyler's Linux systems will use btrfs going forward, so this kernel can read their disks. Copy-on-write B-trees, the chunk tree (logical -> physical, single and DUP), root/FS/csum trees, crc32c checksums, zlib/lzo/zstd, holes, subvolumes. Still to do: writing (see "Why read-only first"), other checksum types, RAID/multi-device, xattrs. [btrfs on-disk format](https://btrfs.readthedocs.io/en/latest/dev/On-disk-format.html), [btrfs docs](https://btrfs.readthedocs.io/) 8. **NTFS** — last, maybe never. Next, in this order: 1. ~~**Core polish**~~ — done (stage 11): SMP on 4 CPUs, a scheduler with sleep/wake and wait queues, memory freed on every exit, guard pages. Left over for later: priorities, per-CPU run queues, an I/O APIC, a bigger or growable heap. 2. ~~**Networking**~~ — done (stage 12): virtio-net plus [smoltcp](https://github.com/smoltcp-rs/smoltcp), sockets, and `ifconfig`, `ping`, `host`, `fetch`, `httpd`. Left over for later: non-blocking sockets and `poll`, IPv6, a loopback interface, TLS. 3. ~~**More virtio devices**~~ — done (stage 13): virtio-rng, virtio-console (hvc0) and virtio-gpu in 2D mode, with the text console on a framebuffer and `/dev/fb0` for programs. Left over for later: moving virtio-blk and virtio-net to the modern transport, resizing the display, a hardware cursor, virtio-input (keyboard/mouse events for the compositor), 3D (virgl, see above). 4. ~~**The process model**~~ — done (stage 14): `fork` (copy-on-write), `execve` (with the auxiliary vector a libc needs), `mmap` and friends with demand paging, signals, `wait4`, process groups and job control. Left over for later: `MAP_SHARED` file mappings (needs a page cache), `sigaltstack`, queued real-time signals, users and permissions. 5. ~~**The rest of the kernel side of a libc**~~ — done (stage 15): all of the list below except `openat` with C strings, the bigger `stat` and `getuid`, which come with the musl port (they are easier to get right with musl's own test programs). Left over: `SIGTTIN`/`SIGTTOU` (background jobs touching the terminal), edge-triggered epoll, IPv6 loopback, per-file wait queues instead of one global poll counter. The plan as it was: - `poll`/`select` (and later `epoll`), so one program can wait on several fds: the keyboard, a pipe and a socket at once; - **non-blocking sockets** (`O_NONBLOCK`, `EAGAIN`, `connect` in progress) and a **loopback** interface (127.0.0.1), so programs can talk to each other over TCP on one machine; - **futexes and threads**: `clone(CLONE_VM | CLONE_THREAD ...)`, `set_tid_address`, `arch_prctl(ARCH_SET_FS)` for thread-local storage, and `futex` wait/wake, which is what libc mutexes and `pthread_join` are built on; - **ttys and ptys**: a proper terminal layer (termios, sessions, controlling terminals, `/dev/tty`, pseudo-terminals), so hvc0 gets job control and a terminal emulator becomes possible; - smaller things musl expects: `nanosleep`, `readv`/`writev`, `openat` with C strings, a bigger `stat`, `getuid` and friends. 6. **The kernel/OS split and a musl port** (next; this is the OS stage and waits for Tyler's go): user space moves into its own tree (libc, toolchain, programs), and [musl](https://musl.libc.org/) is ported, so ordinary C programs recompiled against Rustbelt's syscalls (numbered like Linux's already) can run. `mmap` of `/dev/fb0` comes with it, so programs can draw without a syscall per rectangle. One kernel job comes first: **saving FPU/SSE state** (`XSAVE`/`FXSAVE` on every switch and in signal frames). Rustbelt's own programs are built soft-float, so the kernel has never needed it, but musl and every ordinary C compiler output use SSE registers freely; without saving them two threads would silently corrupt each other's floating point. Also: paths as C strings (`openat`), Linux's full `struct stat`. 7. **The Hyprland-style compositor** (after that): see below; `tile` is its first sketch, and stage 15's ptys are what its terminal windows will run shells on. **Long term: a graphical desktop.** The goal is a **Hyprland-style tiling compositor**: windows tile automatically, with keyboard-driven layouts and smooth animations. It will draw **natively on the virtio-gpu framebuffer** first, with Rustbelt's own programs as clients, so it needs nothing beyond step 3 plus `mmap` and an input device. **Wayland client support** comes later, once step 6 can run ordinary Linux programs: the compositor then speaks the Wayland protocol, so existing Wayland apps can open windows on it. Out of scope for now: **drivers for physical hardware** (real disks, NICs, USB). Rustbelt targets QEMU's virtio devices. A **Raspberry Pi / ARM port** is the planned future hardware target. The split into a separate **userland and OS** (its own repo layout, libc, toolchain) comes later. Other ideas, any time: - **Shell extras**: Left/Right cursor movement, Tab completion of file names, `$?`, `export` and `$VAR` expansion (the environment itself is there since stage 14). - **More programs**: `more`/`less` (needs raw mode and screen size), an editor. Small exercises along the way: a timer-based async `sleep` future, speed up the PIT ([OSDev: PIT](https://wiki.osdev.org/Programmable_Interval_Timer)), add handlers for divide error / general protection fault, make `BootInfoFrameAllocator` O(1), add a `mem` stat (bytes allocated) to the allocator. Other good references: [OSDev wiki main page](https://wiki.osdev.org/Expanded_Main_Page), [Intel SDM](https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdms.html), the [blog_os repo](https://github.com/phil-opp/blog_os) (one branch per post, e.g. `post-04`), and for a UEFI/framebuffer setup later, the newer [`bootloader` 0.11](https://github.com/rust-osdev/bootloader) crate. ## Notes / gotchas - The target JSON changes occasionally with new nightlies (e.g. `target-pointer-width` became an integer, `rustc-abi` was added). If a nightly update breaks the build, compare with the latest `x86_64-blog_os.json` in the blog_os repo, or pin a date in `rust-toolchain.toml` (`channel = "nightly-2026-10-29"`). - The "packages contain code that will be rejected by a future version of Rust: core" warning comes from building `core` with build-std and is harmless. - `bootimage` prints "`CARGO_MANIFEST_DIR` env variable not set" when run outside `cargo run`; also harmless. - `cargo test` with build-std breaks ("duplicate lang item in crate `core`: `sized`") if `Cargo.toml` sets `panic = "abort"` in `[profile.*]`. The target JSON already sets `"panic-strategy": "abort"`, so the profile settings were removed. - Stale disk image when switching cargo features: cargo re-uses a previously built kernel ELF with an *old* mtime, so bootimage thinks its cached image is current and boots the wrong build. `scripts/runner.sh` and the scripts `touch` the ELF first to avoid this. If you call `cargo bootimage --features ...` by hand after switching features, `touch target/x86_64-rust_kernel/debug/rustbelt` first. - `build.rs` reruns when `user/`, `disk/`, `disk2/`, `disk3/`, `disk4/`, `build/` or `build.rs` change and rebuilds `target/disk.img` and the ext2, exFAT and btrfs images. QEMU opens them with `snapshot=on`, so the kernel's writes (the tests create and delete files) never reach the files on disk (except in `scripts/ext2-check.sh` and `scripts/exfat-check.sh`). - e2fsprogs: `sudo apt install e2fsprogs`. `mke2fs`, `e2fsck`, `debugfs` and `dumpe2fs` live in `/usr/sbin`, which isn't on a normal user's `PATH` (`build.rs` and the script look there). Newer `mke2fs` dropped `-r 0`; revision 0 is `-E revision=0`. `mke2fs -d` stores all-zero blocks as holes, so `sparse.bin`'s data chunks are deliberately non-zero. - exfatprogs: `sudo apt install exfatprogs`. `mkfs.exfat`, `fsck.exfat` and `dump.exfat` live in `/usr/sbin` too. Mounting an exFAT image to fill it needs root and a loop device, and `exfat-fuse` can't mount here either (`fusermount3: unknown option 'user=box'`; with sudo, "Block device required"), hence `build/exfat_populate.rs`. `dump.exfat -d PATH` in exfatprogs 1.2.9 can't find files inside some directories (e.g. `/NewFolder/Note.txt`), even on images that only mkfs and the populator touched; `dump.exfat -s DIR` lists them fine, so `exfat-check.sh` uses that. - btrfs-progs: `sudo apt install btrfs-progs` (6.14 here; `mkfs.btrfs`, `btrfs`, `btrfs-convert` are in `/usr/bin`). `mkfs.btrfs --rootdir` writes sparse files as real data (no holes, no prealloc) and numbers `-u` subvolumes in its own order (`-u projects -u ro:archive` gives `archive` 256 and `projects` 257). Compressed extents are at most 128 KiB, uncompressed ones 1 MiB, and data that doesn't compress is stored plain. `btrfs-convert` refuses ext2 with 1 KiB blocks and fails *silently* on a 64 MiB image (256 MiB works). `btrfs inspect-internal dump-tree` and `dump-super` on an image show exactly what the driver parses; they were the main debugging tool. - The VGA console is ASCII-only: `Ünïcödé café.txt` shows as `■n■c■d■ caf■.txt` on screen (serial shows it right). Mapping UTF-8 to code page 437 in `vga_buffer.rs` would be a small fix. - `docs/long-link` points through lots of `..` and `.`, but along a path that also works on Linux (`cat` it after `debugfs -R "rdump /docs /tmp/x"`). The VFS resolves `..` lexically, so a target like `file.txt/..` would work in Rustbelt but not on Linux; the test image avoids that. - Debug builds use a lot of stack: deep call chains (syscall -> VFS -> fatfs -> ELF loader) needed more than 16 KiB. If you see "kernel stack overflow in thread", that's the guard page catching it: make the stack bigger (`STACK_SIZE` in `src/stack.rs`) or the call chain shallower. - **SMP pitfalls hit in stage 11:** - *`join` returned before the memory was free.* The dead thread was removed from the table first and its stack freed a moment later, on another CPU, so the leak test saw 7 extra frames now and then. Now the stack and process reference are dropped first, then the thread is removed. - *Mixed-up output.* With two programs printing on two CPUs, lines came out as `count = CPU900000000`: `println!` sent each formatted piece as its own `write`, and the console printed one write piece by piece. Now `ulib`'s `println!` formats a whole line into a buffer first, and the console holds a lock for the whole of one write. - *A lost wake-up through the back door.* If a thread blocks but there's nothing else to run, it halts in place; a wake from another CPU then marks it Ready *and* queues it while it's still running here. Another CPU could then pick it up too. `done_waiting` takes it back off the queue. - *A process exiting before its spawner stored its thread id.* On one CPU that couldn't happen; on four it can. `reap` waits for the id. - Freeing a kernel stack's page tables would need every CPU to flush its paging-structure caches; keeping them and reusing the slot avoids it. - **Networking pitfalls hit in stage 12:** - *A double fault after `httpd` exited, now and then.* The cause was a **spurious interrupt**. When an interrupt line drops before the PIC can deliver it (the level-triggered virtio-net line, here), the PIC still raises *something*: IRQ 7 on the first chip or IRQ 15 on the second. Our IDT had no entry for IRQ 15, and an empty IDT entry doesn't give a friendly error. It gives a "segment not present" fault, and with no handler for that either, a double fault. Now IRQ 7 and 15 have handlers that check nothing and send no end-of-interrupt to the chip that didn't really interrupt (for IRQ 15, only the first chip gets one, because it did see the cascade). - *`[net] DHCP lease lost` at boot.* smoltcp's DHCP socket starts by reporting "deconfigured". It's only printed now if there was a lease. - *Slirp's TFTP server answers from port 69*, not from a fresh port like RFC 1350 says. `tests/net.rs` replies to wherever the data came from. - QEMU's default network card is an e1000. Adding `-netdev`/`-device virtio-net-pci` replaces it, so the PCI device count stayed 10 (stage 13's GPU, rng and console make it 12). - **Pitfalls hit in stage 15:** - *`write` must return what the caller asked for, not what went out.* With ONLCR a 6-byte `"hello\n"` becomes 7 bytes on the wire. The tty returned 7; the shell's `write_all` then sliced past the end of its buffer, panicked, and its panic handler overflowed the stack. Now the tty returns the caller's count (and maps a partial write back). - *Sleeping is only as fine as the timer.* `nanosleep(1 ms)` and a 50 ms futex timeout could return early or late by up to one 55 ms PIT tick, depending on where in the tick they started. Deadlines now add a tick (never early), and `nanosleep` tops up by yielding until the TSC clock says the time has really passed. - *A closed slave means hang-up.* `ptytest` closed its copy of the slave after forking; for a moment no one had it open, the master reported `POLLHUP`, and the test read EOF. Keep the slave open in the parent (Linux behaves the same). - *Read one byte at a time in raw mode if you might `exec`.* The shell's line editor read 64 bytes at once; when commands were pasted into hvc0, the lines after the first were sitting in its buffer and were lost when it ran the first. Reading one byte per call leaves the rest in the tty, where the next reader finds it. - *Lazily started kernel threads look like leaks.* The hvc0 input thread started on first use, so its stack (16 frames) appeared between the headless self-test's two frame counts. It is now started at boot when a virtio console is present. - **Pitfalls hit in stage 14:** - *The kernel wrote straight through copy-on-write.* Ring 0 ignores read-only page flags unless `CR0.WP` is set, so a `read()` into a forked child's buffer changed the parent's memory too. `memory::init` now sets `CR0.WP`, and every syscall runs user buffers through the fault logic first (`mm::user_access_ok`). - *`sysretq` can't return from a signal handler.* It loads RIP and RFLAGS from `rcx` and `r11`, so those two registers can never be restored. Syscalls now return through `iretq` like interrupts (`src/trap.rs`). - *`siginfo` field offsets.* For SIGCHLD, Linux puts `si_pid` at offset 16, `si_uid` at 20 and `si_status` at 24. `ulib`'s struct had the status one `i32` too far, so `sigtest` saw 0 instead of the exit code. - *Leaked file descriptors across exec.* The shell's script file was open as fd 3 when it forked, so every program inherited it and `run --trace` showed `open(...) = 4` instead of 3. The shell now opens scripts, pipes and redirections with `O_CLOEXEC`, as real shells do. - *Tracing `execve`.* The trace line was built after the call, when the path string lived in an address space that no longer existed (``). It's now formatted before the call. - *`kill %1` without job control.* In a script (no terminal), jobs aren't in their own process group, so `kill -pgid` found nobody (`ESRCH`); it now signals the job's pid there. - *A fork bomb must leave nothing behind.* `MAX_PROCESSES` (64) makes the 65th fork fail with `EAGAIN`. The test counts frames and processes before and after killing the bomb, and after a warm-up run, so lazily allocated kernel structures don't look like leaks. - *`AT_EXECFN` was just `forktest`* when started by `spawn`, which only passed the basename along. `spawn_image` now gets the full path. - **Pitfalls hit in stage 13:** - *`memory allocation of 65536 bytes failed` in `tile`.* User programs have a fixed 64 KiB heap (`user/ulib.rs`), so a 64 KiB row buffer can never fit. The wallpaper now goes out 8 rows (32 KiB) at a time; a growable heap needs `mmap` or `brk`, next stage. - *virtio-rng's PCI class is 0x00 ("unclassified") and the console's is 0x07 (communication)*; `pci::class_name` didn't know them. With the GPU, rng and console the PCI bus now has 12 devices instead of 10. - *virtio-gpu has no legacy interface*, so the stage-6 style "one I/O port BAR" driver couldn't work: the modern transport (capabilities, memory BARs, a 64-bit BAR for the GPU) had to come first. - *The log mirror must never deadlock.* Every `serial_print!` also goes to hvc0. `serial::_print` releases the serial lock first and calls the console with interrupts off, so an interrupt handler that prints can't find the hvc0 lock already held on its own CPU. - *A test counted mounts*: `/dev` is a new mount, so `tests/vfs.rs` expects 4, and the root listing has `dev` in it. - `-vga none -device virtio-vga` keeps the old VGA text memory (virtio-vga *is* VGA-compatible), so the console tests that read `0xb8000` still pass, and the bootloader's text output still shows until the switch. - *A boot that sat at `A:/ $` for 90 s* turned out to be a stale, non-headless image (the headless build had failed to compile), waiting for keyboard input; not a driver bug. Check the build before the boot. - **The `\ |` key**: pc-keyboard 0.7's scancode set 1 decodes it (scancode 0x2B) as `KeyCode::Oem7`, but its US 104-key layout only maps `Oem5` to `\` and `|`. So that key typed nothing, which went unnoticed until the shell needed `|`. `task/keyboard.rs` renames `Oem7` to `Oem5` before decoding. - User programs: the linker drops empty output sections, and their `ALIGN(4K)` with them. A program without `.data` then had its `.bss` on the read-only `.rodata` page (see "ELF loading"). `user/link.ld` uses `. = ALIGN(4K);` before `.data`, and the loader now rejects such files.