rust-kernel

@@ -175,7 +175,7 @@ dependencies = [  [[package]] name = "rustbelt"-version = "0.10.0"+version = "0.11.0" dependencies = [  "bootloader",  "conquer-once",

@@ -1,6 +1,6 @@ [package] name = "rustbelt"-version = "0.10.0"+version = "0.11.0" edition = "2024"  [dependencies]@@ -78,6 +78,8 @@ harness = false # Extra QEMU arguments used by `cargo run` (via `bootimage runner`). [package.metadata.bootimage] run-args = [+    # Four CPU cores (smp.rs starts the other three).+    "-smp", "4",     # Second disk: the FAT32 image build.rs makes (target/disk.img), as a     # virtio-blk PCI device. snapshot=on: writes go to a temporary overlay,     # so the image is the same at every boot (and tests can't corrupt it).@@ -93,6 +95,7 @@ run-args = [ ] # Used by `cargo test`: headless, results on serial, exit code 33 = pass. test-args = [+    "-smp", "4",   # four CPU cores (tests that call init_smp use them all)     # Second disk: the FAT32 image build.rs makes (target/disk.img), as a     # virtio-blk PCI device. snapshot=on: writes go to a temporary overlay,     # so the image is the same at every boot (and tests can't corrupt it).

@@ -13,17 +13,25 @@ Under the shell:   to the serial port (COM1). - **CPU exceptions.** It handles breakpoints, page faults, and double faults   (on their own stack).-- **Timer and keyboard.** The timer drives an uptime clock and preemption. The+- **Several CPUs.** It starts every CPU core QEMU gives it (`-smp 4`)+  through the local APIC, and its scheduler runs threads and programs on+  all of them at once.+- **Timers and keyboard.** The PIT drives an uptime clock and wakes+  sleeping threads; each CPU's local APIC timer drives preemption. The   PS/2 keyboard feeds a tty with cooked and raw modes. - **Memory.** It walks and edits its own page tables. It has a kernel heap-  with a fixed-size-block allocator.+  with a fixed-size-block allocator, and frees every frame a process or+  thread used when it exits. Every stack has an unmapped **guard page**+  under it, so an overflow is caught and reported instead of silently+  corrupting memory. - **Multitasking.** It multitasks two ways: an **async/await executor** and-  **preemptive kernel threads**.+  **preemptive kernel threads**. Threads can sleep and wait for events+  (wait queues) without using any CPU while they wait. - **User programs.** Small Rust ELF binaries run in **ring 3** as processes,   each with its own address space. Their memory is freed when they exit. They-  talk to the kernel through about 26 `syscall`s, numbered like Linux's:-  `read`, `write`, `open`, `stat`, `getdents64`, `pipe`, `uname`... Two are-  our own: `spawn` and `wait`.+  talk to the kernel through about 27 `syscall`s, numbered like Linux's:+  `read`, `write`, `open`, `stat`, `getdents64`, `pipe`, `uname`... A few+  are our own: `spawn`, `wait`, `mounts` and `sysinfo`. - **Files.** Files live behind a Linux-style **VFS**: a ramfs at `/` and   `/tmp`, a **FAT32** partition mounted at `/mnt/disk0`, an **ext2**   disk at `/mnt/disk1`, an **exFAT** disk at `/mnt/disk2` and a **btrfs**@@ -45,9 +53,73 @@ Most code follows the blog_os sources (MIT / Apache-2.0) closely, with extra comments. Everything else is new: the serial mirroring, uptime clock, demo features and scripts, and everything past chapter 12. That covers preemptive threads, user mode, syscalls, ELF loading, the VFS, PCI, virtio-blk,-partitions, FAT32, the shell, and the ext2, exFAT and btrfs drivers.--![Rustbelt shell session on the btrfs drive E:: ls -l with a symlink and two subvolumes, cat of an inline file through a symlink and of a compressed file, cp from E: to D:, and a write refused with "Read-only file system"](boot.png)+partitions, FAT32, the shell, the ext2, exFAT and btrfs drivers, and+multi-core (SMP) support.++![Rustbelt on 4 CPUs: ver shows 0.11.0 (stage 11: core polish); two spin programs run in the background; sysinfo reports 4 CPUs online with two of them at 100% load; overflow is killed with "stack overflow"](boot.png)++## New to Rust and C? Start here++Rustbelt is meant to be read. You don't need to know Rust or C well to+follow it; here is a path through it, and the handful of ideas that make+kernel code look strange at first.++**A reading order.** Each file starts with a comment explaining what it is+for. Read them roughly in the order the kernel uses them:++1. `src/main.rs` -- `kernel_main` is the first Rust code that runs. It+   calls the `init` functions below in order, then starts the shell.+2. `src/vga_buffer.rs`, `src/serial.rs` -- printing text: the screen is just+   memory at address `0xb8000`, the serial port is a few I/O ports.+3. `src/interrupts.rs`, `src/gdt.rs` -- what happens when the CPU is+   interrupted (a key press, a timer, a crash).+4. `src/memory.rs`, `src/allocator.rs` -- page tables (how addresses are+   translated) and the heap (`Box`, `Vec`).+5. `src/thread.rs` -- threads, the scheduler and sleeping. `src/smp.rs` and+   `src/apic.rs` -- how the other CPUs get started.+6. `src/process.rs`, `src/syscall.rs`, `src/elf.rs` -- user programs.+7. `user/hello.rs`, then `user/sh.rs` -- what a program on Rustbelt looks like.+8. `src/fs/` -- the file systems, once the rest makes sense.++**Rust in kernel code, in five ideas:**++- **`#![no_std]`**: there is no operating system underneath, so Rust's+  standard library (files, threads, `println!`) isn't available. Only+  `core` (basic types) and, once there's a heap, `alloc` (`Box`, `Vec`,+  `String`). Rustbelt defines its own `println!`.+- **`unsafe`**: Rust normally guarantees memory safety; inside an `unsafe {}`+  block *you* promise it instead. Kernels need it to touch hardware (raw+  addresses, port I/O, CPU registers). Rustbelt keeps these blocks small and+  says in a comment why each one is OK.+- **Ownership and `Drop`**: every value has one owner, and when the owner+  goes away Rust runs its `Drop`. Rustbelt uses that for cleanup: dropping a+  `KernelStack` unmaps it and frees its memory (`src/stack.rs`); dropping+  the last `Arc<Process>` frees the whole address space.+- **`Mutex` and `Arc`**: `spin::Mutex<T>` is a lock: `.lock()` gives you the+  `T` and unlocks automatically when you're done. `Arc<T>` is a shared+  pointer with a reference count, so several threads can hold the same+  process.+- **`global_asm!` / `asm!`**: a few things (switching stacks, entering ring+  3, starting a CPU) need raw assembly. Each block is commented line by line.++**C ideas you'll see** (the hardware docs and OSDev wiki are written in C+terms): a *pointer* is just an address; `#[repr(C)]` lays out a struct the+way C would, so it matches what the CPU or a disk format expects; *volatile*+reads and writes tell the compiler a memory access really has to happen+(it's a device, not ordinary memory).++**Kernel words, briefly:** a *page* is 4 KiB of memory; a *frame* is the+physical memory behind it. An *interrupt* stops the CPU and runs a handler.+*Ring 0* is the kernel, *ring 3* user programs. A *syscall* is a user program+asking the kernel for something. *SMP* means several CPUs sharing memory; the+first one to boot is the *BSP*, the others *APs*.++**Try things.** `cargo run`, then in the shell: `help`, `ver`, `sysinfo`,+`spin 200 > A:/tmp/spin.txt &` followed by `sysinfo` (watch one CPU go to 100%), `overflow`+(a stack overflow, caught). Change something small -- the message in+`user/overflow.rs`, the bar width in `user/sysinfo.rs` -- and run+`cargo test` to check nothing broke. When something crashes, the `[kernel]`+lines on the serial output tell you which handler caught it and where.  ## What's installed (on this box) @@ -148,7 +220,8 @@ only, so the VGA screen shows just the shell session (and kernel messages such as a killed process).  ```-Rustbelt 0.10.0 -- a hobby x86_64 kernel in Rust+Rustbelt 0.11.0 -- a hobby x86_64 kernel in Rust+[smp] 4 CPUs online [pci] 00:00.0 8086:1237 host bridge ... (10 PCI devices) [pci] 10 devices on the PCI bus@@ -175,7 +248,7 @@ Rustbelt 0.10.0 -- a hobby x86_64 kernel in Rust A:/ $ # Rustbelt shell self-test: `cargo run --features headless` runs this with A:/ $ # `sh /mnt/disk0/scripts/selftest.sh` (B:/scripts/selftest.sh) and scripts/run-headless.sh checks the output. A:/ $ ver-Rustbelt 0.10.0 (stage 10: btrfs read-only) x86_64+Rustbelt 0.11.0 (stage 11: core polish) x86_64 A:/ $ drives A:  /            ramfs B:  /mnt/disk0   fat32@@ -437,10 +510,46 @@ mkdir: newdir: Read-only file system E:/ $ run evil [process] started pid 53 (evil), entry 0x80004000f0 [evil] pid 53: reading kernel memory at 0x200000...-[kernel] user page fault: address Ok(VirtAddr(0x200000)), error PageFaultErrorCode(PROTECTION_VIOLATION | USER_MODE)+[kernel] user page fault: address 0x200000, error PageFaultErrorCode(PROTECTION_VIOLATION | USER_MODE) [kernel] page fault in user mode at rip 0x800040017e [process] pid 53 (evil) killed: page fault [exit code 139]+E:/ $ # Stage 11: a stack overflow hits the guard zone and is killed cleanly;+E:/ $ # then two CPU hogs in the background, and sysinfo shows the load per CPU.+E:/ $ run overflow+[process] started pid 54 (overflow), entry 0x8000400130+[overflow] pid 54: recursing until the stack runs out...+[kernel] user page fault: address 0xffffffae88, error PageFaultErrorCode(CAUSED_BY_WRITE | USER_MODE)+[kernel] stack overflow in user mode at rip 0x8000400021+[process] pid 54 (overflow) killed: stack overflow+[exit code 139]+E:/ $ start spin+[process] started pid 55 (spin), entry 0x8000400400+[pid 55] spin started in the background+[spin] pid 55: spinning without syscalls...+E:/ $ start spin+[process] started pid 56 (spin), entry 0x8000400400+[pid 56] spin started in the background+[spin] pid 56: spinning without syscalls...+E:/ $ sysinfo+[process] started pid 57 (sysinfo), entry 0x8000400420+[spin] round 1 done (count = 30000000)+[spin] round 1 done (count = 30000000)+[spin] round 2 done (count = 60000000)+[spin] round 2 done (count = 60000000)+4 CPUs online (this program ran on CPU 1), up 11 s+10 threads, 4 processes, 1296 KiB of memory frames in use+[spin] round 3 done (count = 90000000)+[process] pid 55 (spin) exited with code 0+CPU load over the last quarter second:+  CPU0 [####################] 100%+  CPU1 [....................]   0%+  CPU2 [####################] 100%+  CPU3 [....................]   0%+[process] pid 57 (sysinfo) exited with code 0+[done] pid 55 (start spin), exit code 0+E:/ $ wait+... E:/ $ nosuchprogram sh: nosuchprogram: command not found (try `help`) E:/ $ cd Q:@@ -451,15 +560,20 @@ A:/tmp $ pwd A:/tmp $ exit 0 [process] pid 1 (sh) exited with code 0 [ok] shell self-test finished: Exited(0)-[memory] 40 frames in use-[run-headless] all 41+4 expected shell lines found+[memory] 171 frames in use before the shell, 171 after it and all its children exited+[run-headless] frames in use returned to the baseline+[run-headless] all 45+4 expected shell lines found [run-headless] kernel exited OK ``` -The run ends with `[memory] 40 frames in use`, even though 53 processes have-come and gone. That's exactly what was in use right after boot (mostly the-four virtio disks' queues and buffers, 10 frames each), so exited processes-give their page tables, code and stacks back.+(Which CPU each program lands on, and the exact order of lines printed by+programs running at the same time, change from run to run.)++The run ends with the same number of frames in use before and after the+shell, even though 57 processes have come and gone; `run-headless.sh`+fails if they differ. Exited processes give back their page tables, code,+user stacks and kernel stacks. (The 171 at boot are mostly the four virtio+disks' queues and buffers, the kernel stack area and the other CPUs' stacks.)  The kernel boots from the bootimage on the first (IDE) disk; the scripts and `Cargo.toml` attach a **second** disk, `target/disk.img` (GPT + FAT32), a@@ -515,6 +629,8 @@ heap_allocation::simple_allocation...               [ok] | `tests/exfat.rs` | both exFAT images are mounted (`/mnt/disk2`, `/mnt/disk6`), label and cluster size; long, mixed-case and Unicode names listed as written; case-insensitive lookup (`ÜNÏCÖDÉ CAFÉ.TXT`, nested paths); `numbers.txt` at a cluster boundary and at its end; the FAT-chained `frag-chain.txt` on 4 KiB and 512-byte clusters; bad names (`EINVAL`, `ENAMETOOLONG`, `EEXIST` for a name in another case); on each image: mkdir, create, write across clusters, append, truncate, a write past the end (zero gap), `ENOTEMPTY`, unlink, rmdir, and the bitmap and PercentInUse are back where they started; a contiguous file turns into a FAT chain; 200 files grow a directory; the root grows; copies between FAT, ext2 and exFAT; a last test leaves files for `scripts/exfat-check.sh`'s `fsck.exfat` | | `tests/btrfs.rs` | the four btrfs images are mounted (`/mnt/disk3`, `disk7`-`disk9`) and described (label, node size, DUP/single, subvolume count); **every file on every image is read and its size and CRC-32 compared with build.rs's manifest of the host's tree** (zlib, lzo with a two-level tree, zstd with 64 KiB nodes, and the converted image); reads at odd offsets across compressed and plain extent boundaries; holes in the converted `sparse.bin`; `ext2_saved/image` starts like an ext2 disk; hash lookups are exact (`HELLO.TXT` is `ENOENT`); symlinks, including one leaving a subvolume; subvolumes listed with ids and read-only flags; the sha256 image is refused with a reason; create/write/truncate/mkdir/remove give `EROFS`; and, last, damage on purpose: copy 0 of a DUP node (copy 1 is used), a data sector (`EIO` for that block only), then both copies (`EIO`). `scripts/btrfs-check.sh` runs `btrfs check --readonly` and compares the same kernel's output with the host's files | | `tests/threads.rs` | two busy-looping threads that never yield both make progress while the main thread busy-waits too (only possible with preemption); a thread can exit and the others keep running |+| `tests/smp.rs` | **multi-core and the scheduler**: all 4 CPUs are online and take timer ticks; 4 busy threads run on 2+ CPUs *at the same time* (per-CPU busy ticks); 3 `spin` processes run on 2+ CPUs; 4 threads on 4 CPUs add 200,000 to a locked counter without losing one; `sleep_ticks(10)` lasts 10-12 ticks; sleepers wake in deadline order (3, 6, 9); while 4 threads sleep the CPUs are idle (<10% busy, so no busy-polling); two threads ping-pong 1,000 times through a `WaitQueue`; **leak checks**: 200 threads and 40 processes spawned and exited leave `frames_in_use`, the thread count, the process count and the number of kernel stacks exactly where they were |+| `tests/guard_pages.rs` | the guard page sits right below each kernel stack, and the user stack's 64 KiB guard zone has the right bounds; a kernel thread that recurses forever is killed with "kernel stack overflow" three times in a row (the rest of the kernel carries on, and its stack is freed); the user `overflow` program ends with `Killed("stack overflow")` and leaves no frames behind |  How it works: `#![feature(custom_test_frameworks)]` collects every `#[test_case]` function and passes them to `rustbelt::test_runner` (in `src/lib.rs`), which@@ -535,14 +651,16 @@ Cargo.toml                deps, features, test settings, QEMU run/test args src/lib.rs                the kernel library: module list, init(), kprintln!, test framework, exit_qemu src/main.rs               the bootable binary: kernel_main (init, then a thread that runs the shell and                           restarts it when it exits), the `demos` feature's demos, panic handler-src/gdt.rs                GDT (kernel + user segments) + TSS (double-fault IST stack, rsp0 per thread)+src/gdt.rs                GDT (kernel + user segments) + TSS (double-fault IST stack, rsp0 per thread);+                          one GDT + TSS per CPU src/interrupts.rs         IDT; breakpoint / page fault / GPF / invalid opcode / double fault handlers-                          (faults from ring 3 kill the process); 8259 PIC;-                          timer handler (uptime, ticks, calls the thread scheduler),+                          (faults from ring 3 kill the process; guard-page hits kill with+                          "stack overflow"); 8259 PIC; local APIC timer, reschedule IPI;+                          PIT handler (uptime, ticks, wakes sleeping threads),                           keyboard handler (just queues the scancode) src/memory.rs             OffsetPageTable setup from CR3, example mapping, BootInfoFrameAllocator,-                          global frame allocator + free list, per-process address spaces (and freeing-                          them), user pointer checks+                          global frame allocator + free list (free_frame), per-process address spaces+                          (and freeing them), user pointer checks, kernel stack mappings src/allocator.rs          heap location/size, init_heap (maps heap pages), #[global_allocator], Locked<A> src/allocator/bump.rs     bump allocator (not active, for learning) src/allocator/fixed_size_block.rs  fixed-size-block allocator (ACTIVE), falls back to linked_list_allocator@@ -550,8 +668,14 @@ src/task/mod.rs           async: Task (pinned boxed future + id), yield_now() src/task/simple_executor.rs  first executor: polls every task in a loop (no wakers) src/task/executor.rs      real executor: task queue + wakers, sleeps with hlt when idle src/task/keyboard.rs      lock-free scancode queue, ScancodeStream, async keyboard task (keys -> tty)-src/thread.rs             preemptive threads: spawn, round-robin schedule(), asm context switch,-                          per-thread kernel stack (TSS.rsp0) + CR3 switching+src/thread.rs             threads + the multi-core scheduler: spawn, one shared run queue, per-CPU idle+                          threads, asm context switch, sleep/wake, WaitQueue, join, stack-overflow kills+src/stack.rs              kernel stacks: 64 KiB each with an unmapped guard page below, in their own+                          region (L4 slot 200), slots reused when threads exit+src/smp.rs                multi-core bring-up: AP trampoline (real mode -> long mode, in assembly),+                          INIT/SIPI, per-CPU data (swapgs), CPU numbering, waking idle CPUs+src/apic.rs               local APIC: enable, timer (calibrated against the PIT), EOI, IPIs+src/acpi.rs               finds the ACPI RSDP/RSDT/MADT to list the CPUs src/syscall.rs            syscall/sysret MSR setup, asm entry stub, dispatch + all syscalls (table at the top) src/process.rs            Process (pid, parent, cwd, args, fd table, exit status), spawn from a path or                           ELF bytes, iretq into ring 3, exit/kill, wait (reaping)@@ -600,6 +724,8 @@ user/sh.rs                the shell: line editor, builtins, aliases, PATH, pipes user/shparse.rs           the shell's parser + drive letters (also compiled into tests/shell.rs) user/{ls,cat,wc,grep,mkdir,rm,touch,cp,sleep}.rs   the shell's tools user/{hello,evil,spin,demo,child}.rs  test/demo programs (all programs go to /mnt/disk0/bin)+user/sysinfo.rs           CPUs, per-CPU load bars, threads, processes, memory (the sysinfo syscall)+user/overflow.rs          recurses forever: killed with "stack overflow" by the user stack guard zone src/vga_buffer.rs         VGA text-mode writer, print!/println!, print_at (status line), backspace src/serial.rs             COM1 UART writer + serial_print!/serial_println! tests/                    integration tests, one QEMU-booted kernel each@@ -707,9 +833,10 @@ cheap (just return from `poll`), and all tasks share one stack.  ### Preemptive: kernel threads (`src/thread.rs`, beyond the blog series) -Each thread has its own 64 KiB stack (a `Vec<u8>` on the heap). On every timer-interrupt (~18 times a second) the handler sends the EOI to the PIC and then-calls `thread::schedule()`, which picks the next thread round-robin and calls+Each thread has its own 64 KiB stack (a `KernelStack`, with a guard page+under it). On every local APIC timer interrupt (~100 times a second, on+each CPU) the handler sends the EOI and then calls `thread::schedule()`,+which picks the next ready thread round-robin and calls `switch_context(&mut old.rsp, new.rsp)` (a dozen lines of `global_asm!`):  ```@@ -756,8 +883,8 @@ and allocates, so `GlobalAlloc::alloc`/`dealloc` now take the heap lock inside **Blocking (stage 6).** A thread can be *blocked*: the scheduler skips it until someone wakes it. Two ways to be woken: -- `sleep_ticks(n)` sets `wake_at = now + n`; `schedule()` (run on every timer-  tick) unblocks it once that tick has passed (the `sleep_ms` syscall rounds+- `sleep_ticks(n)` marks the thread sleeping until tick `now + n`; the PIT+  handler wakes it once that tick has passed (the `sleep_ms` syscall rounds   up to 55 ms ticks). - A `WaitQueue` (a list of waiting thread ids). `wait_until(|| condition)`   turns interrupts off, checks the condition, and if it's false puts the@@ -771,18 +898,150 @@ someone wakes it. Two ways to be woken: If every thread is blocked, the scheduler idles with `sti; hlt` until an interrupt makes something runnable. -**Stack size and the canary.** Stage 6 first used 16 KiB stacks and crashed-with a ring-0 jump to address 0: the syscall -> VFS -> FAT -> ELF loader path in-an unoptimized debug build used more than that, overflowed into the heap and-smashed a neighbour. Now stacks are 64 KiB, and the lowest 8 bytes of each-hold a canary value (`0x5AFE_57AC_C0DE_CAFE`). `schedule()` checks it on every-switch and panics with "kernel stack overflow in thread" if it changed: a-cheap stand-in for a real guard page. Finished threads' stacks are freed (the-next `spawn` drops them, since a thread can't free the stack it's running on).--Known simplifications (good exercises): no guard page under thread stacks (the-canary only catches overflows after the fact); no priorities; a single CPU,-so `spin::Mutex` is enough.+**Stack size.** Stage 6 first used 16 KiB stacks and crashed with a ring-0+jump to address 0: the syscall -> VFS -> FAT -> ELF loader path in an+unoptimized debug build used more than that, overflowed into the heap and+smashed a neighbour. Stacks are 64 KiB now, and since stage 11 each has a+real guard page under it (see "Multiple CPUs" below), which replaced the+old canary check.++Stage 11 rewrote the scheduler for several CPUs; the next section explains+what changed.++## Multiple CPUs, sleep/wake and guard pages (stage 11)++Until stage 10, Rustbelt used one CPU. QEMU now runs with `-smp 4`, and the+kernel starts all four.++### Starting the other CPUs (`src/smp.rs`, `src/apic.rs`, `src/acpi.rs`)++At power-on only one CPU runs: the *bootstrap processor* (BSP, CPU 0). The+others (*application processors*, APs) wait, halted, until the BSP sends+them messages through its **local APIC**, a small interrupt controller+built into every CPU core:++1. **Which CPUs exist?** The firmware describes them in ACPI tables. `acpi.rs`+   finds the RSDP (a signature in the BIOS area), follows it to the RSDT, then+   to the MADT, which lists one entry per CPU with its APIC id.+2. **Trampoline.** An AP starts in 16-bit *real mode*, like an 8086, at an+   address below 1 MiB. So `smp.rs` copies a small assembly trampoline to+   physical address `0x8000`, plus parameters: the page table (CR3), the+   control registers, a fresh kernel stack, the CPU number and the Rust+   function to call. The trampoline jumps straight from real mode into+   64-bit long mode using the BSP's settings.+3. **INIT, then STARTUP.** The BSP sends an *INIT* inter-processor interrupt+   (IPI), waits ~110 ms (two PIT ticks), then a *STARTUP* IPI whose vector+   (`0x08`) means "start at `0x8000`"; a second STARTUP if the AP didn't+   answer. Then it waits for the AP to say it's ready, one CPU at a time.+4. **AP setup** (`ap_entry`): its own GDT and TSS (with its own double-fault+   stack), the shared IDT, the syscall MSRs, its local APIC and timer,+   and then it becomes an idle thread in the scheduler.++**Per-CPU data.** Some things must exist once per CPU: the TSS (it holds the+stack used when an interrupt arrives from ring 3), the "current thread", the+syscall entry's scratch slots. `cpu_id()` reads the local APIC id and maps it+to 0..3. The syscall entry can't call Rust before it has a stack, so it uses+`swapgs`: the `KERNEL_GS_BASE` MSR of each CPU points at that CPU's+`PerCpu { syscall_kernel_rsp, syscall_user_rsp }`, and `swapgs` makes+`gs:[0]`/`gs:[8]` reach them.++**Timers.** The old PIT (on the 8259 PIC) still runs, on CPU 0 only: it+keeps the uptime and wakes sleeping threads. Each CPU's local APIC timer,+calibrated against the PIT to ~10 ms, preempts whatever that CPU runs.+**No I/O APIC**: it routes device interrupts to chosen CPUs, but+Rustbelt's devices are polled (virtio-blk) or happy on CPU 0 (keyboard,+PIT), so the 8259 is kept and wired to CPU 0's local APIC (LINT0 in+"ExtINT" mode). Moving to the I/O APIC is a good exercise once+interrupt-driven virtio needs it.++### The scheduler (`src/thread.rs`)++- **One shared ready queue** (a `VecDeque`) under one lock. Each CPU has its+  own *current* thread and its own *idle* thread, which runs `sti; hlt` when+  there's nothing to do. Simple and fair; per-CPU queues with work stealing+  would scale better, but four CPUs don't need them.+- **Thread states:** `Ready`, `Running`, `Blocked` (on a wait queue),+  `Sleeping(until tick)`, `Finished`.+- **Handing a thread to another CPU safely.** When CPU 0 switches away from+  thread T, T stays marked `on_cpu` until CPU 0 has fully left T's stack+  (`finish_switch`, right after the switch). Until then no other CPU may+  pick T, or two CPUs would run on the same stack.+- **Sleeping without polling.** `sleep_ticks(n)` marks the thread+  `Sleeping(now + n)` and switches away; it's not on the ready queue, so it+  costs nothing. The PIT handler wakes threads whose time has come+  (`wake_sleepers`) and, if some CPU is idle, sends it a *reschedule IPI*+  (vector `0x41`) so it picks the thread up at once instead of at its next+  timer tick.+- **Wait queues.** `queue.wait_until(|| condition)` blocks until the+  closure returns `Some`; `queue.wake_all()` wakes the waiters. The+  classic *lost wake-up* (the event happens between "check" and "block",+  possibly on another CPU) is prevented by joining the queue *before*+  checking, plus a `woken` flag: a wake that arrives early makes the block+  return immediately. Console input, pipes, `wait`, `join` and the async+  executor (when it has nothing to do) all use wait queues.+- **Statistics.** Each local APIC tick counts as *busy* or *idle* for that+  CPU; the `sysinfo` syscall (502) returns these, and `user/sysinfo.rs`+  turns two samples into load bars.++**The locking rule now matters more:** a lock that is ever taken with+interrupts off must always be taken with interrupts off. On one CPU breaking+it deadlocks only sometimes; on four, a thread spinning with interrupts off+on CPU 1 can wait forever for a lock held by a preempted thread. The+scheduler, frame allocator, heap, process table and kernel page-table locks+all follow it.++### Memory cleanup++Everything a thread or process had is given back when it ends:++- A finished thread can't free the stack it's running on, so the *next*+  thread on that CPU frees it (`finish_switch`), before removing the thread+  from the table. So when `join` returns, the memory really is free.+- A process's address space (its page tables at every level, code, data,+  heap, user stack) is freed in `Drop for Process`, when the last `Arc` to it+  goes away. `wait` joins the process's thread before reaping, so the parent+  holds the last reference.+- Children whose parent exits first are *orphaned* and clean themselves up+  when they exit.++`memory::frames_in_use()` counts frames; the headless run and `tests/smp.rs`+check it returns to exactly the same number after hundreds of threads and+processes.++### Guard pages (`src/stack.rs`, `src/process.rs`)++A stack grows down. If it grows past its end it would silently overwrite+whatever lies below. A **guard page** is an address range under the stack+that is deliberately *not mapped*: touching it causes a page fault, so the+overflow is caught at the first byte.++- **Kernel stacks** live in their own region of the address space (L4 slot+  200, `0x0000_6400_0000_0000`): 4 KiB guard page, 64 KiB stack, guard page,+  stack, ... When a kernel stack overflows, the CPU can't even push the+  page-fault frame (it would land in the guard page too), so it raises a+  *double fault*, which runs on its own IST stack. That handler sees the+  faulting address is a guard page and prints+  `[kernel] kernel stack overflow in thread 'NAME' (hit guard page 0x...)`,+  then kills that thread (or its process) and keeps going.+- **Stack probes.** A function with a big local array could jump *over* a+  4 KiB guard page. `"stack-probes": {"kind": "inline"}` in the target JSON+  makes the compiler touch each page of a big frame in order, so it can't.+- **User stacks** (16 KiB) get a 64 KiB never-mapped zone underneath, and+  the ELF loader refuses programs whose segments reach into it. A fault+  there kills the program with "stack overflow" (exit code 139). Try+  `overflow` in the shell.++Kernel stacks are reused: a freed stack's slot goes on a free list and its+page tables stay, so they never need freeing. **Why no TLB shootdown?** When+a stack is freed, other CPUs may still have its old addresses cached in their+TLB (the CPU's cache of page translations). Normally the freeing CPU would+send them all an IPI to flush it. Here nothing can use a freed slot until a+*new* thread gets it, so each thread instead flushes its stack's pages+(`invlpg`) on each CPU the first time it runs there. Cheaper and simpler,+and only possible because nobody else ever touches a thread's stack.++Known simplifications (good exercises): no priorities; one shared run+queue; no CPU affinity; device interrupts all go to CPU 0.  ## User mode, syscalls and processes @@ -828,7 +1087,8 @@ be adjacent in that order. The TSS has two jobs: IST[0] is the double-fault stack (stage 2), and **`rsp0`** is the stack the CPU switches to when an interrupt arrives while in ring 3. Each thread has its own kernel stack, so the scheduler writes the next thread's stack top into `TSS.rsp0` on every switch-(`gdt::set_kernel_stack`).+(`gdt::set_kernel_stack`). Since stage 11 every CPU has its own GDT and TSS+(the TSS is per CPU by nature: it says where *this* CPU's interrupts land).  ### The syscall path (`src/syscall.rs`) @@ -846,8 +1106,9 @@ One call, step by step:    also save `r10, r8, r9`) and executes `syscall`. 2. CPU: `rcx` = user RIP, `r11` = user RFLAGS, CS/SS = kernel, RFLAGS &= ~SFMASK,    RIP = LSTAR. RSP is still the *user* stack.-3. `syscall_entry` (asm): stash user RSP in a global, load this thread's kernel-   stack top (set by the scheduler), push user RSP, `rcx`, `r11` and the args.+3. `syscall_entry` (asm): `swapgs` to reach this CPU's `PerCpu` data,+   stash user RSP there, load this thread's kernel stack top (set by the+   scheduler), push user RSP, `swapgs` back, push `rcx`, `r11` and the args. 4. Shuffle into the C ABI and call `dispatch(nr, a1..a5)` in Rust with    interrupts re-enabled (so a long syscall, e.g. a blocking `read`, can be preempted). 5. `dispatch` validates user pointers (`user_slice`: range must be inside the@@ -856,8 +1117,9 @@ One call, step by step: 6. `cli`, pop everything back, restore user RSP, `sysretq` (RIP = rcx,    RFLAGS = r11, CPL = 3). -Single CPU, so the "current kernel stack" and "saved user RSP" are plain-globals; SMP kernels use per-CPU data reached with `swapgs`.+With several CPUs, the "current kernel stack" and "saved user RSP" can't be+globals (two CPUs can be in a syscall at once), so they're per-CPU, reached+with `swapgs` (see "Multiple CPUs" below).  Numbers are Linux's wherever Linux has the same call, so man pages and `strace` habits mostly apply. Args 4 and 5 go in `r10` and `r8` like Linux.@@ -878,7 +1140,7 @@ Numbers are Linux's wherever Linux has the same call, so man pages and | 39 | `getpid` | | | | 60 | `exit` | code | | | 61 | `wait` | pid, flags | **simpler than Linux `wait4`**: blocks, returns the exit code (139 = killed, like a shell shows SIGSEGV), reaps the child (freeing its memory); `ECHILD` if not your child; flags `WNOHANG` (1): don't block, `EAGAIN` while it's still running |-| 63 | `uname` | *utsname | Linux's struct: 6 NUL-padded 65-byte fields: "Rustbelt", "qemu", "0.10.0", "stage 10: btrfs read-only", "x86_64", "(none)" |+| 63 | `uname` | *utsname | Linux's struct: 6 NUL-padded 65-byte fields: "Rustbelt", "qemu", "0.11.0", "stage 11: core polish", "x86_64", "(none)" | | 79 | `getcwd` | buf, len | returns the length (no NUL); `ERANGE` if too small | | 80 | `chdir` | path, path_len | per-process cwd, inherited by spawned children | | 83 | `mkdir` | path, path_len | |@@ -888,6 +1150,7 @@ Numbers are Linux's wherever Linux has the same call, so man pages and | 217 | `getdents64` | fd, buf, len | real Linux `struct linux_dirent64` records (ino, off, reclen, type, name\0) | | 500 | `spawn` | path, path_len, argv, argv_len, *SpawnOptions | **ours**: start a program, returns its pid | | 501 | `mounts` | buf, len | **ours** (Linux reads `/proc/mounts`): the mount table as text, one `path fstype` line per mount, in mount order; returns the length, `ERANGE` if too small. The shell assigns drive letters from it |+| 502 | `sysinfo` | *SysInfo | **ours**: CPU count, the CPU you're on, thread/process/frame counts, uptime ticks, and per-CPU timer ticks and busy ticks (sample twice for the load) |  Deviations from Linux worth knowing: @@ -1906,8 +2169,9 @@ script checks again independently, from the host's files with Python's Done: all 12 chapters of the 2nd edition, plus preemptive kernel threads, user mode + syscalls, ELF loading, a VFS with a ramfs, PCI, virtio-blk, MBR/GPT partitions, FAT32, blocking threads, pipes, spawn/wait, process-memory cleanup, the shell, and hand-written ext2, exFAT and (read-only)-btrfs drivers.+memory cleanup, the shell, hand-written ext2, exFAT and (read-only)+btrfs drivers, and multi-core support with a sleep/wake scheduler and+guard pages.  - [x] Freestanding binary — [phil-opp](https://os.phil-opp.com/freestanding-rust-binary/) - [x] Minimal kernel booting in QEMU — [phil-opp](https://os.phil-opp.com/minimal-rust-kernel/), [OSDev: Bare Bones](https://wiki.osdev.org/Bare_Bones)@@ -1937,6 +2201,7 @@ btrfs drivers. - [x] Process cleanup: frame free list, address spaces freed on reap - [x] ext2, hand-written: read + write, indirect blocks, symlinks, 1/4 KiB blocks, revision 0/1, checked with `e2fsck` — [OSDev: Ext2](https://wiki.osdev.org/Ext2), [The Second Extended File System (Dave Poirier)](https://www.nongnu.org/ext2-doc/ext2.html) - [x] exFAT, hand-written: read + write, entry sets + checksums, up-case table, NoFatChain and FAT chains, checked with `fsck.exfat` — [Microsoft exFAT specification](https://learn.microsoft.com/en-us/windows/win32/fileio/exfat-specification)+- [x] SMP (stage 11): ACPI MADT, local APIC + timer, INIT/SIPI trampoline, per-CPU GDT/TSS/`swapgs` data, a scheduler for 4 CPUs with sleep/wake and wait queues, frames freed on every exit, guard pages under kernel and user stacks — [OSDev: SMP](https://wiki.osdev.org/Symmetric_Multiprocessing), [OSDev: APIC](https://wiki.osdev.org/APIC), [OSDev: MADT](https://wiki.osdev.org/MADT), [OSDev: APIC timer](https://wiki.osdev.org/APIC_Timer) - [x] btrfs, hand-written, read-only: chunk tree (single/DUP), root/FS/csum trees, crc32c everywhere, zlib/lzo/zstd, holes, subvolumes, checked against `btrfs check` and the host's files — [btrfs on-disk format](https://btrfs.readthedocs.io/en/latest/dev/On-disk-format.html)  ### Filesystem roadmap (Tyler's plan)@@ -1976,13 +2241,11 @@ whose `root()` returns `Inode`s; mounting it is one `fs::mount` call.  Next, in this order: -1. **Core polish**: SMP (bring up the other CPUs through the APIC, per-CPU-   data, locks that work across cores), a memory cleanup (a real physical-   frame allocator instead of the boot-time walk plus free list, a bigger-   or growable heap), guard pages under kernel and thread stacks, and a-   proper scheduler (priorities, sleeping without polling, per-CPU run-   queues).-2. **Networking**: a virtio-net driver plus [smoltcp](https://github.com/smoltcp-rs/smoltcp)+1. ~~**Core polish**~~ — done (stage 11): SMP on 4 CPUs, a scheduler with+   sleep/wake and wait queues, memory freed on every exit, guard pages.+   Left over for later: priorities, per-CPU run queues, an I/O APIC, a+   bigger or growable heap.+2. **Networking** (next): a virtio-net driver plus [smoltcp](https://github.com/smoltcp-rs/smoltcp)    for TCP/IP. [virtio spec](https://docs.oasis-open.org/virtio/virtio/v1.2/virtio-v1.2.html) 3. **More virtio devices**: virtio-rng (entropy), virtio-console, and    virtio-gpu (a framebuffer instead of VGA text mode).@@ -1990,6 +2253,14 @@ Next, in this order:    [musl](https://musl.libc.org/) port, so ordinary C programs recompiled    against Rustbelt's syscalls (numbered like Linux's already) can run. +**Long term: a graphical desktop.** The goal is a **Hyprland-style tiling+compositor**: windows tile automatically, with keyboard-driven layouts and+smooth animations. It will draw **natively on the virtio-gpu framebuffer**+first, with Rustbelt's own programs as clients, so it needs nothing beyond+step 3. **Wayland client support** comes later, once step 4 can run+ordinary Linux programs: the compositor then speaks the Wayland protocol, so+existing Wayland apps can open windows on it.+ Out of scope for now: **drivers for physical hardware** (real disks, NICs, USB). Rustbelt targets QEMU's virtio devices. A **Raspberry Pi / ARM port** is the planned future hardware target. The split into a separate@@ -2067,8 +2338,27 @@ and for a UEFI/framebuffer setup later, the newer [`bootloader` 0.11](https://gi   The VFS resolves `..` lexically, so a target like `file.txt/..` would   work in Rustbelt but not on Linux; the test image avoids that. - Debug builds use a lot of stack: deep call chains (syscall -> VFS -> fatfs-  -> ELF loader) needed more than 16 KiB. If you see a weird ring-0 fault at-  address 0 or "kernel stack overflow in thread", suspect the stack first.+  -> ELF loader) needed more than 16 KiB. If you see "kernel stack overflow+  in thread", that's the guard page catching it: make the stack bigger+  (`STACK_SIZE` in `src/stack.rs`) or the call chain shallower.+- **SMP pitfalls hit in stage 11:**+  - *`join` returned before the memory was free.* The dead thread was+    removed from the table first and its stack freed a moment later, on+    another CPU, so the leak test saw 7 extra frames now and then. Now the+    stack and process reference are dropped first, then the thread is removed.+  - *Mixed-up output.* With two programs printing on two CPUs, lines came+    out as `count =   CPU900000000`: `println!` sent each formatted piece as+    its own `write`, and the console printed one write piece by piece. Now+    `ulib`'s `println!` formats a whole line into a buffer first, and the+    console holds a lock for the whole of one write.+  - *A lost wake-up through the back door.* If a thread blocks but there's+    nothing else to run, it halts in place; a wake from another CPU then+    marks it Ready *and* queues it while it's still running here. Another+    CPU could then pick it up too. `done_waiting` takes it back off the queue.+  - *A process exiting before its spawner stored its thread id.* On one CPU+    that couldn't happen; on four it can. `reap` waits for the id.+  - Freeing a kernel stack's page tables would need every CPU to flush its+    paging-structure caches; keeping them and reusing the slot avoids it. - **The `\ |` key**: pc-keyboard 0.7's scancode set 1 decodes it (scancode   0x2B) as `KeyCode::Oem7`, but its US 104-key layout only maps `Oem5` to   `\` and `|`. So that key typed nothing, which went unnoticed until the

Binary file changed.

@@ -72,6 +72,7 @@ mod exfat_populate; const PROGRAMS: &[&str] = &[     "sh", "ls", "cat", "wc", "grep", "mkdir", "rm", "touch", "cp", "sleep", // the shell + tools     "hello", "evil", "spin", "demo", "child", // stage 5/6 demos and tests+    "sysinfo", "overflow", // stage 11: CPU load, and a stack overflow (guard pages) ];  const DISK_SECTORS: u64 = 64 * 1024 * 1024 / 512;

@@ -61,6 +61,13 @@ echo hi > new.txt rm hello.txt mkdir newdir run evil+# Stage 11: a stack overflow hits the guard zone and is killed cleanly;+# then two CPU hogs in the background, and sysinfo shows the load per CPU.+run overflow+start spin+start spin+sysinfo+wait nosuchprogram cd Q: cd A:/tmp

@@ -37,7 +37,7 @@ for img in disk ext2 exfat btrfs ext2-4k ext2-rev0 exfat-512 btrfs-lzo btrfs-zst     drives+=(-drive "file=target/$img.img,format=raw,if=virtio,snapshot=on") done set +e-timeout 120 qemu-system-x86_64 -drive format=raw,file="$KERNEL" -no-reboot "${drives[@]}" \+timeout 120 qemu-system-x86_64 -smp 4 -drive format=raw,file="$KERNEL" -no-reboot "${drives[@]}" \     -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none > $LOG code=$? set -e

@@ -46,7 +46,7 @@ echo "== run the tests/exfat.rs kernel on target/fsck/tests-exfat*.img" KERNEL=$(ls -t target/x86_64-rust_kernel/debug/build/rustbelt/*/out/bootimage-exfat-*.bin 2>/dev/null | head -1 || true) [ -n "$KERNEL" ] || { echo "no test kernel: run 'cargo test --test exfat' first"; exit 1; } set +e-timeout 120 qemu-system-x86_64 -drive format=raw,file="$KERNEL" -no-reboot \+timeout 120 qemu-system-x86_64 -smp 4 -drive format=raw,file="$KERNEL" -no-reboot \     -drive file=target/disk.img,format=raw,if=virtio,snapshot=on \     -drive file=target/ext2.img,format=raw,if=virtio,snapshot=on \     -drive file=target/fsck/tests-exfat.img,format=raw,if=virtio \

@@ -42,7 +42,7 @@ echo "== run the tests/ext2.rs kernel on target/fsck/tests-*.img" KERNEL=$(ls -t target/x86_64-rust_kernel/debug/build/rustbelt/*/out/bootimage-ext2-*.bin 2>/dev/null | head -1 || true) [ -n "$KERNEL" ] || { echo "no test kernel: run 'cargo test --test ext2' first"; exit 1; } set +e-timeout 120 qemu-system-x86_64 -drive format=raw,file="$KERNEL" -no-reboot \+timeout 120 qemu-system-x86_64 -smp 4 -drive format=raw,file="$KERNEL" -no-reboot \     -drive file=target/disk.img,format=raw,if=virtio,snapshot=on \     -drive file=target/fsck/tests-ext2.img,format=raw,if=virtio \     -drive file=target/exfat.img,format=raw,if=virtio,snapshot=on \

@@ -43,7 +43,7 @@ BTRFS_DRIVE="file=${BTRFS_IMG:-target/btrfs.img},format=raw,if=virtio,snapshot=o  LOG=target/headless-serial.log set +e-timeout 90 qemu-system-x86_64 \+timeout 90 qemu-system-x86_64 -smp 4 \     -drive format=raw,file="$IMG" \     -drive file=target/disk.img,format=raw,if=virtio,snapshot=on \     -drive "$EXT2_DRIVE" \@@ -57,7 +57,8 @@ set -e # Lines the shell self-test must have printed (only for a plain boot). if [ -z "${1:-}" ] && [ "$code" = 33 ]; then     expect=(-        "Rustbelt 0.10.0 (stage 10: btrfs read-only) x86_64" # ver+        "Rustbelt 0.11.0 (stage 11: core polish) x86_64" # ver+        "[smp] 4 CPUs online"                          # smp.rs started 3 more cores         "B:  /mnt/disk0   fat32"                       # drives         "C:  /mnt/disk1   ext2"         "D:  /mnt/disk2   exfat"@@ -95,6 +96,9 @@ if [ -z "${1:-}" ] && [ "$code" = 33 ]; then         "rm: hello.txt: Read-only file system"         "mkdir: newdir: Read-only file system"         "[exit code 139]"                              # run evil+        "(overflow) killed: stack overflow"            # user stack guard zone+        "4 CPUs online (this program ran on CPU"       # sysinfo+        "  CPU3 ["         "sh: nosuchprogram: command not found"         "cd: there is no drive Q:"         "A:/tmp $ pwd"@@ -103,12 +107,19 @@ if [ -z "${1:-}" ] && [ "$code" = 33 ]; then     for line in "${expect[@]}"; do         grep -qF -- "$line" "$LOG" || { echo "[run-headless] MISSING: $line"; missing=1; }     done-    # Pipe results: `... | grep -i EXT | wc -l` -> 1, `dir bin | wc -l` -> 15,+    # Pipe results: `... | grep -i EXT | wc -l` -> 1, `dir bin | wc -l` -> 17,     # `cat C:/chunks.txt | wc -l` (copied from E:) -> 3, and     # `type D:/COW.TXT | wc -l` (copied from E:) -> 4.-    for n in 1 15 3 4; do+    for n in 1 17 3 4; do         grep -qx -- "$n" "$LOG" || { echo "[run-headless] MISSING pipe result: $n"; missing=1; }     done+    # No memory leaks: once the shell and all ~60 programs it ran have+    # exited, exactly as many frames must be in use as before it started.+    if grep -qE '^\[memory\] ([0-9]+) frames in use before the shell, \1 after' "$LOG"; then+        echo "[run-headless] frames in use returned to the baseline"+    else+        echo "[run-headless] MISSING: frames back at the baseline after the shell"; missing=1+    fi     [ $missing = 0 ] && echo "[run-headless] all ${#expect[@]}+4 expected shell lines found" || code=35 fi 

@@ -21,25 +21,25 @@ OUT="${1:-$PWD/boot.png}" SESSION="${2:-}" shift $(( $# > 2 ? 2 : $# )) if [ -z "$SESSION" ]; then-    # The default session (what boot.png shows). "lx{BS}s" types a typo-    # and fixes it.+    # The default session (what boot.png shows): two CPU hogs in the+    # background (output to files), sysinfo showing them on two of the four+    # CPUs, and a stack overflow caught by the guard zone. "vx{BS}er" types+    # a typo and fixes it.     SESSION=$(mktemp /tmp/session.XXXXXX)     cat > "$SESSION" <<'TXT'-cd e:-lx{BS}s -l-cat hello-link-cat docs/btrfs/chunks.txt-cp docs/btrfs/cow.txt D:/cow.txt-echo hi > new.txt+vx{BS}er+spin 150 > A:/tmp/spin1.txt &+spin 150 > A:/tmp/spin2.txt &+sysinfo+overflow TXT-    set -- "E:/ $ ls -l" "-rw-r--r--    2160000  big.txt" "hello-link -> hello.txt" \-        "Hello from btrfs! (an inline extent" "E:/ $ cat docs/btrfs/chunks.txt" \-        "sh: new.txt: Read-only file system"+    set -- "Rustbelt 0.11.0 (stage 11: core polish) x86_64" "4 CPUs online (this program ran on CPU" \+        "CPU load over the last quarter second:" "killed: stack overflow" fi SOCK=$(mktemp -u /tmp/qemu-mon.XXXXXX) SERIAL_LOG=$(mktemp /tmp/serial.XXXXXX) -qemu-system-x86_64 -drive format=raw,file="$IMG" \+qemu-system-x86_64 -smp 4 -drive format=raw,file="$IMG" \     -drive file=target/disk.img,format=raw,if=virtio,snapshot=on \     -drive file=target/ext2.img,format=raw,if=virtio,snapshot=on \     -drive file=target/exfat.img,format=raw,if=virtio,snapshot=on \

@@ -0,0 +1,79 @@+//! Just enough ACPI to find the CPUs.+//!+//! The firmware describes the machine in ACPI tables. We follow the chain:+//!+//!   RSDP ("RSD PTR ", found by scanning the BIOS area)+//!     -> RSDT (Root System Description Table: a list of table addresses)+//!       -> MADT (signature "APIC": one entry per interrupt controller)+//!+//! Each MADT entry of type 0 is a CPU's local APIC; we collect the APIC IDs+//! of the enabled ones. That's the list of CPUs smp.rs will start.++use crate::memory::phys_to_virt;+use alloc::vec::Vec;+use x86_64::PhysAddr;++/// Read a value of type T at physical address `addr`.+fn read<T: Copy>(addr: u64) -> T {+    unsafe { core::ptr::read_unaligned(phys_to_virt(PhysAddr::new(addr)).as_ptr::<T>()) }+}++fn bytes(addr: u64, len: usize) -> &'static [u8] {+    unsafe { core::slice::from_raw_parts(phys_to_virt(PhysAddr::new(addr)).as_ptr::<u8>(), len) }+}++/// ACPI tables are valid if all their bytes add up to 0 (mod 256).+fn checksum_ok(addr: u64, len: usize) -> bool {+    bytes(addr, len).iter().fold(0u8, |sum, b| sum.wrapping_add(*b)) == 0+}++/// Find the RSDP: in the first KiB of the EBDA (Extended BIOS Data Area)+/// or in the BIOS ROM area 0xE0000-0xFFFFF, on a 16-byte boundary.+fn find_rsdp() -> Option<u64> {+    let ebda = (read::<u16>(0x40E) as u64) << 4;+    let areas = [(ebda, ebda + 1024), (0xE0000, 0x100000)];+    for (start, end) in areas {+        if start == 0 {+            continue;+        }+        for addr in (start..end).step_by(16) {+            if bytes(addr, 8) == b"RSD PTR " && checksum_ok(addr, 20) {+                return Some(addr);+            }+        }+    }+    None+}++/// The APIC IDs of all enabled CPUs, or None if there's no usable MADT.+pub fn cpu_apic_ids() -> Option<Vec<u32>> {+    let rsdp = find_rsdp()?;+    let rsdt = read::<u32>(rsdp + 16) as u64; // works for ACPI 1.0 and 2.0++    let rsdt_len = read::<u32>(rsdt + 4) as u64;+    // Table header is 36 bytes, then 4-byte physical addresses.+    for i in 0..(rsdt_len - 36) / 4 {+        let table = read::<u32>(rsdt + 36 + i * 4) as u64;+        if bytes(table, 4) != b"APIC" {+            continue;+        }+        let len = read::<u32>(table + 4) as u64;+        let mut ids = Vec::new();+        // MADT: header (36) + local APIC address (4) + flags (4), then+        // variable-length entries: [type, length, ...].+        let mut entry = table + 44;+        while entry + 2 <= table + len {+            let (kind, entry_len) = (read::<u8>(entry), read::<u8>(entry + 1) as u64);+            if entry_len < 2 {+                break; // broken table: stop rather than loop forever+            }+            // Type 0 = processor local APIC: [0, 8, acpi id, apic id, flags u32]+            // flags bit 0 = enabled, bit 1 = "can be enabled later".+            if kind == 0 && read::<u32>(entry + 4) & 1 != 0 {+                ids.push(read::<u8>(entry + 3) as u32);+            }+            entry += entry_len;+        }+        return Some(ids);+    }+    None+}

@@ -0,0 +1,166 @@+//! The local APIC: every CPU's own interrupt controller.+//!+//! Each x86 CPU core has a "local APIC" (Advanced Programmable Interrupt+//! Controller). We use three of its features:+//!+//! * **Inter-processor interrupts (IPIs).** One CPU can interrupt another+//!   by writing the target's APIC ID and a vector into its ICR register.+//!   That's how CPU 0 wakes the others up at boot (INIT + "startup" IPIs,+//!   see smp.rs) and how a CPU pokes an idle one when new work arrives.+//! * **The APIC timer.** A per-CPU countdown timer. We program it to fire+//!   every ~10 ms on *every* CPU, which drives preemption (time slices).+//!   Its speed isn't known in advance, so we measure it against the PIT.+//! * **Its ID**, which tells a CPU which CPU it is (see `smp::cpu_id`).+//!+//! The registers are memory-mapped at physical 0xFEE0_0000 (each CPU sees+//! its *own* APIC at that same address). We reach them through the+//! physical-memory map. (On real hardware that range is uncached thanks to+//! the firmware's MTRR settings; QEMU doesn't care either way.)+//!+//! The old 8259 PIC stays in charge of device interrupts (PIT timer,+//! keyboard) and delivers them to CPU 0 only, through CPU 0's LINT0 pin+//! ("virtual wire mode"). Routing device interrupts to any CPU would need+//! the I/O APIC; we don't need that yet.++use crate::interrupts::{self, InterruptIndex};+use crate::memory;+use core::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};+use x86_64::PhysAddr;++// Register offsets (bytes from the APIC base).+const ID: usize = 0x20;+const TPR: usize = 0x80; // task priority: 0 = accept everything+const EOI: usize = 0xB0;+const SVR: usize = 0xF0; // spurious vector + "APIC enabled" bit 8+const ICR_LOW: usize = 0x300;+const ICR_HIGH: usize = 0x310;+const LVT_TIMER: usize = 0x320;+const LVT_LINT0: usize = 0x350;+const LVT_LINT1: usize = 0x360;+const TIMER_INITIAL: usize = 0x380;+const TIMER_CURRENT: usize = 0x390;+const TIMER_DIVIDE: usize = 0x3E0;++const MASKED: u32 = 1 << 16;+const TIMER_PERIODIC: u32 = 1 << 17;+const DIVIDE_BY_16: u32 = 0b0011;++/// Virtual address of the APIC registers (0 = not set up yet).+static BASE: AtomicU64 = AtomicU64::new(0);+/// Timer count for one 10 ms slice (measured once, on CPU 0).+static TIMER_COUNT: AtomicU32 = AtomicU32::new(0);+/// Is CPU 0's APIC timer running? (Until then the PIT drives preemption.)+static TIMER_ON: AtomicBool = AtomicBool::new(false);++fn read(reg: usize) -> u32 {+    unsafe { core::ptr::read_volatile((BASE.load(Ordering::Relaxed) as usize + reg) as *const u32) }+}++fn write(reg: usize, value: u32) {+    unsafe { core::ptr::write_volatile((BASE.load(Ordering::Relaxed) as usize + reg) as *mut u32, value) }+}++/// Is the APIC usable yet?+pub fn is_ready() -> bool {+    BASE.load(Ordering::Relaxed) != 0+}++/// This CPU's APIC ID (0 before `init_bsp`).+pub fn id() -> u32 {+    if is_ready() { read(ID) >> 24 } else { 0 }+}++/// "End of interrupt": tell the APIC we handled its interrupt (needed for+/// the APIC timer and IPIs; PIC interrupts get their EOI at the PIC).+pub fn eoi() {+    write(EOI, 0);+}++/// Is the APIC timer driving preemption?+pub fn timer_running() -> bool {+    TIMER_ON.load(Ordering::Relaxed)+}++/// CPU 0: find and enable the APIC, measure its timer, start it.+/// Needs interrupts on (it waits for PIT ticks).+pub fn init_bsp() {+    use x86_64::registers::model_specific::Msr;+    // IA32_APIC_BASE MSR: physical base address in bits 12 and up.+    let phys = unsafe { Msr::new(0x1B).read() } & 0xF_FFFF_F000;+    BASE.store(memory::phys_to_virt(PhysAddr::new(phys)).as_u64(), Ordering::Relaxed);+    enable();+    // Keep the PIC's interrupts flowing into this CPU: LINT0 = ExtINT+    // ("ask the PIC for the vector"), LINT1 = NMI.+    write(LVT_LINT0, 0x700);+    write(LVT_LINT1, 0x400);+    calibrate();+    start_timer();+    TIMER_ON.store(true, Ordering::Relaxed);+}++/// Another CPU: enable its APIC and start its timer. Its LINT0 stays+/// masked (the PIC talks to CPU 0 only).+pub fn init_ap() {+    enable();+    write(LVT_LINT0, MASKED);+    write(LVT_LINT1, MASKED);+    start_timer();+}++fn enable() {+    write(SVR, 0x100 | InterruptIndex::ApicSpurious as u32);+    write(TPR, 0);+}++/// Measure how far the APIC timer counts in 2 PIT ticks (~110 ms), and+/// work out the count for 10 ms.+fn calibrate() {+    write(TIMER_DIVIDE, DIVIDE_BY_16);+    write(LVT_TIMER, MASKED); // count down, but don't interrupt+    let start = interrupts::ticks();+    while interrupts::ticks() == start {+        x86_64::instructions::hlt(); // line up with a tick edge+    }+    write(TIMER_INITIAL, u32::MAX);+    let start = interrupts::ticks();+    while interrupts::ticks() < start + 2 {+        x86_64::instructions::hlt();+    }+    let per_tick = (u32::MAX - read(TIMER_CURRENT)) / 2;+    write(TIMER_INITIAL, 0);+    // One PIT tick is ~54.9 ms; we want 10 ms.+    let count = (per_tick as u64 * 10 / 55).max(1000) as u32;+    TIMER_COUNT.store(count, Ordering::Relaxed);+}++fn start_timer() {+    write(TIMER_DIVIDE, DIVIDE_BY_16);+    write(LVT_TIMER, InterruptIndex::ApicTimer as u32 | TIMER_PERIODIC);+    write(TIMER_INITIAL, TIMER_COUNT.load(Ordering::Relaxed));+}++/// Write the interrupt command register: send `command` to APIC `target`,+/// then wait until the APIC has delivered it.+fn send(target: u32, command: u32) {+    write(ICR_HIGH, target << 24);+    write(ICR_LOW, command); // writing the low half sends it+    while read(ICR_LOW) & (1 << 12) != 0 {+        core::hint::spin_loop(); // "delivery pending"+    }+}++/// Interrupt CPU `target` with `vector` (a normal, "fixed" interrupt).+pub fn send_ipi(target: u32, vector: u8) {+    send(target, 0x4000 | vector as u32);+}++/// INIT IPI: reset CPU `target` into its "wait for startup IPI" state.+pub fn send_init(target: u32) {+    send(target, 0x4500);+}++/// Startup IPI: CPU `target` starts in 16-bit real mode at physical+/// address `page * 4096`.+pub fn send_startup(target: u32, page: u8) {+    send(target, 0x4600 | page as u32);+}

@@ -117,7 +117,8 @@ pub fn load(data: &[u8], l4: PhysFrame) -> Result<u64, ElfError> {         if seg.filesz > seg.memsz || file_end > data.len() as u64 {             return Err(ElfError::BadProgramHeader);         }-        if seg.vaddr < USER_START || mem_end > USER_END {+        // Segments must stay below the user stack and its guard zone.+        if seg.vaddr < USER_START || mem_end > crate::process::USER_STACK_ZONE_START {             return Err(ElfError::OutsideUserSpace);         }         if seg.memsz == 0 {

@@ -22,8 +22,19 @@ //! //!   0x00 null | 0x08 kernel code | 0x10 kernel data | 0x18 user data | //!   0x20 user code | 0x28 TSS (16 bytes, two slots)+//!+//! **One GDT and TSS per CPU.** Each CPU runs a different thread, so each+//! needs its own RSP0 and its own double-fault stack, i.e. its own TSS. (A+//! TSS descriptor is also marked "busy" when loaded, so two CPUs can't share+//! one GDT entry either.) CPU 0 uses the static tables below, which work+//! before the heap exists; the other CPUs get heap-allocated copies from+//! `init_ap`. The selectors are the same on every CPU. -use core::ptr::{addr_of, addr_of_mut};+use crate::smp::MAX_CPUS;+use crate::stack::KernelStack;+use alloc::boxed::Box;+use core::ptr::{addr_of, addr_of_mut, null_mut};+use core::sync::atomic::{AtomicPtr, Ordering}; use lazy_static::lazy_static; use x86_64::VirtAddr; use x86_64::structures::gdt::{Descriptor, GlobalDescriptorTable, SegmentSelector};@@ -32,17 +43,21 @@ use x86_64::structures::tss::TaskStateSegment; /// Which IST slot (0..=6) the double-fault handler uses. pub const DOUBLE_FAULT_IST_INDEX: u16 = 0; -/// The TSS. A `static mut` because the scheduler rewrites RSP0 on every+/// CPU 0's TSS. A `static mut` because the scheduler rewrites RSP0 on every /// context switch (the CPU re-reads it from memory on each ring 3 -> ring 0-/// transition, so writing the field is enough). Only touched with interrupts-/// disabled, on a single CPU.+/// transition, so writing the field is enough). Only touched with+/// interrupts disabled, by CPU 0 itself. static mut TSS: TaskStateSegment = TaskStateSegment::new(); -/// Stack for the double-fault handler (static: no heap needed. It has no-/// guard page, so don't do anything recursive in the double-fault handler!)-const DOUBLE_FAULT_STACK_SIZE: usize = 4096 * 5;+/// CPU 0's double-fault stack (static: no heap needed this early). 64 KiB,+/// because after a kernel stack overflow the double-fault handler runs the+/// code that kills the thread on it.+const DOUBLE_FAULT_STACK_SIZE: usize = 64 * 1024; static mut DOUBLE_FAULT_STACK: [u8; DOUBLE_FAULT_STACK_SIZE] = [0; DOUBLE_FAULT_STACK_SIZE]; +/// Each CPU's TSS (CPU 0: the static one, others: leaked boxes).+static TSS_OF: [AtomicPtr<TaskStateSegment>; MAX_CPUS] = [const { AtomicPtr::new(null_mut()) }; MAX_CPUS];+ /// Selectors (GDT offsets + requested privilege level) of our segments. pub struct Selectors {     pub kernel_code: SegmentSelector,@@ -52,54 +67,77 @@ pub struct Selectors {     tss: SegmentSelector, } +/// A GDT in our standard layout whose TSS entry points at `tss`.+fn build_gdt(tss: &'static TaskStateSegment) -> (GlobalDescriptorTable, Selectors) {+    let mut gdt = GlobalDescriptorTable::new();+    // Order matters for syscall/sysret, see the module comment.+    let kernel_code = gdt.append(Descriptor::kernel_code_segment());+    let kernel_data = gdt.append(Descriptor::kernel_data_segment());+    let user_data = gdt.append(Descriptor::user_data_segment());+    let user_code = gdt.append(Descriptor::user_code_segment());+    let tss = gdt.append(Descriptor::tss_segment(tss));+    (gdt, Selectors { kernel_code, kernel_data, user_data, user_code, tss })+}+ lazy_static! {-    static ref GDT: (GlobalDescriptorTable, Selectors) = {-        let mut gdt = GlobalDescriptorTable::new();-        // Order matters for syscall/sysret, see the module comment.-        let kernel_code = gdt.append(Descriptor::kernel_code_segment());-        let kernel_data = gdt.append(Descriptor::kernel_data_segment());-        let user_data = gdt.append(Descriptor::user_data_segment());-        let user_code = gdt.append(Descriptor::user_code_segment());-        // Safety: TSS is a static, so the reference lives forever.-        let tss = gdt.append(Descriptor::tss_segment(unsafe { &*addr_of!(TSS) }));-        (gdt, Selectors { kernel_code, kernel_data, user_data, user_code, tss })-    };+    // Safety: TSS is a static, so the reference lives forever.+    static ref GDT: (GlobalDescriptorTable, Selectors) = build_gdt(unsafe { &*addr_of!(TSS) }); }  /// The segment selectors, e.g. to build an `iretq` frame into ring 3.+/// (Identical on every CPU.) pub fn selectors() -> &'static Selectors {     &GDT.1 } -/// Load our GDT, reload the segment registers and load the TSS.-pub fn init() {+/// Load `gdt`, reload the segment registers and load the TSS.+fn load(gdt: &'static (GlobalDescriptorTable, Selectors)) {     use x86_64::instructions::segmentation::{CS, DS, ES, SS, Segment};     use x86_64::instructions::tables::load_tss;+    gdt.0.load();+    let s = &gdt.1;+    unsafe {+        // The old selectors point into the previous GDT (the bootloader's, or+        // the AP trampoline's), so reload them. (`iretq` checks SS.)+        CS::set_reg(s.kernel_code);+        SS::set_reg(s.kernel_data);+        DS::set_reg(s.kernel_data);+        ES::set_reg(s.kernel_data);+        load_tss(s.tss);+    }+} +/// CPU 0: load the static GDT and TSS (works without a heap).+pub fn init() {     unsafe {         // Stacks grow downwards on x86, so hand the CPU the *end* address.         let start = VirtAddr::from_ptr(addr_of!(DOUBLE_FAULT_STACK));         (*addr_of_mut!(TSS)).interrupt_stack_table[DOUBLE_FAULT_IST_INDEX as usize] =             start + DOUBLE_FAULT_STACK_SIZE as u64;     }+    TSS_OF[0].store(addr_of_mut!(TSS), Ordering::Release);+    load(&GDT);+} -    GDT.0.load();-    let s = &GDT.1;-    unsafe {-        // The old selectors point into the bootloader's GDT, which we just-        // replaced, so reload them. (`iretq` checks SS, so it must be valid.)-        CS::set_reg(s.kernel_code);-        SS::set_reg(s.kernel_data);-        DS::set_reg(s.kernel_data);-        ES::set_reg(s.kernel_data);-        load_tss(s.tss);-    }+/// Another CPU (`cpu` > 0): build and load its own GDT and TSS, with a+/// guard-paged double-fault stack. These live forever (CPUs never go away).+pub fn init_ap(cpu: usize) {+    let df_stack = KernelStack::new().expect("no memory for a double-fault stack");+    let mut tss = TaskStateSegment::new();+    tss.interrupt_stack_table[DOUBLE_FAULT_IST_INDEX as usize] = VirtAddr::new(df_stack.top());+    core::mem::forget(df_stack); // never freed+    let tss: *mut TaskStateSegment = Box::into_raw(Box::new(tss));+    let gdt = Box::leak(Box::new(build_gdt(unsafe { &*tss })));+    TSS_OF[cpu].store(tss, Ordering::Release);+    load(gdt); } -/// Set RSP0: the kernel stack the CPU switches to when an interrupt arrives-/// while user code runs. Called by the scheduler on every context switch.+/// Set RSP0 of the current CPU: the kernel stack it switches to when an+/// interrupt arrives while user code runs. Called by the scheduler on+/// every context switch (with interrupts off). pub fn set_kernel_stack(stack_top: u64) {-    unsafe {-        (*addr_of_mut!(TSS)).privilege_stack_table[0] = VirtAddr::new(stack_top);+    let tss = TSS_OF[crate::smp::cpu_id()].load(Ordering::Acquire);+    if !tss.is_null() {+        unsafe { (*tss).privilege_stack_table[0] = VirtAddr::new(stack_top) };     } }

@@ -4,6 +4,11 @@ //! a handler function: //!   * vectors 0..=31  are CPU exceptions (divide error, page fault, ...) //!   * vectors 32..=47 are where we remap the 8259 PIC's 16 hardware IRQs+//!   * 0x40, 0x41, 0xFF come from the local APIC (see apic.rs): its timer,+//!     the "reschedule" IPI one CPU sends another, and spurious interrupts+//!+//! All CPUs share this one IDT. Device IRQs (PIT, keyboard) only ever+//! arrive on CPU 0; APIC timer ticks and IPIs arrive on every CPU. //! //! Handlers use the special `extern "x86-interrupt"` calling convention: the //! compiler saves/restores all registers and returns with `iretq`.@@ -33,6 +38,9 @@ pub static PICS: spin::Mutex<ChainedPics> = pub enum InterruptIndex {     Timer = PIC_1_OFFSET, // IRQ 0: Programmable Interval Timer, ~18.2 Hz by default     Keyboard,             // IRQ 1: PS/2 keyboard+    ApicTimer = 0x40,     // each CPU's local APIC timer: time slices (~100 Hz)+    Reschedule = 0x41,    // IPI: "wake up, there's work for you"+    ApicSpurious = 0xFF,  // the APIC's "never mind" interrupt }  impl InterruptIndex {@@ -61,11 +69,14 @@ lazy_static! {         }         idt[InterruptIndex::Timer.as_u8()].set_handler_fn(timer_interrupt_handler);         idt[InterruptIndex::Keyboard.as_u8()].set_handler_fn(keyboard_interrupt_handler);+        idt[InterruptIndex::ApicTimer.as_u8()].set_handler_fn(apic_timer_handler);+        idt[InterruptIndex::Reschedule.as_u8()].set_handler_fn(reschedule_handler);+        idt[InterruptIndex::ApicSpurious.as_u8()].set_handler_fn(spurious_handler);         idt     }; } -/// Load the IDT into the CPU (`lidt`).+/// Load the IDT into the CPU (`lidt`). Every CPU does this once. pub fn init_idt() {     IDT.load(); }@@ -121,10 +132,19 @@ extern "x86-interrupt" fn page_fault_handler(     error_code: PageFaultErrorCode, ) {     use x86_64::registers::control::Cr2;+    let addr = Cr2::read_raw();     if error_code.contains(PageFaultErrorCode::USER_MODE) {-        serial_println!("[kernel] user page fault: address {:?}, error {:?}", Cr2::read(), error_code);+        serial_println!("[kernel] user page fault: address {:#x}, error {:?}", addr, error_code);+        // Just below the user stack is a guard zone that's never mapped.+        if crate::process::in_user_stack_guard(addr) {+            kill_if_user(&stack_frame, "stack overflow");+        }         kill_if_user(&stack_frame, "page fault");     }+    // The kernel touched a kernel stack's guard page: that thread overflowed.+    if crate::stack::is_guard_page(addr) {+        crate::thread::die_of_stack_overflow(addr);+    }     println!("EXCEPTION: PAGE FAULT at {:?}", Cr2::read());     serial_println!(         "EXCEPTION: PAGE FAULT\nAccessed address: {:?}\nError code: {:?}\n{:#?}",@@ -149,6 +169,14 @@ extern "x86-interrupt" fn double_fault_handler(     stack_frame: InterruptStackFrame,     _error_code: u64, // always 0 for double faults ) -> ! {+    // The usual way a kernel stack overflow shows up: the thread's stack+    // pointer ran into the guard page, the page fault couldn't push its+    // frame there, and that second fault became a double fault. CR2 still+    // holds the guard page address. Kill just that thread.+    let addr = x86_64::registers::control::Cr2::read_raw();+    if crate::stack::is_guard_page(addr) {+        crate::thread::die_of_stack_overflow(addr);+    }     serial_println!("EXCEPTION: DOUBLE FAULT\n{:#?}", stack_frame);      // In the deliberate stack-overflow test, reaching here means success.@@ -168,7 +196,13 @@ extern "x86-interrupt" fn double_fault_handler( /// Timer ticks since boot. Atomic, so it's safe to touch from an interrupt. pub static TICKS: AtomicU64 = AtomicU64::new(0); -/// IRQ 0. Fires ~18.2 times per second (the PIT's power-on default).+/// PIT ticks since boot (~18.2 per second, ~55 ms each).+pub fn ticks() -> u64 {+    TICKS.load(Ordering::Relaxed)+}++/// IRQ 0 (CPU 0 only). Fires ~18.2 times per second (the PIT's power-on+/// default). It's the system clock: uptime, and waking sleeping threads. extern "x86-interrupt" fn timer_interrupt_handler(_stack_frame: InterruptStackFrame) {     let ticks = TICKS.fetch_add(1, Ordering::Relaxed) + 1; @@ -191,10 +225,31 @@ extern "x86-interrupt" fn timer_interrupt_handler(_stack_frame: InterruptStackFr         PICS.lock().notify_end_of_interrupt(InterruptIndex::Timer.as_u8());     } -    // Preemption: maybe switch to another kernel thread (see thread.rs).+    // Wake threads whose sleep is over (they're queued for any CPU).+    crate::thread::wake_sleepers(ticks);++    // Preemption is the APIC timer's job; before it runs, the PIT does it.+    if !crate::apic::timer_running() {+        crate::thread::schedule();+    }+}++/// Every CPU's APIC timer, ~100 times a second: time's up for the running+/// thread, let the next one have a go (see thread.rs).+extern "x86-interrupt" fn apic_timer_handler(_stack_frame: InterruptStackFrame) {+    crate::apic::eoi(); // before switching, as above+    crate::thread::timer_tick();+}++/// Another CPU queued work and saw we were idle.+extern "x86-interrupt" fn reschedule_handler(_stack_frame: InterruptStackFrame) {+    crate::apic::eoi();     crate::thread::schedule(); } +/// Spurious APIC interrupt: ignore it (and no EOI, by the spec).+extern "x86-interrupt" fn spurious_handler(_stack_frame: InterruptStackFrame) {}+ /// IRQ 1. The keyboard controller has a scancode waiting at I/O port 0x60. /// We must read it (or no more keyboard interrupts arrive), then just queue /// it for the async keyboard task (task/keyboard.rs) to decode and print.

@@ -21,7 +21,9 @@ extern crate alloc;  use core::panic::PanicInfo; +pub mod acpi; // ACPI tables: just enough to list the CPUs (MADT) pub mod allocator; // heap: maps heap pages + the #[global_allocator]+pub mod apic; // local APIC: per-CPU timer, inter-processor interrupts pub mod block; // block devices: BlockDevice trait, virtio-blk, partitions pub mod elf; // ELF64 loader for user programs pub mod errno; // Linux-compatible error numbers@@ -32,9 +34,11 @@ pub mod memory; // paging, frame allocator, per-process address spaces pub mod pci; // PCI bus enumeration (config space via ports 0xCF8/0xCFC) pub mod process; // processes: address space + thread + fd table, ring 3 entry pub mod serial; // COM1 serial port+pub mod smp; // multi-core: start the other CPUs, per-CPU data+pub mod stack; // kernel stacks with guard pages pub mod syscall; // syscall/sysret entry + system call implementations pub mod task; // cooperative multitasking: async tasks + executor-pub mod thread; // preemptive kernel threads + context switch+pub mod thread; // threads + the multi-core scheduler (sleep/wake, wait queues) pub mod tty; // console: keyboard line discipline + fd 0/1/2 pub mod vga_buffer; // 80x25 VGA text mode screen @@ -59,6 +63,13 @@ pub fn init_memory(boot_info: &'static bootloader::BootInfo) {     memory::init_frame_allocator(frame_allocator); } +/// Multi-core bring-up: the local APIC and its timer (preemption), CPU 0's+/// idle thread, and every other CPU the ACPI tables list (see smp.rs).+/// Needs `init` and `init_memory`. Returns the number of running CPUs.+pub fn init_smp() -> usize {+    smp::init()+}+ /// Storage bring-up: scan the PCI bus, start the disk drivers, read the /// partition tables and mount FAT volumes under /mnt. Needs the heap, the /// frame allocator and `fs::init()`. Returns (PCI devices, log lines).@@ -147,6 +158,7 @@ bootloader::entry_point!(test_kernel_main); fn test_kernel_main(boot_info: &'static bootloader::BootInfo) -> ! {     init();     init_memory(boot_info); // a heap, for unit tests that use Vec (btrfs, lzo)+    init_smp();     test_main();     hlt_loop(); }

@@ -60,6 +60,11 @@ fn kernel_main(boot_info: &'static BootInfo) -> ! {     #[cfg(not(feature = "demos"))]     rustbelt::init_memory(boot_info); +    // --- More CPUs (smp.rs, apic.rs) -----------------------------------------+    // Start the other CPU cores; from now on threads run on all of them.+    let cpus = rustbelt::init_smp();+    kprintln!("[smp] {} CPU{} online", cpus, if cpus == 1 { "" } else { "s" });+     // --- Preemptive threads demo (thread.rs) --------------------------------     // From now on the timer interrupt switches between threads every tick.     #[cfg(feature = "demos")]@@ -119,6 +124,8 @@ fn shell_supervisor() {         return;     }     loop {+        #[cfg(feature = "headless")]+        let frames_before = rustbelt::memory::frames_in_use();         #[allow(unused_mut)]         let mut args = process::SpawnArgs::kernel("sh");         #[cfg(feature = "headless")]@@ -136,7 +143,13 @@ fn shell_supervisor() {         {             let ok = status == process::ExitStatus::Exited(0);             serial_println!("[{}] shell self-test finished: {:?}", if ok { "ok" } else { "failed" }, status);-            serial_println!("[memory] {} frames in use", rustbelt::memory::frames_in_use());+            // Every process the self-test ran is gone: all their memory+            // (page tables, user pages, kernel stacks) must be back.+            serial_println!(+                "[memory] {} frames in use before the shell, {} after it and all its children exited",+                frames_before,+                rustbelt::memory::frames_in_use()+            );             rustbelt::exit_qemu(if ok { rustbelt::QemuExitCode::Success } else { rustbelt::QemuExitCode::Failed });         }         kprintln!("[kernel] shell exited ({:?}); starting a new one", status);

@@ -20,6 +20,7 @@ use bootloader::bootinfo::{MemoryMap, MemoryRegionType}; use core::sync::atomic::{AtomicU64, Ordering}; use spin::Mutex;+use x86_64::instructions::interrupts::without_interrupts; use x86_64::{     PhysAddr, VirtAddr,     registers::control::Cr3,@@ -132,7 +133,8 @@ pub fn usable_memory_bytes(memory_map: &MemoryMap) -> u64 { /// Hand the frame allocator over to the kernel (after the heap is set up), /// so processes can get page tables and pages later. pub fn init_frame_allocator(frame_allocator: BootInfoFrameAllocator) {-    *FRAME_ALLOCATOR.lock() = Some(frame_allocator);+    without_interrupts(|| *FRAME_ALLOCATOR.lock() = Some(frame_allocator));+    init_kernel_stack_area(); // must happen before any process exists }  /// Frames given back with `free_frame`, as a linked list threaded through@@ -148,22 +150,27 @@ static BOOT_FRAMES_USED: AtomicU64 = AtomicU64::new(0);  /// One physical frame (contents undefined), or None if out of memory / not /// initialized. Recycled frames first, then fresh ones from the memory map.+///+/// The frame locks are only ever taken with interrupts off (see the locking+/// rule in thread.rs): the scheduler and interrupt handlers free frames too. pub fn alloc_frame() -> Option<PhysFrame> {-    {-        let mut free = FREE_FRAMES.lock();-        if free.head != 0 {-            let frame = PhysFrame::containing_address(PhysAddr::new(free.head));-            free.head = unsafe { *phys_to_virt(frame.start_address()).as_ptr::<u64>() };-            free.len -= 1;-            return Some(frame);+    without_interrupts(|| {+        {+            let mut free = FREE_FRAMES.lock();+            if free.head != 0 {+                let frame = PhysFrame::containing_address(PhysAddr::new(free.head));+                free.head = unsafe { *phys_to_virt(frame.start_address()).as_ptr::<u64>() };+                free.len -= 1;+                return Some(frame);+            }         }-    }-    fresh_frame()+        fresh_frame()+    }) }  /// A never-used frame straight from the boot allocator. fn fresh_frame() -> Option<PhysFrame> {-    let frame = FRAME_ALLOCATOR.lock().as_mut()?.allocate_frame()?;+    let frame = without_interrupts(|| FRAME_ALLOCATOR.lock().as_mut()?.allocate_frame())?;     BOOT_FRAMES_USED.fetch_add(1, Ordering::Relaxed);     Some(frame) }@@ -171,15 +178,17 @@ fn fresh_frame() -> Option<PhysFrame> { /// Give a frame back. The caller promises nothing uses it any more (no page /// table maps it, no device writes to it). pub fn free_frame(frame: PhysFrame) {-    let mut free = FREE_FRAMES.lock();-    unsafe { *phys_to_virt(frame.start_address()).as_mut_ptr::<u64>() = free.head };-    free.head = frame.start_address().as_u64();-    free.len += 1;+    without_interrupts(|| {+        let mut free = FREE_FRAMES.lock();+        unsafe { *phys_to_virt(frame.start_address()).as_mut_ptr::<u64>() = free.head };+        free.head = frame.start_address().as_u64();+        free.len += 1;+    }) }  /// How many frames are in use right now (handed out and not freed). pub fn frames_in_use() -> usize {-    BOOT_FRAMES_USED.load(Ordering::Relaxed) as usize - FREE_FRAMES.lock().len+    without_interrupts(|| BOOT_FRAMES_USED.load(Ordering::Relaxed) as usize - FREE_FRAMES.lock().len) }  /// `FrameAllocator` adapter around the global allocator (for `map_to`, which@@ -356,3 +365,66 @@ pub fn user_range_ok(addr: u64, len: u64, write: bool) -> bool {         _ => false,     }) }++// ---------------------------------------------------------------------------+// Kernel-space mappings for kernel stacks (see stack.rs)+// ---------------------------------------------------------------------------++/// The L4 slot that holds every kernel stack (0x0000_6400_0000_0000).+pub const STACKS_L4_INDEX: usize = 200;++/// Protects the kernel's own page tables. Taken with interrupts off.+static KERNEL_TABLES: Mutex<()> = Mutex::new(());++/// Give the kernel-stack slot its level-3 table *now*, before any process+/// address space is created. New address spaces copy the kernel's L4+/// entries, so they all point at this one L3 table, and every stack mapped+/// later shows up in every address space automatically.+pub fn init_kernel_stack_area() {+    without_interrupts(|| {+        let _lock = KERNEL_TABLES.lock();+        let l4: &mut PageTable = unsafe { &mut *phys_to_virt(kernel_l4_frame().start_address()).as_mut_ptr() };+        if l4[STACKS_L4_INDEX].is_unused() {+            let l3 = alloc_zeroed_frame().expect("no memory for the kernel stack area");+            l4[STACKS_L4_INDEX].set_frame(l3, PageTableFlags::PRESENT | PageTableFlags::WRITABLE);+        }+    })+}++/// Map `page` (inside the kernel stack area) to a fresh zeroed frame:+/// present, writable, not executable, kernel-only.+pub fn map_kernel_page(page: Page) -> Option<()> {+    let frame = alloc_zeroed_frame()?;+    let mut flags = PageTableFlags::PRESENT | PageTableFlags::WRITABLE;+    if nx_enabled() {+        flags |= PageTableFlags::NO_EXECUTE;+    }+    let ok = without_interrupts(|| {+        let _lock = KERNEL_TABLES.lock();+        let mut mapper = unsafe { mapper_for(kernel_l4_frame()) };+        match unsafe { mapper.map_to(page, frame, flags, &mut GlobalFrameAllocator) } {+            Ok(flush) => {+                flush.flush(); // drop any stale TLB entry on *this* CPU+                true+            }+            Err(_) => false,+        }+    });+    if !ok {+        free_frame(frame);+    }+    ok.then_some(())+}++/// Unmap a page mapped by `map_kernel_page` and free its frame. (The page+/// tables stay: the next stack in the same place reuses them.)+pub fn unmap_kernel_page(page: Page) {+    without_interrupts(|| {+        let _lock = KERNEL_TABLES.lock();+        let mut mapper = unsafe { mapper_for(kernel_l4_frame()) };+        if let Ok((frame, flush)) = mapper.unmap(page) {+            flush.flush();+            free_frame(frame);+        }+    })+}

@@ -19,12 +19,21 @@ //! Unix shells do between fork and exec, is done with spawn's stdio options. //! //! Exit and wait: a process's exit status is stored in it and its `exited`-//! wait queue is woken; the parent's `wait(pid)` blocks on that queue, then-//! removes the child from the process table ("reaps" it -- until then it's-//! what Unix calls a zombie).+//! wait queue is woken; the parent's `wait(pid)` blocks on that queue, waits+//! until the thread is completely gone, then removes the child from the+//! process table ("reaps" it -- until then it's what Unix calls a zombie). //!-//! Not done yet (exercises): freeing a dead process's frames and page tables,-//! reparenting orphans, signals/kill, more than one thread per process.+//! Memory: a process's frames and page tables are freed when the last+//! `Arc<Process>` goes away (`impl Drop`). That's always after its thread is+//! gone: the thread holds an Arc until the scheduler frees it, which happens+//! only once the CPU has switched to another thread and address space. So+//! no CPU can still be using the page tables we free.+//!+//! Orphans: when a parent exits first, its running children become orphans+//! (parent = ORPHAN); nobody will wait for them, so they remove themselves+//! from the table when they exit. Unix hands them to `init` instead.+//!+//! Not done yet: signals/kill, more than one thread per process.  use crate::elf::{self, ElfError}; use crate::errno::Errno;@@ -32,8 +41,8 @@ use crate::fs::{self, file::File}; use crate::memory::{self, USER_END}; use crate::thread::{self, WaitQueue}; use crate::{serial_println, tty::Console};-use alloc::{boxed::Box, collections::BTreeMap, string::String, sync::Arc, vec, vec::Vec};-use core::sync::atomic::{AtomicU64, Ordering};+use alloc::{collections::BTreeMap, string::String, sync::Arc, vec, vec::Vec};+use core::sync::atomic::{AtomicU32, AtomicU64, AtomicUsize, Ordering}; use spin::Mutex; use x86_64::VirtAddr; use x86_64::instructions::interrupts::without_interrupts;@@ -42,6 +51,20 @@ use x86_64::structures::paging::{Page, PageTableFlags, PhysFrame, Size4KiB}; /// The user stack: 16 KiB, ending one (unmapped) page below USER_END. pub const USER_STACK_TOP: u64 = USER_END - 0x1000; pub const USER_STACK_SIZE: u64 = 16 * 1024;+/// Below the stack: a 64 KiB guard zone that is never mapped, so a runaway+/// recursion hits it and gets killed with "stack overflow" (instead of+/// scribbling over the program's data). Big, so that one large stack frame+/// can't jump over it.+pub const USER_STACK_GUARD: u64 = 64 * 1024;+/// Lowest address of stack + guard: program segments must end below this.+pub const USER_STACK_ZONE_START: u64 = USER_STACK_TOP - USER_STACK_SIZE - USER_STACK_GUARD;+/// `parent` of a process whose parent exited first.+pub const ORPHAN: u64 = u64::MAX;++/// Is user address `addr` in the guard zone under the user stack?+pub fn in_user_stack_guard(addr: u64) -> bool {+    (USER_STACK_ZONE_START..USER_STACK_TOP - USER_STACK_SIZE).contains(&addr)+} /// Max open files per process. const MAX_FDS: usize = 16; /// Max total size of a program's arguments (they go on its 16 KiB stack).@@ -70,16 +93,20 @@ impl ExitStatus {  pub struct Process {     pub pid: u64,-    /// Parent pid; 0 = started by the kernel itself.-    pub parent: u64,+    /// Parent pid; 0 = started by the kernel itself, ORPHAN = parent exited.+    pub parent: AtomicU64,     pub name: String,     /// Print every syscall this process makes (`spawn` flag; `run --trace`).     pub trace: bool,     /// Current directory (always a normalized absolute path).     pub cwd: Mutex<String>,     entry: u64,-    /// Level-4 page table of its address space (freed when it's reaped).+    /// Level-4 page table of its address space (freed in `drop`).     l4: PhysFrame,+    /// Its thread's id (set right after the thread is created).+    tid: AtomicUsize,+    /// CPUs its thread has run on (bit per CPU; kept up by the scheduler).+    pub ran_on: AtomicU32,     args: Vec<String>,     fds: Mutex<Vec<Option<Arc<dyn File>>>>,     /// Locked with interrupts off only (it's read in a wait_until condition).@@ -88,7 +115,8 @@ pub struct Process { }  static NEXT_PID: AtomicU64 = AtomicU64::new(1);-/// All processes that haven't been reaped yet, by pid.+/// All processes that haven't been reaped yet, by pid. Locked with+/// interrupts off only (exit_current takes it with interrupts off). static PROCESSES: Mutex<BTreeMap<u64, Arc<Process>>> = Mutex::new(BTreeMap::new());  /// Everything needed to start a process besides the program itself.@@ -162,22 +190,23 @@ fn start_process(name: &str, l4: PhysFrame, entry: u64, spawn: SpawnArgs) -> Arc     let [stdin, stdout, stderr] = spawn.stdio;     let process = Arc::new(Process {         pid: NEXT_PID.fetch_add(1, Ordering::Relaxed),-        parent: spawn.parent,+        parent: AtomicU64::new(spawn.parent),         name: String::from(name),         trace: spawn.trace,         cwd: Mutex::new(spawn.cwd),         entry,         l4,+        tid: AtomicUsize::new(usize::MAX),+        ran_on: AtomicU32::new(0),         args: spawn.args,         fds: Mutex::new(vec![Some(stdin), Some(stdout), Some(stderr)]),         status: Mutex::new(None),         exited: WaitQueue::new(),     });-    PROCESSES.lock().insert(process.pid, process.clone());+    without_interrupts(|| PROCESSES.lock().insert(process.pid, process.clone()));     serial_println!("[process] started pid {} ({}), entry {:#x}", process.pid, name, entry);-    // Thread names are &'static str; leaking a few bytes per process is fine here.-    let thread_name: &'static str = Box::leak(String::from(name).into_boxed_str());-    thread::spawn_with(thread_name, user_thread_entry, l4, Some(process.clone()));+    let tid = thread::spawn_with(name, user_thread_entry, l4, Some(process.clone()));+    process.tid.store(tid, Ordering::Release);     process } @@ -286,9 +315,22 @@ impl Process {     } } +impl Drop for Process {+    /// The last reference is gone, so its thread is gone too: give the+    /// address space's frames and page tables back.+    fn drop(&mut self) {+        unsafe { memory::free_address_space(self.l4) };+    }+}+ /// Look up a live (or not yet reaped) process. pub fn get(pid: u64) -> Option<Arc<Process>> {-    PROCESSES.lock().get(&pid).cloned()+    without_interrupts(|| PROCESSES.lock().get(&pid).cloned())+}++/// How many processes are in the table (running or not yet reaped).+pub fn count() -> usize {+    without_interrupts(|| PROCESSES.lock().len()) }  /// End the current process (exit syscall, or killed by an exception).@@ -307,32 +349,61 @@ pub fn exit_current(status: ExitStatus) -> ! {                 crate::kprintln!("[process] pid {} ({}) killed: {}", process.pid, process.name, why)             }         }-        // From here until this thread is gone, no preemption: once the status-        // is set, the parent may reap us and free our address space -- which-        // must not happen while we could still be switched back to.+        // No preemption from here on: we're about to disappear anyway.         x86_64::instructions::interrupts::disable();-        *process.status.lock() = Some(status);+        let removed = {+            // The table lock makes "set my status" and "look at my parent"+            // one step, so a parent exiting at the same time on another CPU+            // either sees our status (and reaps us) or orphans us first (and+            // we reap ourselves). Never both, never neither.+            let mut table = PROCESSES.lock();+            *process.status.lock() = Some(status);+            let mut removed = Vec::new();+            let children: Vec<Arc<Process>> =+                table.values().filter(|c| c.parent.load(Ordering::Relaxed) == process.pid).cloned().collect();+            for child in children {+                if child.status().is_some() {+                    removed.extend(table.remove(&child.pid)); // exited: reap it+                } else {+                    child.parent.store(ORPHAN, Ordering::Relaxed); // still running+                }+            }+            if process.parent.load(Ordering::Relaxed) == ORPHAN {+                removed.extend(table.remove(&process.pid)); // nobody will wait for us+            }+            removed+        };+        drop(removed); // (frees their memory if they're completely gone)         process.exited.wake_all(); // the parent may be blocked in wait()         drop(process);     }     thread::exit_current() } -/// Block (the calling thread) until `process` has ended, then remove it from-/// the process table. Used by the kernel and by the `wait` syscall.+/// Block (the calling thread) until `process` has ended and its thread is+/// gone, then remove it from the process table. Used by the kernel and by+/// the `wait` syscall. Once the caller drops its own `Arc`, all of the+/// process's memory is free again. pub fn wait(process: &Process) -> ExitStatus {     let status = process.exited.wait_until(|| *process.status.lock());     reap(process);     status } -/// Remove an ended process from the table and free its memory.+/// Wait for the (exited) process's thread to be freed, then remove the+/// process from the table. fn reap(process: &Process) {-    if PROCESSES.lock().remove(&process.pid).is_some() {-        // Its thread has finished (status is set with interrupts off right-        // before the thread exits), so nothing runs in this address space.-        unsafe { memory::free_address_space(process.l4) };-    }+    // (The tid is stored right after the thread is created; on another CPU+    // the thread may have run and exited before that.)+    let tid = loop {+        match process.tid.load(Ordering::Acquire) {+            usize::MAX => thread::yield_now(),+            tid => break tid,+        }+    };+    thread::join(tid);+    let removed = without_interrupts(|| PROCESSES.lock().remove(&process.pid));+    drop(removed); }  /// `wait` flag: don't block; `EAGAIN` if the child is still running.@@ -340,7 +411,7 @@ pub const WNOHANG: u64 = 1;  /// The `wait(pid, flags)` syscall: only a process's parent may wait for it. pub fn wait_child(parent: u64, pid: u64, flags: u64) -> Result<ExitStatus, Errno> {-    let child = get(pid).filter(|c| c.parent == parent).ok_or(Errno::ECHILD)?;+    let child = get(pid).filter(|c| c.parent.load(Ordering::Relaxed) == parent).ok_or(Errno::ECHILD)?;     if flags & WNOHANG != 0 {         let status = child.status().ok_or(Errno::EAGAIN)?;         reap(&child);

@@ -0,0 +1,294 @@+//! SMP (symmetric multiprocessing): starting the other CPUs, and per-CPU data.+//!+//! At power-on only one CPU runs: the BSP (bootstrap processor, our CPU 0).+//! The others, the APs (application processors), sit halted, waiting for+//! a wake-up call. To start one:+//!+//!  1. Copy a small piece of 16-bit code, the **trampoline**, to physical+//!     address 0x8000 (below 1 MiB: a starting AP is in 16-bit "real mode",+//!     the mode the original 8086 had, and can only reach the first MiB).+//!  2. Send it an INIT IPI (reset), wait 10+ ms, then a STARTUP IPI with+//!     vector 0x08: the AP starts executing at 0x08 * 4096 = 0x8000.+//!  3. The trampoline switches straight to 64-bit long mode using the BSP's+//!     own settings (page tables, CR0/CR4/EFER) and calls `ap_entry` in+//!     Rust on a fresh stack.+//!  4. `ap_entry` gives the CPU its own GDT/TSS, loads the shared IDT, sets+//!     up `syscall` and its APIC timer, and turns the CPU's current+//!     execution into its *idle thread*. From then on the scheduler hands+//!     it threads like any other CPU.+//!+//! Which CPUs exist? The ACPI MADT table lists them (see acpi.rs).+//!+//! **Which CPU am I?** Every CPU reads the APIC ID register of its own local+//! APIC; a table maps APIC IDs to our CPU numbers 0, 1, 2, ...++use crate::{acpi, apic, gdt, interrupts, memory, stack::KernelStack, thread};+use core::arch::global_asm;+use core::sync::atomic::{AtomicBool, AtomicU8, AtomicU32, AtomicU64, AtomicUsize, Ordering};+use spin::Mutex;+use x86_64::instructions::interrupts::without_interrupts;++/// Most CPUs we handle (bitmasks are u32; 16 keeps the arrays small).+pub const MAX_CPUS: usize = 16;++/// APIC ID -> CPU number.+static APIC_TO_CPU: [AtomicU8; 256] = [const { AtomicU8::new(0) }; 256];+/// CPU number -> APIC ID.+static CPU_TO_APIC: [AtomicU32; MAX_CPUS] = [const { AtomicU32::new(0) }; MAX_CPUS];+/// How many CPUs are running (CPU 0 plus started APs).+static CPU_COUNT: AtomicUsize = AtomicUsize::new(1);+/// CPUs halted in their idle loop (bit per CPU), so new work can wake one.+static IDLE_CPUS: AtomicU32 = AtomicU32::new(0);++/// Per-CPU data the syscall entry reaches through GS (see syscall.rs).+/// The field order is part of the assembly code: offsets 0 and 8.+#[repr(C)]+pub struct PerCpu {+    /// Top of the kernel stack of the thread running on this CPU.+    pub syscall_kernel_rsp: AtomicU64,+    /// Scratch: the user RSP during the first instructions of a syscall.+    pub syscall_user_rsp: AtomicU64,+}+static PER_CPU: [PerCpu; MAX_CPUS] = [const {+    PerCpu { syscall_kernel_rsp: AtomicU64::new(0), syscall_user_rsp: AtomicU64::new(0) }+}; MAX_CPUS];++/// The number (0, 1, ...) of the CPU this code runs on. Only meaningful+/// with interrupts off, or as a hint: the scheduler may move a thread to+/// another CPU at any timer tick.+pub fn cpu_id() -> usize {+    APIC_TO_CPU[apic::id() as usize & 0xFF].load(Ordering::Relaxed) as usize+}++/// How many CPUs are running.+pub fn cpu_count() -> usize {+    CPU_COUNT.load(Ordering::Acquire)+}++/// This CPU's `PerCpu` block.+pub fn this_cpu() -> &'static PerCpu {+    &PER_CPU[cpu_id()]+}++/// Mark this CPU as idle (about to `hlt`) or busy.+pub fn set_idle(cpu: usize, idle: bool) {+    if idle {+        IDLE_CPUS.fetch_or(1 << cpu, Ordering::SeqCst);+    } else {+        IDLE_CPUS.fetch_and(!(1 << cpu), Ordering::SeqCst);+    }+}++/// New work is ready: if another CPU is idle, interrupt it so it picks the+/// work up now rather than at its next timer tick (up to 10 ms later).+pub fn kick_idle_cpu() {+    let idle = IDLE_CPUS.load(Ordering::SeqCst) & !(1 << cpu_id());+    if idle != 0 {+        let cpu = idle.trailing_zeros() as usize;+        let target = CPU_TO_APIC[cpu].load(Ordering::Relaxed);+        without_interrupts(|| apic::send_ipi(target, interrupts::InterruptIndex::Reschedule as u8));+    }+}++// ---------------------------------------------------------------------------+// Starting the other CPUs+// ---------------------------------------------------------------------------++/// Physical address the trampoline is copied to (must be below 1 MiB and+/// 4 KiB-aligned; the STARTUP IPI's vector is this / 4096).+const TRAMPOLINE: u64 = 0x8000;++/// The trampoline's parameter block (filled in by the BSP for each AP).+/// Must match the `.quad`s at `ap_trampoline_params` below.+#[repr(C)]+struct ApParams {+    cr3: u64,+    cr4: u64,+    efer: u64,+    cr0: u64,+    stack_top: u64,+    cpu: u64,+    entry: u64,+}++unsafe extern "C" {+    static ap_trampoline_start: u8;+    static ap_trampoline_params: u8;+    static ap_trampoline_end: u8;+}++// The trampoline. It runs at physical 0x8000 (not where the linker put it),+// so every address is computed as 0x8000 + (label - ap_trampoline_start).+// AT&T syntax, because that's what most 16-bit examples use.+global_asm!(+    r#"+.section .text.ap_trampoline, "ax"+.code16+.global ap_trampoline_start+ap_trampoline_start:+    cli+    cld+    xorw %ax, %ax+    movw %ax, %ds                 # data segment 0: plain physical addresses+    # A temporary GDT with a 64-bit code segment and a data segment.+    lgdtl (ap_gdt_ptr - ap_trampoline_start + 0x8000)+    # Turn on the same CPU features as the BSP: CR4 (PAE: 64-bit page+    # tables, ...), CR3 (the kernel's page tables), EFER (LME = long mode+    # enable, NXE = no-execute bit, SCE = syscall).+    movl (ap_cr4 - ap_trampoline_start + 0x8000), %eax+    movl %eax, %cr4+    movl (ap_cr3 - ap_trampoline_start + 0x8000), %eax+    movl %eax, %cr3+    movl $0xC0000080, %ecx        # EFER's MSR number+    movl (ap_efer - ap_trampoline_start + 0x8000), %eax+    xorl %edx, %edx+    wrmsr+    # CR0: protection + paging at once. With EFER.LME set this goes straight+    # from real mode to long mode (skipping 32-bit protected mode works on+    # QEMU and real CPUs, though the manuals describe the longer route).+    movl (ap_cr0 - ap_trampoline_start + 0x8000), %eax+    movl %eax, %cr0+    # Far jump into the 64-bit code segment (selector 0x08).+    ljmpl $0x08, $(ap_long_mode - ap_trampoline_start + 0x8000)++.code64+ap_long_mode:+    movw $0x10, %ax+    movw %ax, %ds+    movw %ax, %es+    movw %ax, %ss+    xorw %ax, %ax+    movw %ax, %fs+    movw %ax, %gs+    movq (ap_stack - ap_trampoline_start + 0x8000), %rsp+    movq (ap_cpu - ap_trampoline_start + 0x8000), %rdi+    movq (ap_entry - ap_trampoline_start + 0x8000), %rax+    callq *%rax                   # ap_entry(cpu): never returns+1:  hlt+    jmp 1b++.balign 8+ap_gdt:+    .quad 0+    .quad 0x00af9a000000ffff      # 0x08: 64-bit kernel code+    .quad 0x00cf92000000ffff      # 0x10: kernel data+ap_gdt_ptr:+    .word 3 * 8 - 1+    .long ap_gdt - ap_trampoline_start + 0x8000+.balign 8+.global ap_trampoline_params+ap_trampoline_params:+ap_cr3:   .quad 0+ap_cr4:   .quad 0+ap_efer:  .quad 0+ap_cr0:   .quad 0+ap_stack: .quad 0+ap_cpu:   .quad 0+ap_entry: .quad 0+.global ap_trampoline_end+ap_trampoline_end:+.text+"#,+    options(att_syntax)+);++/// The boot stack of the AP being started (it becomes its idle thread's).+static AP_STACK: Mutex<Option<KernelStack>> = Mutex::new(None);+/// Set by a starting AP as soon as it reaches Rust code.+static AP_ALIVE: AtomicBool = AtomicBool::new(false);+/// Set by a starting AP when it's fully set up.+static AP_READY: AtomicBool = AtomicBool::new(false);++/// Bring up multi-core: enable CPU 0's APIC and timer, create CPU 0's idle+/// thread, then start every other CPU listed in the ACPI tables, one by+/// one. Needs the heap and interrupts on. Returns the number of CPUs.+pub fn init() -> usize {+    apic::init_bsp();+    CPU_TO_APIC[0].store(apic::id(), Ordering::Relaxed);+    APIC_TO_CPU[apic::id() as usize].store(0, Ordering::Relaxed);+    thread::init();++    let ids = acpi::cpu_apic_ids().unwrap_or_default();+    copy_trampoline();+    for id in ids {+        let cpu = cpu_count();+        if id == apic::id() || cpu == MAX_CPUS || id > 255 {+            continue;+        }+        APIC_TO_CPU[id as usize].store(cpu as u8, Ordering::Relaxed);+        CPU_TO_APIC[cpu].store(id, Ordering::Relaxed);+        if start_ap(id, cpu) {+            CPU_COUNT.store(cpu + 1, Ordering::Release);+        } else {+            crate::kprintln!("[smp] CPU with APIC id {} did not start", id);+        }+    }+    cpu_count()+}++fn copy_trampoline() {+    let (start, end) = unsafe { (&ap_trampoline_start as *const u8, &ap_trampoline_end as *const u8) };+    let len = end as usize - start as usize;+    assert!(len <= 4096, "AP trampoline too big");+    let dest: *mut u8 = memory::phys_to_virt(x86_64::PhysAddr::new(TRAMPOLINE)).as_mut_ptr();+    unsafe { core::ptr::copy_nonoverlapping(start, dest, len) };+}++/// Start one AP and wait (up to ~1 s) until it reports in.+fn start_ap(apic_id: u32, cpu: usize) -> bool {+    use x86_64::registers::control::{Cr0, Cr4};+    use x86_64::registers::model_specific::Efer;+    let Some(stack) = KernelStack::new() else { return false };+    let cr3 = memory::kernel_l4_frame().start_address().as_u64();+    assert!(cr3 < 1 << 32, "the trampoline loads CR3 with a 32-bit value");+    let params = ApParams {+        cr3,+        cr4: Cr4::read_raw(),+        efer: Efer::read_raw() & !(1 << 10), // not LMA ("long mode active"): the CPU sets that itself+        cr0: Cr0::read_raw(),+        stack_top: stack.top(),+        cpu: cpu as u64,+        entry: ap_entry as *const () as u64,+    };+    let offset = unsafe { &ap_trampoline_params as *const u8 as u64 - &ap_trampoline_start as *const u8 as u64 };+    let dest: *mut ApParams = memory::phys_to_virt(x86_64::PhysAddr::new(TRAMPOLINE + offset)).as_mut_ptr();+    unsafe { dest.write(params) };+    without_interrupts(|| *AP_STACK.lock() = Some(stack));+    AP_ALIVE.store(false, Ordering::SeqCst);+    AP_READY.store(false, Ordering::SeqCst);++    apic::send_init(apic_id);+    wait_ticks(2, || false); // the spec asks for 10 ms; 2 PIT ticks is >= 55 ms+    apic::send_startup(apic_id, (TRAMPOLINE / 4096) as u8);+    // The spec suggests a second STARTUP IPI if the first one got lost.+    if !wait_ticks(1, || AP_ALIVE.load(Ordering::SeqCst)) {+        apic::send_startup(apic_id, (TRAMPOLINE / 4096) as u8);+    }+    wait_ticks(18, || AP_READY.load(Ordering::SeqCst))+}++/// Wait up to `ticks` PIT ticks for `done()`; returns its final value.+fn wait_ticks(ticks: u64, done: impl Fn() -> bool) -> bool {+    let end = interrupts::ticks() + ticks;+    while interrupts::ticks() < end {+        if done() {+            return true;+        }+        core::hint::spin_loop();+    }+    done()+}++/// Where each AP arrives from the trampoline: 64-bit mode, interrupts off,+/// on its boot stack, with the trampoline's temporary GDT.+extern "C" fn ap_entry(cpu: u64) -> ! {+    let cpu = cpu as usize;+    AP_ALIVE.store(true, Ordering::SeqCst);+    gdt::init_ap(cpu);+    interrupts::init_idt();+    crate::syscall::init();+    apic::init_ap();+    let stack = without_interrupts(|| AP_STACK.lock().take()).expect("AP boot stack missing");+    AP_READY.store(true, Ordering::SeqCst);+    thread::run_ap(cpu, stack) // become this CPU's idle thread+}

@@ -0,0 +1,123 @@+//! Kernel stacks with guard pages.+//!+//! Every kernel thread (and every CPU's double-fault handler) needs its own+//! stack. Stack overflows are nasty: the stack grows *down* into whatever+//! lies below it and silently corrupts it. So each stack gets a **guard+//! page** underneath: a page that is deliberately never mapped. Overflowing+//! the stack touches the guard page, the CPU raises a page fault (and,+//! because pushing the fault frame onto the broken stack fails too, a double+//! fault on a separate emergency stack), and interrupts.rs prints a clear+//! "kernel stack overflow" message and kills the thread.+//!+//! All stacks live in one 512 GiB region (L4 slot 200), cut into slots:+//!+//! ```text+//!   slot n:  [ guard page (unmapped) ][ 16 pages = 64 KiB of stack ]+//!            ^ STACKS_START + n * SLOT_SIZE                         ^ top+//! ```+//!+//! **Reusing a slot safely on several CPUs.** When a stack is freed we unmap+//! its pages and give the frames back. Another CPU may still have the old+//! translations cached in its TLB (translation lookaside buffer). If the+//! slot got a new stack and a thread ran on that other CPU, it could use the+//! stale entries and write into frames that now belong to someone else.+//! Real kernels send a "TLB shootdown" interrupt to every CPU. We use a+//! simpler trick: the scheduler remembers, per thread, which CPUs have+//! flushed that thread's stack from their TLB, and flushes it (`invlpg`,+//! 16 instructions) the first time the thread runs on a CPU. Nothing else+//! ever touches another thread's stack, so that's enough.++use crate::memory;+use alloc::vec::Vec;+use core::sync::atomic::{AtomicU64, Ordering};+use spin::Mutex;+use x86_64::instructions::interrupts::without_interrupts;+use x86_64::{VirtAddr, structures::paging::Page};++/// Usable size of every kernel stack.+pub const STACK_SIZE: u64 = 64 * 1024;+const PAGE: u64 = 4096;+/// One guard page plus the stack.+const SLOT_SIZE: u64 = PAGE + STACK_SIZE;+/// Start of the stack region (L4 slot 200 = 200 * 512 GiB).+pub const STACKS_START: u64 = (memory::STACKS_L4_INDEX as u64) << 39;+const STACKS_END: u64 = STACKS_START + (1 << 39);++/// Next never-used slot number.+static NEXT_SLOT: AtomicU64 = AtomicU64::new(0);+/// Slots whose stacks were freed, ready for reuse (taken with interrupts off).+static FREE_SLOTS: Mutex<Vec<u64>> = Mutex::new(Vec::new());+/// How many stacks exist right now (for statistics and tests).+static LIVE: AtomicU64 = AtomicU64::new(0);++/// An owned, mapped kernel stack. Dropping it unmaps the pages and frees+/// the frames.+pub struct KernelStack {+    slot: u64,+}++impl KernelStack {+    /// A new 64 KiB stack (zeroed), or None if we're out of memory.+    pub fn new() -> Option<KernelStack> {+        let slot = without_interrupts(|| FREE_SLOTS.lock().pop())+            .unwrap_or_else(|| NEXT_SLOT.fetch_add(1, Ordering::Relaxed));+        let stack = KernelStack { slot };+        LIVE.fetch_add(1, Ordering::Relaxed);+        assert!(stack.top() <= STACKS_END, "kernel stack area exhausted");+        for page in stack.pages() {+            if memory::map_kernel_page(page).is_none() {+                return None; // `stack` is dropped here: what we mapped is undone+            }+        }+        Some(stack)+    }++    /// Lowest usable address (just above the guard page).+    pub fn bottom(&self) -> u64 {+        STACKS_START + self.slot * SLOT_SIZE + PAGE+    }++    /// One past the highest usable address: the initial stack pointer.+    pub fn top(&self) -> u64 {+        self.bottom() + STACK_SIZE+    }++    /// Address of the guard page under this stack.+    pub fn guard_page(&self) -> u64 {+        self.bottom() - PAGE+    }++    fn pages(&self) -> impl Iterator<Item = Page> {+        let bottom = self.bottom();+        (0..STACK_SIZE / PAGE).map(move |i| Page::containing_address(VirtAddr::new(bottom + i * PAGE)))+    }++    /// Forget any TLB entries *this* CPU might hold for this stack's+    /// addresses (left over from an older stack in the same slot).+    pub fn flush_local_tlb(&self) {+        for page in self.pages() {+            x86_64::instructions::tlb::flush(page.start_address());+        }+    }+}++impl Drop for KernelStack {+    fn drop(&mut self) {+        for page in self.pages() {+            memory::unmap_kernel_page(page); // skips pages that never got mapped+        }+        LIVE.fetch_sub(1, Ordering::Relaxed);+        let slot = self.slot;+        without_interrupts(|| FREE_SLOTS.lock().push(slot));+    }+}++/// Is `addr` inside the guard page of some kernel stack?+pub fn is_guard_page(addr: u64) -> bool {+    (STACKS_START..STACKS_END).contains(&addr) && (addr - STACKS_START) % SLOT_SIZE < PAGE+}++/// How many kernel stacks are allocated right now.+pub fn live_stacks() -> u64 {+    LIVE.load(Ordering::Relaxed)+}

@@ -35,6 +35,7 @@ //! | 217 | getdents64  | fd, buf, len                        | Linux `struct linux_dirent64` records | //! | 500 | spawn       | path, path_len, argv, argv_len, *SpawnOptions | ours: start a program, returns its pid (see process.rs for why not fork/exec) | //! | 501 | mounts      | buf, len                            | ours: "path fstype\n" per mount (Linux: read /proc/mounts) |+//! | 502 | sysinfo     | *SysInfo                            | ours: CPUs, threads, memory, per-CPU busy time (Linux: /proc/stat etc.) | //! //! spawn's `argv` is all arguments separated by NUL bytes ("ls\0-l\0/bin"); //! empty means just [program name]. `SpawnOptions` (or a null pointer for@@ -51,8 +52,14 @@ //!   save registers, re-enable interrupts and call `syscall_dispatch` in Rust. //!   `sysretq` reverses it: RIP = rcx, RFLAGS = r11, CPL = 3. //!-//! Single CPU only: the "current kernel stack" and the user-RSP scratch slot-//! are plain globals. (Multi-core kernels use per-CPU data via `swapgs`.)+//! **Several CPUs.** The "kernel stack of the running thread" differs per+//! CPU, but at the entry we have no free register to compute which CPU we+//! are on. The trick every x86_64 kernel uses: the GS segment base. Each CPU+//! keeps a pointer to its own `smp::PerCpu` block in the KERNEL_GS_BASE MSR;+//! `swapgs` swaps that with the (user's) GS base, so for a few instructions+//! `gs:[0]` is *this CPU's* kernel stack and `gs:[8]` a scratch slot. We+//! swap back right after switching stacks, so the thread can be moved to+//! another CPU later without any GS confusion. //! //! Why not `int 0x80`? It also works (and is simpler: the CPU switches stacks //! via TSS.RSP0 for us), but `syscall` is what Linux and every modern x86_64@@ -64,7 +71,7 @@ use crate::fs::{self, InodeKind}; use crate::{memory, process, thread}; use alloc::{string::String, sync::Arc, vec::Vec}; use core::arch::global_asm;-use core::sync::atomic::{AtomicU64, Ordering};+use core::sync::atomic::Ordering;  pub const SYS_READ: u64 = 0; pub const SYS_WRITE: u64 = 1;@@ -90,6 +97,7 @@ pub const SYS_READLINK: u64 = 89; pub const SYS_GETDENTS64: u64 = 217; pub const SYS_SPAWN: u64 = 500; pub const SYS_MOUNTS: u64 = 501;+pub const SYS_SYSINFO: u64 = 502;  pub const O_CREAT: u64 = 0o100; pub const O_TRUNC: u64 = 0o1000;@@ -125,6 +133,7 @@ pub fn name(nr: u64) -> &'static str {         SYS_GETDENTS64 => "getdents64",         SYS_SPAWN => "spawn",         SYS_MOUNTS => "mounts",+        SYS_SYSINFO => "sysinfo",         _ => "?",     } }@@ -139,22 +148,17 @@ pub struct SpawnOptions {     pub flags: u64, } -/// Kernel stack top of the running thread (set by the scheduler).-#[unsafe(no_mangle)]-static SYSCALL_KERNEL_RSP: AtomicU64 = AtomicU64::new(0);-/// Scratch slot for the user RSP during the first instructions of the entry.-#[unsafe(no_mangle)]-static SYSCALL_USER_RSP: AtomicU64 = AtomicU64::new(0);--/// Called by the scheduler when switching threads.+/// Called by the scheduler when switching threads: the kernel stack that+/// `syscall` on this CPU should use from now on. pub fn set_kernel_stack(stack_top: u64) {-    SYSCALL_KERNEL_RSP.store(stack_top, Ordering::Relaxed);+    crate::smp::this_cpu().syscall_kernel_rsp.store(stack_top, Ordering::Relaxed); } -/// Program the MSRs that make `syscall` work.+/// Program the MSRs that make `syscall` work (on the calling CPU; every CPU+/// runs this once). pub fn init() {     use x86_64::VirtAddr;-    use x86_64::registers::model_specific::{Efer, EferFlags, LStar, SFMask, Star};+    use x86_64::registers::model_specific::{Efer, EferFlags, KernelGsBase, LStar, SFMask, Star};     use x86_64::registers::rflags::RFlags;      let s = crate::gdt::selectors();@@ -171,6 +175,8 @@ pub fn init() {     // the kernel stack. DF: Rust expects the direction flag clear. TF: no     // single-stepping into the kernel.     SFMask::write(RFlags::INTERRUPT_FLAG | RFlags::DIRECTION_FLAG | RFlags::TRAP_FLAG);+    // KERNEL_GS_BASE: this CPU's PerCpu block, for `swapgs` in the entry.+    KernelGsBase::write(VirtAddr::from_ptr(crate::smp::this_cpu())); }  unsafe extern "C" {@@ -180,10 +186,13 @@ unsafe extern "C" { global_asm!(     ".global syscall_entry",     "syscall_entry:",-    // Interrupts are off (SFMASK). Switch to this thread's kernel stack.-    "    mov [rip + SYSCALL_USER_RSP], rsp",-    "    mov rsp, [rip + SYSCALL_KERNEL_RSP]",-    "    push qword ptr [rip + SYSCALL_USER_RSP]", // user rsp+    // Interrupts are off (SFMASK). Switch to this thread's kernel stack,+    // found in this CPU's PerCpu block (gs:[0]; gs:[8] is scratch).+    "    swapgs",+    "    mov gs:[8], rsp",+    "    mov rsp, gs:[0]",+    "    push qword ptr gs:[8]", // user rsp+    "    swapgs",                // GS back to normal right away     "    push rcx",                                // user rip     "    push r11",                                // user rflags     // Preserve the argument registers for the user (Rust may clobber them).@@ -262,6 +271,7 @@ extern "C" fn syscall_dispatch(nr: u64, a1: u64, a2: u64, a3: u64, a4: u64, a5:          SYS_GETDENTS64 => sys_getdents64(a1, a2, a3),         SYS_SPAWN => sys_spawn(a1, a2, a3, a4, a5),         SYS_MOUNTS => sys_mounts(a1, a2),+        SYS_SYSINFO => sys_sysinfo(a1),         _ => Err(Errno::ENOSYS),     };     let ret = match result {@@ -493,7 +503,7 @@ fn sys_uname(buf: u64) -> Result<u64, Errno> {         "Rustbelt",                     // sysname (`uname -s`)         "qemu",                         // nodename (hostname)         env!("CARGO_PKG_VERSION"),      // release (`uname -r`)-        "stage 10: btrfs read-only",    // version+        "stage 11: core polish",        // version         "x86_64",                       // machine         "(none)",                       // domainname     ];@@ -505,6 +515,43 @@ fn sys_uname(buf: u64) -> Result<u64, Errno> {     Ok(0) } +/// What `sysinfo` fills in (ours; the user side is in user/ulib.rs).+#[repr(C)]+#[derive(Default)]+pub struct SysInfo {+    pub cpus: u64,+    pub this_cpu: u64,+    pub threads: u64,+    pub processes: u64,+    pub frames_in_use: u64,+    pub uptime_ticks: u64,+    /// Per CPU: local-timer ticks so far, and how many found it busy+    /// (running something other than its idle thread).+    pub ticks: [u64; 16],+    pub busy_ticks: [u64; 16],+}++fn sys_sysinfo(buf: u64) -> Result<u64, Errno> {+    let out = user_slice_mut(buf, core::mem::size_of::<SysInfo>() as u64)?;+    let mut info = SysInfo {+        cpus: crate::smp::cpu_count() as u64,+        this_cpu: crate::smp::cpu_id() as u64,+        threads: thread::thread_count() as u64,+        processes: process::count() as u64,+        frames_in_use: memory::frames_in_use() as u64,+        uptime_ticks: crate::interrupts::ticks(),+        ..Default::default()+    };+    for cpu in 0..info.cpus as usize {+        (info.ticks[cpu], info.busy_ticks[cpu]) = thread::cpu_ticks(cpu);+    }+    let bytes = unsafe {+        core::slice::from_raw_parts(&info as *const SysInfo as *const u8, core::mem::size_of::<SysInfo>())+    };+    out.copy_from_slice(bytes);+    Ok(0)+}+ /// The mount table as text, one "path fstype" line per mount. fn sys_mounts(buf: u64, len: u64) -> Result<u64, Errno> {     let mut text = String::new();

@@ -79,6 +79,13 @@ impl Executor {     /// atomically (interrupts are only recognised after the `hlt` starts).     fn sleep_if_idle(&self) {         use x86_64::instructions::interrupts::{self, enable_and_hlt};+        if crate::thread::scheduler_started() {+            // Block this thread until a waker fires (e.g. from the keyboard+            // interrupt): other threads get the CPU meanwhile.+            EXECUTOR_IDLE.wait_until(|| (!self.task_queue.is_empty()).then_some(()));+            return;+        }+        // No scheduler (some tests): just halt until the next interrupt.         interrupts::disable();         if self.task_queue.is_empty() {             enable_and_hlt();@@ -108,9 +115,13 @@ impl TaskWaker {      fn wake_task(&self) {         self.task_queue.push(self.task_id).expect("task_queue full");+        EXECUTOR_IDLE.wake_all();     } } +/// Where an idle executor thread blocks until a task is woken.+static EXECUTOR_IDLE: crate::thread::WaitQueue = crate::thread::WaitQueue::new();+ impl Wake for TaskWaker {     fn wake(self: Arc<Self>) {         self.wake_task();

@@ -1,10 +1,12 @@-//! Preemptive kernel threads (beyond the phil-opp series).+//! Threads and the scheduler: preemptive, multi-core, with sleep/wake. //!-//! Each thread has its own stack. The timer interrupt calls `schedule()`,-//! which picks the next thread round-robin and calls `switch_context` to swap-//! stacks. Threads don't have to cooperate: even `loop {}` gets interrupted.+//! Each thread has its own kernel stack (64 KiB with a guard page, see+//! stack.rs). Every CPU's APIC timer fires ~100 times a second and calls+//! `schedule()`, which puts the running thread at the back of the shared+//! **ready queue** and switches to the thread at the front. Threads don't+//! have to cooperate: even `loop {}` gets interrupted. //!-//! How a switch works (all inside the timer interrupt, interrupts off):+//! How a switch works (inside the timer interrupt, interrupts off): //! //!   thread A running -> timer IRQ -> CPU pushes A's RIP/RSP/RFLAGS (the //!   interrupt frame) on A's stack -> the `x86-interrupt` handler saves the@@ -18,94 +20,289 @@ //!   ... which returns into B's timer handler, which restores B's caller-saved //!   registers and `iretq`s back to wherever B was interrupted. //!-//! So a sleeping thread's whole CPU state is simply sitting on its own stack,+//! So a waiting thread's whole CPU state is simply sitting on its own stack, //! and all we have to remember per thread is its stack pointer. //!-//! Threads that run user code (processes, see process.rs) additionally carry-//! an address space (CR3) and a kernel stack top. On every switch the-//! scheduler loads the next thread's CR3 and points the TSS's RSP0 and the-//! syscall entry at its kernel stack, so interrupts and syscalls from ring 3-//! land on the right stack.+//! Threads that run user code (processes, see process.rs) also carry an+//! address space (CR3). On every switch the scheduler loads the next+//! thread's CR3 and points this CPU's TSS.RSP0 and syscall stack at the+//! thread's kernel stack. //!-//! Blocking: a thread waiting for something (keyboard input, a child-//! process, a pipe, a timeout) is marked `blocked` and the scheduler skips it-//! until someone calls `wake` (or its sleep deadline passes). That's what-//! `WaitQueue` is for. If *every* thread is blocked, the CPU idles with `hlt`-//! until an interrupt wakes somebody up.+//! **States.** A thread is Ready (in the queue), Running (on some CPU),+//! Blocked (waiting on a `WaitQueue`: keyboard input, a pipe, a child...),+//! Sleeping (until a PIT tick; the PIT handler wakes it) or Finished. Blocked+//! and sleeping threads are not in the queue at all, so they cost no CPU+//! time: no busy-polling anywhere. //!-//! Locking rule (single CPU!): any lock that is also taken with interrupts-//! *off* (the scheduler's, a wait queue's, a pipe buffer's...) must *always*-//! be taken with interrupts off. Otherwise a thread could be preempted while-//! holding it, and the next thread would spin on it forever, unable to be-//! preempted itself.+//! **Idle threads.** Each CPU has one, run when the queue is empty. It+//! halts the CPU (`hlt`) until an interrupt arrives. A CPU that queues work+//! while another is idle sends it a "reschedule" IPI (smp.rs). //!-//! A *new* thread has never been interrupted, so `spawn` builds a fake stack-//! that looks like `switch_context` saved it, with the "return address"-//! pointing at `thread_trampoline`, which enables interrupts and calls the-//! thread's function.+//! **Several CPUs, one queue.** The scheduler's data is behind one spinlock.+//! Two subtle points:+//!   * A thread that was just preempted on CPU A is put in the queue *before*+//!     A has finished switching away from its stack. CPU B must not start+//!     running it on the same stack at the same time! So each thread has an+//!     `on_cpu` flag, cleared only after the switch (`finish_switch`), and+//!     the scheduler skips threads that still have it set.+//!   * Likewise a finished thread's stack (and maybe its process's memory)+//!     can only be freed once no CPU is on it: also in `finish_switch`.+//!+//! **Locking rule.** Any lock that is ever taken with interrupts *off* (the+//! scheduler's, a wait queue's, a pipe buffer's, the frame allocator's...)+//! must *always* be taken with interrupts off. Otherwise a thread could be+//! preempted while holding it, and an interrupt handler on the same CPU+//! that wants it would spin forever: the holder can't run until the+//! handler returns. (Spinning briefly while *another* CPU holds a lock is+//! fine: that CPU keeps running and will release it.)+//!+//! A *new* thread has never been switched away from, so `spawn` builds a+//! fake stack that looks like `switch_context` saved it, with the "return+//! address" pointing at `thread_trampoline`, which calls the thread's function.  use crate::process::Process;-use alloc::{boxed::Box, sync::Arc, vec, vec::Vec};+use crate::smp::{self, MAX_CPUS};+use crate::stack::KernelStack;+use alloc::{+    boxed::Box,+    collections::{BTreeMap, VecDeque},+    string::String,+    sync::Arc,+    vec::Vec,+}; use core::arch::global_asm;+use core::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use spin::Mutex; use x86_64::instructions::interrupts; use x86_64::registers::control::Cr3; use x86_64::structures::paging::PhysFrame; -/// 64 KiB per thread stack. There's no guard page (stacks are plain heap-/// memory), so overflowing one silently overwrites whatever lies below it.-/// We at least *detect* that afterwards: the lowest 8 bytes hold-/// `STACK_CANARY`, checked whenever the scheduler switches away from a-/// thread. (16 KiB was enough until syscalls went through the FAT driver:-/// unoptimized debug builds use big stack frames.)-const STACK_SIZE: usize = 64 * 1024;-const STACK_CANARY: u64 = 0x5AFE_57AC_C0DE_CAFE;+/// Thread id. Never reused.+pub type Tid = usize;++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum State {+    Ready,+    Running,+    Blocked,+    /// Sleeping until this PIT tick.+    Sleeping(u64),+    Finished,+}  struct Thread {-    name: &'static str,+    name: String,     /// Saved stack pointer while the thread is not running.     rsp: u64,-    /// The thread's stack. `None` for the boot thread (uses the bootloader's stack).-    stack: Option<Vec<u8>>,-    /// Top of that stack (0 for the boot thread). Becomes TSS.RSP0 and the-    /// syscall stack while this thread runs.-    kernel_stack_top: u64,+    /// The thread's stack. `None` for the boot thread "main", which runs on+    /// the stack the bootloader gave us (that one has a guard page too).+    stack: Option<KernelStack>,     /// Level-4 page table to run with (the kernel's, or a process's own).     cr3: PhysFrame,     /// The process this thread belongs to (None = pure kernel thread).     process: Option<Arc<Process>>,-    finished: bool,-    /// Waiting for something; the scheduler skips blocked threads.-    blocked: bool,-    /// If blocked by `sleep_ticks`: wake up at this tick.-    wake_at: Option<u64>,+    state: State,+    /// Some CPU is (still) running on this thread's stack.+    on_cpu: bool,+    /// A wake-up arrived while it was running: its next block returns at once.+    woken: bool,+    /// One of the per-CPU idle threads (never in the ready queue).+    idle: bool,+    /// CPUs that have flushed this thread's stack from their TLB (stack.rs).+    tlb_clean_on: u32,+    /// CPUs this thread has run on, one bit each (for tests and curiosity).+    ran_on: u32, }  struct Scheduler {-    /// Boxed so each Thread has a fixed address even if the Vec grows: we-    /// hand a raw pointer to `rsp` to `switch_context`.-    threads: Vec<Box<Thread>>,-    current: usize,+    /// All threads. Boxed so each Thread has a fixed address while the map+    /// changes: `switch_context` gets a raw pointer to `rsp`.+    threads: BTreeMap<Tid, Box<Thread>>,+    /// Threads waiting for a CPU, in order.+    ready: VecDeque<Tid>,+    /// Per CPU: the running thread, the idle thread, and the thread we just+    /// switched away from (for `finish_switch`).+    current: [Option<Tid>; MAX_CPUS],+    idle: [Option<Tid>; MAX_CPUS],+    prev: [Option<Tid>; MAX_CPUS],+    next_id: Tid,+}++static SCHEDULER: Mutex<Scheduler> = Mutex::new(Scheduler {+    threads: BTreeMap::new(),+    ready: VecDeque::new(),+    current: [None; MAX_CPUS],+    idle: [None; MAX_CPUS],+    prev: [None; MAX_CPUS],+    next_id: 0,+});++/// Lock the scheduler. Interrupts must be off (locking rule above).+fn lock() -> spin::MutexGuard<'static, Scheduler> {+    debug_assert!(!interrupts::are_enabled(), "scheduler lock with interrupts on");+    SCHEDULER.lock()+}++/// Per-CPU timer statistics: ticks seen, and ticks that found the CPU busy.+static CPU_TICKS: [AtomicU64; MAX_CPUS] = [const { AtomicU64::new(0) }; MAX_CPUS];+static CPU_BUSY_TICKS: [AtomicU64; MAX_CPUS] = [const { AtomicU64::new(0) }; MAX_CPUS];+/// Is CPU n running its idle thread right now?+static CPU_IDLING: [AtomicBool; MAX_CPUS] = [const { AtomicBool::new(false) }; MAX_CPUS];+/// Kernel stack overflows caught so far.+static STACK_OVERFLOWS: AtomicU64 = AtomicU64::new(0);+/// Woken whenever a thread is gone for good (for `join`).+static THREAD_EXITED: WaitQueue = WaitQueue::new();++impl Scheduler {+    fn thread(&mut self, id: Tid) -> &mut Thread {+        self.threads.get_mut(&id).expect("no such thread")+    }++    fn add(&mut self, thread: Thread) -> Tid {+        let id = self.next_id;+        self.next_id += 1;+        self.threads.insert(id, Box::new(thread));+        id+    }++    /// The thread running on `cpu`. On first use on CPU 0 that's the boot+    /// code, which we adopt as thread "main" (its rsp is saved on its first+    /// switch). APs are registered in `run_ap` before they schedule.+    fn current(&mut self, cpu: usize) -> Tid {+        if let Some(id) = self.current[cpu] {+            return id;+        }+        let mut main = new_thread("main", None, Cr3::read().0, None);+        main.state = State::Running;+        main.on_cpu = true;+        let id = self.add(main);+        self.current[cpu] = Some(id);+        id+    }++    /// Next runnable thread from the queue, skipping threads another CPU is+    /// still switching away from (they stay queued).+    fn pop_ready(&mut self) -> Option<Tid> {+        for _ in 0..self.ready.len() {+            let id = self.ready.pop_front()?;+            match self.threads.get(&id) {+                Some(t) if t.state == State::Ready && !t.on_cpu => return Some(id),+                Some(t) if t.state == State::Ready => self.ready.push_back(id),+                _ => {} // stale entry (can't really happen): drop it+            }+        }+        None+    }++    /// Make a blocked/sleeping thread ready. Returns true if it was queued.+    fn make_ready(&mut self, id: Tid) -> bool {+        let Some(t) = self.threads.get_mut(&id) else { return false };+        match t.state {+            State::Blocked | State::Sleeping(_) => {+                t.state = State::Ready;+                self.ready.push_back(id);+                true+            }+            // Not asleep (yet): it may be just about to block. Make sure it+            // doesn't, or this wake-up would be lost.+            State::Running | State::Ready => {+                t.woken = true;+                false+            }+            State::Finished => false,+        }+    }+}++fn new_thread(name: &str, stack: Option<KernelStack>, cr3: PhysFrame, process: Option<Arc<Process>>) -> Thread {+    Thread {+        name: String::from(name),+        rsp: 0,+        stack,+        cr3,+        process,+        state: State::Ready,+        on_cpu: false,+        woken: false,+        idle: false,+        tlb_clean_on: 0,+        ran_on: 0,+    }+}++/// Create CPU 0's idle thread (smp::init calls this). Before this, a CPU+/// with nothing to run just halts inside whichever thread is blocking.+pub fn init() {+    let id = new_kernel_thread("idle0", idle_entry, crate::memory::kernel_l4_frame(), None, false);+    interrupts::without_interrupts(|| {+        let mut s = lock();+        s.current(0);+        s.thread(id).idle = true;+        s.idle[0] = Some(id);+    }); } -static SCHEDULER: Mutex<Scheduler> = Mutex::new(Scheduler { threads: Vec::new(), current: 0 });+fn idle_entry() {+    idle_loop();+} -/// Start a new kernel thread running `entry`. Needs the heap.-pub fn spawn(name: &'static str, entry: fn()) {-    spawn_with(name, entry, crate::memory::kernel_l4_frame(), None);+/// An AP (from smp.rs) turns its boot code into its idle thread, on the+/// boot stack it was given. Interrupts are off.+pub fn run_ap(cpu: usize, stack: KernelStack) -> ! {+    {+        let mut s = lock();+        let mut idle = new_thread("idle", Some(stack), crate::memory::kernel_l4_frame(), None);+        idle.name = alloc::format!("idle{}", cpu);+        idle.idle = true;+        idle.state = State::Running;+        idle.on_cpu = true;+        idle.tlb_clean_on = 1 << cpu;+        let id = s.add(idle);+        s.current[cpu] = Some(id);+        s.idle[cpu] = Some(id);+    }+    CPU_IDLING[cpu].store(true, Ordering::Relaxed);+    idle_loop()+}++/// What a CPU does when there's nothing to do: run whatever becomes ready,+/// otherwise sleep (`hlt`) until an interrupt.+fn idle_loop() -> ! {+    loop {+        interrupts::disable();+        schedule(); // returns when the queue is empty (again)+        let cpu = smp::cpu_id();+        // Say "I'm idle" *before* the last look at the queue: whoever queues+        // work after that look sees the flag and sends us an IPI.+        smp::set_idle(cpu, true);+        if interrupts::without_interrupts(|| lock().ready.is_empty()) {+            interrupts::enable_and_hlt(); // `sti; hlt` with no gap for a lost IPI+        } else {+            interrupts::enable();+        }+        smp::set_idle(cpu, false);+    }+}++/// Start a new kernel thread running `entry`. Needs the heap and memory.+pub fn spawn(name: &str, entry: fn()) -> Tid {+    spawn_with(name, entry, crate::memory::kernel_l4_frame(), None) }  /// Start a thread for `process`: runs `entry` (in the kernel) with the /// process's address space `cr3` active; `entry` then drops to ring 3.-pub fn spawn_with(name: &'static str, entry: fn(), cr3: PhysFrame, process: Option<Arc<Process>>) {-    let mut stack = vec![0u8; STACK_SIZE];-    stack[..8].copy_from_slice(&STACK_CANARY.to_ne_bytes());+pub fn spawn_with(name: &str, entry: fn(), cr3: PhysFrame, process: Option<Arc<Process>>) -> Tid {+    new_kernel_thread(name, entry, cr3, process, true)+}++fn new_kernel_thread(name: &str, entry: fn(), cr3: PhysFrame, process: Option<Arc<Process>>, queue: bool) -> Tid {+    let stack = KernelStack::new().expect("out of memory for a kernel stack");      // Build the initial stack, top down (stacks grow towards lower addresses):     //   [return address = thread_trampoline]     //   rbp, rbx, r12 (= entry), r13, r14, r15     <- rsp points at r15     // exactly what `switch_context` pops before its `ret`.-    let top = (stack.as_mut_ptr() as u64 + STACK_SIZE as u64) & !0xf; // 16-byte aligned+    let top = stack.top();     let initial: [u64; 7] = [         0,                                // r15         0,                                // r14@@ -118,93 +315,70 @@ pub fn spawn_with(name: &'static str, entry: fn(), cr3: PhysFrame, process: Opti     let rsp = top - 8 * initial.len() as u64;     unsafe { core::ptr::copy_nonoverlapping(initial.as_ptr(), rsp as *mut u64, initial.len()) }; -    let thread = Box::new(Thread {-        name,-        rsp,-        stack: Some(stack),-        kernel_stack_top: top,-        cr3,-        process,-        finished: false,-        blocked: false,-        wake_at: None,-    });--    // The timer handler also takes this lock, so keep interrupts off while we-    // hold it -- otherwise the handler would spin forever on our lock.-    interrupts::without_interrupts(|| {-        let mut scheduler = SCHEDULER.lock();-        register_main_thread(&mut scheduler);-        // Free the stacks of threads that have finished (we can't do that in-        // exit_current: the thread is still running on its stack there).-        let current = scheduler.current;-        for (i, t) in scheduler.threads.iter_mut().enumerate() {-            if t.finished && i != current {-                t.stack = None;-                t.process = None;-            }+    let mut thread = new_thread(name, Some(stack), cr3, process);+    thread.rsp = rsp;+    let id = interrupts::without_interrupts(|| {+        let mut s = lock();+        s.current(smp::cpu_id()); // adopt "main" if this is the first spawn+        let id = s.add(thread);+        if queue {+            s.ready.push_back(id);         }-        scheduler.threads.push(thread);+        id     });-}--/// On first use, register the code that's running right now (the boot/main-/// thread) as thread 0. Its rsp is filled in the first time we switch away.-fn register_main_thread(scheduler: &mut Scheduler) {-    if scheduler.threads.is_empty() {-        scheduler.threads.push(Box::new(Thread {-            name: "main",-            rsp: 0,-            stack: None,-            kernel_stack_top: 0,-            cr3: Cr3::read().0,-            process: None,-            finished: false,-            blocked: false,-            wake_at: None,-        }));+    if queue {+        smp::kick_idle_cpu();     }+    id } -/// Called from the timer interrupt (interrupts disabled, EOI already sent).-/// Switches to the next runnable thread, round-robin. Returns when this-/// thread is scheduled again.+/// Switch this CPU to the next thread in the queue, if there is one.+/// Called with interrupts off: from the timer interrupt, `yield_now`, or a+/// thread that just blocked. Returns when this thread runs again (maybe on+/// another CPU!), or right away if there's nothing else to run. pub fn schedule() {+    let cpu = smp::cpu_id();     let (old_rsp, new_rsp) = {-        let mut s = SCHEDULER.lock();-        let n = s.threads.len();-        if n < 2 {-            return; // no threads spawned yet-        }-        let current = s.current;-        // Wake up sleepers whose time has come.-        let now = crate::interrupts::TICKS.load(core::sync::atomic::Ordering::Relaxed);-        for t in s.threads.iter_mut() {-            if t.wake_at.is_some_and(|at| now >= at) {-                t.wake_at = None;-                t.blocked = false;-            }-        }-        // Find the next thread that can run.-        let next = (1..=n)-            .map(|i| (current + i) % n)-            .find(|&i| !s.threads[i].finished && !s.threads[i].blocked);-        let next = match next {-            Some(next) if next != current => next,-            _ => return, // nobody else to run+        let mut s = lock();+        let Some(cur) = s.current[cpu] else { return }; // no threads yet+        let cur_state = s.thread(cur).state;+        let cur_idle = s.thread(cur).idle;+        let next = match s.pop_ready() {+            Some(next) => next,+            // Nobody waiting: a running thread just carries on...+            None if cur_state == State::Running => return,+            // ...one that blocked or finished hands over to the idle thread.+            None => match s.idle[cpu] {+                Some(idle) if idle != cur => idle,+                _ => return, // no idle thread yet: the caller halts instead+            },         };-        if let Some(stack) = &s.threads[current].stack {-            if stack[..8] != STACK_CANARY.to_ne_bytes() {-                panic!("kernel stack overflow in thread '{}'", s.threads[current].name);+        // Back of the line for the current thread (if it can still run).+        if cur_state == State::Running {+            if cur_idle {+                s.thread(cur).state = State::Ready; // idle threads aren't queued+            } else {+                s.thread(cur).state = State::Ready;+                s.ready.push_back(cur);             }         }-        s.current = next; -        // Prepare the CPU for the next thread *before* switching stacks.-        let t = &s.threads[next];-        if t.kernel_stack_top != 0 {-            crate::gdt::set_kernel_stack(t.kernel_stack_top); // interrupts from ring 3-            crate::syscall::set_kernel_stack(t.kernel_stack_top); // `syscall` from ring 3+        let bit = 1u32 << cpu;+        let t = s.thread(next);+        t.state = State::Running;+        t.on_cpu = true;+        t.ran_on |= bit;+        if let Some(process) = &t.process {+            process.ran_on.fetch_or(bit, core::sync::atomic::Ordering::Relaxed);+        }+        if let Some(stack) = &t.stack {+            if t.tlb_clean_on & bit == 0 {+                stack.flush_local_tlb(); // see "Reusing a slot" in stack.rs+                t.tlb_clean_on |= bit;+            }+            // Where interrupts (TSS.RSP0) and `syscall` from ring 3 land.+            crate::gdt::set_kernel_stack(stack.top());+            crate::syscall::set_kernel_stack(stack.top());         }         if Cr3::read().0 != t.cr3 {             // Switch address space. Safe for the code running right now:@@ -212,13 +386,63 @@ pub fn schedule() {             // every address space. Writing CR3 also flushes the TLB.             unsafe { Cr3::write(t.cr3, Cr3::read().1) };         }--        let old_rsp: *mut u64 = &mut s.threads[current].rsp;-        (old_rsp, s.threads[next].rsp)+        CPU_IDLING[cpu].store(t.idle, Ordering::Relaxed);+        let new_rsp = t.rsp;+        s.current[cpu] = Some(next);+        s.prev[cpu] = Some(cur);+        let old_rsp: *mut u64 = &mut s.thread(cur).rsp;+        (old_rsp, new_rsp)         // The lock is released here, *before* switching: the next thread         // will never come back to release a lock we still held.     };     unsafe { switch_context(old_rsp, new_rsp) };+    finish_switch();+}++/// Right after a switch, on the new thread: the previous thread's stack is+/// no longer in use. Clear its `on_cpu` so other CPUs may run it, and if it+/// had finished, remove it and free its stack (and, if it was the last user+/// of its process, the process's memory, via `Drop for Process`).+fn finish_switch() {+    let cpu = smp::cpu_id();+    let (leftovers, requeued) = {+        let mut s = lock();+        let Some(prev) = s.prev[cpu].take() else { return };+        let t = s.thread(prev);+        t.on_cpu = false;+        let state = t.state;+        // A finished thread: take out what it owns (its stack, and its+        // share of the process) to free below, outside the lock.+        let leftovers = (state == State::Finished).then(|| (prev, t.stack.take(), t.process.take()));+        (leftovers, state == State::Ready)+    };+    if requeued {+        smp::kick_idle_cpu(); // it may have been skipped while on_cpu was set+    }+    if let Some((dead, stack, process)) = leftovers {+        // Free first, *then* remove it from the table: `join` waits for the+        // removal, so once `join` returns the memory really is free (the+        // leak tests count frames right after joining).+        drop(stack); // with interrupts still off: the allocators allow that+        drop(process);+        lock().threads.remove(&dead);+        THREAD_EXITED.wake_all();+    }+}++/// Every CPU's APIC timer tick: count it for the statistics, then preempt.+pub fn timer_tick() {+    let cpu = smp::cpu_id();+    CPU_TICKS[cpu].fetch_add(1, Ordering::Relaxed);+    if !CPU_IDLING[cpu].load(Ordering::Relaxed) {+        CPU_BUSY_TICKS[cpu].fetch_add(1, Ordering::Relaxed);+    }+    schedule();+}++/// (ticks, busy ticks) of CPU `cpu`'s APIC timer so far.+pub fn cpu_ticks(cpu: usize) -> (u64, u64) {+    (CPU_TICKS[cpu].load(Ordering::Relaxed), CPU_BUSY_TICKS[cpu].load(Ordering::Relaxed)) }  /// Give up the rest of this time slice (also callable outside interrupts).@@ -229,98 +453,193 @@ pub fn yield_now() { /// The process the running thread belongs to, if any. pub fn current_process() -> Option<Arc<Process>> {     interrupts::without_interrupts(|| {-        let s = SCHEDULER.lock();-        s.threads.get(s.current).and_then(|t| t.process.clone())+        let mut s = lock();+        let cur = s.current(smp::cpu_id());+        s.thread(cur).process.clone()     }) } -/// End the running thread: mark it finished and switch away for good.-/// (Its stack is kept: we're still running on it until the switch.)+/// End the running thread: mark it finished and switch away for good. The+/// next thread on this CPU frees its stack (see `finish_switch`). pub fn exit_current() -> ! {     interrupts::disable();     {-        let mut s = SCHEDULER.lock();-        let current = s.current;-        s.threads[current].finished = true;+        let mut s = lock();+        let cur = s.current(smp::cpu_id());+        s.thread(cur).state = State::Finished;+    }+    loop {+        schedule(); // never returns once there's an idle thread+        // No idle thread (scheduler not fully started): wait for anything.+        interrupts::enable_and_hlt();+        interrupts::disable();     }-    schedule(); // never returns: finished threads aren't picked again-    crate::hlt_loop(); // only reached if nothing else exists to run }  /// Name of the running thread (for printing).-pub fn current_name() -> &'static str {+pub fn current_name() -> String {     interrupts::without_interrupts(|| {-        let s = SCHEDULER.lock();-        s.threads.get(s.current).map_or("main", |t| t.name)+        let mut s = lock();+        let cur = s.current(smp::cpu_id());+        s.thread(cur).name.clone()     }) } -/// Id of the running thread (its index in the thread list; ids are never reused).-pub fn current_id() -> usize {-    interrupts::without_interrupts(|| {-        let mut s = SCHEDULER.lock();-        register_main_thread(&mut s);-        s.current-    })+/// Id of the running thread.+pub fn current_id() -> Tid {+    interrupts::without_interrupts(|| lock().current(smp::cpu_id())) } -/// Make thread `id` runnable again (no-op if it isn't blocked).-pub fn wake(id: usize) {-    interrupts::without_interrupts(|| {-        if let Some(t) = SCHEDULER.lock().threads.get_mut(id) {-            t.blocked = false;-            t.wake_at = None;-        }-    });+/// Is the scheduler fully running (CPU 0 has its idle thread)?+pub fn scheduler_started() -> bool {+    interrupts::without_interrupts(|| lock().idle[0].is_some())+}++/// Does thread `id` still exist (not finished, or finished but its stack+/// not freed yet)?+pub fn exists(id: Tid) -> bool {+    interrupts::without_interrupts(|| lock().threads.contains_key(&id))+}++/// Wait until thread `id` is completely gone (finished and freed).+pub fn join(id: Tid) {+    THREAD_EXITED.wait_until(|| (!lock().threads.contains_key(&id)).then_some(()));+}++/// Number of threads (including idle threads and "main").+pub fn thread_count() -> usize {+    interrupts::without_interrupts(|| lock().threads.len())+}++/// CPUs thread `id` has run on so far (bit n = CPU n), if it exists.+pub fn ran_on(id: Tid) -> Option<u32> {+    interrupts::without_interrupts(|| lock().threads.get(&id).map(|t| t.ran_on)) } -fn is_blocked(id: usize) -> bool {-    SCHEDULER.lock().threads[id].blocked+/// How many kernel stack overflows have been caught.+pub fn stack_overflows() -> u64 {+    STACK_OVERFLOWS.load(Ordering::Relaxed) } -/// Mark the running thread blocked, then run other threads until somebody-/// wakes it. Must be called with interrupts *off* (so a wake-up can't slip in-/// between the caller checking its condition and us blocking); returns with-/// interrupts still off.-fn block_current(wake_at: Option<u64>) {+/// Make thread `id` runnable again (if it's blocked or sleeping), and poke+/// an idle CPU to run it.+pub fn wake(id: Tid) {+    if interrupts::without_interrupts(|| lock().make_ready(id)) {+        smp::kick_idle_cpu();+    }+}++/// PIT tick (CPU 0's timer interrupt): wake every thread whose sleep is over.+pub fn wake_sleepers(now: u64) {+    let woke = {+        let mut s = lock();+        let due: Vec<Tid> = s+            .threads+            .iter()+            .filter(|(_, t)| matches!(t.state, State::Sleeping(at) if at <= now))+            .map(|(id, _)| *id)+            .collect();+        for &id in &due {+            s.make_ready(id);+        }+        !due.is_empty()+    };+    if woke {+        smp::kick_idle_cpu();+    }+}++/// Block the running thread (until `wake`, or until PIT tick `until`), and+/// run other threads meanwhile. Must be called with interrupts *off*, so a+/// wake-up can't slip in between the caller checking its condition and us+/// blocking; returns with interrupts still off. Returns at once if a+/// wake-up already arrived (`woken`).+fn block_current(until: Option<u64>) {     let me = {-        let mut s = SCHEDULER.lock();-        register_main_thread(&mut s);-        let me = s.current;-        s.threads[me].blocked = true;-        s.threads[me].wake_at = wake_at;+        let mut s = lock();+        let me = s.current(smp::cpu_id());+        let t = s.thread(me);+        if t.woken {+            t.woken = false;+            return;+        }+        t.state = match until {+            Some(at) => State::Sleeping(at),+            None => State::Blocked,+        };         me     };     loop {-        schedule(); // returns once we're picked again -- or if nobody else can run-        if !is_blocked(me) {+        schedule(); // returns once we've been woken and picked again+        if done_waiting(me) {             return;         }-        // Every thread is blocked: idle until an interrupt (timer, keyboard)-        // possibly wakes someone. `enable_and_hlt` = `sti; hlt` with no gap,-        // so the interrupt can't sneak in before the hlt.+        // We're still here: there was nothing (not even an idle thread) to+        // switch to. Halt until an interrupt possibly wakes us.+        // `enable_and_hlt` = `sti; hlt` with no gap for the interrupt.         interrupts::enable_and_hlt();         interrupts::disable();-        if !is_blocked(me) {+        if done_waiting(me) {             return;         }     } } -/// Sleep for `ticks` timer ticks (~55 ms each). The thread is *blocked*-/// meanwhile, so it costs no CPU time; the scheduler wakes it up.+/// After `schedule` returned to a thread that blocked: has it been woken?+/// If it was woken but never left the CPU, it's Ready and queued: take it+/// back out of the queue, it's running right here.+fn done_waiting(me: Tid) -> bool {+    let mut s = lock();+    match s.thread(me).state {+        State::Blocked | State::Sleeping(_) => false,+        State::Ready => {+            s.thread(me).state = State::Running;+            s.ready.retain(|&id| id != me);+            true+        }+        _ => true,+    }+}++/// Clear a stale wake-up flag before starting to wait for something new.+fn clear_woken() {+    let mut s = lock();+    let me = s.current(smp::cpu_id());+    s.thread(me).woken = false;+}++/// Sleep for `ticks` PIT ticks (~55 ms each). The thread is off the run+/// queue meanwhile, so it costs no CPU time; the PIT handler wakes it. pub fn sleep_ticks(ticks: u64) {-    use crate::interrupts::TICKS;-    use core::sync::atomic::Ordering;     let was_enabled = interrupts::are_enabled();     interrupts::disable();-    let wake_at = TICKS.load(Ordering::Relaxed) + ticks.max(1);-    block_current(Some(wake_at));+    let wake_at = crate::interrupts::ticks() + ticks.max(1);+    clear_woken();+    while crate::interrupts::ticks() < wake_at {+        block_current(Some(wake_at)); // may return early on a stray wake-up+    }     if was_enabled {         interrupts::enable();     } } +/// The running thread overflowed its kernel stack (the page-fault or+/// double-fault handler saw an access to a guard page). Say so, and kill+/// it: its process if it has one, else just the thread.+pub fn die_of_stack_overflow(addr: u64) -> ! {+    STACK_OVERFLOWS.fetch_add(1, Ordering::Relaxed);+    let name = current_name();+    crate::kprintln!(+        "[kernel] kernel stack overflow in thread '{}' (hit guard page {:#x}); killing it",+        name,+        addr & !0xfff+    );+    if current_process().is_some() {+        crate::process::exit_current(crate::process::ExitStatus::Killed("kernel stack overflow"));+    }+    exit_current()+}+ /// Threads waiting for some event (like a Linux `wait_queue_head_t`). /// /// Usage: `queue.wait_until(|| condition)` blocks until the closure returns@@ -328,7 +647,7 @@ pub fn sleep_ticks(ticks: u64) { /// The closure runs with interrupts off, so it must be quick and only touch /// locks that follow the locking rule at the top of this file. pub struct WaitQueue {-    waiting: Mutex<Vec<usize>>,+    waiting: Mutex<Vec<Tid>>, }  impl WaitQueue {@@ -340,14 +659,18 @@ impl WaitQueue {         let was_enabled = interrupts::are_enabled();         let result = loop {             interrupts::disable();-            // Check and go to sleep without interrupts in between: otherwise-            // a wake_all() could happen right after the check, and we'd-            // sleep forever (the classic "lost wake-up" bug).+            // The order matters (the classic "lost wake-up" bug): put+            // ourselves on the list *first*, then check. If the condition+            // becomes true on another CPU after our check, its wake_all()+            // finds us on the list and sets `woken`, so block_current+            // returns straight away instead of sleeping forever.+            let me = current_id();+            clear_woken();+            self.waiting.lock().push(me);             if let Some(value) = condition() {+                self.waiting.lock().retain(|&id| id != me);                 break value;             }-            let me = current_id();-            self.waiting.lock().push(me);             block_current(None);             // Woken up: loop and check the condition again (somebody else may             // have consumed what we were waiting for).@@ -359,11 +682,10 @@ impl WaitQueue {     }      pub fn wake_all(&self) {-        interrupts::without_interrupts(|| {-            for id in self.waiting.lock().drain(..) {-                wake(id);-            }-        });+        let waiting: Vec<Tid> = interrupts::without_interrupts(|| self.waiting.lock().drain(..).collect());+        for id in waiting {+            wake(id);+        }     } } @@ -410,8 +732,11 @@ global_asm!(  /// Rust side of a new thread's life. extern "C" fn thread_start(entry: usize) -> ! {-    // We arrived here from inside the timer interrupt, where interrupts are-    // off. Turn them back on, or this thread could never be preempted.+    // We arrived here from inside `schedule`, like any switched-to thread:+    // do the same clean-up as `schedule` does after its switch.+    finish_switch();+    // Interrupts are off in `schedule`. Turn them back on, or this thread+    // could never be preempted.     interrupts::enable();      let entry: fn() = unsafe { core::mem::transmute(entry) };

@@ -128,6 +128,9 @@ pub fn input_key(key: KeyCode) { /// screen and serial port. pub struct Console; +/// Held for the whole of one console `write` (see `Console::write`).+static CONSOLE_WRITE: spin::Mutex<()> = spin::Mutex::new(());+ impl File for Console {     /// Blocks until input is available. Cooked: returns up to `buf.len()`     /// bytes of a line, stopping after its '\n'. Raw: whatever keys are there.@@ -153,6 +156,15 @@ impl File for Console {     }      fn write(&self, buf: &[u8]) -> Result<usize, Errno> {+        // One write = one unbroken piece of output, even when programs on+        // other CPUs write at the same time. Taken in thread context only+        // (never in an interrupt handler), so while waiting we just yield.+        let _turn = loop {+            match CONSOLE_WRITE.try_lock() {+                Some(guard) => break guard,+                None => crate::thread::yield_now(),+            }+        };         // Print valid UTF-8 as is, anything else as '?'. Control characters         // are handled one by one.         for chunk in buf.utf8_chunks() {

@@ -20,6 +20,7 @@ entry_point!(main); fn main(boot_info: &'static BootInfo) -> ! {     rustbelt::init();     rustbelt::init_memory(boot_info);+    rustbelt::init_smp(); // all 4 CPUs run threads     block::init(&pci::scan());     test_main();     rustbelt::hlt_loop();

@@ -38,6 +38,7 @@ entry_point!(main); fn main(boot_info: &'static BootInfo) -> ! {     rustbelt::init();     rustbelt::init_memory(boot_info);+    rustbelt::init_smp(); // all 4 CPUs run threads     fs::init();     rustbelt::init_storage();     test_main();

@@ -26,6 +26,7 @@ entry_point!(main); fn main(boot_info: &'static BootInfo) -> ! {     rustbelt::init();     rustbelt::init_memory(boot_info);+    rustbelt::init_smp(); // all 4 CPUs run threads     fs::init();     rustbelt::init_storage();     test_main();

@@ -30,6 +30,7 @@ entry_point!(main); fn main(boot_info: &'static BootInfo) -> ! {     rustbelt::init();     rustbelt::init_memory(boot_info);+    rustbelt::init_smp(); // all 4 CPUs run threads     fs::init();     rustbelt::init_storage();     test_main();

@@ -24,6 +24,7 @@ entry_point!(main); fn main(boot_info: &'static BootInfo) -> ! {     rustbelt::init();     rustbelt::init_memory(boot_info);+    rustbelt::init_smp(); // all 4 CPUs run threads     fs::init();     rustbelt::init_storage();     test_main();

@@ -0,0 +1,99 @@+//! Stage 11: guard pages. Below every kernel stack is an unmapped 4 KiB+//! page, and below every user stack a 64 KiB unmapped zone. Running off the+//! end of a stack hits them, and the kernel kills just the culprit with a+//! clear "stack overflow" message, instead of corrupting other memory.+#![no_std]+#![no_main]+#![feature(custom_test_frameworks)]+#![test_runner(rustbelt::test_runner)]+#![reexport_test_harness_main = "test_main"]++extern crate alloc;++use bootloader::{BootInfo, entry_point};+use core::panic::PanicInfo;+use core::sync::atomic::{AtomicBool, Ordering};+use rustbelt::process::{self, ExitStatus, SpawnArgs};+use rustbelt::stack::{self, KernelStack, STACK_SIZE};+use rustbelt::{fs, memory, thread};++entry_point!(main);++fn main(boot_info: &'static BootInfo) -> ! {+    rustbelt::init();+    rustbelt::init_memory(boot_info);+    rustbelt::init_smp();+    fs::init();+    rustbelt::init_storage();+    test_main();+    rustbelt::hlt_loop();+}++#[test_case]+fn guard_page_sits_right_below_each_kernel_stack() {+    let s = KernelStack::new().expect("no stack");+    assert_eq!(s.top() - s.bottom(), STACK_SIZE);+    assert_eq!(s.guard_page(), s.bottom() - 4096);+    assert!(stack::is_guard_page(s.guard_page()));+    assert!(stack::is_guard_page(s.bottom() - 1));+    assert!(!stack::is_guard_page(s.bottom()));+    assert!(!stack::is_guard_page(s.top() - 8));+}++#[test_case]+fn user_stack_guard_zone_bounds() {+    let stack_bottom = process::USER_STACK_TOP - process::USER_STACK_SIZE;+    assert!(process::in_user_stack_guard(stack_bottom - 1));+    assert!(process::in_user_stack_guard(process::USER_STACK_ZONE_START));+    assert!(!process::in_user_stack_guard(stack_bottom));+    assert!(!process::in_user_stack_guard(process::USER_STACK_ZONE_START - 1));+}++static REACHED_END: AtomicBool = AtomicBool::new(false);++#[allow(unconditional_recursion)]+fn recurse(depth: u64) -> u64 {+    let buf = core::hint::black_box([depth as u8; 256]);+    recurse(depth + 1) + buf[0] as u64+}++fn overflowing_thread() {+    recurse(0);+    REACHED_END.store(true, Ordering::Relaxed);+}++#[test_case]+fn kernel_stack_overflow_kills_only_that_thread() {+    let frames = memory::frames_in_use();+    let stacks = stack::live_stacks();+    let caught = thread::stack_overflows();+    for _ in 0..3 {+        // Three times: the handler must keep working (on any CPU).+        let id = thread::spawn("recurser", overflowing_thread);+        thread::join(id);+    }+    assert!(!REACHED_END.load(Ordering::Relaxed));+    assert_eq!(thread::stack_overflows(), caught + 3);+    // Its stack was freed like any other thread's, and we're still running.+    assert_eq!(stack::live_stacks(), stacks);+    assert_eq!(memory::frames_in_use(), frames);+}++#[test_case]+fn user_stack_overflow_kills_the_process() {+    let run = || {+        let spawn = SpawnArgs::kernel("overflow");+        let p = process::spawn_path("/mnt/disk0/bin/overflow", spawn).expect("spawn failed");+        process::wait(&p)+    };+    assert_eq!(run(), ExitStatus::Killed("stack overflow")); // warm up+    let frames = memory::frames_in_use();+    assert_eq!(run(), ExitStatus::Killed("stack overflow"));+    // Everything it had (incl. the 16 KiB of stack it filled) is free again.+    assert_eq!(memory::frames_in_use(), frames);+}++#[panic_handler]+fn panic(info: &PanicInfo) -> ! {+    rustbelt::test_panic_handler(info)+}

@@ -24,6 +24,7 @@ entry_point!(main); fn main(boot_info: &'static BootInfo) -> ! {     rustbelt::init();     rustbelt::init_memory(boot_info);+    rustbelt::init_smp(); // all 4 CPUs run threads     fs::init();     rustbelt::init_storage();     test_main();@@ -175,7 +176,7 @@ fn scripted_shell_session() {         "piped text\n",         "wc -l\n2\n",         "saved\nmore\n",-        "Rustbelt 0.10.0",+        "Rustbelt 0.11.0",         "B:/ $ exit 3\n",     ] {         assert!(out.contains(expected), "missing {:?} in:\n{}", expected, out);

@@ -0,0 +1,268 @@+//! Stage 11: several CPUs, and the scheduler that shares them out.+//! - all 4 CPUs come up and take timer interrupts+//! - threads and processes really run on more than one CPU at a time+//! - locks still work when two CPUs race for them+//! - sleeping threads wake on time, in order, and cost no CPU while asleep+//! - wait queues block and wake without busy-polling+//! - spawning and exiting threads/processes in a loop leaks no memory+#![no_std]+#![no_main]+#![feature(custom_test_frameworks)]+#![test_runner(rustbelt::test_runner)]+#![reexport_test_harness_main = "test_main"]++extern crate alloc;++use alloc::{string::String, sync::Arc, vec::Vec};+use bootloader::{BootInfo, entry_point};+use core::panic::PanicInfo;+use core::sync::atomic::{AtomicBool, AtomicU64, Ordering};+use rustbelt::interrupts::ticks;+use rustbelt::process::{self, ExitStatus, SpawnArgs};+use rustbelt::thread::{self, WaitQueue};+use rustbelt::{fs, memory, smp, stack};+use spin::Mutex;+use x86_64::instructions::interrupts::without_interrupts;++entry_point!(main);++fn main(boot_info: &'static BootInfo) -> ! {+    rustbelt::init();+    rustbelt::init_memory(boot_info);+    rustbelt::init_smp();+    fs::init();+    rustbelt::init_storage();+    test_main();+    rustbelt::hlt_loop();+}++/// Start /mnt/disk0/bin/<name> with these arguments.+fn run(name: &str, extra_args: &[&str]) -> Arc<process::Process> {+    let mut spawn = SpawnArgs::kernel(name);+    spawn.args.extend(extra_args.iter().map(|a| String::from(*a)));+    process::spawn_path(&alloc::format!("/mnt/disk0/bin/{}", name), spawn).expect("spawn failed")+}++/// (ticks, busy ticks) of every CPU right now.+fn cpu_ticks() -> Vec<(u64, u64)> {+    (0..smp::cpu_count()).map(thread::cpu_ticks).collect()+}++#[test_case]+fn four_cpus_are_online_and_ticking() {+    assert_eq!(smp::cpu_count(), 4, "QEMU runs with -smp 4");+    let before = cpu_ticks();+    thread::sleep_ticks(4);+    let after = cpu_ticks();+    for cpu in 0..4 {+        assert!(after[cpu].0 > before[cpu].0, "CPU {} got no timer ticks", cpu);+    }+}++// --- work runs on several CPUs at once --------------------------------------++static STOP: AtomicBool = AtomicBool::new(false);++fn spin_until_stopped() {+    while !STOP.load(Ordering::Relaxed) {+        core::hint::spin_loop();+    }+}++#[test_case]+fn busy_threads_run_on_several_cpus_in_parallel() {+    STOP.store(false, Ordering::Relaxed);+    let ids: Vec<_> = (0..4).map(|_| thread::spawn("spinner", spin_until_stopped)).collect();+    thread::sleep_ticks(2); // let them spread out+    let before = cpu_ticks();+    thread::sleep_ticks(6); // ~330 ms; we sleep, the 4 spinners get 4 CPUs+    let after = cpu_ticks();+    let mut ran_on = 0;+    for &id in &ids {+        ran_on |= thread::ran_on(id).expect("spinner vanished");+    }+    STOP.store(true, Ordering::Relaxed);+    for id in ids {+        thread::join(id);+    }+    // At least two CPUs were busy (>= half their ticks) at the same time.+    let busy_cpus = (0..4)+        .filter(|&c| {+            let total = after[c].0 - before[c].0;+            let busy = after[c].1 - before[c].1;+            total > 0 && busy * 2 >= total+        })+        .count();+    assert!(ran_on.count_ones() >= 2, "spinners ran on CPUs {:#b} only", ran_on);+    assert!(busy_cpus >= 2, "only {} CPUs were busy: {:?} -> {:?}", busy_cpus, before, after);+}++#[test_case]+fn user_processes_run_on_several_cpus() {+    let ps: Vec<_> = (0..3).map(|_| run("spin", &[])).collect();+    let mut cpus = 0;+    for p in &ps {+        assert_eq!(process::wait(p), ExitStatus::Exited(0));+        cpus |= p.ran_on.load(Ordering::Relaxed);+    }+    assert!(cpus.count_ones() >= 2, "3 spin processes ran on CPUs {:#b} only", cpus);+}++// --- locks across CPUs --------------------------------------------------------++static COUNTER: Mutex<u64> = Mutex::new(0);++fn add_a_lot() {+    for _ in 0..50_000 {+        // Read-modify-write under the lock: without it, two CPUs would+        // overwrite each other's updates and the total would come up short.+        let mut c = COUNTER.lock();+        *c = core::hint::black_box(*c) + 1;+    }+}++#[test_case]+fn a_lock_shared_by_four_cpus_loses_no_updates() {+    *COUNTER.lock() = 0;+    let ids: Vec<_> = (0..4).map(|_| thread::spawn("adder", add_a_lot)).collect();+    for id in ids {+        thread::join(id);+    }+    assert_eq!(*COUNTER.lock(), 200_000);+}++// --- sleep -------------------------------------------------------------------++#[test_case]+fn sleep_lasts_as_long_as_asked() {+    let start = ticks();+    thread::sleep_ticks(10);+    let slept = ticks() - start;+    assert!((10..=12).contains(&slept), "asked for 10 ticks, slept {}", slept);+}++static WOKE: Mutex<Vec<u64>> = Mutex::new(Vec::new());++fn sleeper(n: u64) {+    thread::sleep_ticks(n);+    without_interrupts(|| WOKE.lock().push(n));+}+fn sleep_9() {+    sleeper(9)+}+fn sleep_3() {+    sleeper(3)+}+fn sleep_6() {+    sleeper(6)+}++#[test_case]+fn sleepers_wake_in_deadline_order() {+    WOKE.lock().clear();+    let ids = [thread::spawn("s9", sleep_9), thread::spawn("s3", sleep_3), thread::spawn("s6", sleep_6)];+    for id in ids {+        thread::join(id);+    }+    assert_eq!(*WOKE.lock(), [3, 6, 9]);+}++#[test_case]+fn sleeping_threads_use_no_cpu() {+    // Four threads asleep (plus us): every CPU should be idle nearly all+    // the time, because sleepers are off the run queue, not polling.+    let ids: Vec<_> = (0..4).map(|_| thread::spawn("s9", sleep_9)).collect();+    thread::sleep_ticks(1);+    let before = cpu_ticks();+    thread::sleep_ticks(6);+    let after = cpu_ticks();+    for id in ids {+        thread::join(id);+    }+    let total: u64 = (0..4).map(|c| after[c].0 - before[c].0).sum();+    let busy: u64 = (0..4).map(|c| after[c].1 - before[c].1).sum();+    assert!(total > 0);+    assert!(busy * 10 <= total, "{} of {} ticks busy while everyone slept", busy, total);+}++// --- wait queues ---------------------------------------------------------------++static TURN: AtomicU64 = AtomicU64::new(0);+static TURN_CHANGED: WaitQueue = WaitQueue::new();+const ROUNDS: u64 = 500;++/// Take a turn whenever TURN has parity `mine`, ROUNDS times.+fn take_turns(mine: u64) {+    for _ in 0..ROUNDS {+        TURN_CHANGED.wait_until(|| (TURN.load(Ordering::SeqCst) % 2 == mine).then_some(()));+        TURN.fetch_add(1, Ordering::SeqCst);+        TURN_CHANGED.wake_all();+    }+}+fn ping() {+    take_turns(0)+}+fn pong() {+    take_turns(1)+}++#[test_case]+fn wait_queue_ping_pong_between_cpus() {+    TURN.store(0, Ordering::SeqCst);+    let a = thread::spawn("ping", ping);+    let b = thread::spawn("pong", pong);+    thread::join(a);+    thread::join(b);+    assert_eq!(TURN.load(Ordering::SeqCst), 2 * ROUNDS);+}++// --- no leaks --------------------------------------------------------------------++fn nothing() {}++/// Spawn 8 do-nothing threads at once and join them.+fn spawn_and_join_8() {+    let ids: Vec<_> = (0..8).map(|_| thread::spawn("nop", nothing)).collect();+    for id in ids {+        thread::join(id);+    }+}++#[test_case]+fn spawning_and_exiting_threads_leaks_no_frames() {+    spawn_and_join_8(); // warm up: the first time may add stack page tables+    let frames = memory::frames_in_use();+    let threads = thread::thread_count();+    let stacks = stack::live_stacks();+    for _ in 0..25 {+        spawn_and_join_8();+    }+    assert_eq!(memory::frames_in_use(), frames, "frames leaked by 200 threads");+    assert_eq!(thread::thread_count(), threads);+    assert_eq!(stack::live_stacks(), stacks);+}++/// Run 4 `child 0` processes at once and wait for all of them.+fn run_4_children() {+    let ps: Vec<_> = (0..4).map(|_| run("child", &["0"])).collect();+    for p in ps {+        assert_eq!(process::wait(&p), ExitStatus::Exited(0));+    } // each Arc dropped here: the last reference, so the memory is freed+}++#[test_case]+fn spawning_and_exiting_processes_leaks_no_frames() {+    run_4_children(); // warm up (file caches, stack slots)+    let frames = memory::frames_in_use();+    let processes = process::count();+    for _ in 0..10 {+        run_4_children();+    }+    assert_eq!(process::count(), processes);+    assert_eq!(memory::frames_in_use(), frames, "frames leaked by 40 processes");+}++#[panic_handler]+fn panic(info: &PanicInfo) -> ! {+    rustbelt::test_panic_handler(info)+}

@@ -9,20 +9,15 @@ use bootloader::{BootInfo, entry_point}; use core::panic::PanicInfo; use core::sync::atomic::{AtomicBool, AtomicU64, Ordering};-use rustbelt::allocator; use rustbelt::interrupts::TICKS;-use rustbelt::memory::{self, BootInfoFrameAllocator}; use rustbelt::thread;-use x86_64::VirtAddr;  entry_point!(main);  fn main(boot_info: &'static BootInfo) -> ! {     rustbelt::init();-    let phys_mem_offset = VirtAddr::new(boot_info.physical_memory_offset);-    let mut mapper = unsafe { memory::init(phys_mem_offset) };-    let mut frame_allocator = unsafe { BootInfoFrameAllocator::init(&boot_info.memory_map) };-    allocator::init_heap(&mut mapper, &mut frame_allocator).expect("heap initialization failed");+    rustbelt::init_memory(boot_info);+    rustbelt::init_smp();      test_main();     rustbelt::hlt_loop();

@@ -27,6 +27,7 @@ entry_point!(main); fn main(boot_info: &'static BootInfo) -> ! {     rustbelt::init();     rustbelt::init_memory(boot_info);+    rustbelt::init_smp(); // all 4 CPUs run threads     fs::init();     rustbelt::init_storage();     test_main();

@@ -19,6 +19,7 @@ entry_point!(main); fn main(boot_info: &'static BootInfo) -> ! {     rustbelt::init();     rustbelt::init_memory(boot_info);+    rustbelt::init_smp(); // all 4 CPUs run threads     fs::init();      test_main();

@@ -0,0 +1,29 @@+//! A buggy program: a function that calls itself forever. Every call needs+//! a little more stack, so the stack grows down until it runs into the+//! *guard zone* below it, which is never mapped. The CPU raises a page+//! fault there and the kernel kills just this program, saying+//! "killed: stack overflow", instead of letting the stack quietly grow over+//! other memory.+#![no_std]+#![no_main]++#[macro_use]+#[path = "ulib.rs"]+mod ulib;++/// Uses 1 KiB of stack per call, then calls itself again.+fn recurse(depth: u64) -> u64 {+    let mut buf = [0u8; 1024];+    // `black_box` stops the compiler from removing the array or turning+    // the recursion into a loop.+    buf[0] = depth as u8;+    let buf = core::hint::black_box(buf);+    recurse(depth + 1) + buf[0] as u64+}++fn main() -> i64 {+    println!("[overflow] pid {}: recursing until the stack runs out...", ulib::getpid());+    let n = recurse(0);+    println!("[overflow] returned {} -- this must never be printed!", n);+    1+}

@@ -36,8 +36,8 @@ const HELP: &str = "\ Rustbelt shell commands   builtins   cd [dir]  pwd  echo [text]  ver  drives  alias [name=value]              history  jobs  wait [pid]  cls / clear  help  exit [code]-  programs   ls [-l]  cat  wc  grep [-i] [-v]  mkdir  rm  touch  cp  sleep-  (bin dirs) hello  demo  evil  spin  child  sh+  programs   ls [-l]  cat  wc  grep [-i] [-v]  mkdir  rm  touch  cp  sleep  sysinfo+  (bin dirs) hello  demo  evil  spin [rounds]  overflow  child  sh   run [--trace] prog args   run a program (--trace prints its syscalls)   start prog args           run it in the background (or: prog args &)   a | b | c                 pipes;  > file  >> file  < file  redirection

@@ -1,5 +1,6 @@ //! A CPU hog: busy-loops without ever making a syscall in between. The timer //! interrupt still preempts it, so everything else keeps running.+//! `spin` does 3 rounds of 30 million additions; `spin <n>` does n rounds. #![no_std] #![no_main] @@ -9,8 +10,9 @@ mod ulib;  fn main() -> i64 {     println!("[spin] pid {}: spinning without syscalls...", ulib::getpid());+    let rounds: u64 = ulib::arg(1).and_then(|a| a.parse().ok()).unwrap_or(3);     let mut count: u64 = 0;-    for round in 1..=3 {+    for round in 1..=rounds {         for _ in 0..30_000_000 {             count = core::hint::black_box(count + 1);         }

@@ -0,0 +1,48 @@+//! `sysinfo`: how many CPUs there are, how busy each one is, and how many+//! threads, processes and memory frames are in use.+//!+//! The load is measured, not guessed: each CPU counts its ~10 ms timer+//! ticks, and how many of them interrupted real work rather than its idle+//! thread. We read those counters twice, a quarter of a second apart; busy ticks+//! divided by all ticks in between is that CPU's load.+#![no_std]+#![no_main]++#[macro_use]+#[path = "ulib.rs"]+mod ulib;++fn main() -> i64 {+    let before = ulib::sysinfo();+    ulib::sleep_ms(250); // sleeping, not spinning: we don't count as load+    let after = ulib::sysinfo();++    let seconds = after.uptime_ticks * 10 / 182; // the PIT ticks ~18.2 times a second+    println!(+        "{} CPU{} online (this program ran on CPU {}), up {} s",+        after.cpus,+        if after.cpus == 1 { "" } else { "s" },+        after.this_cpu,+        seconds+    );+    println!(+        "{} threads, {} processes, {} KiB of memory frames in use",+        after.threads,+        after.processes,+        after.frames_in_use * 4+    );+    println!("CPU load over the last quarter second:");+    for cpu in 0..(after.cpus as usize).min(16) {+        let ticks = after.ticks[cpu] - before.ticks[cpu];+        let busy = after.busy_ticks[cpu] - before.busy_ticks[cpu];+        let percent = if ticks == 0 { 0 } else { busy * 100 / ticks };+        // A 20-character bar: one '#' per 5%.+        let mut bar = [b'.'; 20];+        for c in bar.iter_mut().take((percent / 5) as usize) {+            *c = b'#';+        }+        let bar = core::str::from_utf8(&bar).unwrap_or("");+        println!("  CPU{} [{}] {:3}%", cpu, bar, percent);+    }+    0+}

@@ -41,6 +41,7 @@ pub const SYS_READLINK: u64 = 89; pub const SYS_GETDENTS64: u64 = 217; pub const SYS_SPAWN: u64 = 500; pub const SYS_MOUNTS: u64 = 501;+pub const SYS_SYSINFO: u64 = 502;  pub const O_CREAT: u64 = 0o100; pub const O_TRUNC: u64 = 0o1000;@@ -199,6 +200,28 @@ pub fn uname(buf: &mut [u8; 6 * 65]) -> [&str; 6] { pub fn mounts(buf: &mut [u8]) -> i64 {     syscall3(SYS_MOUNTS, buf.as_mut_ptr() as u64, buf.len() as u64, 0) }+/// What `sysinfo` reports (same layout as the kernel's syscall::SysInfo).+/// `ticks[cpu]` counts that CPU's ~10 ms timer ticks, `busy_ticks[cpu]` the+/// ones where it was running a real thread rather than idling: sample twice+/// and subtract to get a CPU's load.+#[repr(C)]+#[derive(Clone, Copy, Default)]+pub struct SysInfo {+    pub cpus: u64,+    pub this_cpu: u64,+    pub threads: u64,+    pub processes: u64,+    pub frames_in_use: u64,+    pub uptime_ticks: u64,+    pub ticks: [u64; 16],+    pub busy_ticks: [u64; 16],+}+/// Ask the kernel for CPU, thread, process and memory counts.+pub fn sysinfo() -> SysInfo {+    let mut info = SysInfo::default();+    syscall3(SYS_SYSINFO, &mut info as *mut SysInfo as u64, 0, 0);+    info+} /// Current directory into `buf`; returns its length (or -errno). pub fn getcwd(buf: &mut [u8]) -> i64 {     syscall3(SYS_GETCWD, buf.as_mut_ptr() as u64, buf.len() as u64, 0)@@ -355,10 +378,54 @@ impl fmt::Write for Stdout {     } } +/// Collects the pieces of one `print!`/`println!` so the whole line goes to+/// the kernel in a single `write`. With several CPUs, two programs printing+/// at the same moment would otherwise have their pieces mixed up mid-line.+/// Flushes when full and when dropped (at the end of the macro).+pub struct LineWriter {+    fd: i64,+    buf: [u8; 256],+    len: usize,+}++impl LineWriter {+    pub fn new(fd: i64) -> LineWriter {+        LineWriter { fd, buf: [0; 256], len: 0 }+    }+    pub fn flush(&mut self) {+        if self.len > 0 {+            write_all(self.fd, &self.buf[..self.len]);+            self.len = 0;+        }+    }+}++impl fmt::Write for LineWriter {+    fn write_str(&mut self, s: &str) -> fmt::Result {+        if self.len + s.len() > self.buf.len() {+            self.flush();+        }+        if s.len() > self.buf.len() {+            write_all(self.fd, s.as_bytes()); // too big to buffer: send as is+        } else {+            self.buf[self.len..self.len + s.len()].copy_from_slice(s.as_bytes());+            self.len += s.len();+        }+        Ok(())+    }+}++impl Drop for LineWriter {+    fn drop(&mut self) {+        self.flush();+    }+}+ macro_rules! print {     ($($arg:tt)*) => {{         use core::fmt::Write;-        let _ = write!($crate::ulib::Stdout, $($arg)*);+        let mut out = $crate::ulib::LineWriter::new(1);+        let _ = write!(out, $($arg)*);     }}; } @@ -384,14 +451,16 @@ impl fmt::Write for Fd { macro_rules! eprintln {     ($($arg:tt)*) => {{         use core::fmt::Write;-        let _ = writeln!($crate::ulib::Stderr, $($arg)*);+        let mut out = $crate::ulib::LineWriter::new(2);+        let _ = writeln!(out, $($arg)*);     }}; }  macro_rules! println {     ($($arg:tt)*) => {{         use core::fmt::Write;-        let _ = writeln!($crate::ulib::Stdout, $($arg)*);+        let mut out = $crate::ulib::LineWriter::new(1);+        let _ = writeln!(out, $($arg)*);     }}; } 

@@ -11,6 +11,7 @@     "linker": "rust-lld",     "panic-strategy": "abort",     "disable-redzone": true,+    "stack-probes": { "kind": "inline" },     "features": "-mmx,-sse,+soft-float",     "rustc-abi": "softfloat" }