foundry

# Proposal: split Rustbelt's userland into `rustbelt-os`, then port musl

*Status: draft for Tyler, 2026-10-31. Read-only study of `/workspace/rust-kernel`
at `master` 58fc4fd (Rustbelt 0.15.1), **re-tested on 0.16.0 (63fcb86),
0.17.0 (fc5e78a) and 0.18.0 (dea72d3)**: see §2.6, §3b, §3c and §3d. Nothing in the kernel repo was changed.
`/workspace/rustbelt-os` does not exist yet; creating it is step 1 below, after you approve.*

---

## 0. The short version

- **Today** the kernel repo builds everything. Its `build.rs` compiles the 36
  user programs in `user/`, which are 35 Rust programs and 1 C program. It
  writes them into a FAT32 disk image, `target/disk.img`, under `/bin`. The
  kernel mounts that disk at `/mnt/disk0` and runs `/mnt/disk0/bin/sh`.
- **The idea:** move the programs, the disk-image builder and (new) the C
  library into their own repo, `rustbelt-os`. That repo produces **one disk image**.
  The kernel only boots it. The contract between the two is a short document
  that says what the image must contain.
- **Good news from testing.** I built musl 1.2.5 in a scratch area, compiled C
  programs against it, and booted them on an unmodified 0.15.1 kernel (my own
  clone, my own target dir). A **static musl `printf`/`malloc` hello world
  works today**. So do stdio, files, `opendir`, `fork`+`exec`, `pthread_create`,
  signals and `double` maths. The **dash shell** also runs scripts: variables,
  `$((...))`, functions, pipes, `$(...)`, redirection, `read`, `trap`.
- **Update, 0.17.0:** in a full `--trace` run the only ENOSYS left is
  `mremap` (25). Links, chown, statfs, CPU time, timers, ids and the small
  server socket calls all work, and `rm -r` is fixed (§2.6). The remaining
  gaps are `AF_UNIX`/`socketpair`, `/proc` and two wrong values (§8, K10).
- **Update, 0.18.0:** Unix sockets (stream, datagram, seqpacket; path,
  abstract and autobind names; `socketpair`), fd passing with `SCM_RIGHTS`,
  `SO_PEERCRED`/`SCM_CREDENTIALS`, `/proc/self/fd` and `/proc/self/exe` all
  pass by value, and the timer fixes are confirmed. **Correction:** my
  0.17 "mount point `st_dev`" finding was a bug in my probe, not the kernel
  (§2.6). Two small new issues: a tty fd's `fstat` doesn't match
  `/dev/console`, so `ttyname()` fails, and stream reads stop *before* a
  message carrying fds where Linux reads into it (§8, K11).
- **0.15.1's blocker is gone in 0.16.0.** Edward's stage 16 added
  `rt_sigsuspend` (130), `umask`, the rlimits and `sched_getaffinity`, and
  orphan handling. On 0.16.0 my whole dash suite passes, including `wait`,
  `umask` and `ulimit -n`, and nothing leaks. So dash needs **no more kernel
  work**. What is left for coreutils/busybox is links (`symlink` 88, `link`
  86), `chown` (92–94), `statfs` (137/138), timers and clocks (37, 38, 222,
  229, 98, 100), ids (105, 106, 115, 118) and some socket calls (§2.6).
- **M1 needs no kernel change at all.** QEMU can simply be given the OS
  repo's disk image in place of the kernel's.

---

## 1. Words used in this document

| Term | Meaning |
|---|---|
| **userland / user space** | Everything that is not the kernel: the shell, `ls`, the C library. It runs in *ring 3*, the CPU's unprivileged mode. |
| **libc** | The C standard library (`printf`, `malloc`, `fopen`...). It turns those functions into **syscalls**, the requests a program makes to the kernel. |
| **musl** | A small, clean libc used by Alpine Linux. It is easy to read and easy to link **statically**. |
| **static binary** | A program with the libc copied into the file itself, so it needs no `.so` files and no dynamic loader at run time. Rustbelt has no dynamic loader yet, so static is the only option for now. |
| **ELF** | The executable file format on Linux (and on Rustbelt). `ET_EXEC` is a fixed-address executable; `ET_DYN` is a position-independent one (a shared library, or a PIE). |
| **PIE / static-PIE** | Position-Independent Executable: a program that can be loaded at any address. A *static-PIE* is static **and** position-independent. It fixes up its own addresses at startup (musl's `rcrt1.o` does this). |
| **cross toolchain** | A compiler setup that builds programs for a *different* system than the one it runs on. Here it is Debian's `gcc` aimed at musl and Rustbelt in place of glibc and Linux. |
| **sysroot** | A folder holding the target's headers (`include/`) and libraries (`lib/`). The compiler reads those instead of the host's. |
| **rootfs** | The filesystem mounted at `/`, holding `/bin`, `/etc` and so on. |
| **disk image** | A plain file holding the bytes of a whole virtual disk (partition table + filesystem). QEMU uses it as a disk. |
| **GPT** | The modern partition table format. Each partition has a *type GUID* saying what it is for. |
| **tarball** | A `.tar.gz`/`.tar.xz` archive of source code, as released by a project. |
| **vendoring** | Keeping a copy of a dependency's source (here, the release tarball) inside your own repo, so builds never need the network. |
| **`no_std`** | Rust without its standard library: only `core` (and `alloc` once there is a heap). That is how Rustbelt's Rust programs are built today. |

---

## 2. How things work today (findings)

### 2.1 Repo and build layout

```
rust-kernel/
├── Cargo.toml            one package "rustbelt" (lib + bin), no Cargo workspace; bootloader 0.9 (BIOS)
├── .cargo/config.toml    build-std core/alloc, custom target x86_64-rust_kernel.json, runner scripts/runner.sh
├── rust-toolchain.toml   nightly + rust-src + llvm-tools-preview; also installs x86_64-unknown-none
├── build.rs              ★ builds ALL user programs and ALL disk images (details below)
├── build/exfat_populate.rs  build.rs's own exFAT writer
├── src/                  the kernel
├── user/                 ★ the userland: 35 *.rs programs + ulib.rs + shparse.rs + rawabi.c + link.ld
├── disk/                 files for the FAT32 disk (readme, notes/, scripts/selftest.sh)
├── disk2/ disk3/ disk4/  files for the ext2 / exFAT / btrfs test disks
├── tests/                cargo test: each test boots the kernel in QEMU
└── scripts/              run-headless.sh, screenshot.sh, *-check.sh, http-reply.sh, hvc0-session.txt
```

There is no Makefile or xtask. Everything goes through `cargo bootimage` /
`cargo run` / `cargo test` plus the shell scripts.

### 2.2 How user programs are compiled, stored and loaded

1. **Rust programs** (`build.rs` → `build_program`). Each `user/X.rs` is a
   single file compiled by plain `rustc` for the built-in
   `x86_64-unknown-none` target. That target ships a precompiled
   `core`/`alloc`, so it needs no build-std and uses soft-float. The flags are
   `panic=abort`, `opt-level=2`, `relocation-model=static`,
   `code-model=large` and `-T user/link.ld`. Each program pulls in the
   tiny runtime with `#[path = "ulib.rs"] mod ulib;`. That runtime holds the
   syscall wrappers, `println!`, a `brk` heap and `_start`.
2. **C program** (`build_c_program`). There is one, `user/rawabi.c`, built
   with the host's `gcc -static -nostdlib -fpie -no-pie
   -Wl,-Ttext-segment=0x8000400000`. That means no libc at all, just raw
   Linux syscalls.
3. **Link address.** User code starts at **`0x80_0040_0000`**
   (`USER_START = 0x80_0000_0000` in `src/memory.rs`). That is above 4 GiB,
   where normal Linux binaries (at `0x400000`) are not allowed.
4. **Storage.** The programs used to be embedded with `include_bytes!`, but
   since stage 6 they **live on a disk image**. `build.rs` makes
   `target/disk.img`: 64 MiB, a GPT from `sfdisk`, one partition at LBA 2048,
   **FAT32** from `mkfs.vfat -F 32 -s 1 -n TYLERDISK`, filled by mtools
   `mmd`/`mcopy` (no root needed). `disk/` goes to the root of the volume and
   the programs go to `/bin`.
5. **Other disks.** `build.rs` also makes the ext2 (`mke2fs -d`), exFAT
   (`mkfs.exfat` + its own populator) and btrfs (`mkfs.btrfs --rootdir`)
   images. These are **kernel filesystem-driver test fixtures**, not userland.
6. **No initrd, no cpio.** There is no initial ramdisk. `/` is a **ramfs**
   built in code (`fs::init`: `/hello.txt`, `/etc/motd`, `/tmp`, `/mnt`,
   `/dev`). Each disk is probed by magic number (ext2 → exFAT → btrfs → FAT)
   and mounted at `/mnt/diskN` in PCI order.
7. **ELF loader** (`src/elf.rs`). It accepts only static **`ET_EXEC`** files
   whose segments sit inside the user range. It refuses two segments with
   different permissions on the same page. `execve` builds Linux's
   argv/envp/auxv stack, including `AT_PHDR`, `AT_RANDOM`, `AT_EXECFN` and
   `AT_SECURE=0`, with no vDSO.

### 2.3 How the kernel finds "init"

There is no init process yet. In `src/main.rs`, a **kernel thread**
(`shell_supervisor`) spawns the hard-coded `SHELL = "/mnt/disk0/bin/sh"`,
waits for it, and starts a new one when it exits. With `--features headless`
it passes `/mnt/disk0/scripts/selftest.sh` as an argument instead. When the
shell exits, it reports the memory frames still in use and exits QEMU via
`isa-debug-exit` (33 = pass). The default environment is
`PATH=/bin:/mnt/disk0/bin HOME=/ TERM=linux` (`src/process.rs`).

### 2.4 The shell

`user/sh.rs` (786 lines) and `user/shparse.rs` (236 lines, pure functions)
are written in **`no_std` Rust**. The shell is a hybrid:

- **Unix side:** `cd`/`pwd`, pipes, `<` `>` `>>`, quotes, `&`, job control
  (process groups, Ctrl+C/Ctrl+Z, `jobs`/`fg`/`bg`/`kill`), `history`,
  `alias`, PATH lookup, and scripts (`sh file`, or commands on stdin).
  Programs are started with `fork` + `execve`.
- **DOS side:** `run [--trace] prog` (`--trace` is strace-like), `start
  prog` (runs it in the background), aliases `dir`=`ls`, `type`=`cat`,
  `del`=`rm`, `md`=`mkdir`, `copy`=`cp`, plus `cls`, `ver` and `drives`.
- **Drive letters** are shell syntax only. `A:` = `/`, and `B:`, `C:`...
  map to the `/mnt/diskN` mounts in mount order (B: FAT32, C: ext2, D: exFAT,
  E: btrfs), read through Rustbelt's own `mounts` syscall (501).
  `shparse::translate` rewrites words; the prompt shows `B:/bin`.
- There are no variables or `$?` yet.
- `tests/shell.rs` compiles `shparse.rs` straight into a kernel test with
  `#[path = "../user/shparse.rs"]`.

### 2.5 Coupling that the split has to handle

- **The kernel's own `cargo test` suite needs the userland.** The tests in
  `tests/*.rs` run `sh`, `ls`, `cat`, `wc`, `grep`, `tile`, `ping`,
  `fetch`, `random`, `child`, `forktest`... from `/mnt/disk0/bin`.
- **`run-headless.sh`** checks about 75 lines of shell output, including
  "`dir bin | wc -l` → 36". That is really an *OS* test.
- **Three paths are hard-coded in the kernel:** `SHELL`, `SELFTEST` and the
  `PATH` default.

### 2.6 Syscalls: what is there and what is missing (updated for 0.18.0)

*Re-tested on **Rustbelt 0.18.0 (master dea72d3)**, 2026-10-31, from real
musl 1.2.5 programs: `mprobe2`, now 219 value checks (group D is new, for
stage 18); `stdev`, a mount-point `st_dev` check with one `stat` per
statement; `ttyp`, a tty check; `rmr`, a naive `rm -r`; `talarm`. dash
0.5.12 was tested too. Every new check was first calibrated on the box's
Linux host (as a normal user, never root). See §3d. Older results are in
§3, §3b and §3c.*

Rustbelt has the Linux syscalls by Linux number, plus its own 500–505
(`spawn`, `mounts`, `rb_sysinfo` 502, `resolve`, `netinfo`, `trace_me`).
Anything it doesn't know returns `-ENOSYS` (errno 38), and `run --trace`
prints it as `syscall_N(...)`.

**The headline: in a full `--trace` run of mprobe2 the only ENOSYS calls
are `syscall_25` (`mremap`) and `syscall_283` (`timerfd_create`, which the
probe calls on purpose as a known gap).** musl's `realloc` copes with the
missing `mremap`.

**New in 0.18.0, checked by value:**

| Area | Calls (x86_64 number) | Checked |
|---|---|---|
| Unix sockets | `socket` 41 / `socketpair` 53 with `AF_UNIX` and `SOCK_STREAM`, `SOCK_DGRAM`, `SOCK_SEQPACKET`; `bind` 49, `listen` 50, `accept` 43, `connect` 42, `sendto` 44 / `recvfrom` 45 | data both ways; a stream merges two writes; datagram and seqpacket keep message borders (3, 5, then an empty datagram); a short buffer gives `MSG_TRUNC`, and `recv(MSG_TRUNC)` returns the real length; `fstat` says `S_IFSOCK`; `SO_TYPE`, `SO_DOMAIN`; `getpeername` on a pair is the family alone |
| names | path, abstract, autobind | a path `bind` makes an `S_IFSOCK` file on ramfs and ext2 and `EPERM` on FAT; `open` on it is `ENXIO`; a second `bind` is `EADDRINUSE`; the file stays after close; `connect` after `unlink` is `ENOENT`; an abstract name has the leading NUL, is freed on close, and can be bound again; autobind gives an 8-byte name (`@00000`); `recvfrom` shows the sender's autobind name, and a reply to it arrives |
| connections | the "embryo" | `connect` and `send` work before `accept`; `getsockname` and `getpeername` show the path |
| EOF / HUP | `shutdown` 48, `poll` 7, `epoll_pwait` 281 | after the peer's `SHUT_WR`: `POLLIN|POLLRDHUP` (0x2001) with no `POLLHUP`, `EPOLLRDHUP`, then `read` returns 0; the half-closed side can still receive; after the peer closes: `POLLHUP` (0x2011), `EPOLLHUP`, `EPIPE` plus SIGPIPE, and `MSG_NOSIGNAL` gives `EPIPE` without the signal. All the same values as Linux |
| fd passing | `sendmsg` 46 / `recvmsg` 47 with `SCM_RIGHTS` | a passed pipe end can be written through, and the pipe sees EOF when it is closed (no hidden reference); a passed file shares its offset; passing to a `fork`ed child works; `MSG_CMSG_CLOEXEC` sets `FD_CLOEXEC` (and it's clear without it); 3 fds into `CMSG_SPACE(4)` give 2 fds plus `MSG_CTRUNC`, and the third is closed; no control buffer gives `MSG_CTRUNC` and closes the fd; 254 fds is `EINVAL` and 253 is accepted; 253 fds into a 64-slot fd table installs 57 with `MSG_CTRUNC`, and every extra reference is closed; a socket fd can be passed too |
| in-flight fds | — | **a pipe end that is sent but never received keeps the pipe open while it is queued, and is closed when the receiving socket closes** (the reader gets EOF), for stream and datagram |
| credentials | `SO_PEERCRED`, `SO_PASSCRED`, `SCM_CREDENTIALS` | `SO_PEERCRED` gives your own pid/uid/gid on a pair, and the *child's* pid when a forked child connects; with `SO_PASSCRED` every message carries `SCM_CREDENTIALS`; sending your own credentials explicitly works |
| backpressure | `SO_SNDBUF`/`SO_RCVBUF` | a nonblocking writer stops with `EAGAIN` at 212992 bytes (Linux: 180224, which also counts per-buffer overhead) and `POLLOUT` goes away; after draining `POLLOUT` is back; `SO_SNDBUF` 4096 reads back as 8192; a blocking writer pushes 1 MiB through a slow reader |
| /proc | `/proc` (magic `0x9fa0`), `/proc/self/exe`, `/proc/self/fd` | `/proc/self/exe` is `/mnt/disk0/bin/mprobe2`, also in a child; the `/proc/self/fd` listing matches the open fds exactly (plus the directory's own fd); `readlink` gives the path for a file, `pipe:[4011]` (same number as `st_ino`, the same for both ends), `socket:[1032]` (a different inode for each end), `anon_inode:[eventpoll]`, and `/dev/console` for 0–2; opening `/proc/self/fd/N` gives a fresh offset for a file, the same pipe for a pipe, and `ENXIO` for a socket; a closed fd gives `ENOENT`; `realpath` works |
| timers (fixes) | `timer_getoverrun` 225 | 0 before the first delivery; 0 while the next signal is pending (the last delivery's value, like Linux); after delivery 26 for a window that actually lasted 0.541 s (27 expirations − 1; Linux gives 24 for exactly 0.500 s); a second 0.1 s round gives 4, so the count restarts at each delivery |
| frame report | — | 183 before and 183 after the shell in **every** 0.18 boot, including the ones with `alarm`: timerd now starts at boot |

**Working, and checked by value (stage 16 + stage 17, re-checked on 0.18.0):**


| Area | Calls (x86_64 number) | Checked |
|---|---|---|
| signals | `rt_sigsuspend` 130, `rt_sigpending` 127, `rt_sigtimedwait` 128, `rt_sigqueueinfo` 129, `sigaltstack` 131 | dash `wait`/`wait $!` statuses; `SA_ONSTACK` runs on the alt stack; `--trace` now shows `= -4 EINTR` (no more -514) |
| umask / limits / CPUs | `umask` 95, `getrlimit` 97, `setrlimit` 160, `prlimit64` 302, `sched_getaffinity` 204 | `umask` 0022; `ulimit -n` 64; `EMFILE` under a lowered limit; nproc 4 |
| file size / mode / times | `truncate` 76, `ftruncate` 77, `fsync` 74, `fdatasync` 75, `chmod` 90, `fchmod` 91, `fchmodat` 268, `utimensat` 280 | exact read-back on ramfs |
| locks | `fcntl` `F_SETLK`/`F_GETLK`, `F_GETFL`/`F_SETFL`, **`flock` 73** | `flock`: a second `open()` gets `EWOULDBLOCK`; the lock is shared through a `fork`ed fd; `LOCK_UN` releases it |
| links | **`link` 86, `linkat` 265, `symlink` 88, `symlinkat` 266** | `st_nlink` 2 with the same inode; writing through one name shows in the other; dangling symlinks; `AT_SYMLINK_FOLLOW`; `lstat`/`readlink`; `EXDEV` across mounts; `EPERM` on FAT |
| rename / readdir | `rename` moves entries, `getdents64` | rename of a full directory; **`rm -r` with unlink-while-readdir removes all 311 entries on ramfs *and* ext2**; `d_ino == st_ino`; nftw `FTW_DEPTH` delete ok |
| owners | **`chown` 92, `fchown` 93, `lchown` 94, `fchownat` 260** | uid/gid read back; `-1` keeps a field; `lchown` changes the link, not the target; FAT accepts root:root and gives `EPERM` for 5:5; new files are 0:0 |
| fs info | **`statfs` 137, `fstatfs` 138, `statx` 332** | ramfs `0x858458f6`, FAT `0x4d44`, pipe `PIPEFS`; disk0 is 124990 × 512 B ≈ 61 MiB with free ≤ total; `statvfs` ok; `statx` size and mode |
| CPU time | **`times` 100, `getrusage` 98, `clock_getres` 229, `CLOCK_PROCESS_CPUTIME_ID`, `CLOCK_THREAD_CPUTIME_ID`** | a 0.5 s busy loop adds 50 ticks to `times()` and about 500000 to `clock()`; a child's CPU time shows up after `wait`; resolution 1 µs (wall clocks) and 10 ms (CPU clocks) |
| system info | **Linux `sysinfo` 99** | uptime, 120 MB total RAM, free ≤ total, procs, loads |
| timers | **`alarm` 37, `setitimer`/`getitimer` 38/36 (`ITIMER_REAL`), `timer_create`…`timer_delete` 222–226** | `alarm(1)` fires after 1.088 s; 3 × 100 ms periodic ticks take 0.372 s; a POSIX timer fires after 0.258 s for 200 ms with `SI_TIMER` and the right `si_value`; `alarm(0)` returns the seconds left |
| ids / priority / host | **`setuid` 105, `setgid` 106, `setresuid` 117, `getresuid` 118, `getgroups`/`setgroups` 115/116, `getpriority`/`setpriority` 140/141, `sethostname` 170** | the saved-id rules hold (`setresuid(1000,1000,0)` → `setuid(0)` works; after `setuid(1000)` the switch back is `EPERM`); a non-root process lowering its nice value gets `EACCES`; `uname` shows the new host name |
| sockets | **`getsockname` 51, `getpeername` 52, `shutdown` 48, `getsockopt` 55**, more `setsockopt` 54 options | `SO_REUSEADDR`, `TCP_NODELAY` and `SO_KEEPALIVE` accepted and read back; `SO_TYPE` and `SO_ACCEPTCONN`; an unknown option gives `ENOPROTOOPT`; a loopback connect/accept shows matching ports on both sides; after `SHUT_WR` the peer reads EOF but the half-closed side can still read, and a write gives `EPIPE`; a refused connect reports `SO_ERROR` = `ECONNREFUSED` once, then 0 |
| vector / copy I/O | **`preadv` 295, `pwritev` 296, `sendfile` 40** | offsets honoured and the file position unchanged; `sendfile` works file→file (with and without an offset) and file→pipe |
| misc | `madvise` 28, `set_robust_list` 273, `prctl` 157 | — |

**Still missing (known, Edward's list):**

| Gap | Effect |
|---|---|
| `mremap` 25 (ENOSYS) | big `realloc` copies instead; slower but correct |
| `AF_INET6` (EAFNOSUPPORT) | IPv4 only |
| `ITIMER_VIRTUAL`/`ITIMER_PROF`, `timer_create` on CPU clocks (EINVAL) | profilers, `ulimit -t`-style CPU watchdogs |
| `timerfd_*` 283/286/287 | event-loop programs (not used by dash/busybox core) |
| `/proc/<pid>` (ENOENT), `/proc/mounts`, `/proc/meminfo`, `/proc/self/cwd` …; `/proc/self` is a directory, not a symlink | `ps`, `top`, `pidof`, `mount` (listing), `free`, `df` without arguments, `pwdx` |
| no garbage collector for Unix socket cycles (Edward's note; not tested, since it leaks by design) | only programs that send a socket through itself; rare |
| `sendmsg`/`recvmsg` control messages on INET sockets | `IP_PKTINFO`-style servers |
| permissions/ids **not enforced**; nice ignored by the scheduler | fine for a single-user hobby OS |
| no `/etc/passwd` → `getpwuid(0)` fails | **OS-repo job** (`rootfs/etc/passwd`, `group`) |

**Retraction: the 0.17.0 "mount point `st_dev`" finding was my probe's bug.**
Edward was right. My 0.17 probe printed the value like this:

```c
printf("... /tmp=%lx /mnt/disk0=%lx /=%lx\n", (long)st.st_dev, (long)st2.st_dev,
       (stat("/", &st), (long)st.st_dev));
```

The third argument calls `stat("/", &st)` and overwrites `st`, and C does
not say in which order a function's arguments are evaluated. gcc evaluated
the third one first, so "/tmp=1" was really `/`'s device. (A three-line
test with the same pattern prints the overwritten value with gcc at both
`-O0` and `-O2`: `exp18/order.c`.) I checked it again properly, with one
`stat`, `lstat` or `fstat` per statement and each result in its own struct
(`exp18/stdev.c`). I ran this **on 0.17.0 as well as 0.18.0**, in a
separate worktree at fc5e78a, to be sure the kernel had always been right:

| path | 0.17.0 `st_dev` | 0.18.0 `st_dev` |
|---|---|---|
| `/`, `/hello.txt` | 1 | 1 |
| `/tmp`, `/tmp/`, `/tmp/.`, a file in `/tmp` | **2** | **2** |
| `/dev` | 3 | 3 |
| `/proc` | — | 4 (new) |
| `/mnt/disk0`, `/mnt/disk0/bin` | 4 | 5 |
| `/mnt/disk1` (ext2) | 5 | 6 |

`stat`, `lstat` and `fstat` agree on every row. The mount point reports
the mounted file system, as on Linux. The fixed `mprobe2` now checks this
with separate statements. (The numbers after `/dev` moved up by one
because `/proc` is now mounted first; that is expected.)

**Wrong values on 0.18.0 (new; details in §8, K11):**

1. **`ttyname(0)` fails with `ENODEV`, so `tty` would too.** musl's
   `ttyname` reads `/proc/self/fd/0` (`/dev/console`), then checks that
   `stat("/dev/console")` and `fstat(0)` give the same `st_dev`/`st_ino`.
   They don't: `fstat(0)` says dev 1, ino 75059993789296, mode 020620;
   `stat("/dev/console")` says dev 3, ino 11760308497919119911, mode
   020666. Each new `open("/dev/console")` also gets yet another inode,
   and `st_rdev` is 0 for all of them (Linux: 5,1). `isatty(0)` and
   `tcgetattr` work, and `/dev/null` is consistent (dev 6, ino 4, rdev
   1,3 for both `fstat` and `stat`). So only the console's open file
   reports the wrong identity. (`fstat` 5 versus `stat` 4 / `newfstatat` 262.)
2. **A stream read stops *before* a message carrying fds; Linux reads
   into it.** Take a stream socket with `"AB"` written plainly and then
   `"CD"` sent with one fd. On Linux, both `read` and `recvmsg` return
   `"ABCD"`. `recvmsg` also returns the fd; a plain `read` closes it.
   On 0.18.0 both return `"AB"`, and the next `recvmsg` returns `"CD"`
   with the fd. The README describes the 0.18 behaviour on purpose, but
   it also says "all Linux's behaviour". Nothing loses data and no fd
   leaks; real programs (libwayland, D-Bus) loop on `recvmsg`, so this
   is low priority. A program that expects the fd on the first
   `recvmsg` would see it one call late. (`read` 0, `recvmsg` 47.)
3. **Not a bug, noted:** the autobind name is `@00000` (Linux picks a
   varied number); `/proc/self/cwd` doesn't exist (`getcwd` works).
4. **Time accounting note, not a bug** (from 0.17): with no vDSO, a loop
   calling `clock_gettime` spends most of its time in the kernel.

*Fixed in 0.18.0 and confirmed:* the 0.17 overrun undercount, and the
16-frame timerd one-off (both above).

### 2.7 Toolchain on this box

| Tool | Status |
|---|---|
| rustup + nightly 1.101 (2026-10-29) | ✅ targets `x86_64-unknown-linux-gnu`, `x86_64-unknown-none`; `rust-lld` and llvm tools present |
| `bootimage` 0.10.5 | ✅ |
| gcc 14.2 (Debian, **default-PIE**), GNU ld 2.44 | ✅ |
| QEMU 10.0.13 | ✅ |
| dosfstools, mtools, sfdisk, e2fsprogs (`mke2fs`, `debugfs`, `e2fsck`), exfatprogs, btrfs-progs | ✅ |
| git, curl, wget, python3, patch, xz, perl, gpg | ✅ |
| **make** | ❌ was missing. **I installed it** (`apt install make`, 4.4.1) to build musl for the experiment below |
| musl-gcc / musl-tools | ❌ not installed (Debian has musl-tools 1.2.5 if wanted; I built musl from source instead) |
| clang, ld.lld, cmake, ninja, meson, autoconf/automake, bison/flex, pkg-config, **bzip2**, cpio | ❌. bzip2 is needed to unpack busybox's `.tar.bz2`; autotools are only needed if we build from git snapshots instead of release tarballs |

`sudo` works without a password, so installing packages is easy when you OK it.

---

## 3. What I tried (in `/workspace/clarke-scratch`, never in the kernel repo)

All of this used a **local `git clone`** of the kernel at
`/workspace/clarke-scratch/rk`, with its `origin` remote removed so a push
is impossible. It has its own `target/`. Arthur's `/workspace/rust-kernel/target`
was never touched.

1. **Built musl 1.2.5** from the official tarball (sha256 `a9a118bb…c75e4`)
   with `./configure --disable-shared` into `clarke-scratch/musl-sysroot`.
   It took about 5 seconds.
2. **Found a link problem with high addresses.** musl's startup code does
   `lea _DYNAMIC(%rip)` on a *weak, undefined* symbol, which means address 0.
   Linked at `0x80_0040_0000`, that relative jump is too far ("relocation
   truncated to fit"), and ld's GOT relaxation fails as well. The stock
   `musl-gcc` wrapper always picks `Scrt1.o`, so it cannot be used as is.
   **The workaround that works** is a 15-line wrapper, `rb-cc`
   (saved at `clarke-scratch/exp/rb-cc`). It uses `crt1.o`, `-fPIE` code,
   `-static -no-pie`, `-Ttext-segment=0x8000400000`,
   `--defsym=_DYNAMIC=0x8000400000`, `--no-relax`, and no gcc
   `crtbegin/crtend`. The result is an ordinary static `ET_EXEC` that runs
   both on Linux and on Rustbelt.
3. **Booted it on Rustbelt** (`run --trace mhello a b`). These are the exact
   syscalls musl's startup + stdio + malloc made, and all succeeded:

   ```
   arch_prctl(ARCH_SET_FS) = 0         <- thread-local storage for the main thread
   set_tid_address(...)    = 2
   uname(...)              = 0
   ioctl(1, TIOCGWINSZ)    = 0         <- stdio checks "is stdout a terminal?" to pick line buffering
   writev(1, ..., 2)       = 43        <- stdio writes buffer + new text in one call
   brk(0), brk(+8K)                    <- mallocng's metadata
   mmap(.., PROT_NONE, MAP_FIXED)      <- guard page
   mmap(0, 100K, RW, ANON) / munmap    <- the big malloc
   exit_group(0)
   ```

4. **`mprobe`, a broader musl test.** These pass on Rustbelt: TIOCGWINSZ,
   `tcgetattr`, `isatty`, 5 MiB `malloc` + 9 MiB `realloc` (musl falls back
   from the missing `mremap` to copy), `fopen`/`fgets`/`fprintf`, `stat`,
   `opendir`/`readdir`, `getcwd`, `clock_gettime`, `gmtime`,
   `signal`+`raise`, `fork`+`execl`+`waitpid`, `pthread_create`/`join`, and
   SSE `double` maths.
   **These fail with ENOSYS:** `getrlimit`, `chmod`, `ftruncate`,
   `utimensat`, `symlink`, `sigpending`, `sigaltstack`.
   `sysconf(_SC_NPROCESSORS_ONLN)` says **1** CPU, not 4, because
   `sched_getaffinity` is missing.
5. **dash 0.5.12** (Debian's release tarball) built with `CC=rb-cc
   ./configure --host=x86_64-linux-musl && make` in about 2 seconds. The
   stripped binary is 180 KB. On Rustbelt, a test script passed:
   variables, arithmetic, `for`, functions, `case`, `$(ls | wc -l)`,
   `2>/dev/null`, `read < file`, `&` + `$!`, `cd`, `test -d/-x`,
   `command -v`, `trap` + `kill -USR1 $$`, and `exit 5` → `[exit code 5]`.
   - **`wait` hangs forever.** dash blocks all signals and calls
     `sigsuspend` to wait for SIGCHLD. That returns ENOSYS at once, so dash
     spins (23 000+ ENOSYS calls in the trace) and the SIGCHLD stays
     blocked. The headless watchdog killed it after 60 s.
   - **`umask`** prints `37777777777` and **`ulimit -n`** prints
     `549755813889`. Both are garbage from the ENOSYS returns.
   - **Leak:** a background `sleep` that nobody waited for left
     **27 frames** in use after the shell exited (157 before, 184 after).
     The kernel does not yet hand orphans to an init process to reap.
   - In the `--trace` output, unknown syscalls show up as `?(...)`
     **without their number**, which makes it harder to spot what is
     missing.

Logs: `clarke-scratch/exp/qemu-musl.log`, `qemu-dash*.log`. The two-file
experiment patch to the *clone's* `build.rs` (an env var that copies extra
binaries into `/bin`) is in `exp/kernel-clone-experiment.diff`.

---

## 3b. Re-test on Rustbelt 0.16.0 (master 63fcb86)

I updated the scratch clone `rk` to 63fcb86 with a one-off `git fetch
/workspace/rust-kernel master`. That is a fetch from a path, so no remote was
added; `git remote -v` is empty. I rebuilt with `CARGO_TARGET_DIR=rk/target`
and booted headless four times. The same musl 1.2.5 sysroot and `rb-cc`
wrapper were used; the musl binaries needed no change.

| Boot | What ran | Result |
|---|---|---|
| `probes` | `mhello`, `mprobe`, `mprobe2` (new: about 80 checks that compare **values**, not just return codes) | exit 33. Startup/stdio/malloc unchanged. All the stage-16 items pass; the remaining failures are exactly the "still missing" table in §2.6 |
| `dash` | dash suite `t2.sh` (about 40 checks), then `sleep 3` so orphans finish | everything correct: `wait` / `wait $!` / `wait` all, `umask`, `ulimit -n` = 64, `ulimit -n 32`, getopts, heredoc, traps, `EXIT` trap, exit status 5. **Frames 157 → 157** |
| `noreap` | the same suite, then the shell exits **at once** while the orphans are still sleeping | frames 157 → 211. This is **not a leak**: the two orphans are still alive when the kernel takes its count. The headless frame report only waits for pid 1, not for orphans that are auto-reaped later. |
| `trace` | `run --trace mprobe2` and `run --trace dash t2.sh` | dash makes **no ENOSYS calls at all**. mprobe2's ENOSYS list is in §2.6: `syscall_25 38 40 48 51 53 55 73 86 88 92 93 94 98 99 100 105 106 115 118 137 138 140 141 170 222 229 295 296` |

Logs are in `clarke-scratch/exp16/` (`qemu-probes.log`, `qemu-dash.log`,
`qemu-noreap.log`, `qemu-trace.log`), with the probe source `mprobe2.c`, the
dash suite `t2.sh`, and `boot.sh`, which rebuilds the disk with a given
self-test and boots it.

## 3c. Re-test on Rustbelt 0.17.0 (master fc5e78a)

Same method as §3b. I fetched once from the local path (no remote left
behind), rebuilt in `rk/target`, and booted headless with disk 0 (FAT32)
plus the kernel's ext2 test image as disk 1 (`/mnt/disk1`, snapshot mode).

| Boot | What ran | Result |
|---|---|---|
| `probes`, `probes2` | `mhello`, extended `mprobe2` (114 checks: group A = stage 16 regression, B = stage 17, C = known gaps) | **108 / 114 pass; the 6 failures are exactly the known gaps** (socketpair/AF_UNIX, AF_INET6, mremap, /proc ×2, /etc/passwd). The new checks found the overrun problem. (They also reported an `st_dev` problem, which was my probe's bug and is retracted in §2.6.) |
| `twice`, `alarm` | `mprobe2` twice; `talarm` alone | 167 → 183 frames each time: the one-off cost of starting timerd, not a leak |
| `dash` | dash suite `t2.sh`, then `rmr.sh` (builds 5 dirs × 60 files + subdirs on `/tmp` and on ext2 `/mnt/disk1`, deletes with a naive `rm -r`) | dash suite unchanged and correct; **rm -r removes all 311 entries on both file systems**, nothing left; frames 167 → 167, even with orphans still running when dash exits |
| `trace` | `run --trace` of `mprobe2`, `rmr` and the dash suite | **only `syscall_25` (mremap) is ENOSYS.** Every other error is one the probe asked for (`EXDEV`, `EPERM` on FAT, `EWOULDBLOCK`, `EPIPE`, `ECONNREFUSED`…). No `-512`/`-514` any more; interrupted calls show `-4 EINTR` |

Files: `clarke-scratch/exp17/` holds `mprobe2.c` (the 0.16 version is kept
as `mprobe2-016.c`), `rmr.c`, `rmr.sh`, `talarm.c`, `boot.sh` and the
`qemu-*.log` files.

## 3d. Re-test on Rustbelt 0.18.0 (master dea72d3)

Same method as §3c: one fetch from the local path (no remote left behind),
a rebuild in `rk/target`, and headless boots with disk 0 (FAT32) plus the
ext2 test image as disk 1. For the `st_dev` question I also built 0.17.0
again in a separate scratch worktree (`clarke-scratch/rk17`, own target).

| Boot | What ran | Result |
|---|---|---|
| `stdev-017` (0.17.0) and `probes` (0.18.0) | `stdev`: `stat`/`lstat`/`fstat` of 11 paths, one call per statement | mount points report the mounted fs on **both** versions → my 0.17 finding retracted (§2.6) |
| `probes`, `probes2` | `ttyp`, then `mprobe2` (219 checks: A = stage 16, B = stage 17 with the fixed `st_dev` block and the new overrun checks, **D = stage 18**: 98 checks for Unix sockets, fd passing, credentials, HUP, backpressure and `/proc/self`; C = known gaps) | **210 / 219 pass.** 6 failures are the known gaps (AF_INET6, mremap 25, `/proc/<pid>`, timerfd 283, `/proc/mounts`, `/etc/passwd`). 3 are the fd-boundary difference (2 checks, plus 1 that follows from them). `ttyname` fails (printed, not counted) |
| `dash` | dash suite `t2.sh`, `rmr.sh` on `/tmp` and ext2, `talarm` | dash output identical to 0.17 (except the program count, 43 → 47); rm -r removed 311, failed 0, on both; `alarm` fires; **frames 183 → 183** |
| `trace` | `run --trace mprobe2` | ENOSYS only for `syscall_25` (mremap) and `syscall_283` (timerfd, called on purpose); 100 distinct calls in use |

On the Linux host (as a normal user) the new group D passes everything
except what the host doesn't have (FAT/ext2 mounts, the exe path,
`/proc/self` being a symlink), so the checks themselves are right.

Files: `clarke-scratch/exp18/` holds `mprobe2.c` + `unix18.c` (the 0.17
version is kept as `mprobe2-017.c`), `stdev.c`, `ttyp.c`, `order.c`,
`boot.sh`, `host.log` and the `qemu-*.log` files.

---

## 4. Proposed layout of `rustbelt-os`

```
rustbelt-os/
├── README.md                 what this is, quick start (make image / make run / make test)
├── Makefile                  ★ the single entry point (targets in §6)
├── config.mk                 KERNEL_DIR, KERNEL_REV, ARCH=x86_64, IMAGE_SIZE, musl/port versions
├── docs/
│   ├── boot-contract.md      the kernel ↔ OS contract (§5), versioned
│   ├── syscall-gaps.md       living list: which program needs which missing call
│   └── porting.md            how to add a C program ("a port")
│
├── rust/                     Rust userland (no_std, x86_64-unknown-none), a Cargo workspace
│   ├── Cargo.toml            [workspace] members = [...]
│   ├── .cargo/config.toml    target = x86_64-unknown-none, rustflags: static, code-model=large, -T link.ld
│   ├── link.ld               moved from kernel user/link.ld
│   ├── rbrt/                 "Rustbelt runtime": today's ulib.rs as a library crate
│   ├── rbsh/                 Tyler's shell: src/main.rs (sh.rs) + src/parse.rs (shparse.rs)
│   │                         └─ the parser's unit tests run on the host with plain `cargo test`
│   ├── coreutils-rs/         src/bin/{ls,cat,wc,grep,mkdir,rm,touch,cp,sleep}.rs
│   ├── net/                  src/bin/{ifconfig,ping,host,fetch,httpd}.rs
│   ├── sys/                  src/bin/{getty,tile,random,sysinfo}.rs
│   └── init/                 (M3) /sbin/init: runs /etc/rc, respawns shells on console + hvc0, reaps orphans
│
├── toolchain/
│   ├── musl/
│   │   ├── musl-1.2.5.tar.gz         vendored release tarball (1.1 MB)
│   │   ├── musl-1.2.5.tar.gz.asc     upstream signature
│   │   ├── SHA256SUMS
│   │   └── patches/                  0001-*.patch, applied in order (empty at first)
│   ├── build-musl.sh                 unpack → patch → configure → make → install into out/sysroot
│   └── rb-cc                         compiler wrapper (gcc + musl sysroot + Rustbelt link flags)
│
├── c/                        small C programs of our own (hello-musl.c, mprobe.c = libc smoke tests)
│
├── ports/                    third-party C programs, one directory each
│   ├── dash/                 dash-0.5.12.tar.gz, SHA256SUMS, patches/, build.sh
│   ├── sbase/                (M4) suckless coreutils
│   └── busybox/              (M5) busybox-1.37.x tarball, busybox.config, patches/
│
├── rootfs/                   files copied into the image as-is
│   ├── readme.txt  notes/    (from kernel disk/)
│   ├── etc/                  motd, passwd, group, hosts, resolv.conf (nameserver 10.0.2.3), profile, rc
│   └── scripts/selftest.sh   the headless self-test (from kernel disk/scripts/)
│
├── scripts/
│   ├── mkimage.sh            staging dir → GPT + FAT32 (v1) or GPT + ext2 (v2) image
│   ├── build-kernel.sh       builds the kernel bootimage from KERNEL_DIR into out/kernel-target
│   ├── run-qemu.sh           interactive boot
│   ├── run-headless.sh       boot + selftest + expected-output check (moved from the kernel)
│   └── compare-image.sh      M1 check: file list + bytes vs the kernel's own disk.img
│
└── out/                      (git-ignored) sysroot/, staging/, disk.img, kernel-target/, logs
```

**Why this shape?**

- `rust/` and `ports/` are separate on purpose. Our Rust programs talk to the
  kernel directly through `rbrt`. Ported C programs go through musl. They
  are built in different ways and fail in different ways.
- One directory per port, holding the tarball, checksum, patches and a short
  `build.sh`, is how Alpine (APKBUILD) and the BSD ports trees do it.
  Adding a program then means "copy a directory and edit it".
- `out/` holds everything generated, so `make clean` is just
  `rm -rf out`.

### Where musl goes: vendored tarball + patches (recommended), not a submodule

| Option | For | Against |
|---|---|---|
| **Vendored release tarball + `patches/` (recommended)** | Works offline, with no remote (you said local only). The exact bytes are pinned by SHA256 and the upstream `.asc` signature. Patches are small readable files. Upgrading means dropping in a new tarball and re-running. | 1.1 MB in git (fine; dash 0.2 MB, busybox about 2.5 MB) |
| git submodule | Full upstream history | A submodule needs a URL to clone from. With local-only rules it would point at another local clone, which is confusing for a beginner and easy to get wrong. Editing musl in place mixes our changes with theirs. |
| `apt install musl-tools` | Zero work | Ties us to Debian's build. Its `musl-gcc` wrapper hard-codes `Scrt1.o` (see §3), and we can't patch it. |

My expectation is that musl will need **no patches at first**. Every fix so
far belongs either in the kernel or in our `rb-cc` flags.

---

## 5. The boot-image contract (how the kernel and the OS connect)

The contract is a short, versioned document (`docs/boot-contract.md`) that
both repos agree on. The kernel never needs to know how the image was built.

### Contract v1: "today, written down" (used for M1–M2, needs no kernel change)

1. The OS image is attached to QEMU as the **first virtio-blk disk** (it
   becomes `/mnt/disk0`, drive B:).
2. The image is **GPT** with one partition starting at LBA 2048, formatted
   **FAT32**. The label `TYLERDISK` is kept for now.
3. It must contain **`/bin/sh`**, and `/scripts/selftest.sh` for headless
   runs. Programs live in `/bin` and are static ELF `ET_EXEC` files linked at
   or above `0x80_0040_0000`.
4. The kernel provides `/` (ramfs), `/tmp`, `/dev` and `/mnt/diskN`. It runs
   `/mnt/disk0/bin/sh` with `PATH=/bin:/mnt/disk0/bin`. In headless builds
   it passes `/mnt/disk0/scripts/selftest.sh` and exits QEMU with the
   shell's status (33 = pass).
5. The kernel's other test disks (ext2/exFAT/btrfs) stay **kernel-owned**.
   The OS run script may attach them after disk 0, built by the kernel into
   the OS repo's own `out/kernel-target`, so B:–E: keep their letters.

With this contract, M1 is just "the OS repo makes `disk.img`, and
`run-qemu.sh` passes it to QEMU in place of the kernel's".

### Contract v2: "a real root filesystem" (from M3, needs Edward's K5)

1. The OS image is a **GPT** disk with a partition whose type GUID is
   **Linux root (x86-64)**, `4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709`.
   This GUID comes from the freedesktop *Discoverable Partitions
   Specification*, the same rule systemd uses. The kernel then does not have
   to guess which disk is "the OS".
2. The partition is formatted **ext2**, built with `mke2fs -d staging/`.
   ext2 is the better fit for a root filesystem:
   - it has Unix permission bits and executable flags,
   - it has **symlinks** (busybox installs its 300 applets as symlinks),
   - names are case-sensitive,
   - Rustbelt's ext2 driver is its most complete read-write one, and the
     host's `e2fsck` can check it.

   FAT32 has none of the first three.
3. The kernel mounts that partition at **`/`**. It keeps a ramfs at `/tmp`
   and devfs at `/dev`, and mounts the other disks at `/mnt/diskN` as today.
4. The kernel runs the first that exists of **`/sbin/init`, `/bin/init`,
   `/bin/sh`** as **pid 1**, in the same order Linux tries. The argument
   vector is `["init"]`; in headless builds it is `["init", "selftest"]`,
   or an env var `RUSTBELT_HEADLESS=1`.
   - The kernel does **not** respawn pid 1. Respawning becomes init's job.
   - In headless builds, the pid-1 exit status ends the QEMU run (33/35 as
     today).
5. **Orphans are handed to pid 1**, which reaps them.
6. **Fallback:** if there is no root partition, the kernel behaves as in v1.
   Old images keep booting, and kernel `cargo test` keeps working.
7. Drive letters stay shell-only. `A:` = `/` is now the ext2 root, and B:...
   are the `/mnt/diskN` disks.

---

## 6. How userland gets built: `make image`

```
make            # = make image
make image      # → out/disk.img   (everything below, in order)
make rust       #   cargo build --release for rust/ (no_std, x86_64-unknown-none)
make musl       #   toolchain/build-musl.sh → out/sysroot (once; about 5 s)
make ports      #   each ports/*/build.sh with CC=toolchain/rb-cc, DESTDIR=out/staging
make staging    #   out/staging = rootfs/ + rust binaries in /bin + ports
make kernel     #   scripts/build-kernel.sh (cd $KERNEL_DIR; CARGO_TARGET_DIR=out/kernel-target cargo bootimage)
make run        #   QEMU window, our disk as disk 0
make test       #   headless boot + self-test + expected-line checks (+ host unit tests for rbsh's parser)
make compare    #   M1: our /bin vs the kernel's disk.img, file by file
make clean      #   rm -rf out
```

Notes for a beginner:

- **Why `make`?** It is the standard tool for "build these steps when their
  inputs change". musl, dash and busybox all build with it already, and it
  is now installed on the box. cargo stays in charge of the Rust part.
- **Cargo looks for `.cargo/config.toml` and `rust-toolchain.toml` in the
  folder you run it from**, not next to `--manifest-path`. That is why
  `build-kernel.sh` must `cd $KERNEL_DIR` first, and why it sets its own
  `CARGO_TARGET_DIR` so it never writes into the kernel's `target/`.
- The Rust userland needs only the precompiled `x86_64-unknown-none` target.
  It probably builds on **stable** Rust too (no `#![feature]`s are used), so
  that is worth testing.
- `rb-cc` works with ordinary `./configure && make`. The binaries it makes
  **also run on the Linux host**, so configure's test programs run fine. One
  catch: configure then detects *Linux* features, not Rustbelt's. Watch for
  that in `docs/syscall-gaps.md`.

---

## 7. Migration steps (small, safe, in order)

Each step ends with something you can check. The kernel repo is never edited
by the OS work; kernel changes are requests to Edward (§8).

| # | Step | Check |
|---|---|---|
| 0 | Pick a kernel commit to pin (`KERNEL_REV=58fc4fd`) and record it in `config.mk`. | — |
| 1 | `git init /workspace/rustbelt-os` (no remote). **Copy, not move,** `user/`, `disk/` and `scripts/hvc0-session.txt` out of the kernel at that commit with `git -C rust-kernel archive 58fc4fd user disk \| tar -x`. That reads git's store only, so nothing in the kernel tree or `target/` is touched. Commit with "imported from rust-kernel 58fc4fd". *(If you want the history of `user/` kept: `git subtree split -P user` in a throwaway clone, then pull that branch in.)* | the files match the kernel commit |
| 2 | Write `Makefile` + `scripts/mkimage.sh` that do exactly what `build.rs` does: the same `rustc` flags per program, the same `gcc` line for `rawabi`, the same `mkfs.vfat`/`mtools`/`sfdisk`. | `make compare`: same file list; same bytes for every `/bin` file (expected, since it is the same compiler and flags) |
| 3 | `scripts/build-kernel.sh` + `run-headless.sh`: build the pinned kernel in a clone or worktree under `out/`, boot it with **our** `disk.img` as disk 0. | the existing about 75 expected lines + "36 programs" all pass |
| 4 | **M1 done.** The kernel's `user/` still exists and still works on its own; nothing broke. | both repos green |
| 5 | Convert `rust/` to a Cargo workspace: `ulib.rs` → `rbrt` crate (`#[macro_export]` `println!`, `#[panic_handler]`, `#[global_allocator]`, `_start` calling the program's `main`). Move `shparse` tests to the host. | `make test` green; `cargo test -p rbsh` runs on the host in under a second |
| 6 | Vendor musl, add `build-musl.sh`, `rb-cc`, `c/hello-musl.c`, `c/mprobe.c`; add their expected lines to the self-test. | **M2**: "hello from musl on Rustbelt" in the headless log |
| 7 | (Edward K1) the kernel can take an external disk 0, so kernel `cargo test` can use the OS image. The ABI fixtures stay in the kernel (§8). | kernel `cargo test` green both ways |
| 8 | (Edward K5; K3 and K4 landed in 0.16.0) contract v2. OS side: ext2 image, `rust/init`, `rootfs/etc/*`, `ports/dash`. | **M3** (§10) |
| 9 | Only once both sides have been green for a while: Edward deletes the moved programs from the kernel's `user/`, keeping the ABI test fixtures. | kernel tests still pass |
| 10 | M4, M5 as in §10, each port adding its syscall needs to `docs/syscall-gaps.md` → requests to Edward. | — |

**What stays in the kernel repo for good** (my recommendation). These are
**ABI test fixtures** that check kernel behaviour, not OS programs:
`hello evil spin demo child overflow forktest cowtest mmaptest sigtest
threads echoserver ptytest nbtest fputest pathtest rawabi`, plus a frozen
copy of `ulib.rs` for them. Your call; see Q2.

**What moves:** `sh` + `shparse`, `ls cat wc grep mkdir rm touch cp sleep`,
`ifconfig ping host fetch httpd`, `getty tile random sysinfo`, `disk/`
(readme, notes, selftest) and `run-headless.sh`'s shell checks.

---

## 8. Kernel-side requests for Edward

These are written as requests. The kernel is Edward's lane.

**Status after 0.18.0 (dea72d3):**

- ✅ **Done:** K0 (trace numbers), K3 (sigsuspend, umask, rlimits, CPUs),
  K4 (in the subreaper / real init / auto-reap form; that works for us,
  and `rust/init` should call `prctl(PR_SET_CHILD_SUBREAPER)` or be named
  `init`), **K6** (everything listed is in; permissions are stored but not
  enforced), **K9** (`SO_REUSEADDR`, `times`, CPU clocks, trace `EINTR`,
  and the frame report waits for all tasks), **K10 (b) and (c)** (overrun
  and timerd, confirmed on 0.18.0). **K10 (a) is withdrawn: it was my
  probe's bug**, not the kernel's (§2.6).
- 🟡 **K7, mostly done:** Linux `sysinfo` 99, `sethostname`,
  `getsockopt(SO_ERROR)`, `getsockname`/`getpeername`/`shutdown`,
  `alarm`/`setitimer`/POSIX timers, `rt_sigtimedwait`, and now
  **`socketpair`/`AF_UNIX` and `/proc/self/fd` + `/proc/self/exe`** are
  in. Still open: `/proc/<pid>`, `/proc/mounts`, `/proc/meminfo`,
  `reboot` 169, `mount`/`umount`.
- ⬜ **Open:** K1 (external disk 0), K2 (static-PIE), K5 (rootfs v2 +
  `/sbin/init`), K8 (`/dev/fb0` mmap), K10 (d) (the remaining gaps), and
  the new **K11** below.

| ID | Needed by | Request | Why |
|---|---|---|---|
| **K0** | anytime, tiny | `--trace`: print the **number** of unknown syscalls (`syscall_130(...) = -38 ENOSYS`, not `?(...)`). | It makes the porting work much faster. |
| **K1** | M1b, nice to have | `build.rs`: if `RUSTBELT_DISK0=/path/disk.img` is set, copy that image to `$CARGO_TARGET_DIR/disk.img` and skip building the moved programs. The default stays unchanged. | Kernel `cargo test` can run against the OS image; there is one source of truth for programs. |
| **K2** | M2, recommended | ELF loader: accept **static-PIE**, meaning `ET_DYN` **without** `PT_INTERP`. Load it at a bias, e.g. `USER_START + 0x40_0000`; set `AT_PHDR`/`AT_ENTRY` with the bias and `AT_BASE=0`. Refuse `PT_INTERP` (dynamic programs) with a clear `ENOEXEC` log line. | Removes the `rb-cc` link hacks. Stock `musl-gcc -static-pie` works. C++ works too: today's workaround drops gcc's `crtbegin`, so C++ constructors would break. Rust's own `x86_64-unknown-linux-musl` target (std, static-PIE by default) can then run on Rustbelt. |
| **K3** | **M3 blocker** | **`rt_sigsuspend` 130** (atomically set the mask and sleep until a signal). Also `umask` 95 (keep a per-process value, inherited by fork), `prlimit64` 302 + `getrlimit` 97 (report `RLIMIT_NOFILE` = 64 = `MAX_FDS`, and stack and so on), and `sched_getaffinity` 204 (so nproc = 4). | dash/ash `wait` spins forever; `umask`/`ulimit` print garbage; `nproc` says 1. All seen in §3. |
| **K4** | M3 | **Reparent orphans to pid 1** and let pid 1 reap them. | 27 frames leaked in the dash test. Also needed for a real init. |
| **K5** | M3 | Boot contract **v2** (§5): find the GPT "Linux root x86-64" partition, mount it at `/`, exec `/sbin/init` → `/bin/init` → `/bin/sh` as pid 1, headless via argv/env, and the pid-1 exit status → QEMU exit. Fall back to v1. Default `PATH=/sbin:/bin:/usr/bin:/mnt/disk0/bin`. | It makes the OS repo the owner of what runs at boot. |
| **K6** | M4 | File metadata on ext2: `chmod`/`fchmod`/`fchmodat`; the `chown` family (store it, or accept it for uid 0); `utimensat` + **real `st_atime/mtime/ctime`** in `stat`; `truncate`/`ftruncate`; `fsync`/`fdatasync` (returning 0 is fine); `symlink`/`symlinkat` and `link`/`linkat`; `statfs`/`fstatfs`; `getrusage`/`times`; `rt_sigpending`; `sigaltstack`; `clock_getres`; `getgroups`/`setuid` family (uid 0 no-ops). | `chmod`, `touch`, `ln`, `truncate`, `dd`, `df`, `cp -p`, `ls -l` dates, `time`, `id` |
| **K7** | M5 | Linux `sysinfo` 99; **`/proc`** (`/proc/mounts`, `/proc/self/fd/N`, `/proc/<pid>/stat` + `cmdline`, `/proc/meminfo`, `/proc/uptime`); `reboot` 169 (POWER_OFF → QEMU exit); `sethostname` 170; `getsockopt(SO_ERROR)`, `getsockname`, `getpeername`, `shutdown`, `socketpair`; `setitimer`/`alarm`; `rt_sigtimedwait`; later `AF_UNIX`, `mount`/`umount`. | `ps`, `top`, `free`, `uptime`, `mount`, `tty`, `poweroff`, busybox `init`, `wget` (nonblocking connect checks `SO_ERROR`), `timeout`, `syslogd` |
| **K9** | M4–M5, new | Small fixes found on 0.16.0: accept `setsockopt(SO_REUSEADDR)` (returns `ENOPROTOOPT` 92 today); make `times` 100 exist (musl hands its raw `-ENOSYS` back as a tick count); `CLOCK_PROCESS_CPUTIME_ID` / `CLOCK_THREAD_CPUTIME_ID` so `clock()` isn't -1; optionally show `ERESTARTNOHAND` as `EINTR` in `--trace`; optionally have the headless frame report wait for (or count) auto-reaped orphans. | servers, `time`, `clock()` |
| **K10** | M4–M5 (0.17.0 findings) | ~~(a) mount-point `st_dev`~~ **withdrawn: it was my probe's bug** (unspecified argument order in one `printf`); the kernel was right on 0.17.0 too. ✅ (b) `timer_getoverrun` accumulates and latches (fixed in 0.18.0, checked). ✅ (c) timerd starts at boot (fixed in 0.18.0; frames 183 → 183). (d) Still open: `mremap` 25 (the last ENOSYS my programs hit by accident), `/proc/<pid>` + `/proc/mounts` + `/proc/meminfo`, `timerfd` 283/286/287, `ITIMER_VIRTUAL/PROF`, `AF_INET6`, a socket-cycle GC. | programs that use overruns to catch up; clean frame numbers; `ps`, `free`, event loops |
| **K11** | M4–M5, new (0.18.0 findings), small | (a) **Console identity:** `fstat` on a `/dev/console` fd should give the same `st_dev`/`st_ino` as `stat("/dev/console")` (today 1/75059993789296 vs 3/11760308497919119911, and a new inode per `open`), and `st_rdev` should be 5,1. Then musl's `ttyname` (and `tty`, and anything that checks "is this my terminal") works; it gives `ENODEV` today. `/dev/null` already does this right. (b) Optional: match Linux on stream sockets when an fd message is queued behind plain bytes. Linux's `read`/`recvmsg` reads *into* the fd message and stops after it (returns `"ABCD"`, fds on `recvmsg`, closed by `read`); 0.18 stops *before* it (returns `"AB"`). (c) Optional: `/proc/self/cwd`. | `tty`, `ttyname`, `ps`-style tty columns, job-control shells that check their terminal; strict fd-passing protocols |
| K8 | later | `mmap` of `/dev/fb0` (already on the roadmap). | C graphics programs and the compositor |

---

## 9. musl porting risks: what musl needs vs. what is missing

### Static startup (`crt1.o` → `__libc_start_main`): ✅ all present (verified)

`arch_prctl(ARCH_SET_FS)` for TLS, `set_tid_address`, and the auxv entries
`AT_PHDR` (musl finds its `PT_TLS` segment through it), `AT_RANDOM` (stack
protector canary), `AT_PAGESZ` and `AT_SECURE`. With no vDSO, musl falls
back to the real `clock_gettime` syscall.

### stdio: ✅

`ioctl(TIOCGWINSZ)` (line-buffer stdout on a tty), `writev`, `read`,
`lseek`, `close`.

### malloc (mallocng): ✅ with fallbacks

It uses `brk`, `mmap`, `munmap` and `mmap(PROT_NONE, MAP_FIXED)` guard pages.
`mremap` 25 is missing, so big `realloc`s copy (slower, but correct).
`madvise` 28 failing is ignored.

### Risk list

| Risk | Status / fix |
|---|---|
| High link address vs musl's crt + ld relaxation | **Seen.** Worked around in `rb-cc`; the clean fix is K2 (static-PIE). |
| Silent wrong answers from ENOSYS | `umask`, `getrlimit`, `nproc` and `stat` times are **fixed in 0.16.0**. Still wrong: `times()` returns -38, `clock()` returns -1. K9; track each in `docs/syscall-gaps.md`. |
| Blocking waits use `sigsuspend` | **Fixed in 0.16.0** (verified with dash `wait`). |
| Orphans not reaped → memory leak | **Fixed in 0.16.0** (auto-reaped; 157 → 157 frames). |
| Permissions stored (0.16.0) but not enforced: `test -x` is always true, `access()` always 0 | Fine for now; matters once there are multiple users (not planned). |
| `/etc/passwd`, `/etc/group`, `/etc/resolv.conf` missing | The OS repo supplies them in `rootfs/etc/`. musl's resolver then does DNS itself over UDP to 10.0.2.3; Rustbelt has UDP, but this is untested. |
| DNS over TCP fallback needs `sendmsg`/`recvmsg` | Only for big replies; low priority. |
| `fcntl` record locks / `flock` | POSIX record locks are **in 0.16.0**; `flock` 73 and `F_OFD_*` are still missing. |
| Rename isn't atomic; non-empty directories can't move | `mv dir` across names fails with EXDEV; tools fall back to copying. |
| 60 s headless watchdog | Long self-tests must stay under it, or ask for a feature flag to raise it. |
| `MAX_FDS` = 64 | Fine for dash/busybox (they dup to fds ≥ 10). |
| C++ | Needs gcc's `crtbegin/crtend`, which conflict with the high-address hack → wait for K2. |

---

## 10. Milestones

| | Goal | Done when | Kernel needs |
|---|---|---|---|
| **M1** | The split builds the same image | `rustbelt-os` `make image` produces a disk 0 whose `/bin` matches the kernel's file for file, and the existing headless self-test passes booting it | none (K1 optional) |
| **M1b** | Cargo workspace + `rbrt` crate | same tests pass; parser unit tests run on the host | none |
| **M2** | musl hello, static | `/bin/hello-musl` and `/bin/mprobe` in the image; the self-test checks "hello from musl on Rustbelt 0.15.1" and mprobe's ok lines | none (**already works in my test**); K2 to drop the hacks; K0 |
| **M3** | A shell on musl | dash as a port in `/bin/dash`; `wait`, `umask`, `ulimit` correct; ext2 root (contract v2) with `/sbin/init` (Rust) that respawns your shell on the console and hvc0; no leaked frames after the self-test | K5 (K3, K4 done in 0.16.0) |
| **M4** | Coreutils-style tools | **sbase** (suckless, MIT, small and easy-to-read C, plain Makefile) builds and runs: `ls -l`, `cp -p`, `touch`, `ln -s`, `chmod`, `mv`, `sort`, `uniq`, `head`/`tail`, `find`, `xargs`, `sed`, `tr`, `date`, `df`... Each tool goes in the gap list until green | K6 |
| **M5** | busybox | busybox 1.37 with a curated `.config` (start with about 40 applets: ash, vi, less, find, grep, sed, awk, tar, gzip, wget...), installed as `/bin/busybox` + symlinks on the ext2 root; `busybox ash` as an alternative shell; later `busybox init` | K7 (step by step); `apt install bzip2` to unpack the tarball |

Why **sbase before busybox**? Each sbase tool is one short C file, so when
one fails you can read the whole thing. busybox is one 1 MB+ binary with
shared helpers. It is more useful, but harder to debug first. **GNU
coreutils is not recommended early**: it is huge (gnulib + autotools) and
uses many more syscalls. **toybox** (0BSD) is a reasonable alternative to
sbase.

Also possible after K2: **Rust programs with the full standard library**,
built with `rustup target add x86_64-unknown-linux-musl`. Your Rust tools
could then use `std::fs`, `String` and `HashMap` like normal Rust.

---

## 11. Open questions for Tyler

1. **Name and place.** Is `/workspace/rustbelt-os` right? Plain copy at
   58fc4fd, or keep `user/`'s git history via `git subtree split`?
2. **What stays in the kernel.** Is my list in §7 right? (The ABI test
   programs stay as kernel fixtures; the shell, tools and net/sys programs
   move.)
3. **Which shell is `/bin/sh`?** I suggest your Rust shell stays the
   interactive login shell, with its DOS commands and drive letters, named
   e.g. `rbsh`. dash (later busybox ash) becomes `/bin/sh` for POSIX
   scripts. Or should "M3 shell on musl" mean your shell itself moves to Rust
   std on musl (needs K2)?
4. **Root filesystem for v2.** OK to switch the OS disk to **ext2**? That
   changes the letters: A: = `/` becomes the OS disk. Does today's B: FAT32
   disk stay as a data disk?
5. **Name clashes.** When sbase/busybox bring their own `ls`, `cat`... do
   your Rust versions stay the default, move to e.g. `/usr/lib/rustbelt/bin`,
   or get a prefix?
6. **Vendored tarballs in git.** OK to commit them (about 4 MB total by M5),
   or keep a git-ignored download cache pinned by SHA256?
7. **Which kernel does the OS build use?** A pinned commit of
   `/workspace/rust-kernel` (recommended), or whatever checkout
   `KERNEL_DIR` points at? Edward's `edward/syscalls` worktree is a natural
   choice once K3+ land there.
8. **Box packages.** I installed `make` for the experiment. Is it OK to
   install `bzip2` (busybox) and maybe `musl-tools` / `autoconf` later?
9. **Who sends §8 to Edward, and in what order?** I suggest K0, K3, K4 first:
   small, and they unblock M3.

---

## Appendix: the scratch artifacts

All under `/workspace/clarke-scratch/` (safe to delete):

- `dl/`: musl 1.2.5 and dash 0.5.12 tarballs and their build trees
- `musl-sysroot/`: the musl install (headers, `libc.a`, crt files)
- `tc/rb-cc`, `exp/rb-cc`: the compiler wrapper
- `exp/hello.c`, `exp/mprobe.c`, `exp/dash-t.sh`: the test programs
- `exp/qemu-musl.log`, `exp/qemu-dash*.log`: boot logs
- `exp/kernel-clone-experiment.diff`: the clone-only `build.rs` tweak
- `rk/`: the kernel clone (no remote, own `target/`), now at 0.16.0 63fcb86 plus my experiment changes
- `exp18/`: the 0.18.0 re-test (`mprobe2.c` + `unix18.c` with 219 checks, `stdev.c`, `ttyp.c`, `order.c`, `boot.sh`, `host.log`, `qemu-*.log` including `qemu-stdev-017.log` from the 0.17.0 worktree `rk17/`, and a copy of this proposal before the update)
- `exp17/`: the 0.17.0 re-test (`mprobe2.c` with 114 checks, `rmr.c`, `rmr.sh`, `talarm.c`, `boot.sh`, `qemu-*.log`, and a copy of this proposal before the update)
- `exp16/`: the 0.16.0 re-test (`mprobe2.c`, `t2.sh`, `boot.sh`, four QEMU logs, and a copy of this proposal before the update)
- `extra-bin/`: the binaries copied into the clone's disk