foundry

@@ -0,0 +1,803 @@+# Proposal: split Rustbelt's userland into `rustbelt-os`, then port musl++*Status: draft for Tyler, 2026-10-31. Read-only study of `/workspace/rust-kernel`+at `master` 58fc4fd (Rustbelt 0.15.1), **re-tested on 0.16.0 (63fcb86),+0.17.0 (fc5e78a) and 0.18.0 (dea72d3)**: see §2.6, §3b, §3c and §3d. Nothing in the kernel repo was changed.+`/workspace/rustbelt-os` does not exist yet; creating it is step 1 below, after you approve.*++---++## 0. The short version++- **Today** the kernel repo builds everything. Its `build.rs` compiles the 36+  user programs in `user/`, which are 35 Rust programs and 1 C program. It+  writes them into a FAT32 disk image, `target/disk.img`, under `/bin`. The+  kernel mounts that disk at `/mnt/disk0` and runs `/mnt/disk0/bin/sh`.+- **The idea:** move the programs, the disk-image builder and (new) the C+  library into their own repo, `rustbelt-os`. That repo produces **one disk image**.+  The kernel only boots it. The contract between the two is a short document+  that says what the image must contain.+- **Good news from testing.** I built musl 1.2.5 in a scratch area, compiled C+  programs against it, and booted them on an unmodified 0.15.1 kernel (my own+  clone, my own target dir). A **static musl `printf`/`malloc` hello world+  works today**. So do stdio, files, `opendir`, `fork`+`exec`, `pthread_create`,+  signals and `double` maths. The **dash shell** also runs scripts: variables,+  `$((...))`, functions, pipes, `$(...)`, redirection, `read`, `trap`.+- **Update, 0.17.0:** in a full `--trace` run the only ENOSYS left is+  `mremap` (25). Links, chown, statfs, CPU time, timers, ids and the small+  server socket calls all work, and `rm -r` is fixed (§2.6). The remaining+  gaps are `AF_UNIX`/`socketpair`, `/proc` and two wrong values (§8, K10).+- **Update, 0.18.0:** Unix sockets (stream, datagram, seqpacket; path,+  abstract and autobind names; `socketpair`), fd passing with `SCM_RIGHTS`,+  `SO_PEERCRED`/`SCM_CREDENTIALS`, `/proc/self/fd` and `/proc/self/exe` all+  pass by value, and the timer fixes are confirmed. **Correction:** my+  0.17 "mount point `st_dev`" finding was a bug in my probe, not the kernel+  (§2.6). Two small new issues: a tty fd's `fstat` doesn't match+  `/dev/console`, so `ttyname()` fails, and stream reads stop *before* a+  message carrying fds where Linux reads into it (§8, K11).+- **0.15.1's blocker is gone in 0.16.0.** Edward's stage 16 added+  `rt_sigsuspend` (130), `umask`, the rlimits and `sched_getaffinity`, and+  orphan handling. On 0.16.0 my whole dash suite passes, including `wait`,+  `umask` and `ulimit -n`, and nothing leaks. So dash needs **no more kernel+  work**. What is left for coreutils/busybox is links (`symlink` 88, `link`+  86), `chown` (92–94), `statfs` (137/138), timers and clocks (37, 38, 222,+  229, 98, 100), ids (105, 106, 115, 118) and some socket calls (§2.6).+- **M1 needs no kernel change at all.** QEMU can simply be given the OS+  repo's disk image in place of the kernel's.++---++## 1. Words used in this document++| Term | Meaning |+|---|---|+| **userland / user space** | Everything that is not the kernel: the shell, `ls`, the C library. It runs in *ring 3*, the CPU's unprivileged mode. |+| **libc** | The C standard library (`printf`, `malloc`, `fopen`...). It turns those functions into **syscalls**, the requests a program makes to the kernel. |+| **musl** | A small, clean libc used by Alpine Linux. It is easy to read and easy to link **statically**. |+| **static binary** | A program with the libc copied into the file itself, so it needs no `.so` files and no dynamic loader at run time. Rustbelt has no dynamic loader yet, so static is the only option for now. |+| **ELF** | The executable file format on Linux (and on Rustbelt). `ET_EXEC` is a fixed-address executable; `ET_DYN` is a position-independent one (a shared library, or a PIE). |+| **PIE / static-PIE** | Position-Independent Executable: a program that can be loaded at any address. A *static-PIE* is static **and** position-independent. It fixes up its own addresses at startup (musl's `rcrt1.o` does this). |+| **cross toolchain** | A compiler setup that builds programs for a *different* system than the one it runs on. Here it is Debian's `gcc` aimed at musl and Rustbelt in place of glibc and Linux. |+| **sysroot** | A folder holding the target's headers (`include/`) and libraries (`lib/`). The compiler reads those instead of the host's. |+| **rootfs** | The filesystem mounted at `/`, holding `/bin`, `/etc` and so on. |+| **disk image** | A plain file holding the bytes of a whole virtual disk (partition table + filesystem). QEMU uses it as a disk. |+| **GPT** | The modern partition table format. Each partition has a *type GUID* saying what it is for. |+| **tarball** | A `.tar.gz`/`.tar.xz` archive of source code, as released by a project. |+| **vendoring** | Keeping a copy of a dependency's source (here, the release tarball) inside your own repo, so builds never need the network. |+| **`no_std`** | Rust without its standard library: only `core` (and `alloc` once there is a heap). That is how Rustbelt's Rust programs are built today. |++---++## 2. How things work today (findings)++### 2.1 Repo and build layout++```+rust-kernel/+├── Cargo.toml            one package "rustbelt" (lib + bin), no Cargo workspace; bootloader 0.9 (BIOS)+├── .cargo/config.toml    build-std core/alloc, custom target x86_64-rust_kernel.json, runner scripts/runner.sh+├── rust-toolchain.toml   nightly + rust-src + llvm-tools-preview; also installs x86_64-unknown-none+├── build.rs              ★ builds ALL user programs and ALL disk images (details below)+├── build/exfat_populate.rs  build.rs's own exFAT writer+├── src/                  the kernel+├── user/                 ★ the userland: 35 *.rs programs + ulib.rs + shparse.rs + rawabi.c + link.ld+├── disk/                 files for the FAT32 disk (readme, notes/, scripts/selftest.sh)+├── disk2/ disk3/ disk4/  files for the ext2 / exFAT / btrfs test disks+├── tests/                cargo test: each test boots the kernel in QEMU+└── scripts/              run-headless.sh, screenshot.sh, *-check.sh, http-reply.sh, hvc0-session.txt+```++There is no Makefile or xtask. Everything goes through `cargo bootimage` /+`cargo run` / `cargo test` plus the shell scripts.++### 2.2 How user programs are compiled, stored and loaded++1. **Rust programs** (`build.rs` → `build_program`). Each `user/X.rs` is a+   single file compiled by plain `rustc` for the built-in+   `x86_64-unknown-none` target. That target ships a precompiled+   `core`/`alloc`, so it needs no build-std and uses soft-float. The flags are+   `panic=abort`, `opt-level=2`, `relocation-model=static`,+   `code-model=large` and `-T user/link.ld`. Each program pulls in the+   tiny runtime with `#[path = "ulib.rs"] mod ulib;`. That runtime holds the+   syscall wrappers, `println!`, a `brk` heap and `_start`.+2. **C program** (`build_c_program`). There is one, `user/rawabi.c`, built+   with the host's `gcc -static -nostdlib -fpie -no-pie+   -Wl,-Ttext-segment=0x8000400000`. That means no libc at all, just raw+   Linux syscalls.+3. **Link address.** User code starts at **`0x80_0040_0000`**+   (`USER_START = 0x80_0000_0000` in `src/memory.rs`). That is above 4 GiB,+   where normal Linux binaries (at `0x400000`) are not allowed.+4. **Storage.** The programs used to be embedded with `include_bytes!`, but+   since stage 6 they **live on a disk image**. `build.rs` makes+   `target/disk.img`: 64 MiB, a GPT from `sfdisk`, one partition at LBA 2048,+   **FAT32** from `mkfs.vfat -F 32 -s 1 -n TYLERDISK`, filled by mtools+   `mmd`/`mcopy` (no root needed). `disk/` goes to the root of the volume and+   the programs go to `/bin`.+5. **Other disks.** `build.rs` also makes the ext2 (`mke2fs -d`), exFAT+   (`mkfs.exfat` + its own populator) and btrfs (`mkfs.btrfs --rootdir`)+   images. These are **kernel filesystem-driver test fixtures**, not userland.+6. **No initrd, no cpio.** There is no initial ramdisk. `/` is a **ramfs**+   built in code (`fs::init`: `/hello.txt`, `/etc/motd`, `/tmp`, `/mnt`,+   `/dev`). Each disk is probed by magic number (ext2 → exFAT → btrfs → FAT)+   and mounted at `/mnt/diskN` in PCI order.+7. **ELF loader** (`src/elf.rs`). It accepts only static **`ET_EXEC`** files+   whose segments sit inside the user range. It refuses two segments with+   different permissions on the same page. `execve` builds Linux's+   argv/envp/auxv stack, including `AT_PHDR`, `AT_RANDOM`, `AT_EXECFN` and+   `AT_SECURE=0`, with no vDSO.++### 2.3 How the kernel finds "init"++There is no init process yet. In `src/main.rs`, a **kernel thread**+(`shell_supervisor`) spawns the hard-coded `SHELL = "/mnt/disk0/bin/sh"`,+waits for it, and starts a new one when it exits. With `--features headless`+it passes `/mnt/disk0/scripts/selftest.sh` as an argument instead. When the+shell exits, it reports the memory frames still in use and exits QEMU via+`isa-debug-exit` (33 = pass). The default environment is+`PATH=/bin:/mnt/disk0/bin HOME=/ TERM=linux` (`src/process.rs`).++### 2.4 The shell++`user/sh.rs` (786 lines) and `user/shparse.rs` (236 lines, pure functions)+are written in **`no_std` Rust**. The shell is a hybrid:++- **Unix side:** `cd`/`pwd`, pipes, `<` `>` `>>`, quotes, `&`, job control+  (process groups, Ctrl+C/Ctrl+Z, `jobs`/`fg`/`bg`/`kill`), `history`,+  `alias`, PATH lookup, and scripts (`sh file`, or commands on stdin).+  Programs are started with `fork` + `execve`.+- **DOS side:** `run [--trace] prog` (`--trace` is strace-like), `start+  prog` (runs it in the background), aliases `dir`=`ls`, `type`=`cat`,+  `del`=`rm`, `md`=`mkdir`, `copy`=`cp`, plus `cls`, `ver` and `drives`.+- **Drive letters** are shell syntax only. `A:` = `/`, and `B:`, `C:`...+  map to the `/mnt/diskN` mounts in mount order (B: FAT32, C: ext2, D: exFAT,+  E: btrfs), read through Rustbelt's own `mounts` syscall (501).+  `shparse::translate` rewrites words; the prompt shows `B:/bin`.+- There are no variables or `$?` yet.+- `tests/shell.rs` compiles `shparse.rs` straight into a kernel test with+  `#[path = "../user/shparse.rs"]`.++### 2.5 Coupling that the split has to handle++- **The kernel's own `cargo test` suite needs the userland.** The tests in+  `tests/*.rs` run `sh`, `ls`, `cat`, `wc`, `grep`, `tile`, `ping`,+  `fetch`, `random`, `child`, `forktest`... from `/mnt/disk0/bin`.+- **`run-headless.sh`** checks about 75 lines of shell output, including+  "`dir bin | wc -l` → 36". That is really an *OS* test.+- **Three paths are hard-coded in the kernel:** `SHELL`, `SELFTEST` and the+  `PATH` default.++### 2.6 Syscalls: what is there and what is missing (updated for 0.18.0)++*Re-tested on **Rustbelt 0.18.0 (master dea72d3)**, 2026-10-31, from real+musl 1.2.5 programs: `mprobe2`, now 219 value checks (group D is new, for+stage 18); `stdev`, a mount-point `st_dev` check with one `stat` per+statement; `ttyp`, a tty check; `rmr`, a naive `rm -r`; `talarm`. dash+0.5.12 was tested too. Every new check was first calibrated on the box's+Linux host (as a normal user, never root). See §3d. Older results are in+§3, §3b and §3c.*++Rustbelt has the Linux syscalls by Linux number, plus its own 500–505+(`spawn`, `mounts`, `rb_sysinfo` 502, `resolve`, `netinfo`, `trace_me`).+Anything it doesn't know returns `-ENOSYS` (errno 38), and `run --trace`+prints it as `syscall_N(...)`.++**The headline: in a full `--trace` run of mprobe2 the only ENOSYS calls+are `syscall_25` (`mremap`) and `syscall_283` (`timerfd_create`, which the+probe calls on purpose as a known gap).** musl's `realloc` copes with the+missing `mremap`.++**New in 0.18.0, checked by value:**++| Area | Calls (x86_64 number) | Checked |+|---|---|---|+| Unix sockets | `socket` 41 / `socketpair` 53 with `AF_UNIX` and `SOCK_STREAM`, `SOCK_DGRAM`, `SOCK_SEQPACKET`; `bind` 49, `listen` 50, `accept` 43, `connect` 42, `sendto` 44 / `recvfrom` 45 | data both ways; a stream merges two writes; datagram and seqpacket keep message borders (3, 5, then an empty datagram); a short buffer gives `MSG_TRUNC`, and `recv(MSG_TRUNC)` returns the real length; `fstat` says `S_IFSOCK`; `SO_TYPE`, `SO_DOMAIN`; `getpeername` on a pair is the family alone |+| names | path, abstract, autobind | a path `bind` makes an `S_IFSOCK` file on ramfs and ext2 and `EPERM` on FAT; `open` on it is `ENXIO`; a second `bind` is `EADDRINUSE`; the file stays after close; `connect` after `unlink` is `ENOENT`; an abstract name has the leading NUL, is freed on close, and can be bound again; autobind gives an 8-byte name (`@00000`); `recvfrom` shows the sender's autobind name, and a reply to it arrives |+| connections | the "embryo" | `connect` and `send` work before `accept`; `getsockname` and `getpeername` show the path |+| EOF / HUP | `shutdown` 48, `poll` 7, `epoll_pwait` 281 | after the peer's `SHUT_WR`: `POLLIN|POLLRDHUP` (0x2001) with no `POLLHUP`, `EPOLLRDHUP`, then `read` returns 0; the half-closed side can still receive; after the peer closes: `POLLHUP` (0x2011), `EPOLLHUP`, `EPIPE` plus SIGPIPE, and `MSG_NOSIGNAL` gives `EPIPE` without the signal. All the same values as Linux |+| fd passing | `sendmsg` 46 / `recvmsg` 47 with `SCM_RIGHTS` | a passed pipe end can be written through, and the pipe sees EOF when it is closed (no hidden reference); a passed file shares its offset; passing to a `fork`ed child works; `MSG_CMSG_CLOEXEC` sets `FD_CLOEXEC` (and it's clear without it); 3 fds into `CMSG_SPACE(4)` give 2 fds plus `MSG_CTRUNC`, and the third is closed; no control buffer gives `MSG_CTRUNC` and closes the fd; 254 fds is `EINVAL` and 253 is accepted; 253 fds into a 64-slot fd table installs 57 with `MSG_CTRUNC`, and every extra reference is closed; a socket fd can be passed too |+| in-flight fds | — | **a pipe end that is sent but never received keeps the pipe open while it is queued, and is closed when the receiving socket closes** (the reader gets EOF), for stream and datagram |+| credentials | `SO_PEERCRED`, `SO_PASSCRED`, `SCM_CREDENTIALS` | `SO_PEERCRED` gives your own pid/uid/gid on a pair, and the *child's* pid when a forked child connects; with `SO_PASSCRED` every message carries `SCM_CREDENTIALS`; sending your own credentials explicitly works |+| backpressure | `SO_SNDBUF`/`SO_RCVBUF` | a nonblocking writer stops with `EAGAIN` at 212992 bytes (Linux: 180224, which also counts per-buffer overhead) and `POLLOUT` goes away; after draining `POLLOUT` is back; `SO_SNDBUF` 4096 reads back as 8192; a blocking writer pushes 1 MiB through a slow reader |+| /proc | `/proc` (magic `0x9fa0`), `/proc/self/exe`, `/proc/self/fd` | `/proc/self/exe` is `/mnt/disk0/bin/mprobe2`, also in a child; the `/proc/self/fd` listing matches the open fds exactly (plus the directory's own fd); `readlink` gives the path for a file, `pipe:[4011]` (same number as `st_ino`, the same for both ends), `socket:[1032]` (a different inode for each end), `anon_inode:[eventpoll]`, and `/dev/console` for 0–2; opening `/proc/self/fd/N` gives a fresh offset for a file, the same pipe for a pipe, and `ENXIO` for a socket; a closed fd gives `ENOENT`; `realpath` works |+| timers (fixes) | `timer_getoverrun` 225 | 0 before the first delivery; 0 while the next signal is pending (the last delivery's value, like Linux); after delivery 26 for a window that actually lasted 0.541 s (27 expirations − 1; Linux gives 24 for exactly 0.500 s); a second 0.1 s round gives 4, so the count restarts at each delivery |+| frame report | — | 183 before and 183 after the shell in **every** 0.18 boot, including the ones with `alarm`: timerd now starts at boot |++**Working, and checked by value (stage 16 + stage 17, re-checked on 0.18.0):**+++| Area | Calls (x86_64 number) | Checked |+|---|---|---|+| signals | `rt_sigsuspend` 130, `rt_sigpending` 127, `rt_sigtimedwait` 128, `rt_sigqueueinfo` 129, `sigaltstack` 131 | dash `wait`/`wait $!` statuses; `SA_ONSTACK` runs on the alt stack; `--trace` now shows `= -4 EINTR` (no more -514) |+| umask / limits / CPUs | `umask` 95, `getrlimit` 97, `setrlimit` 160, `prlimit64` 302, `sched_getaffinity` 204 | `umask` 0022; `ulimit -n` 64; `EMFILE` under a lowered limit; nproc 4 |+| file size / mode / times | `truncate` 76, `ftruncate` 77, `fsync` 74, `fdatasync` 75, `chmod` 90, `fchmod` 91, `fchmodat` 268, `utimensat` 280 | exact read-back on ramfs |+| locks | `fcntl` `F_SETLK`/`F_GETLK`, `F_GETFL`/`F_SETFL`, **`flock` 73** | `flock`: a second `open()` gets `EWOULDBLOCK`; the lock is shared through a `fork`ed fd; `LOCK_UN` releases it |+| links | **`link` 86, `linkat` 265, `symlink` 88, `symlinkat` 266** | `st_nlink` 2 with the same inode; writing through one name shows in the other; dangling symlinks; `AT_SYMLINK_FOLLOW`; `lstat`/`readlink`; `EXDEV` across mounts; `EPERM` on FAT |+| rename / readdir | `rename` moves entries, `getdents64` | rename of a full directory; **`rm -r` with unlink-while-readdir removes all 311 entries on ramfs *and* ext2**; `d_ino == st_ino`; nftw `FTW_DEPTH` delete ok |+| owners | **`chown` 92, `fchown` 93, `lchown` 94, `fchownat` 260** | uid/gid read back; `-1` keeps a field; `lchown` changes the link, not the target; FAT accepts root:root and gives `EPERM` for 5:5; new files are 0:0 |+| fs info | **`statfs` 137, `fstatfs` 138, `statx` 332** | ramfs `0x858458f6`, FAT `0x4d44`, pipe `PIPEFS`; disk0 is 124990 × 512 B ≈ 61 MiB with free ≤ total; `statvfs` ok; `statx` size and mode |+| CPU time | **`times` 100, `getrusage` 98, `clock_getres` 229, `CLOCK_PROCESS_CPUTIME_ID`, `CLOCK_THREAD_CPUTIME_ID`** | a 0.5 s busy loop adds 50 ticks to `times()` and about 500000 to `clock()`; a child's CPU time shows up after `wait`; resolution 1 µs (wall clocks) and 10 ms (CPU clocks) |+| system info | **Linux `sysinfo` 99** | uptime, 120 MB total RAM, free ≤ total, procs, loads |+| timers | **`alarm` 37, `setitimer`/`getitimer` 38/36 (`ITIMER_REAL`), `timer_create`…`timer_delete` 222–226** | `alarm(1)` fires after 1.088 s; 3 × 100 ms periodic ticks take 0.372 s; a POSIX timer fires after 0.258 s for 200 ms with `SI_TIMER` and the right `si_value`; `alarm(0)` returns the seconds left |+| ids / priority / host | **`setuid` 105, `setgid` 106, `setresuid` 117, `getresuid` 118, `getgroups`/`setgroups` 115/116, `getpriority`/`setpriority` 140/141, `sethostname` 170** | the saved-id rules hold (`setresuid(1000,1000,0)` → `setuid(0)` works; after `setuid(1000)` the switch back is `EPERM`); a non-root process lowering its nice value gets `EACCES`; `uname` shows the new host name |+| sockets | **`getsockname` 51, `getpeername` 52, `shutdown` 48, `getsockopt` 55**, more `setsockopt` 54 options | `SO_REUSEADDR`, `TCP_NODELAY` and `SO_KEEPALIVE` accepted and read back; `SO_TYPE` and `SO_ACCEPTCONN`; an unknown option gives `ENOPROTOOPT`; a loopback connect/accept shows matching ports on both sides; after `SHUT_WR` the peer reads EOF but the half-closed side can still read, and a write gives `EPIPE`; a refused connect reports `SO_ERROR` = `ECONNREFUSED` once, then 0 |+| vector / copy I/O | **`preadv` 295, `pwritev` 296, `sendfile` 40** | offsets honoured and the file position unchanged; `sendfile` works file→file (with and without an offset) and file→pipe |+| misc | `madvise` 28, `set_robust_list` 273, `prctl` 157 | — |++**Still missing (known, Edward's list):**++| Gap | Effect |+|---|---|+| `mremap` 25 (ENOSYS) | big `realloc` copies instead; slower but correct |+| `AF_INET6` (EAFNOSUPPORT) | IPv4 only |+| `ITIMER_VIRTUAL`/`ITIMER_PROF`, `timer_create` on CPU clocks (EINVAL) | profilers, `ulimit -t`-style CPU watchdogs |+| `timerfd_*` 283/286/287 | event-loop programs (not used by dash/busybox core) |+| `/proc/<pid>` (ENOENT), `/proc/mounts`, `/proc/meminfo`, `/proc/self/cwd` …; `/proc/self` is a directory, not a symlink | `ps`, `top`, `pidof`, `mount` (listing), `free`, `df` without arguments, `pwdx` |+| no garbage collector for Unix socket cycles (Edward's note; not tested, since it leaks by design) | only programs that send a socket through itself; rare |+| `sendmsg`/`recvmsg` control messages on INET sockets | `IP_PKTINFO`-style servers |+| permissions/ids **not enforced**; nice ignored by the scheduler | fine for a single-user hobby OS |+| no `/etc/passwd` → `getpwuid(0)` fails | **OS-repo job** (`rootfs/etc/passwd`, `group`) |++**Retraction: the 0.17.0 "mount point `st_dev`" finding was my probe's bug.**+Edward was right. My 0.17 probe printed the value like this:++```c+printf("... /tmp=%lx /mnt/disk0=%lx /=%lx\n", (long)st.st_dev, (long)st2.st_dev,+       (stat("/", &st), (long)st.st_dev));+```++The third argument calls `stat("/", &st)` and overwrites `st`, and C does+not say in which order a function's arguments are evaluated. gcc evaluated+the third one first, so "/tmp=1" was really `/`'s device. (A three-line+test with the same pattern prints the overwritten value with gcc at both+`-O0` and `-O2`: `exp18/order.c`.) I checked it again properly, with one+`stat`, `lstat` or `fstat` per statement and each result in its own struct+(`exp18/stdev.c`). I ran this **on 0.17.0 as well as 0.18.0**, in a+separate worktree at fc5e78a, to be sure the kernel had always been right:++| path | 0.17.0 `st_dev` | 0.18.0 `st_dev` |+|---|---|---|+| `/`, `/hello.txt` | 1 | 1 |+| `/tmp`, `/tmp/`, `/tmp/.`, a file in `/tmp` | **2** | **2** |+| `/dev` | 3 | 3 |+| `/proc` | — | 4 (new) |+| `/mnt/disk0`, `/mnt/disk0/bin` | 4 | 5 |+| `/mnt/disk1` (ext2) | 5 | 6 |++`stat`, `lstat` and `fstat` agree on every row. The mount point reports+the mounted file system, as on Linux. The fixed `mprobe2` now checks this+with separate statements. (The numbers after `/dev` moved up by one+because `/proc` is now mounted first; that is expected.)++**Wrong values on 0.18.0 (new; details in §8, K11):**++1. **`ttyname(0)` fails with `ENODEV`, so `tty` would too.** musl's+   `ttyname` reads `/proc/self/fd/0` (`/dev/console`), then checks that+   `stat("/dev/console")` and `fstat(0)` give the same `st_dev`/`st_ino`.+   They don't: `fstat(0)` says dev 1, ino 75059993789296, mode 020620;+   `stat("/dev/console")` says dev 3, ino 11760308497919119911, mode+   020666. Each new `open("/dev/console")` also gets yet another inode,+   and `st_rdev` is 0 for all of them (Linux: 5,1). `isatty(0)` and+   `tcgetattr` work, and `/dev/null` is consistent (dev 6, ino 4, rdev+   1,3 for both `fstat` and `stat`). So only the console's open file+   reports the wrong identity. (`fstat` 5 versus `stat` 4 / `newfstatat` 262.)+2. **A stream read stops *before* a message carrying fds; Linux reads+   into it.** Take a stream socket with `"AB"` written plainly and then+   `"CD"` sent with one fd. On Linux, both `read` and `recvmsg` return+   `"ABCD"`. `recvmsg` also returns the fd; a plain `read` closes it.+   On 0.18.0 both return `"AB"`, and the next `recvmsg` returns `"CD"`+   with the fd. The README describes the 0.18 behaviour on purpose, but+   it also says "all Linux's behaviour". Nothing loses data and no fd+   leaks; real programs (libwayland, D-Bus) loop on `recvmsg`, so this+   is low priority. A program that expects the fd on the first+   `recvmsg` would see it one call late. (`read` 0, `recvmsg` 47.)+3. **Not a bug, noted:** the autobind name is `@00000` (Linux picks a+   varied number); `/proc/self/cwd` doesn't exist (`getcwd` works).+4. **Time accounting note, not a bug** (from 0.17): with no vDSO, a loop+   calling `clock_gettime` spends most of its time in the kernel.++*Fixed in 0.18.0 and confirmed:* the 0.17 overrun undercount, and the+16-frame timerd one-off (both above).++### 2.7 Toolchain on this box++| Tool | Status |+|---|---|+| rustup + nightly 1.101 (2026-10-29) | ✅ targets `x86_64-unknown-linux-gnu`, `x86_64-unknown-none`; `rust-lld` and llvm tools present |+| `bootimage` 0.10.5 | ✅ |+| gcc 14.2 (Debian, **default-PIE**), GNU ld 2.44 | ✅ |+| QEMU 10.0.13 | ✅ |+| dosfstools, mtools, sfdisk, e2fsprogs (`mke2fs`, `debugfs`, `e2fsck`), exfatprogs, btrfs-progs | ✅ |+| git, curl, wget, python3, patch, xz, perl, gpg | ✅ |+| **make** | ❌ was missing. **I installed it** (`apt install make`, 4.4.1) to build musl for the experiment below |+| musl-gcc / musl-tools | ❌ not installed (Debian has musl-tools 1.2.5 if wanted; I built musl from source instead) |+| clang, ld.lld, cmake, ninja, meson, autoconf/automake, bison/flex, pkg-config, **bzip2**, cpio | ❌. bzip2 is needed to unpack busybox's `.tar.bz2`; autotools are only needed if we build from git snapshots instead of release tarballs |++`sudo` works without a password, so installing packages is easy when you OK it.++---++## 3. What I tried (in `/workspace/clarke-scratch`, never in the kernel repo)++All of this used a **local `git clone`** of the kernel at+`/workspace/clarke-scratch/rk`, with its `origin` remote removed so a push+is impossible. It has its own `target/`. Arthur's `/workspace/rust-kernel/target`+was never touched.++1. **Built musl 1.2.5** from the official tarball (sha256 `a9a118bb…c75e4`)+   with `./configure --disable-shared` into `clarke-scratch/musl-sysroot`.+   It took about 5 seconds.+2. **Found a link problem with high addresses.** musl's startup code does+   `lea _DYNAMIC(%rip)` on a *weak, undefined* symbol, which means address 0.+   Linked at `0x80_0040_0000`, that relative jump is too far ("relocation+   truncated to fit"), and ld's GOT relaxation fails as well. The stock+   `musl-gcc` wrapper always picks `Scrt1.o`, so it cannot be used as is.+   **The workaround that works** is a 15-line wrapper, `rb-cc`+   (saved at `clarke-scratch/exp/rb-cc`). It uses `crt1.o`, `-fPIE` code,+   `-static -no-pie`, `-Ttext-segment=0x8000400000`,+   `--defsym=_DYNAMIC=0x8000400000`, `--no-relax`, and no gcc+   `crtbegin/crtend`. The result is an ordinary static `ET_EXEC` that runs+   both on Linux and on Rustbelt.+3. **Booted it on Rustbelt** (`run --trace mhello a b`). These are the exact+   syscalls musl's startup + stdio + malloc made, and all succeeded:++   ```+   arch_prctl(ARCH_SET_FS) = 0         <- thread-local storage for the main thread+   set_tid_address(...)    = 2+   uname(...)              = 0+   ioctl(1, TIOCGWINSZ)    = 0         <- stdio checks "is stdout a terminal?" to pick line buffering+   writev(1, ..., 2)       = 43        <- stdio writes buffer + new text in one call+   brk(0), brk(+8K)                    <- mallocng's metadata+   mmap(.., PROT_NONE, MAP_FIXED)      <- guard page+   mmap(0, 100K, RW, ANON) / munmap    <- the big malloc+   exit_group(0)+   ```++4. **`mprobe`, a broader musl test.** These pass on Rustbelt: TIOCGWINSZ,+   `tcgetattr`, `isatty`, 5 MiB `malloc` + 9 MiB `realloc` (musl falls back+   from the missing `mremap` to copy), `fopen`/`fgets`/`fprintf`, `stat`,+   `opendir`/`readdir`, `getcwd`, `clock_gettime`, `gmtime`,+   `signal`+`raise`, `fork`+`execl`+`waitpid`, `pthread_create`/`join`, and+   SSE `double` maths.+   **These fail with ENOSYS:** `getrlimit`, `chmod`, `ftruncate`,+   `utimensat`, `symlink`, `sigpending`, `sigaltstack`.+   `sysconf(_SC_NPROCESSORS_ONLN)` says **1** CPU, not 4, because+   `sched_getaffinity` is missing.+5. **dash 0.5.12** (Debian's release tarball) built with `CC=rb-cc+   ./configure --host=x86_64-linux-musl && make` in about 2 seconds. The+   stripped binary is 180 KB. On Rustbelt, a test script passed:+   variables, arithmetic, `for`, functions, `case`, `$(ls | wc -l)`,+   `2>/dev/null`, `read < file`, `&` + `$!`, `cd`, `test -d/-x`,+   `command -v`, `trap` + `kill -USR1 $$`, and `exit 5` → `[exit code 5]`.+   - **`wait` hangs forever.** dash blocks all signals and calls+     `sigsuspend` to wait for SIGCHLD. That returns ENOSYS at once, so dash+     spins (23 000+ ENOSYS calls in the trace) and the SIGCHLD stays+     blocked. The headless watchdog killed it after 60 s.+   - **`umask`** prints `37777777777` and **`ulimit -n`** prints+     `549755813889`. Both are garbage from the ENOSYS returns.+   - **Leak:** a background `sleep` that nobody waited for left+     **27 frames** in use after the shell exited (157 before, 184 after).+     The kernel does not yet hand orphans to an init process to reap.+   - In the `--trace` output, unknown syscalls show up as `?(...)`+     **without their number**, which makes it harder to spot what is+     missing.++Logs: `clarke-scratch/exp/qemu-musl.log`, `qemu-dash*.log`. The two-file+experiment patch to the *clone's* `build.rs` (an env var that copies extra+binaries into `/bin`) is in `exp/kernel-clone-experiment.diff`.++---++## 3b. Re-test on Rustbelt 0.16.0 (master 63fcb86)++I updated the scratch clone `rk` to 63fcb86 with a one-off `git fetch+/workspace/rust-kernel master`. That is a fetch from a path, so no remote was+added; `git remote -v` is empty. I rebuilt with `CARGO_TARGET_DIR=rk/target`+and booted headless four times. The same musl 1.2.5 sysroot and `rb-cc`+wrapper were used; the musl binaries needed no change.++| Boot | What ran | Result |+|---|---|---|+| `probes` | `mhello`, `mprobe`, `mprobe2` (new: about 80 checks that compare **values**, not just return codes) | exit 33. Startup/stdio/malloc unchanged. All the stage-16 items pass; the remaining failures are exactly the "still missing" table in §2.6 |+| `dash` | dash suite `t2.sh` (about 40 checks), then `sleep 3` so orphans finish | everything correct: `wait` / `wait $!` / `wait` all, `umask`, `ulimit -n` = 64, `ulimit -n 32`, getopts, heredoc, traps, `EXIT` trap, exit status 5. **Frames 157 → 157** |+| `noreap` | the same suite, then the shell exits **at once** while the orphans are still sleeping | frames 157 → 211. This is **not a leak**: the two orphans are still alive when the kernel takes its count. The headless frame report only waits for pid 1, not for orphans that are auto-reaped later. |+| `trace` | `run --trace mprobe2` and `run --trace dash t2.sh` | dash makes **no ENOSYS calls at all**. mprobe2's ENOSYS list is in §2.6: `syscall_25 38 40 48 51 53 55 73 86 88 92 93 94 98 99 100 105 106 115 118 137 138 140 141 170 222 229 295 296` |++Logs are in `clarke-scratch/exp16/` (`qemu-probes.log`, `qemu-dash.log`,+`qemu-noreap.log`, `qemu-trace.log`), with the probe source `mprobe2.c`, the+dash suite `t2.sh`, and `boot.sh`, which rebuilds the disk with a given+self-test and boots it.++## 3c. Re-test on Rustbelt 0.17.0 (master fc5e78a)++Same method as §3b. I fetched once from the local path (no remote left+behind), rebuilt in `rk/target`, and booted headless with disk 0 (FAT32)+plus the kernel's ext2 test image as disk 1 (`/mnt/disk1`, snapshot mode).++| Boot | What ran | Result |+|---|---|---|+| `probes`, `probes2` | `mhello`, extended `mprobe2` (114 checks: group A = stage 16 regression, B = stage 17, C = known gaps) | **108 / 114 pass; the 6 failures are exactly the known gaps** (socketpair/AF_UNIX, AF_INET6, mremap, /proc ×2, /etc/passwd). The new checks found the overrun problem. (They also reported an `st_dev` problem, which was my probe's bug and is retracted in §2.6.) |+| `twice`, `alarm` | `mprobe2` twice; `talarm` alone | 167 → 183 frames each time: the one-off cost of starting timerd, not a leak |+| `dash` | dash suite `t2.sh`, then `rmr.sh` (builds 5 dirs × 60 files + subdirs on `/tmp` and on ext2 `/mnt/disk1`, deletes with a naive `rm -r`) | dash suite unchanged and correct; **rm -r removes all 311 entries on both file systems**, nothing left; frames 167 → 167, even with orphans still running when dash exits |+| `trace` | `run --trace` of `mprobe2`, `rmr` and the dash suite | **only `syscall_25` (mremap) is ENOSYS.** Every other error is one the probe asked for (`EXDEV`, `EPERM` on FAT, `EWOULDBLOCK`, `EPIPE`, `ECONNREFUSED`…). No `-512`/`-514` any more; interrupted calls show `-4 EINTR` |++Files: `clarke-scratch/exp17/` holds `mprobe2.c` (the 0.16 version is kept+as `mprobe2-016.c`), `rmr.c`, `rmr.sh`, `talarm.c`, `boot.sh` and the+`qemu-*.log` files.++## 3d. Re-test on Rustbelt 0.18.0 (master dea72d3)++Same method as §3c: one fetch from the local path (no remote left behind),+a rebuild in `rk/target`, and headless boots with disk 0 (FAT32) plus the+ext2 test image as disk 1. For the `st_dev` question I also built 0.17.0+again in a separate scratch worktree (`clarke-scratch/rk17`, own target).++| Boot | What ran | Result |+|---|---|---|+| `stdev-017` (0.17.0) and `probes` (0.18.0) | `stdev`: `stat`/`lstat`/`fstat` of 11 paths, one call per statement | mount points report the mounted fs on **both** versions → my 0.17 finding retracted (§2.6) |+| `probes`, `probes2` | `ttyp`, then `mprobe2` (219 checks: A = stage 16, B = stage 17 with the fixed `st_dev` block and the new overrun checks, **D = stage 18**: 98 checks for Unix sockets, fd passing, credentials, HUP, backpressure and `/proc/self`; C = known gaps) | **210 / 219 pass.** 6 failures are the known gaps (AF_INET6, mremap 25, `/proc/<pid>`, timerfd 283, `/proc/mounts`, `/etc/passwd`). 3 are the fd-boundary difference (2 checks, plus 1 that follows from them). `ttyname` fails (printed, not counted) |+| `dash` | dash suite `t2.sh`, `rmr.sh` on `/tmp` and ext2, `talarm` | dash output identical to 0.17 (except the program count, 43 → 47); rm -r removed 311, failed 0, on both; `alarm` fires; **frames 183 → 183** |+| `trace` | `run --trace mprobe2` | ENOSYS only for `syscall_25` (mremap) and `syscall_283` (timerfd, called on purpose); 100 distinct calls in use |++On the Linux host (as a normal user) the new group D passes everything+except what the host doesn't have (FAT/ext2 mounts, the exe path,+`/proc/self` being a symlink), so the checks themselves are right.++Files: `clarke-scratch/exp18/` holds `mprobe2.c` + `unix18.c` (the 0.17+version is kept as `mprobe2-017.c`), `stdev.c`, `ttyp.c`, `order.c`,+`boot.sh`, `host.log` and the `qemu-*.log` files.++---++## 4. Proposed layout of `rustbelt-os`++```+rustbelt-os/+├── README.md                 what this is, quick start (make image / make run / make test)+├── Makefile                  ★ the single entry point (targets in §6)+├── config.mk                 KERNEL_DIR, KERNEL_REV, ARCH=x86_64, IMAGE_SIZE, musl/port versions+├── docs/+│   ├── boot-contract.md      the kernel ↔ OS contract (§5), versioned+│   ├── syscall-gaps.md       living list: which program needs which missing call+│   └── porting.md            how to add a C program ("a port")+│+├── rust/                     Rust userland (no_std, x86_64-unknown-none), a Cargo workspace+│   ├── Cargo.toml            [workspace] members = [...]+│   ├── .cargo/config.toml    target = x86_64-unknown-none, rustflags: static, code-model=large, -T link.ld+│   ├── link.ld               moved from kernel user/link.ld+│   ├── rbrt/                 "Rustbelt runtime": today's ulib.rs as a library crate+│   ├── rbsh/                 Tyler's shell: src/main.rs (sh.rs) + src/parse.rs (shparse.rs)+│   │                         └─ the parser's unit tests run on the host with plain `cargo test`+│   ├── coreutils-rs/         src/bin/{ls,cat,wc,grep,mkdir,rm,touch,cp,sleep}.rs+│   ├── net/                  src/bin/{ifconfig,ping,host,fetch,httpd}.rs+│   ├── sys/                  src/bin/{getty,tile,random,sysinfo}.rs+│   └── init/                 (M3) /sbin/init: runs /etc/rc, respawns shells on console + hvc0, reaps orphans+│+├── toolchain/+│   ├── musl/+│   │   ├── musl-1.2.5.tar.gz         vendored release tarball (1.1 MB)+│   │   ├── musl-1.2.5.tar.gz.asc     upstream signature+│   │   ├── SHA256SUMS+│   │   └── patches/                  0001-*.patch, applied in order (empty at first)+│   ├── build-musl.sh                 unpack → patch → configure → make → install into out/sysroot+│   └── rb-cc                         compiler wrapper (gcc + musl sysroot + Rustbelt link flags)+│+├── c/                        small C programs of our own (hello-musl.c, mprobe.c = libc smoke tests)+│+├── ports/                    third-party C programs, one directory each+│   ├── dash/                 dash-0.5.12.tar.gz, SHA256SUMS, patches/, build.sh+│   ├── sbase/                (M4) suckless coreutils+│   └── busybox/              (M5) busybox-1.37.x tarball, busybox.config, patches/+│+├── rootfs/                   files copied into the image as-is+│   ├── readme.txt  notes/    (from kernel disk/)+│   ├── etc/                  motd, passwd, group, hosts, resolv.conf (nameserver 10.0.2.3), profile, rc+│   └── scripts/selftest.sh   the headless self-test (from kernel disk/scripts/)+│+├── scripts/+│   ├── mkimage.sh            staging dir → GPT + FAT32 (v1) or GPT + ext2 (v2) image+│   ├── build-kernel.sh       builds the kernel bootimage from KERNEL_DIR into out/kernel-target+│   ├── run-qemu.sh           interactive boot+│   ├── run-headless.sh       boot + selftest + expected-output check (moved from the kernel)+│   └── compare-image.sh      M1 check: file list + bytes vs the kernel's own disk.img+│+└── out/                      (git-ignored) sysroot/, staging/, disk.img, kernel-target/, logs+```++**Why this shape?**++- `rust/` and `ports/` are separate on purpose. Our Rust programs talk to the+  kernel directly through `rbrt`. Ported C programs go through musl. They+  are built in different ways and fail in different ways.+- One directory per port, holding the tarball, checksum, patches and a short+  `build.sh`, is how Alpine (APKBUILD) and the BSD ports trees do it.+  Adding a program then means "copy a directory and edit it".+- `out/` holds everything generated, so `make clean` is just+  `rm -rf out`.++### Where musl goes: vendored tarball + patches (recommended), not a submodule++| Option | For | Against |+|---|---|---|+| **Vendored release tarball + `patches/` (recommended)** | Works offline, with no remote (you said local only). The exact bytes are pinned by SHA256 and the upstream `.asc` signature. Patches are small readable files. Upgrading means dropping in a new tarball and re-running. | 1.1 MB in git (fine; dash 0.2 MB, busybox about 2.5 MB) |+| git submodule | Full upstream history | A submodule needs a URL to clone from. With local-only rules it would point at another local clone, which is confusing for a beginner and easy to get wrong. Editing musl in place mixes our changes with theirs. |+| `apt install musl-tools` | Zero work | Ties us to Debian's build. Its `musl-gcc` wrapper hard-codes `Scrt1.o` (see §3), and we can't patch it. |++My expectation is that musl will need **no patches at first**. Every fix so+far belongs either in the kernel or in our `rb-cc` flags.++---++## 5. The boot-image contract (how the kernel and the OS connect)++The contract is a short, versioned document (`docs/boot-contract.md`) that+both repos agree on. The kernel never needs to know how the image was built.++### Contract v1: "today, written down" (used for M1–M2, needs no kernel change)++1. The OS image is attached to QEMU as the **first virtio-blk disk** (it+   becomes `/mnt/disk0`, drive B:).+2. The image is **GPT** with one partition starting at LBA 2048, formatted+   **FAT32**. The label `TYLERDISK` is kept for now.+3. It must contain **`/bin/sh`**, and `/scripts/selftest.sh` for headless+   runs. Programs live in `/bin` and are static ELF `ET_EXEC` files linked at+   or above `0x80_0040_0000`.+4. The kernel provides `/` (ramfs), `/tmp`, `/dev` and `/mnt/diskN`. It runs+   `/mnt/disk0/bin/sh` with `PATH=/bin:/mnt/disk0/bin`. In headless builds+   it passes `/mnt/disk0/scripts/selftest.sh` and exits QEMU with the+   shell's status (33 = pass).+5. The kernel's other test disks (ext2/exFAT/btrfs) stay **kernel-owned**.+   The OS run script may attach them after disk 0, built by the kernel into+   the OS repo's own `out/kernel-target`, so B:–E: keep their letters.++With this contract, M1 is just "the OS repo makes `disk.img`, and+`run-qemu.sh` passes it to QEMU in place of the kernel's".++### Contract v2: "a real root filesystem" (from M3, needs Edward's K5)++1. The OS image is a **GPT** disk with a partition whose type GUID is+   **Linux root (x86-64)**, `4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709`.+   This GUID comes from the freedesktop *Discoverable Partitions+   Specification*, the same rule systemd uses. The kernel then does not have+   to guess which disk is "the OS".+2. The partition is formatted **ext2**, built with `mke2fs -d staging/`.+   ext2 is the better fit for a root filesystem:+   - it has Unix permission bits and executable flags,+   - it has **symlinks** (busybox installs its 300 applets as symlinks),+   - names are case-sensitive,+   - Rustbelt's ext2 driver is its most complete read-write one, and the+     host's `e2fsck` can check it.++   FAT32 has none of the first three.+3. The kernel mounts that partition at **`/`**. It keeps a ramfs at `/tmp`+   and devfs at `/dev`, and mounts the other disks at `/mnt/diskN` as today.+4. The kernel runs the first that exists of **`/sbin/init`, `/bin/init`,+   `/bin/sh`** as **pid 1**, in the same order Linux tries. The argument+   vector is `["init"]`; in headless builds it is `["init", "selftest"]`,+   or an env var `RUSTBELT_HEADLESS=1`.+   - The kernel does **not** respawn pid 1. Respawning becomes init's job.+   - In headless builds, the pid-1 exit status ends the QEMU run (33/35 as+     today).+5. **Orphans are handed to pid 1**, which reaps them.+6. **Fallback:** if there is no root partition, the kernel behaves as in v1.+   Old images keep booting, and kernel `cargo test` keeps working.+7. Drive letters stay shell-only. `A:` = `/` is now the ext2 root, and B:...+   are the `/mnt/diskN` disks.++---++## 6. How userland gets built: `make image`++```+make            # = make image+make image      # → out/disk.img   (everything below, in order)+make rust       #   cargo build --release for rust/ (no_std, x86_64-unknown-none)+make musl       #   toolchain/build-musl.sh → out/sysroot (once; about 5 s)+make ports      #   each ports/*/build.sh with CC=toolchain/rb-cc, DESTDIR=out/staging+make staging    #   out/staging = rootfs/ + rust binaries in /bin + ports+make kernel     #   scripts/build-kernel.sh (cd $KERNEL_DIR; CARGO_TARGET_DIR=out/kernel-target cargo bootimage)+make run        #   QEMU window, our disk as disk 0+make test       #   headless boot + self-test + expected-line checks (+ host unit tests for rbsh's parser)+make compare    #   M1: our /bin vs the kernel's disk.img, file by file+make clean      #   rm -rf out+```++Notes for a beginner:++- **Why `make`?** It is the standard tool for "build these steps when their+  inputs change". musl, dash and busybox all build with it already, and it+  is now installed on the box. cargo stays in charge of the Rust part.+- **Cargo looks for `.cargo/config.toml` and `rust-toolchain.toml` in the+  folder you run it from**, not next to `--manifest-path`. That is why+  `build-kernel.sh` must `cd $KERNEL_DIR` first, and why it sets its own+  `CARGO_TARGET_DIR` so it never writes into the kernel's `target/`.+- The Rust userland needs only the precompiled `x86_64-unknown-none` target.+  It probably builds on **stable** Rust too (no `#![feature]`s are used), so+  that is worth testing.+- `rb-cc` works with ordinary `./configure && make`. The binaries it makes+  **also run on the Linux host**, so configure's test programs run fine. One+  catch: configure then detects *Linux* features, not Rustbelt's. Watch for+  that in `docs/syscall-gaps.md`.++---++## 7. Migration steps (small, safe, in order)++Each step ends with something you can check. The kernel repo is never edited+by the OS work; kernel changes are requests to Edward (§8).++| # | Step | Check |+|---|---|---|+| 0 | Pick a kernel commit to pin (`KERNEL_REV=58fc4fd`) and record it in `config.mk`. | — |+| 1 | `git init /workspace/rustbelt-os` (no remote). **Copy, not move,** `user/`, `disk/` and `scripts/hvc0-session.txt` out of the kernel at that commit with `git -C rust-kernel archive 58fc4fd user disk \| tar -x`. That reads git's store only, so nothing in the kernel tree or `target/` is touched. Commit with "imported from rust-kernel 58fc4fd". *(If you want the history of `user/` kept: `git subtree split -P user` in a throwaway clone, then pull that branch in.)* | the files match the kernel commit |+| 2 | Write `Makefile` + `scripts/mkimage.sh` that do exactly what `build.rs` does: the same `rustc` flags per program, the same `gcc` line for `rawabi`, the same `mkfs.vfat`/`mtools`/`sfdisk`. | `make compare`: same file list; same bytes for every `/bin` file (expected, since it is the same compiler and flags) |+| 3 | `scripts/build-kernel.sh` + `run-headless.sh`: build the pinned kernel in a clone or worktree under `out/`, boot it with **our** `disk.img` as disk 0. | the existing about 75 expected lines + "36 programs" all pass |+| 4 | **M1 done.** The kernel's `user/` still exists and still works on its own; nothing broke. | both repos green |+| 5 | Convert `rust/` to a Cargo workspace: `ulib.rs` → `rbrt` crate (`#[macro_export]` `println!`, `#[panic_handler]`, `#[global_allocator]`, `_start` calling the program's `main`). Move `shparse` tests to the host. | `make test` green; `cargo test -p rbsh` runs on the host in under a second |+| 6 | Vendor musl, add `build-musl.sh`, `rb-cc`, `c/hello-musl.c`, `c/mprobe.c`; add their expected lines to the self-test. | **M2**: "hello from musl on Rustbelt" in the headless log |+| 7 | (Edward K1) the kernel can take an external disk 0, so kernel `cargo test` can use the OS image. The ABI fixtures stay in the kernel (§8). | kernel `cargo test` green both ways |+| 8 | (Edward K5; K3 and K4 landed in 0.16.0) contract v2. OS side: ext2 image, `rust/init`, `rootfs/etc/*`, `ports/dash`. | **M3** (§10) |+| 9 | Only once both sides have been green for a while: Edward deletes the moved programs from the kernel's `user/`, keeping the ABI test fixtures. | kernel tests still pass |+| 10 | M4, M5 as in §10, each port adding its syscall needs to `docs/syscall-gaps.md` → requests to Edward. | — |++**What stays in the kernel repo for good** (my recommendation). These are+**ABI test fixtures** that check kernel behaviour, not OS programs:+`hello evil spin demo child overflow forktest cowtest mmaptest sigtest+threads echoserver ptytest nbtest fputest pathtest rawabi`, plus a frozen+copy of `ulib.rs` for them. Your call; see Q2.++**What moves:** `sh` + `shparse`, `ls cat wc grep mkdir rm touch cp sleep`,+`ifconfig ping host fetch httpd`, `getty tile random sysinfo`, `disk/`+(readme, notes, selftest) and `run-headless.sh`'s shell checks.++---++## 8. Kernel-side requests for Edward++These are written as requests. The kernel is Edward's lane.++**Status after 0.18.0 (dea72d3):**++- ✅ **Done:** K0 (trace numbers), K3 (sigsuspend, umask, rlimits, CPUs),+  K4 (in the subreaper / real init / auto-reap form; that works for us,+  and `rust/init` should call `prctl(PR_SET_CHILD_SUBREAPER)` or be named+  `init`), **K6** (everything listed is in; permissions are stored but not+  enforced), **K9** (`SO_REUSEADDR`, `times`, CPU clocks, trace `EINTR`,+  and the frame report waits for all tasks), **K10 (b) and (c)** (overrun+  and timerd, confirmed on 0.18.0). **K10 (a) is withdrawn: it was my+  probe's bug**, not the kernel's (§2.6).+- 🟡 **K7, mostly done:** Linux `sysinfo` 99, `sethostname`,+  `getsockopt(SO_ERROR)`, `getsockname`/`getpeername`/`shutdown`,+  `alarm`/`setitimer`/POSIX timers, `rt_sigtimedwait`, and now+  **`socketpair`/`AF_UNIX` and `/proc/self/fd` + `/proc/self/exe`** are+  in. Still open: `/proc/<pid>`, `/proc/mounts`, `/proc/meminfo`,+  `reboot` 169, `mount`/`umount`.+- ⬜ **Open:** K1 (external disk 0), K2 (static-PIE), K5 (rootfs v2 ++  `/sbin/init`), K8 (`/dev/fb0` mmap), K10 (d) (the remaining gaps), and+  the new **K11** below.++| ID | Needed by | Request | Why |+|---|---|---|---|+| **K0** | anytime, tiny | `--trace`: print the **number** of unknown syscalls (`syscall_130(...) = -38 ENOSYS`, not `?(...)`). | It makes the porting work much faster. |+| **K1** | M1b, nice to have | `build.rs`: if `RUSTBELT_DISK0=/path/disk.img` is set, copy that image to `$CARGO_TARGET_DIR/disk.img` and skip building the moved programs. The default stays unchanged. | Kernel `cargo test` can run against the OS image; there is one source of truth for programs. |+| **K2** | M2, recommended | ELF loader: accept **static-PIE**, meaning `ET_DYN` **without** `PT_INTERP`. Load it at a bias, e.g. `USER_START + 0x40_0000`; set `AT_PHDR`/`AT_ENTRY` with the bias and `AT_BASE=0`. Refuse `PT_INTERP` (dynamic programs) with a clear `ENOEXEC` log line. | Removes the `rb-cc` link hacks. Stock `musl-gcc -static-pie` works. C++ works too: today's workaround drops gcc's `crtbegin`, so C++ constructors would break. Rust's own `x86_64-unknown-linux-musl` target (std, static-PIE by default) can then run on Rustbelt. |+| **K3** | **M3 blocker** | **`rt_sigsuspend` 130** (atomically set the mask and sleep until a signal). Also `umask` 95 (keep a per-process value, inherited by fork), `prlimit64` 302 + `getrlimit` 97 (report `RLIMIT_NOFILE` = 64 = `MAX_FDS`, and stack and so on), and `sched_getaffinity` 204 (so nproc = 4). | dash/ash `wait` spins forever; `umask`/`ulimit` print garbage; `nproc` says 1. All seen in §3. |+| **K4** | M3 | **Reparent orphans to pid 1** and let pid 1 reap them. | 27 frames leaked in the dash test. Also needed for a real init. |+| **K5** | M3 | Boot contract **v2** (§5): find the GPT "Linux root x86-64" partition, mount it at `/`, exec `/sbin/init` → `/bin/init` → `/bin/sh` as pid 1, headless via argv/env, and the pid-1 exit status → QEMU exit. Fall back to v1. Default `PATH=/sbin:/bin:/usr/bin:/mnt/disk0/bin`. | It makes the OS repo the owner of what runs at boot. |+| **K6** | M4 | File metadata on ext2: `chmod`/`fchmod`/`fchmodat`; the `chown` family (store it, or accept it for uid 0); `utimensat` + **real `st_atime/mtime/ctime`** in `stat`; `truncate`/`ftruncate`; `fsync`/`fdatasync` (returning 0 is fine); `symlink`/`symlinkat` and `link`/`linkat`; `statfs`/`fstatfs`; `getrusage`/`times`; `rt_sigpending`; `sigaltstack`; `clock_getres`; `getgroups`/`setuid` family (uid 0 no-ops). | `chmod`, `touch`, `ln`, `truncate`, `dd`, `df`, `cp -p`, `ls -l` dates, `time`, `id` |+| **K7** | M5 | Linux `sysinfo` 99; **`/proc`** (`/proc/mounts`, `/proc/self/fd/N`, `/proc/<pid>/stat` + `cmdline`, `/proc/meminfo`, `/proc/uptime`); `reboot` 169 (POWER_OFF → QEMU exit); `sethostname` 170; `getsockopt(SO_ERROR)`, `getsockname`, `getpeername`, `shutdown`, `socketpair`; `setitimer`/`alarm`; `rt_sigtimedwait`; later `AF_UNIX`, `mount`/`umount`. | `ps`, `top`, `free`, `uptime`, `mount`, `tty`, `poweroff`, busybox `init`, `wget` (nonblocking connect checks `SO_ERROR`), `timeout`, `syslogd` |+| **K9** | M4–M5, new | Small fixes found on 0.16.0: accept `setsockopt(SO_REUSEADDR)` (returns `ENOPROTOOPT` 92 today); make `times` 100 exist (musl hands its raw `-ENOSYS` back as a tick count); `CLOCK_PROCESS_CPUTIME_ID` / `CLOCK_THREAD_CPUTIME_ID` so `clock()` isn't -1; optionally show `ERESTARTNOHAND` as `EINTR` in `--trace`; optionally have the headless frame report wait for (or count) auto-reaped orphans. | servers, `time`, `clock()` |+| **K10** | M4–M5 (0.17.0 findings) | ~~(a) mount-point `st_dev`~~ **withdrawn: it was my probe's bug** (unspecified argument order in one `printf`); the kernel was right on 0.17.0 too. ✅ (b) `timer_getoverrun` accumulates and latches (fixed in 0.18.0, checked). ✅ (c) timerd starts at boot (fixed in 0.18.0; frames 183 → 183). (d) Still open: `mremap` 25 (the last ENOSYS my programs hit by accident), `/proc/<pid>` + `/proc/mounts` + `/proc/meminfo`, `timerfd` 283/286/287, `ITIMER_VIRTUAL/PROF`, `AF_INET6`, a socket-cycle GC. | programs that use overruns to catch up; clean frame numbers; `ps`, `free`, event loops |+| **K11** | M4–M5, new (0.18.0 findings), small | (a) **Console identity:** `fstat` on a `/dev/console` fd should give the same `st_dev`/`st_ino` as `stat("/dev/console")` (today 1/75059993789296 vs 3/11760308497919119911, and a new inode per `open`), and `st_rdev` should be 5,1. Then musl's `ttyname` (and `tty`, and anything that checks "is this my terminal") works; it gives `ENODEV` today. `/dev/null` already does this right. (b) Optional: match Linux on stream sockets when an fd message is queued behind plain bytes. Linux's `read`/`recvmsg` reads *into* the fd message and stops after it (returns `"ABCD"`, fds on `recvmsg`, closed by `read`); 0.18 stops *before* it (returns `"AB"`). (c) Optional: `/proc/self/cwd`. | `tty`, `ttyname`, `ps`-style tty columns, job-control shells that check their terminal; strict fd-passing protocols |+| K8 | later | `mmap` of `/dev/fb0` (already on the roadmap). | C graphics programs and the compositor |++---++## 9. musl porting risks: what musl needs vs. what is missing++### Static startup (`crt1.o` → `__libc_start_main`): ✅ all present (verified)++`arch_prctl(ARCH_SET_FS)` for TLS, `set_tid_address`, and the auxv entries+`AT_PHDR` (musl finds its `PT_TLS` segment through it), `AT_RANDOM` (stack+protector canary), `AT_PAGESZ` and `AT_SECURE`. With no vDSO, musl falls+back to the real `clock_gettime` syscall.++### stdio: ✅++`ioctl(TIOCGWINSZ)` (line-buffer stdout on a tty), `writev`, `read`,+`lseek`, `close`.++### malloc (mallocng): ✅ with fallbacks++It uses `brk`, `mmap`, `munmap` and `mmap(PROT_NONE, MAP_FIXED)` guard pages.+`mremap` 25 is missing, so big `realloc`s copy (slower, but correct).+`madvise` 28 failing is ignored.++### Risk list++| Risk | Status / fix |+|---|---|+| High link address vs musl's crt + ld relaxation | **Seen.** Worked around in `rb-cc`; the clean fix is K2 (static-PIE). |+| Silent wrong answers from ENOSYS | `umask`, `getrlimit`, `nproc` and `stat` times are **fixed in 0.16.0**. Still wrong: `times()` returns -38, `clock()` returns -1. K9; track each in `docs/syscall-gaps.md`. |+| Blocking waits use `sigsuspend` | **Fixed in 0.16.0** (verified with dash `wait`). |+| Orphans not reaped → memory leak | **Fixed in 0.16.0** (auto-reaped; 157 → 157 frames). |+| Permissions stored (0.16.0) but not enforced: `test -x` is always true, `access()` always 0 | Fine for now; matters once there are multiple users (not planned). |+| `/etc/passwd`, `/etc/group`, `/etc/resolv.conf` missing | The OS repo supplies them in `rootfs/etc/`. musl's resolver then does DNS itself over UDP to 10.0.2.3; Rustbelt has UDP, but this is untested. |+| DNS over TCP fallback needs `sendmsg`/`recvmsg` | Only for big replies; low priority. |+| `fcntl` record locks / `flock` | POSIX record locks are **in 0.16.0**; `flock` 73 and `F_OFD_*` are still missing. |+| Rename isn't atomic; non-empty directories can't move | `mv dir` across names fails with EXDEV; tools fall back to copying. |+| 60 s headless watchdog | Long self-tests must stay under it, or ask for a feature flag to raise it. |+| `MAX_FDS` = 64 | Fine for dash/busybox (they dup to fds ≥ 10). |+| C++ | Needs gcc's `crtbegin/crtend`, which conflict with the high-address hack → wait for K2. |++---++## 10. Milestones++| | Goal | Done when | Kernel needs |+|---|---|---|---|+| **M1** | The split builds the same image | `rustbelt-os` `make image` produces a disk 0 whose `/bin` matches the kernel's file for file, and the existing headless self-test passes booting it | none (K1 optional) |+| **M1b** | Cargo workspace + `rbrt` crate | same tests pass; parser unit tests run on the host | none |+| **M2** | musl hello, static | `/bin/hello-musl` and `/bin/mprobe` in the image; the self-test checks "hello from musl on Rustbelt 0.15.1" and mprobe's ok lines | none (**already works in my test**); K2 to drop the hacks; K0 |+| **M3** | A shell on musl | dash as a port in `/bin/dash`; `wait`, `umask`, `ulimit` correct; ext2 root (contract v2) with `/sbin/init` (Rust) that respawns your shell on the console and hvc0; no leaked frames after the self-test | K5 (K3, K4 done in 0.16.0) |+| **M4** | Coreutils-style tools | **sbase** (suckless, MIT, small and easy-to-read C, plain Makefile) builds and runs: `ls -l`, `cp -p`, `touch`, `ln -s`, `chmod`, `mv`, `sort`, `uniq`, `head`/`tail`, `find`, `xargs`, `sed`, `tr`, `date`, `df`... Each tool goes in the gap list until green | K6 |+| **M5** | busybox | busybox 1.37 with a curated `.config` (start with about 40 applets: ash, vi, less, find, grep, sed, awk, tar, gzip, wget...), installed as `/bin/busybox` + symlinks on the ext2 root; `busybox ash` as an alternative shell; later `busybox init` | K7 (step by step); `apt install bzip2` to unpack the tarball |++Why **sbase before busybox**? Each sbase tool is one short C file, so when+one fails you can read the whole thing. busybox is one 1 MB+ binary with+shared helpers. It is more useful, but harder to debug first. **GNU+coreutils is not recommended early**: it is huge (gnulib + autotools) and+uses many more syscalls. **toybox** (0BSD) is a reasonable alternative to+sbase.++Also possible after K2: **Rust programs with the full standard library**,+built with `rustup target add x86_64-unknown-linux-musl`. Your Rust tools+could then use `std::fs`, `String` and `HashMap` like normal Rust.++---++## 11. Open questions for Tyler++1. **Name and place.** Is `/workspace/rustbelt-os` right? Plain copy at+   58fc4fd, or keep `user/`'s git history via `git subtree split`?+2. **What stays in the kernel.** Is my list in §7 right? (The ABI test+   programs stay as kernel fixtures; the shell, tools and net/sys programs+   move.)+3. **Which shell is `/bin/sh`?** I suggest your Rust shell stays the+   interactive login shell, with its DOS commands and drive letters, named+   e.g. `rbsh`. dash (later busybox ash) becomes `/bin/sh` for POSIX+   scripts. Or should "M3 shell on musl" mean your shell itself moves to Rust+   std on musl (needs K2)?+4. **Root filesystem for v2.** OK to switch the OS disk to **ext2**? That+   changes the letters: A: = `/` becomes the OS disk. Does today's B: FAT32+   disk stay as a data disk?+5. **Name clashes.** When sbase/busybox bring their own `ls`, `cat`... do+   your Rust versions stay the default, move to e.g. `/usr/lib/rustbelt/bin`,+   or get a prefix?+6. **Vendored tarballs in git.** OK to commit them (about 4 MB total by M5),+   or keep a git-ignored download cache pinned by SHA256?+7. **Which kernel does the OS build use?** A pinned commit of+   `/workspace/rust-kernel` (recommended), or whatever checkout+   `KERNEL_DIR` points at? Edward's `edward/syscalls` worktree is a natural+   choice once K3+ land there.+8. **Box packages.** I installed `make` for the experiment. Is it OK to+   install `bzip2` (busybox) and maybe `musl-tools` / `autoconf` later?+9. **Who sends §8 to Edward, and in what order?** I suggest K0, K3, K4 first:+   small, and they unblock M3.++---++## Appendix: the scratch artifacts++All under `/workspace/clarke-scratch/` (safe to delete):++- `dl/`: musl 1.2.5 and dash 0.5.12 tarballs and their build trees+- `musl-sysroot/`: the musl install (headers, `libc.a`, crt files)+- `tc/rb-cc`, `exp/rb-cc`: the compiler wrapper+- `exp/hello.c`, `exp/mprobe.c`, `exp/dash-t.sh`: the test programs+- `exp/qemu-musl.log`, `exp/qemu-dash*.log`: boot logs+- `exp/kernel-clone-experiment.diff`: the clone-only `build.rs` tweak+- `rk/`: the kernel clone (no remote, own `target/`), now at 0.16.0 63fcb86 plus my experiment changes+- `exp18/`: the 0.18.0 re-test (`mprobe2.c` + `unix18.c` with 219 checks, `stdev.c`, `ttyp.c`, `order.c`, `boot.sh`, `host.log`, `qemu-*.log` including `qemu-stdev-017.log` from the 0.17.0 worktree `rk17/`, and a copy of this proposal before the update)+- `exp17/`: the 0.17.0 re-test (`mprobe2.c` with 114 checks, `rmr.c`, `rmr.sh`, `talarm.c`, `boot.sh`, `qemu-*.log`, and a copy of this proposal before the update)+- `exp16/`: the 0.16.0 re-test (`mprobe2.c`, `t2.sh`, `boot.sh`, four QEMU logs, and a copy of this proposal before the update)+- `extra-bin/`: the binaries copied into the clone's disk