foundry

@@ -46,3 +46,127 @@ Inside the ext2 root: | **M3** | dash as `/bin/sh`, ext2 root (contract v2), Rust `/sbin/init`, exFAT share | **done** on Rustbelt 0.21.0: `make test` (v2) passes, plus host fsck of all three file systems; `make test-v1` still passes | | M4 | sbase tools | needs testing per tool (docs/syscall-gaps.md) | | M5 | busybox in `/usr/lib/busybox` | `bzip2` for the tarball |++## Long-term track: a self-hosted Rust toolchain (after M5, not started)++**The goal:** write a Rust program *on* Foundry OS, compile it there, and+run it there, without the box. Today every program is compiled on the box+(Linux) and copied into the image.++Words used below:++- **Compiler, toolchain.** A compiler turns source code (`main.rs`,+  `hello.c`) into a program the CPU can run. A *toolchain* is the compiler+  plus its helpers: the linker (glues the pieces into one file), the+  standard library, and for Rust, `cargo` (the build tool).+- **Cross-compiling.** Compiling on one system for another one. That is+  what we do now: the box's compilers make programs for Foundry OS.+- **Self-hosted.** A system that can build its own programs (in the end,+  its own compiler) on itself. Linux is self-hosted; Foundry OS isn't yet.+- **Bootstrap.** Getting the first compiler onto a system that has none:+  you cross-compile a compiler on another machine, copy it over, and from+  then on it can build things locally, even newer versions of itself.+- **Dynamic linker (`ld.so`).** Most Linux programs don't carry their+  libraries inside them. They name shared libraries (`.so` files), and a+  small program, the dynamic linker, loads those into memory when the+  program starts. Rustbelt has none today: it refuses such programs+  (`PT_INTERP`), so everything we run is *static* (all in one file).++**Hard parts, honestly:**++- **Size.** rustc is big: the compiler itself (most of it is LLVM, a+  huge C++ library) is about 150-200 MB, and the standard library for one+  target another 100-300 MB. Our whole root partition is 32 MiB today.+- **Memory.** LLVM is hungry: a hello world needs a few hundred MB of RAM,+  a real crate 1 GB or more. Our QEMU runs use QEMU's default of 128 MiB.+- **QEMU speed.** Our tests run QEMU without KVM, i.e. by emulating+  every instruction in software ("TCG"), roughly 5-20x slower than the+  box. With KVM (hardware virtualization) it's near native. Then add the+  kernel's own speed: Rustbelt's FAT driver re-walks the path on every+  read, for example, and nobody has measured page-fault or disk speed.+- **64 file descriptors per process** (`MAX_FDS` in the kernel). cargo,+  rustc and the linker can open more than that at once (many `.rlib`+  files, jobserver pipes, threads).+- **No `root=` option**, and only ext2 roots. A toolchain partition can't+  be chosen at boot. It has to sit on the one ext2 root (so the root must+  grow to a few GB) or be mounted elsewhere, e.g. as another ext2 disk+  that appears at `/mnt/diskN`.++### Stage 1: a C compiler that runs on Foundry OS++*In plain words:* first prove that "compile on the target" works at all,+with a small compiler. **tcc** (Tiny C Compiler, ~300 KB, compiles and+links in one program, needs no separate assembler or linker) is the+easiest first step. Then a real one: **gcc** (with binutils' `as` and+`ld`) or **clang/lld**, built static against our musl.++- *Size:* **M** for tcc (one port, like dash, plus its own small libc+  headers path); **L** for gcc/clang (big builds; gcc needs `as`, `ld`,+  `cc1` and lots of processes; clang is one big LLVM program, a good+  rehearsal for rustc).+- *Done when:* `make test` runs `tcc -static hello.c -o /tmp/hello &&+  /tmp/hello` on Foundry OS and prints the line; later the same with+  `gcc`, and on-target gcc builds `c/fprobe.c` and it passes 40/40.+- *OS-side:* ports/tcc, ports/binutils + ports/gcc (or clang), the musl+  headers and `crt*.o` installed into the root (`/usr/include`,+  `/usr/lib`), a bigger root partition.+- *Kernel (light, maybe nothing):* whatever the ports trip over, found+  the usual way (fprobe-style checks, docs/syscall-gaps.md).++### Stage 2: what rustc and cargo need from the system++*In plain words:* rustc and cargo are ordinary Linux programs, but big+and demanding ones. Before porting them, make sure the system can carry+them: room on disk, enough memory, fast enough files and memory mapping,+lots of threads and child processes, and a way to get libraries+(*crates*).++| Need | Why | Kernel or OS |+|---|---|---|+| **Either** a dynamic linker **or** a fully static rustc | Official rustc builds load `librustc_driver-*.so` at run time. Option A: Rustbelt loads `PT_INTERP` programs and we ship musl's `ld.so` (musl's `libc.so` is its own dynamic linker). Option B: build rustc statically (no `dylib` driver, `crt-static`, LLVM linked in). B needs no kernel work but a custom rustc build. | A: **kernel** (load the interpreter, pass `AT_BASE`/aux vector) + OS (ship `ld.so`). B: OS only |+| RAM: 1-2 GB | LLVM | **kernel**: confirm/raise how much RAM Rustbelt can use; OS: `-m 2G` in our QEMU runs |+| Disk: a few GB of ext2 | toolchain + `target/` dirs | OS (bigger root or a second ext2 disk); **kernel**: ext2 performance and large-file/large-dir behaviour |+| `mmap` of big files, `mremap` | rustc mmaps `.rlib`/metadata files; musl's `realloc` tries `mremap` (25, still ENOSYS, K10) | **kernel** (file-backed `mmap` speed, `mremap`) |+| More than 64 fds | rustc + linker + jobserver | **kernel** (raise `MAX_FDS`) |+| Threads, jobserver pipes | rustc uses threads; cargo hands out build slots through a pipe (`jobserver`) | mostly there (K-tests pass); check `poll` on pipes under load |+| Spawning many processes | cargo starts rustc, rustc starts the linker (`cc`/`ld`); Rust's `Command` uses musl `posix_spawn`, i.e. `clone(CLONE_VM \| CLONE_VFORK)` | **kernel** (check that clone flag set; today's table lists fork-like `SIGCHLD [+ CLONE_VFORK]` and threads) |+| `/proc/self/exe` and more `/proc` | rustc finds its sysroot through `current_exe()`; cargo reads `/proc/cpuinfo`-style info for job counts | `/proc/self/exe` exists; **kernel** for the rest (K7) |+| Crates | `cargo build` downloads from crates.io over **HTTPS** | Option A: TLS (an OS-side port, e.g. rustls in a tool or BearSSL) + DNS, both on our network stack. Option B (simpler first): an **offline vendored registry** (`cargo vendor` on the box, copied in, `--offline`). OS only |++- *Size:* **XL**. Each row is small to medium, but together they are a+  lot, and some (RAM, fds, the dynamic linker) are design changes in the+  kernel.+- *Done when:* a stress program (C or Rust, cross-built) on Foundry OS+  maps a 200 MB file, runs 16 threads, opens 200 fds, spawns 500+  children through `posix_spawn`, and the frame count is flat afterwards.+  Plus `free` shows ≥ 1.5 GB, and `df` shows ≥ 3 GB on the toolchain disk.+- *Kernel requests for Edward (kept light):* RAM limit, `MAX_FDS`,+  `mremap`, `CLONE_VM|CLONE_VFORK`, and (only if we choose option A) the+  `PT_INTERP` loader. Everything else is OS-side.++### Stage 3: rustc and cargo on Foundry OS++*In plain words:* cross-compile rustc and cargo on the box for+`x86_64-unknown-linux-musl` (the same Linux ABI Foundry OS speaks), copy+them onto a Foundry OS disk, and use them there. Our own `no_std`+programs (rbsh, the tools) need only `core`/`alloc` and our `rbrt`, so+they are a good first real test.++Cheaper wins on the way, each its own milestone:++1. `rustc --version` runs on Foundry OS (proves loading, size, memory).+2. `rustc hello.rs` (no cargo) builds a static hello that runs on Foundry+   OS. Needs rustc to find a linker: our on-target gcc/`ld` from stage 1,+   or rustc's bundled `rust-lld`.+3. `cargo build --offline` of a one-file crate.+4. **End:** `cargo build --offline --release` of `rust/rbsh` (our shell)+   **on Foundry OS**, and the resulting `rbsh` runs the v2 self-test.++- *Size:* **XL**. The cross-build of rustc itself is long (hours on the+  box, many GB of disk), and every failure on target is slow to debug+  under TCG.+- *Done when:* step 4 passes in a `make test-selfhost` (allowed to be+  slow, outside the normal `make test`).+- *OS-side:* the cross-build recipe (`ports/rust`, pinned like musl),+  the toolchain disk, the vendored registry, `make test-selfhost`.+- *Kernel:* nothing new beyond stage 2, plus whatever stage 3 finds.