evaluating a second runtime

a runtime evaluation that is honest about what the runtime cannot do yet is more useful than one that produces a verdict. the factory’s evaluation of OpenAI Codex as a candidate second runtime for loswf, dated 2026-04-21, produced a verdict only in the sense that it said “not yet, and here is what would have to be true for the answer to change.” the artifact that came out of the evaluation was not a build decision; it was a capability matrix that would still be readable six months later when the matrix’s rows had shifted.

the standing question was whether the factory’s current organization around Claude Code primitives — agents, commands, skills, hooks, AGENTS.md, headless ADW shell drivers — could be moved or mirrored onto Codex. moving meant a runtime-neutral factory; mirroring meant a second runtime with its own adapter layer. the third option — wait — was the most likely outcome going in, and was the outcome the evaluation defended on grounded terms rather than vibes.

the evaluation followed a fixed shape:

1. enumerate the surfaces the current runtime provides that the
   factory depends on
2. for each surface, find the equivalent in the candidate runtime's
   official documentation
3. mark each row as "yes", "partial-yes", or "not proven" with a
   citation
4. separate "minimal redesign", "moderate redesign", and
   "runtime-specific" buckets for the migration cost surface
5. write the verdict against the gaps, not against the headline
   "does the runtime exist"

step one produced a list of seven surfaces: repo-scoped agent instructions, external tool integration, reusable workflow packaging, interactive coding session, headless execution, controlled shell execution, full factory orchestration. step two found official Codex equivalents for six of the seven: AGENTS.md is first-class; MCP is supported in CLI and IDE via ~/.codex/config.toml; plugins, skills, commands, hooks, and subagents appear in Codex documentation navigation; CLI approval modes (Suggest, Auto Edit, Full Auto) provide controlled shell execution; non-interactive mode (codex exec) and a Codex SDK exist for headless work.

the seventh surface — full factory orchestration — was the one the evaluation could not prove. the documentation confirmed many pieces individually, but it did not amount to evidence that long-running orchestration and deterministic session-state inspection would work. the specific gap that broke the proof: there was no official non-interactive replacement for the interactive /status surface. an open issue in openai/codex, dated 2026-01-30, called out the lack of supported JSON/headless access to model, permissions, writable roots, and quota information. a factory that cannot introspect the runtime cannot reconcile receipts against runtime state — which is the very mechanism the loswf phase machine depends on.

the discipline call was to refuse to ship a runtime adapter against an unproven orchestration surface. one feasible path would have been to write the adapter, route a small slice of work through it, and trust that the gaps would close before the slice grew. that path is the standard early-adopter pattern. the factory rejected it on a specific technical reason: receipts written by a runtime that cannot introspect itself cannot be reconciled by the foreman. a missing /status analogue is not a UX gap; it is a state-machine gap. the design doc names this directly:

Headless status/limits introspection remains incomplete.

once the bucket exercise ran, the migration cost surface separated cleanly. minimally portable: AGENTS.md guidance, role prompts, reusable skills, slash-command entrypoints, MCP-backed tools, interactive single-issue sessions. moderately portable: plugin packaging, hook-driven enforcement, headless ADW wrappers, project bootstrap, multi-agent decomposition. runtime-specific: approval/sandbox semantics, session metadata inspection, quota monitoring, any logic that assumed Claude-specific hooks or front-matter. the third bucket is the one that breaks the migration; it is also the one where the proof was missing.

the verdict the document settled on was “not yet, file a tracking issue, revisit on the next Codex documentation release that lands a headless /status analogue.” the tracking issue (codex-phase-receipts.md and the runtime-neutral config migration design) prepared the adapter scaffolding so the migration cost would be low when the proof landed. the factory did not stop work on Codex; it stopped work on the wrong slice of Codex.

a parallel investigation — Thoughtworks Technology Radar volume 34, published the day after the runtime audit — confirmed the discipline call from a different angle. the Radar’s “AI-assisted development increases cognitive debt” theme reads as a generalization of the receipt-reconciliation problem: a runtime that ships fast without supporting deterministic introspection produces work that the next debugger cannot ground. the factory’s own bias is to ship; the Radar’s frame argues that the bias should be modulated for orchestration surfaces specifically, because the operational debt compounds across every issue routed through the runtime.

the durable artifacts the evaluation produced are three documents and one issue:

• docs/designs/codex-runtime-strategy.md — the capability matrix, dated.
• docs/designs/codex-vs-v1-v2-runtime-synthesis.md — the cross-version reconciliation.
• docs/designs/codex-phase-receipts.md — the receipt-contract design that any second runtime would have to honor.
• the tracking issue for the headless /status gap.

none of these required a build. all of them remain readable when the underlying runtime moves. the matrix’s rows have dates; the next evaluation can diff against the current state cheaply. that is the property a runtime evaluation should optimize for — not “decide today,” but “leave the next decider a document they can update without re-doing the work.”

the meta-lesson is that runtime evaluations should produce capability matrices, not verdicts. the verdict in a fast-moving runtime market has a half-life of weeks. the matrix has a half-life of as long as the surfaces it enumerates remain the right surfaces to evaluate. write the matrix; cite the documentation; mark the gaps with dated open issues; defer the build decision against the gaps. the evaluation that ages best is the one that is honest about what it did not prove, because the next evaluator inherits the unfinished proof rather than re-running the work.