the test that says green on your laptop is not the test that ships your binary

the architectural shift is the frame: khaos-foundation stops being the release orchestrator; khaos-builder takes it over. the 48-hour CI marathon that followed is the teaching beat — not because CI is hard, but because it surfaces the exact gap between what passes locally and what ships a binary. that gap is structural, not accidental.

khaos-builder gets a dashboard

khaos-builder is the release plane for the workspace. it runs as a macOS LaunchAgent on a dedicated Mac Mini, holds the Apple Developer keychain unmediated, and drives the full cross-repo release DAG from tag to signed DMG to Sparkle appcast update. prior to April 3, it was a headless daemon with a CLI surface: operators triggered pipelines via khaos-builder run and watched output in logs.

783ed4c added the embedded web UI — an HTTP server baked into the khaos-builder binary, serving a Vue 3 dashboard via go:embed. 9e11b55 wired the network access layer: the dashboard can proxy API calls to GitHub, report on the status of cloud provider keys, and display cross-repo pipeline state without requiring the operator to SSH into the Mac Mini. 2b414aa added auth so the dashboard is not an open port on the build machine. 6c3caa5 added trigger validation — clicking “run” on a pipeline performs a pre-flight check (required secrets present, target tag exists, dependent repos at compatible states) before dispatching. d851747 added version-bump buttons so operators can increment component versions and trigger the downstream chain directly from the UI, without constructing a git tag manually.

513e101 redesigned the dashboard layout. 682998c added VoltAgent — a diagnostic sub-panel that displays the current active agent step, its inputs and outputs, and whether it is waiting on an external event (keychain prompt, notarization ticket, GitHub API rate limit). 190713f added the build detail view: each pipeline run’s log, timed steps, artifact checksums, and the signing ticket from Apple’s notarization service. b4f8796, 2f4d466, and 299c676 added sourceless pipeline support — pipelines triggered via the GitHub API rather than a local tag event, so the operator can rerun a failed stage without re-tagging.

the CI marathon: what 25 fixes look like in sequence

between 4c9124e and 128d71d, roughly 20 fixes landed across the khaos-builder CI workflows. the session writeup at dbf5189 documents the sequence. each fix is individually small; together they reveal the categories of failure that local validation cannot catch:

keychain unlock. the macOS code-signing step requires the keychain to be unlocked before codesign can access the Developer ID certificate. on the Mac Mini’s LaunchAgent user, the login keychain is not automatically unlocked on boot. the first run of a pipeline that hit the signing step silently hung: codesign waited for a keychain prompt that never appeared. the fix was a security unlock-keychain call at pipeline start, keyed to KEYCHAIN_PASSWORD in the runner’s environment. this is not discoverable by running codesign locally on a logged-in developer machine where the keychain is already unlocked.

foundation paths. khaos-foundation’s build script assumed it ran from the repo root and constructed paths via $(pwd)/build/. on the CI runner, where the working directory was the actions workspace root rather than the foundation submodule, those paths resolved to non-existent locations. three cd calls and two realpath substitutions fixed the majority of path failures.

sed -i quirks. the appcast generation script used sed -i 's/old/new/' — BSD sed syntax valid on macOS but not on the CI runner’s Linux image. the fix was sed -i '' 's/old/new/' for macOS and a conditional for Linux. this class of failure only surfaces when the CI runner’s OS differs from the developer’s machine.

Windows matrix. the khaos-builder release workflow included a Windows build matrix step that had been added speculatively and never validated. when the matrix ran in CI, it failed on path separators and a missing tool. the fix was to remove the Windows matrix entry and file it as a separate future task rather than block the release workflow.

sourceless auto pipelines via GitHub API. pipelines triggered by workflow_dispatch rather than a push event do not have a github.sha in the context. the pipeline’s stage-stamping logic used github.sha as the run identifier. three stages failed with a nil pointer when dispatched sourceless. the fix was a fallback to github.run_id.

glob upload and --generate-notes. gh release create with a glob pattern for artifact upload requires the glob to match at least one file or it exits non-zero. in one pipeline stage, the glob was constructed before the build step confirmed its output path, so the glob matched nothing. --generate-notes had been added to the release command without testing against the GitHub API’s behavior when the tag already had partial notes from a previous failed run — it overwrote them. both were fixed by the end of the marathon, documented in dbf5189.

khaos-foundation’s side of the handover

on the foundation side, the same 48-hour window produced parallel work. 27208c3 fixed the appcast enclosure URL, which had been pointing to the staging CDN rather than the production CDN after a domain migration. 242eed5 moved foundation builds to trigger from a released khaos-manager tag rather than from foundation’s own CI push — this aligns the release cadence with the component that produces the binary being packaged.

11051fc refreshed the landing page with Playwright-driven screenshots of the running application, replacing the hand-authored mockups that had been in place since the initial site launch. the screenshot step runs on every foundation release and captures the current state of the dashboard, the projects view, and the coverage report. e6a4f04 added the stale-stack-artifact guard: if a CloudFormation stack deployment fails mid-run, the subsequent run now detects the partially-deployed stack and rolls back before attempting a fresh deploy. f16a0cb and 4a91b92 completed the design-system refresh on the landing page, pulling in the new @khaos/ui token set.

the structural gap the marathon exposed

local development on a signed-in macOS machine, running ./build.sh in the terminal, does not exercise the keychain unlock path, the foundation path construction, the sed -i portability, or the workflow_dispatch context shape. each of these is only present in the environment where the binary actually ships.

this is not unique to khaos.machine. it is the general shape of the problem: the CI environment is not a faster developer machine. it is a different machine, running a different OS image, under a different user account, with different ambient state, triggered by a different event shape. the CI green on a laptop confirms that the code compiles. it does not confirm that the release pipeline completes.

the khaos-builder dashboard now surfaces the exact failure point for each of these categories as a named diagnostic step. when the next keychain hang occurs — and it will, on the next new Mac Mini or new runner image — the operator sees “keychain unlock” in the step list rather than a 10-minute silent hang followed by a timeout.