ad-hoc dispatch to managed workflow surface, in one afternoon

the single-afternoon timestamp is the lede. between morning coffee and end-of-day on May 15 2026, khaos.machine shipped Step Functions workflow adoption, multi-provider model routing, user-key minting, workflow stage-stamping primitives, a chain-stall watchdog, and the GUI to surface all of it. the breadth is the body — and the breadth traces back to a decision made several weeks earlier about how to separate dispatch from orchestration.

Step Functions: the dispatch architecture change

prior to May 15, khaos-tools dispatched LLM calls ad-hoc: each analysis node in the Reflex DAG called the provider directly, managed its own retry budget, and reported its own status back through the WFL event channel. this worked at low volume. it did not compose. when a chain of 40 scene analyses ran in parallel, the failure mode of one node was invisible to the others until the summarize node ran and found missing output files.

f9ed21e9 landed the Step Functions workflow runner — epics #106, #120–#126 in the khaos-tools backlog. the runner wraps the existing analysis nodes in a state machine managed by AWS Step Functions. each step is discrete, idempotent, and auditable: the Step Functions execution history records every state transition, every retry, and every failure. the Reflex DAG remains the local orchestrator for WFL; Step Functions is the managed surface for the cloud-execution portion of the chain.

3f74b696 and dd96f8dc ported the scene analysis and character analysis nodes to the new runner. a0099e1b added the fanout pattern — parallel step execution for independent scene nodes — with explicit join semantics at the summarize step. bea34768 added the execution history surface to khaos-tools’ --json output, so khaosd can display per-step status without polling the Step Functions API directly.

multi-provider model routing

the model routing arc that landed at d51dc2c5, 3718d4b2, 4851a738, and 978fcd81 addressed a different layer: which LLM provider handles which request, and how that decision changes per-user and per-job.

d51dc2c5 added the GET /v1/models proxy endpoint — an API surface that aggregates the available models from all configured providers and returns them as a unified list. the caller does not need to know which provider holds a given model; it queries the proxy and receives the full catalog. 3718d4b2 added per-user default model: each user can pin a preferred model that applies when no job-level override is present. the preference is stored in the user’s profile and respected by every analysis job the user dispatches.

4851a738 added job-level model override via the /seed endpoint. when a job is seeded, the caller can specify a model identifier; the routing layer resolves the identifier to a provider and bypasses the user’s default. this is the mechanism that allows a user to run a quick analysis against a cheaper model while keeping a higher-capability model as their default for standard work. 978fcd81 added route-by-prefix: model identifiers that begin with a provider prefix (groq/, openrouter/, anthropic/) are routed directly to the named provider without touching the default selection logic. this gives the caller precise control without requiring knowledge of the underlying routing table.

API key minting: kkm_ user keys

8d0afa93 and 8de6efa8 landed the API Keys backend. khaos-tools now mints user-scoped keys prefixed kkm_. the mint operation produces a key that is bound to the calling user’s identity, scoped to a set of allowed operations, and revocable without affecting other keys issued to the same user.

the kkm_ prefix distinguishes user-minted keys from service-to-service keys and from the khs_ storage keys introduced in the Khaos Storage integration. the namespace discipline matters for incident response: a compromised kkm_ key can be revoked immediately by the issuing user without requiring an operator to intervene, and the audit log entry carries the key’s prefix so the affected surface is immediately legible.

the list and revoke endpoints at 8de6efa8 close the CRUD loop. the GUI surfaces the management flow in the user menu (described below).

stage-stamping primitives and the chain-stall watchdog

f0776a30 and 529eeae0 landed the workflow stage-stamping primitives from epics #127–#138. each Step Functions execution step now writes a stamp to a well-known S3 path on entry and on exit. the stamp records: step name, execution ID, input hash, start timestamp, exit code, and output artifact path. stamps are append-only; no step can overwrite another step’s stamp.

the fanout terminal stamp at 529eeae0 is the mechanism that allows the join step to verify that all parallel branches completed successfully before proceeding to summarize. the join step reads the fanout stamps and fails explicitly if any branch’s terminal stamp is absent or carries a non-zero exit code.

c43d2691 added the chain-stall watchdog from epic #161. the watchdog runs as a periodic check alongside the execution: if no new stamp has been written within a configurable window (default: 8 minutes), the watchdog emits a chain.stall event to the WFL event channel and marks the execution as requiring operator attention. this is distinct from a timeout — the execution is not cancelled, and the stall can be resolved by a branch that was delayed by a provider rate limit. the stall signal exists so the operator is not watching a silent progress bar for 45 minutes before discovering that a branch hung.

job files timeline

cf74dc09 added GET /v1/jobs/{id}/files, which returns the run-commit timeline for a job: a time-ordered list of files written by the job’s execution, keyed by step name and timestamp. the endpoint is the data source for the RunFilesTimeline panel in the GUI (described below). the timeline makes the job’s execution concrete: the reviewer can see which step produced which output file, in what order, and whether any file is missing relative to the expected step manifest.

the GUI: surfacing the new dispatch layer

5099496 added multi-provider UI support — the ability to configure Groq and OpenRouter alongside the existing OpenAI surface. the provider forms follow the same pattern established during the March provider expansion: credential input, capability declaration, and a connection test before the configuration is saved. 9f458b0 added the default-model picker: a dropdown in the user preferences panel that presents the unified model catalog from the GET /v1/models proxy and saves the selection as the user’s default. 4430fd7 added the API Keys management panel in the user menu — mint, list, and revoke kkm_ keys without leaving the application.

a2cf3a7 added the RunFilesTimeline panel to the console view. the panel renders the GET /v1/jobs/{id}/files response as a vertical timeline: each step is a row, each output file is an entry under its step, and missing files are highlighted. 8ca9c51 added the dedicated Jobs view — a full-page surface that displays all active and recent jobs, their current step, elapsed time, and a link to the RunFilesTimeline detail. prior to this commit, job status was visible only in the console’s streaming log.

what the single-afternoon shape reveals

each of these arcs had been in flight for weeks before May 15. the Step Functions runner had been in review since late April. the model routing work predated it. the stage-stamping primitives had been specced but not implemented. the reason they all landed on the same day is that the interface contracts between them had been established in advance: the runner writes stamps; the watchdog reads them; the timeline endpoint surfaces them; the GUI renders them. each piece could be built and tested against the contract before the others existed.

the single-afternoon landing is not evidence of exceptional velocity. it is evidence that the sequence was planned correctly — the interfaces were agreed on before the implementations began, so the implementations could merge in any order without breaking each other. that is the reusable pattern: contract-first within a day; implementation-parallel across weeks; merge-anytime when the interfaces hold.