recover means recover — not retry
the structural gap between a retry endpoint and a recover endpoint, and how the first-pass recovery chain resolves it without re-charging work that already succeeded.
before the first-pass recovery chain shipped, the failure surface for a screenplay analysis run looked like this: the job failed, the status endpoint returned a lifecycle string, and the only recovery action available was POST /first-pass/retry — which restarted from the beginning. re-running 115 already-succeeded entities to recover from one failure in entity 116 was the documented behavior.
the problem is not that retry is wrong. retry is the right primitive for recoverable transient errors early in a run. the problem is that retry is the only primitive — which makes it structurally wrong for late-stage failures, for partial failures, and for any run where the cost of repeated LLM synthesis is non-trivial.
the classifier
the recovery chain starts with ClassifyFailure. it reads two signals from the job record: job.Events (the durable event log stored in DDB via AppendEvent) and job.Error (the friendly-wrapped error string). from these it produces a typed failure block: which stage failed, what class of error it was (bearer_expired, llm_failed, push_contention), and whether secondary analyses completed before the failure.
a critical pre-condition was making the classifier’s input signals actually durable. before PR #236, outcome.go published two event types — fanout.finalize_push_contention and fanout.advance_failed — to the in-memory SSE recorder only. the DDB store never saw them. ClassifyFailure scanning job.Events for these kinds returned nothing. the classifier’s push-contention detection was a dead code path.
the consequence of that dead code path was a chain of silent failures with no user-visible signal. when a push-contention event fired at finalize time, outcome.go swallowed the contention error and flipped the stage-parent to Succeeded (line 171’s r.store.Succeed). the grandparent’s job.Error field was empty because the parent had nominally succeeded. the classifier scanned job.Events and found no fanout.finalize_push_contention event because it was never written there. the friendly-wrapped error string also contained nothing useful. from the outside, the job appeared to have completed. the job record in DDB said it succeeded. but the artifacts were not pushed. the user’s dashboard showed a completed analysis backed by a push that silently failed mid-chain.
the outcome.go fix
the fix mirrors the existing precedent at outcome.go:113-114: every publisher.JobEvent(...) call at a classifier-primary event site is paired with a store.AppendEvent(...) call. two lines per site. the classifier now reads what actually happened.
the pairing is the load-bearing detail. the publisher sends the event to the in-memory SSE recorder — the console view, the live job feed, the test recorder in unit tests. store.AppendEvent writes the same event to the DDB job record’s Events slice. the two calls are not alternatives; they serve different consumers. SSE is ephemeral. the DDB log is durable. ClassifyFailure reads DDB. both calls are required for the classifier to work.
the PR adds the paired AppendEvent at two sites: the push-contention swallow (which previously only published SSE) and the advanceStage failure path (same gap). both event kinds are now reachable in ClassifyFailure’s detection logic. the push_contention classification is no longer a dead code path. the git_push_failed classification from an advance_failed event is no longer a dead code path.
the recover endpoint
POST /v1/projects/{owner}/{repo}/first-pass/recover creates a new job, not a mutation of the old one. it takes a ?from=<stage> parameter identifying the first stage to execute. every stage before from is pre-stamped succeeded in the new job’s stage plan before the job starts. no LLM calls for those stages. the new job inherits priorJob.ArtifactPrefix verbatim, so PutResult calls for already-completed entities are idempotent against the existing artifacts.
the gate ordering matters. the recover handler checks in this sequence: no prior job (422), prior already succeeded (409 — there is nothing to recover), a job currently running (409 — race guard via a typed store scan on type = "first-pass" rows, not a raw row cap). only after those checks does it validate from and start the new job. the gate precedence prevents both duplicate work and the harder-to-debug case where a recovery attempt races a still-running analysis.
the from stage defaults to the first failed stage from the classifier output when the caller omits it. this means the common path — “just recover from where it stopped” — requires no parameter. the parameter exists for cases where the operator wants to re-run from an earlier stage, for example after patching a prompt that affected a stage that had already completed in the prior run.
what the GUI sees
the status endpoint now returns a failure block alongside the lifecycle string when the job is in a terminal failure state. the block carries the failure classification, the failed stage, and a secondary_analyses_ready flag. this gives the GUI enough information to offer three distinct actions rather than one: recover from the classifier-recommended stage, recover from an earlier stage, or abandon and restart. the secondary_analyses_ready flag answers the question “can we show partial results while the user decides.”
the pre-stamp loop and the typed store scan are the implementation surface. the user-visible payoff is a recovery action that does what the label says.
related engineering posts: first-pass: failure block, recover, concurrent fan-out