silent failure is the failure

a wrong outcome that looks like a working pipeline is more dangerous than a loud crash. the factory shipped zero PRs in twenty consecutive sweeps; every layer logged success. the pipeline was failing silently at five distinct points, each defect masked by the next one downstream. once the stack was visible, five one-line fixes restored throughput; the factory went from 0 PRs in 20 sweeps to 13 PRs in 10 sweeps in a single session. first noted in the devlog entry.

the bugs, in the order they had to be peeled:

1. the decomposer post-agent hook was looking for a marker the agent no longer emitted. xs/s plans approved by the plan-reviewer never advanced from factory:phase:decomposing to factory:phase:building. children that did spawn re-ran factory:phase:planning despite the parent plan already being approved. fix in #320: decomposer hook accepts the <!-- loswf:no-decompose --> marker; children spawn directly at factory:phase:building.

2. the plan-reviewer verdict matcher used a bare startswith("approved:"). the agent’s required marker block — <!-- loswf:plan-review --> — was emitted on the first line. every verdict therefore looked ambiguous to the hook and got flagged factory:status:needs-attention. fix in #322: anchor the check with lstrip+startswith("approved:"). fix in #331: strip leading HTML markers before the match, so the agent’s required marker block does not defeat the anchored check.

3. the project deny-list blocked git push --force* outright. foreman could not rebase a feature branch without escalating to the operator. fix in #325: narrow the deny rule to git push --force* origin main* (and master*). force-pushes to feature branches are now in-scope for the factory.

4. gh pr merge --delete-branch failed when a sibling worktree had the branch checked out — which was the common case under parallel builders. the reviewer agent’s merge sequence aborted in a state where the PR was merged on github but the local refs were tangled. fix in #330: merge with --delete-branch=false, prune the worktree, then delete the refs separately.

the diagnostic that surfaces the pattern is a phase-state histogram run at the top of each sweep:

for p in intake investigating researching product planning red-team \
         plan-review decomposing building awaiting-children review rollup; do
  n=$(gh issue list --state open --label "factory:phase:$p" --json number -q '. | length')
  echo "$p: $n"
done

the bug-to-symptom mapping the session produced:

each of these is a phase that runs on every sweep, logs success, and moves nothing forward. there is no exception. there is no crash. the receipt the hook writes says the agent completed. the only signal that anything is wrong is the count of items in the next phase staying flat.

the pattern is not specific to a factory pipeline. it is the same shape as any layered system where one stage’s success contract is “i wrote a receipt that says i ran.” if the next stage’s pickup rule does not match what the receipt actually says, the system runs forever, producing receipts, shipping nothing. each subsequent layer’s contract is honored against the receipts it gets, which are valid; it just gets fewer of them than expected. the upstream stage looks healthy. the downstream stage looks idle. the bug lives in the protocol between them, where neither stage owns the integration.

three rules followed from the session:

make zero-progress an alarm. the histogram above is now run on every sweep. when a phase that should be draining stays flat for three consecutive sweeps, the foreman opens an audit ticket. the receipt-says-success contract is no longer trusted; the count of items leaving the phase is the only ground truth.

check marker drift explicitly. the bug class behind #320, #322, and #331 is the same: the agent prompt was updated, the hook matcher was not. every agent’s emitted comment shape is now a hook contract — changes to either require a paired change to the other. the audit pattern: read the last comment on an issue stuck in a phase, then read what the hook looks for.

gh issue view <num> --json comments -q '.comments | reverse | .[0].body' | head -10
grep -B2 -A8 "def post_<phase>" .loswf/tools/loswf_post_agent.py

if the comment does not startswith() what the hook expects, that is the bug. five minutes of work per phase; runs in any factory whose phases write structured comments.

narrow deny-lists. never blanket-block operations. the #325 fix is generalizable: the factory cannot orchestrate itself if its own primitives are denied at the settings layer. deny rules should specify the dangerous combination (operation + target), not the bare operation. a deny on git push --force* blocks every feature-branch rebase; the actual hazard is git push --force* origin main*.

the session also surfaced a fourth rule that did not make the audit guide but is worth recording: dogfood the fix on the next PR. #330’s worktree-safe merge sequence was merged by the reviewer using its own new sequence on the very first try. a fix that catches its own regression on first contact is a fix that has been written correctly; one that requires a follow-up to validate is one whose contract was incomplete. the dogfood test is faster than any CI gate and stronger than any unit test, because it is the entire production path running end to end.

the five-bug stack is now a recurring failure mode in the factory’s audit guide. the histogram is part of every sweep’s preamble. the silent-failure class is no longer silent because the contract for “something is wrong” has been pulled out of the per-stage receipts and into the cross-stage flow rates.