Field note 001 — The 218 worktrees that ate a dispatch
A real incident from a real autonomous engine. Numbers are measured, not estimated.
The symptom
A worker dispatch that should have taken minutes produced four hours of silence. No worker, no log, no error. The orchestrator's own progress reports looked normal the whole time — because the thing reporting progress was the thing that had silently died.
The mechanism
Our engine gives every build worker an isolated git worktree (cx-<issue>). Worktrees are created eagerly and cleaned up... aspirationally. Over six weeks, 218 of them accumulated — each carrying a full node_modules and a copy of every migration file.
The dispatch path included a helper that scans sibling worktrees to allocate the next migration version. At 218 worktrees it was scanning 77,000 migration files. Runtime grew past the orchestrator's 120-second foreground ceiling, the shell killed it mid-flight, and the kill left nothing: no worker, no error file, a zero-byte log.
Four hours of "in progress" that was not real.
The three lessons
- Silence is not progress. A dead sub-process and a slow sub-process look identical from above unless liveness has a heartbeat you can falsify. Our fix class: every launch writes a growing log within seconds, and "log not growing" is treated as dead.
- Hygiene debt is latency debt. The worktrees weren't "just disk." Every file a forgotten workspace holds is something a future scan pays for. Cleanup now rides every merge automatically — not a chore, a pipeline stage.
- Foreground timeouts turn slow into absent. Any step that can exceed the orchestrator's ceiling must run detached with its own completion record, or its failure mode is invisibility.
The receipts
- 218 worktrees, ~77k migration files scanned, 120s ceiling, 4h of false progress.
- Fix: cleanup wired into the merge pass + a path-lint that refuses dispatch briefs pointing at files workers can't read (that class alone caused 5 of our 58 dispatches in one week to burn on arrival).
- Since the fix: cleanup is automatic on every merge. The backlog drained from 218 toward the live-worker floor.
More field notes and the durable-work kit: this repo. The engine that generated this incident merged 16 production PRs unattended the same night.