step.20.txt -- b3ubot U3.7: RETRO closure -- retro pair + ledger row + hash backfill + the execution DAG FROZEN as the step's evidence graph. design.md §4's `CLOSED <--retro pair-- RETRO` arrow ("Everything between the gates is machine: ... retro drafting"), §2.13's third lifecycle row ("the instance is FROZEN as evidence and pointed at by the Step's WorkflowDag row"), and §2.3's last ledger op ("close(retro) -> done row + hash backfill") -- as code. Stops here -- does NOT continue into U3.8 or any later P3 UoW. Step: 20 Parent: end_to_end.md §5 U3.7 bullet verbatim ("RETRO closure: retro pair + ledger row + hash backfill + the execution DAG FROZEN as the step's evidence graph"). design.md §4 (retro drafting is MACHINE work; ratify + escalation are the only human gates), §2.13 (the RETRO->CLOSED freeze row), §2.3 (the close(retro) ledger op), §2.6 ("clean tree per closed UoW; commit messages carry the step trail"), §3 (the WorkflowDag entity's kind/dag_uri/status/frozen_at fields). app/orchestrator/ (U3.1-U3.6) is the package this completes P3's per-state row of; app/ledger/retro.py (U2.4) + table_* (U2.3) are the machinery this composes rather than duplicates; app/tools/gitops.py is the ONLY git path. UoW: U3.7 Date: 2026-07-21 Origin: Operator directive 2026-07-21 (same dispatch shape as step.14-19's): draft AND execute steps/step.20.txt for U3.7 in a single pass (the step.7/9/10/11/14-19 convention, NOT a skeleton-split), with the SAME explicit HARD scope boundary -- stop after U3.7, do not continue into U3.8 or any later UoW; the operator directs P3 one UoW at a time with strict per-UoW checkpoints. ## (0) Status EXECUTED in this pass. Not a skeleton-then-later-execute split -- app/orchestrator/closing.py, the CloseError addition, the instance_is_frozen/refuse_if_frozen freeze wall grown into executing.py (+ its verifying.py call sites), the `b3ubot close` CLI subcommand, tests/test_orchestrator_closing.py, the design.md (§2.3/§2.13) / end_to_end.md / docs/orchestrator_state_machine.md edits, the stale "U3.7 pending" docstring sweep, and this step file were all built together, per the operator's explicit "draft AND execute (single pass)" direction. Per the operator's explicit scope boundary (Origin, above), this session STOPS after U3.7 -- U3.8 (the M1 oracle, incl. automatic stage continuation) is the next UoW but is NOT started here: a closed UoW is terminal and the engine returns control to the human between every phase. ## (1) Ground truth probed at drafting (outputs -> named writers) READ design.md §4 in full (retro drafting sits on the MACHINE side of the gates; the diagram draws NO human gate between RETRO and CLOSED -- ratify and escalation are the only two; "freezes the instance at CLOSED as the step's evidence graph"), §2.13 (the lifecycle table's RETRO->CLOSED row; the WorkflowDag ROW is Track B), §2.3 (the ledger-ops list; "close(retro) -> done row + hash backfill"), §2.6 (clean tree per closed UoW -- commit messages carry the step trail), §3 (the WorkflowDag entity's field list: kind, dag_uri, status, frozen_at -- the precursor row uses it verbatim). READ app/orchestrator/ in full: executing.py (the evidence stamps, _load_instance/roundtrip_verify -- reused, not re-derived; the resume path the freeze wall had to guard), verifying.py (the verification DAG + the verifications.jsonl rows the retro transcribes; the per-pass instance rewrite the freeze wall had to guard; _reset_execution_for_repair -- guarded too), state_machine.py (RETRO->CLOSED exists with a U3.7 rationale string -- NO state-machine change needed, a checked finding; CLOSED is terminal, which already refuses every LEGAL re-entry -- the freeze wall is for the ILLEGAL ones), persistence.py (the history list the retro's state-path line and repair count read), ratify.py/drafting.py (the legality-first lesson). READ app/ledger/retro.py in full (U2.4): the surveyed corpus .diff.txt structural template (title/banner/Executed/4 headings/closing line) and its OWN scoping rule -- narrative content must never be FABRICATED; FILL_IN placeholders for what a template cannot know. The U3.7 drafter fills that same structure with MECHANICAL evidence that DOES exist at closure time -- the honest complement, not a contradiction. run_step_gdiff shells to the SHARED scripts/step_gdiff (read in full: START = parent of the skeleton commit -- the commit that first ADDS step.N.txt; END = newest commit touching it) -- so a toy workspace whose closure work commit IS the skeleton commit gets exactly the closure's own diff. READ app/ledger/table_parser.py/table_transitions.py (U2.3: regex-heading table location -- workspace-ledger detection rides it; transition("done") requires commit=; done is terminal, so a re-run backfills via record_commit -- app/cli.py's ledger_close precedent), app/tools/gitops.py (status/diff/add/commit exist, repo-top verified, git_add requires NAMED paths -- no add-everything verb; push/rewrite still refuse), trail.py (logged_call -- the step_gdiff run and the frozen refusals record through it). CONFIRMED dag-level kv attrs round-trip through the REAL bin/dagModel BEFORE designing the freeze on them: a Dag with (frozen_at = "...") parses and re-renders byte-identically (live probe, this session) -- the frozen kv is substrate-legal, not hoped-so. CONFIRMED `git log --oneline -30` for the 2-commit phrasing (step.19's precedent, 529834f/0e9e608). CONFIRMED baseline: full suite 517 passed / 11 skipped before any edit (matches step.19's reported final). CONFIRMED live CLI smoke before committing (scratch workspaces + isolated $B3UBOT_HOME), two legs: (a) git-init toy ws driven to RETRO, then `b3ubot close` -> both instances frozen, evidence-filled retro, WORK commit + REAL short hash backfilled into §(7), real step_gdiff gdiff, GDIFF commit, porcelain CLEAN, state CLOSED, rc=0; re-close -> the terminal refusal verbatim, rc=1; (b) non-git toy ws -> "(not a git repository)" markers in §(7), gdiff skipped and NAMED, state CLOSED, rc=0. One cosmetic live-smoke finding fixed on sight: the retro title doubled the UoW prefix (title_line already carries it). ## (2) What U3.7 delivers 2.1 **app/orchestrator/closing.py** (new; the package's per-state peer of drafting/ratify/executing/verifying): `close_uow( workspace_root, uow, *, output_fn, home=, scripts_home=)` -- the full arc: state legality FIRST (validate_transition(state, "CLOSED") admits exactly RETRO; zero reads on an illegal move), both evidence instances REQUIRED (a RETRO-state UoW has been executed and verified; a missing instance is a named CloseError), freeze both + WorkflowDag pointer rows, resolve the workspace ledger, draft the evidence-filled retro, stamp §(0) + rewrite §(7), the 2-commit git shape with real-hash backfill (git workspaces), advance RETRO -> CLOSED. Plus CloseError in errors.py; all re-exported through app/orchestrator/__init__. 2.2 **Retro drafting V0 (THE core decision, made and owned): DETERMINISTIC, evidence-filled -- not a PAL call.** Why: (1) closure must be reliable/cheap at the exact moment evidence is sealed -- a provider failure/misnarration there is the worst possible place for one; (2) every fact a V0 retro needs already exists as a machine record (engine-state history, both instances' node kvs, verifications.jsonl, the trails) -- transcribing them IS what makes the retro evidence-based; (3) U2.4's own no-fabricated-reflection rule: a PAL-narrated retro is a genuine QUALITY upgrade (real reflection, not transcription) that rides AFTER M1 proves the arc, as policy, not V0 plumbing. FORM matches the corpus convention: `draft_closure_retro()` fills retro.py's surveyed structural template (title/banner/Executed line/Plan vs actual/Gate evidence/Findings/Numbers/closing line) with mechanical evidence -- per-node outcomes, per-pass gate verdicts with exit codes, the engine-state path, freeze/ledger/gdiff facts, node/gate/pass/repair counts -- and its Executed line NAMES itself MACHINE-DRAFTED with reflection explicitly not synthesized (asserted in-suite: content, not just existence). 2.3 **The freeze (V0 semantics, decided and justified):** a `frozen_at` dag-level kv (design §3's WorkflowDag field name VERBATIM, so Track B lifts it unchanged) stamped into BOTH instances -- `freeze_instance()` is idempotent (a re-run after a partial close keeps the original stamp), written through the trail-recorded tool surface, round-trip verified through the real bin/dagModel -- PLUS the code-enforced no-writes wall: `executing.instance_is_frozen()` (a TEXT sniff on the rendered kv, deliberately -- a guard that itself required the substrate parser could not protect evidence on a parser-less machine) and `refuse_if_frozen()` (refusal trail-recorded as `frozen_instance_refusal`, C-15-B: the trail must show what the engine REFUSED to do). Call sites: execute_uow (resume AND fresh-overwrite paths), verify_uow (the per-pass instance write), _reset_execution_for_repair. NOT filesystem read-only bits: chmod is neither portable policy nor honest protection (any process chmods back); the engine's own modules are the writers the design needs to bind, and terminal CLOSED already refuses the LEGAL routes -- the kv wall is for the ILLEGAL ones (a hand-repaired/divergent engine record -- proven in-suite by tampering one). 2.4 **The WorkflowDag pointer precursor (design §2.13 "pointed at by the Step's WorkflowDag row"; Track B not built):** `/workflow_dags.jsonl` (record_workflow_dag/ read_workflow_dags) -- the established file-based-precursor pattern (egress runs / tool_calls.jsonl / verifications.jsonl), one append-only row per frozen instance: ts / uow / step / kind / dag_uri / status / frozen_at -- design §3's own field list, literally. 2.5 **Git commit at closure (decision, justified): COMMIT-AS-MACHINE-WORK, no approval gate.** design §2.6's "clean tree per closed UoW" makes committing PART of closure; the in-loop human gate already approved the applied change byte-for-byte and the ratify gate approved the plan; git_commit sits on §2.7's NON-destructive side (push/history- rewrite still refuse unconditionally -- nothing leaves the machine); a confirm prompt here would be a third gate design §4 does not draw. The 2-commit corpus shape, machine-run: WORK commit (retro + frozen instances + §(0) closure stamp + §(7) "PENDING" + workspace ledger row -> done/PENDING; paths NAMED from git status --porcelain -- no add-everything verb exists on the surface, deliberately), then the REAL short hash backfilled into §(7) AND the ledger row, the SHARED scripts/step_gdiff run (through logged_call -- outcome recorded AFTER the fact), and the GDIFF commit. Porcelain clean after closure, asserted in-suite. **A NON-git workspace closes honestly**: "(not a git repository)" markers in §(7) -- NEVER a fabricated hash (asserted: no 7-hex token appears) -- and a gdiff skip NAMED in the retro (step_gdiff needs git); closure still completes to CLOSED. 2.6 **Which ledger (decision, justified): the TARGET workspace's.** `/end_to_end.md` detected via U2.3's regex-heading machinery with a row for the closing UoW; the row goes active -> done with commit=PENDING before the work commit and gets the real short hash backfilled in the gdiff commit (a row already `done` -- a re-run after a partial close -- takes record_commit, the ledger_close precedent). No doc / no parseable table / no row => a NAMED skip in the retro, never an error (a toy workspace without a ledger is legitimate) and NEVER this repo's own ledger for a target-workspace UoW (the engine only opens the ledger under workspace_root; own-repo untouched asserted in-suite). 2.7 **Human confirmation (decided against, honestly):** `b3ubot close` runs to completion once invoked -- no --yes, no prompt, no Q-8-B guard, because closure READS NO INPUT. Justified against the diagram's own "gates are few" stance: design §4 draws no human gate between RETRO and CLOSED (ratify + escalation are the only two), and every input closure consumes was human-approved upstream. The human act IS invoking the command -- V0's separate-command convention (draft/ratify/execute/verify/close) keeps the human at the wheel BETWEEN phases; automatic continuation is U3.8's M1. 2.8 **CLI: top-level `b3ubot close`** (the sibling row completed): workspace_root + uow + --scripts-home. Presentation-layer error handling only; close_uow stays pure and raises; unattended runs are honest by construction (no input). 2.9 **tests/test_orchestrator_closing.py** (17 new): the git-init toy-workspace happy path (state CLOSED with the full history; retro pair BOTH files; REAL 7-hex hash in §(7) matching the actual work commit; founding+work+gdiff = exactly 3 commits; porcelain CLEAN); the gdiff records the closure range; retro CONTENT asserted (real node kvs incl. applied=hello.py, real per-pass verdicts with exit codes, the state path, the MACHINE-DRAFTED honesty line, all 4 corpus headings); both instances frozen IN THE FILE; pointer rows with matching frozen_at; §(0) closure stamp once with the U3.5/U3.6 stamps preserved; the non-git honest-marker path; ledger closed with the REAL hash AND the named-skip variant; illegal close refuses with ZERO read_file trail entries and no retro file; close-twice terminal; frozen execute AND verify refusals against a TAMPERED engine record with `frozen_instance_ refusal` trail rows; own steps/ + dag/instances/ untouched; CLI unattended green + error presentation. Hermetic $B3UBOT_HOME throughout (conftest). 2.10 **Docs (same commit):** end_to_end.md §5 U3.7 row -> DONE in the U3.2-U3.6 row style; §11 ledger THROUGH the ledger CLI (transition U3.7 active --step step.20; close U3.7 PENDING; hash backfilled in the gdiff commit -- the step.13-19 shape). design.md §2.3 delivered-state (all four ledger ops now exist), §2.13 delivered-state (the freeze row is live; the pointer precursor named). docs/orchestrator_state_machine.md (U3.7 delivered; edge-table annotation). Every stale "U3.7 pending"/"nothing drafts the retro pair yet" claim updated (verifying.py stamp + completion message + docstring; state_machine.py rationale string; app/orchestrator/ __init__.py). No new docs/*.md file -- this file is the writeup. EXPLICITLY OUT OF SCOPE (named here per the operator's brief, not silently built -- and NOT continued into after this step, per the operator's explicit stop-here scope boundary): - U3.8 M1 (the one-complete-UoW arc, mock AND real provider; automatic stage continuation lives there). -> next - A PAL-narrated retro (the named post-M1 quality upgrade of 2.2's deterministic V0). -> later - T2+/T3/T4 tiers (unchanged from U3.6's boundary). -> later - Auto re-draft on amendments (Q-17-A stays open). -> later - Track B / P4 (WorkflowDag stays the JSONL pointer precursor; no DB row). -> later - Pushing anything anywhere (git_push still refuses; B-3). -> never (without explicit operator direction) ## (3) Verification gates G1 tests/test_orchestrator_closing.py 17/17 PASS live: happy path on a git-init toy workspace, gdiff range, retro evidence CONTENT, freeze in-file, pointer rows, stamp-once, non-git honest markers, both ledger variants, legality, terminality, missing-instance refusal, frozen execute/verify refusals, own-repo hygiene, CLI -- scripted adapters, injected I/O, zero network. G2 The REAL closure artifacts proven, not mocked: the .gdiff.txt comes from the SHARED v3_claude/scripts/step_gdiff run against the toy repo's actual commits; the §(7) hash is the actual `git rev-parse HEAD` short form of the work commit (asserted equal); porcelain clean after closure (design §2.6's own rule, asserted). G3 Frozen-instance enforcement proven at the FILE level, not just the state level: the engine record tampered back to EXECUTING/VERIFYING (the illegal route the state machine cannot see) -> execute_uow/verify_uow refuse loudly naming FROZEN evidence, with `frozen_instance_refusal` rows on the tool trail (C-15-B). G4 Substrate round-trip: dag-level `(frozen_at = "...")` kv confirmed byte-round-trip-safe through the REAL bin/dagModel BEFORE the design committed to it (§(1) CONFIRMED); every frozen instance re-verified through roundtrip_verify at freeze time. G5 Live CLI smoke (not just tests), two legs -- §(1) CONFIRMED: git-init scratch ws close rc=0 (2 commits, real hash, clean porcelain, evidence-filled retro) + terminal re-close rc=1; non-git scratch ws close rc=0 with honest markers. Confirmed live before committing. G6 Full suite before this step's edits: 517 passed / 11 skipped (confirmed live, matching step.19's reported final). After: 534p/11s pre-step-file (+17 closing); 536p/11s once this step.20.txt lands in the corpus (+2 self-inclusive pickups -- the steps 10-19 growth pattern). Zero regressions; zero existing tests edited. G7 Scope boundary held: no U3.8 arc, no automatic verify->close continuation, no PAL retro, no Track B row, no T3/T4 flow, nothing pushed anywhere; this repo's own steps/ and dag/instances/ untouched by every test (asserted in-suite); this session stops after U3.7. G8 §11 ledger row updated THROUGH the existing `ledger` CLI (transition -> active --step step.20, close -> done/PENDING), against the REAL end_to_end.md -- dogfooding, the step.13-19 precedent. G9 The closure convention matches this repo's own: the toy workspace's machine-run closure leaves the SAME artifact set (step.N.txt with §(0) stamps + §(7) real hash, step.N.diff.txt, step.N.gdiff.txt, done ledger row, 2 commits) that steps 13-19 left by hand -- the U3.7 bullet's "indistinguishable in form" bar, checked artifact by artifact in G1/G2. G10 b3ubot porcelain clean after each commit; retro pair via the SHARED v3_claude/scripts/step_gdiff (steps/step.20.gdiff.txt); exactly 2 commits (work + gdiff-with-hash-backfill-folded-in), the step.7-19 precedent. ## (4) LOCKs C-20-A b3ubot porcelain clean after each commit. C-20-B FROZEN EVIDENCE IS NEVER REWRITTEN: after RETRO->CLOSED, executing.py and verifying.py refuse loudly (trail- recorded) before touching either instance -- even under a tampered/divergent engine record. The freeze kv is design §3's `frozen_at`, verbatim, so Track B lifts it unchanged. C-20-C THE RETRO TRANSCRIBES, NEVER INVENTS (U2.4's rule carried into the machine): the V0 closure retro contains ONLY mechanical facts drawn from the real evidence records, names itself MACHINE-DRAFTED, and marks reflective narrative as not synthesized. A PAL-narrated retro is a post-M1 policy upgrade, never smuggled in. C-20-D NO FABRICATED HASHES: a non-git workspace gets the honest "(not a git repository)" marker in §(7) and a NAMED gdiff skip -- never an invented 7-hex token (asserted in-suite). C-20-E COMMIT-AS-MACHINE-WORK STAYS INSIDE §2.7's LINE: closure commits via gitops' non-destructive verbs only, with NAMED paths; git_push/git_rewrite_history still refuse unconditionally; nothing leaves the machine (B-3/B-5). C-20-F THE TARGET WORKSPACE'S LEDGER ONLY: closure never opens this repo's own end_to_end.md for a target-workspace UoW; absence is a NAMED skip, not a fallback. C-20-G NO SILENT SCOPE CREEP: U3.8, the PAL retro, T3/T4, Track B and Q-17-A named out of scope in (2) and confirmed not started (G7); the operator's explicit "stop after U3.7" direction is honored. C-20-H standing locks inherited: B-3 (no disclosure/no push); B-4/B-5 (no provider call in closure -- nothing new leaves the machine; the trails record everything); C-15-B (every closure file op + the step_gdiff run + every frozen refusal on the tool trail); C-18-B/C-19-E (the instance files stay the truth -- the freeze makes them IMMUTABLE truth). ## (5) Open questions Q-20-A Should the freeze also snapshot per-pass verification graphs (step.19's Q-19-C deferred exactly this to U3.7)? DECIDED HERE for V0: no -- the frozen verification instance holds the FINAL pass, verifications.jsonl holds every pass's verdicts, and the retro transcribes both; a per-pass graph-file family would multiply files for evidence the JSONL already carries. Recorded as decided, not left open -- revisit only if Track B's real WorkflowDag rows want per-pass graph pointers. Q-20-B Should `b3ubot close` verify the workspace suite one last time before committing (a paranoid re-check of U3.6's green)? LEAN: no -- VERIFYING->RETRO already required green gates and re-running them would make closure slow and double-spend the oracle; the engine record's history IS the proof the gates ran. Revisit at U3.8 if the M1 arc surfaces a real staleness window between verify and close. Q-20-C Where should a CLOSED UoW's engine-state record eventually retire to (it stays in ~/.b3ubot/orchestrator/ forever today)? LEAN: nowhere until Track B -- the file IS the durable record and P4's Step row will subsume it; premature archival would break the resume reader for no consumer. ## (6) Acceptance - [x] app/orchestrator/closing.py: legality-first close_uow -- freeze + pointer rows + evidence retro + §(0)/§(7) stamps + 2-commit git shape + RETRO->CLOSED (2.1, G1, G2, G5). - [x] Retro drafting V0 decided and owned: deterministic evidence-filled template, corpus FORM, MACHINE-DRAFTED honesty line; PAL retro named post-M1 (2.2, C-20-C). - [x] Freeze semantics decided and enforced: frozen_at dag-level kv (substrate-verified) + refuse_if_frozen in executing AND verifying, refusals trail-recorded, proven against a tampered record (2.3, G3, G4, C-20-B). - [x] WorkflowDag pointer precursor: workflow_dags.jsonl rows with design §3's field list (2.4). - [x] Git-at-closure decided: commit-as-machine-work justified against §2.6/§2.7/§4; real-hash backfill into §(7) AND the ledger row; honest non-git markers, no fabricated hashes (2.5, G2, C-20-D, C-20-E). - [x] Which-ledger decided: the target workspace's, detected; named skip otherwise; own repo never a fallback (2.6, C-20-F). - [x] Closure keeps NO human confirmation once invoked -- justified against design §4's own gate set; `b3ubot close` is the no-input sibling (2.7, 2.8, G5). - [x] 17 new tests, zero regressions, zero edited tests (517p/11s -> 534p/11s -> 536p/11s with this file) (2.9, G1, G6). - [x] design.md §2.3/§2.13; docs/orchestrator_state_machine.md; end_to_end.md §5 row + §11 ledger via the `ledger` CLI; every stale "U3.7 pending" docstring (2.10, G8). - [x] Scope boundary held: U3.8 + PAL retro + T3/T4 + Track B not started (G7, C-20-G) -- this session stops after U3.7. - [x] Retro pair: this file + step.20.diff.txt + step.20.gdiff.txt via the SHARED scripts/step_gdiff (G10). - [x] Exactly 2 commits (work + gdiff-with-backfill), not 3 (G10). ## (7) Hash backfill WORK commit (app/orchestrator/closing.py + errors + the freeze wall in executing.py/verifying.py + state_machine rationale + __init__ + cli + tests + design.md/end_to_end.md/docs + this file): 7cbee41 GDIFF commit (step.20.gdiff.txt via scripts/step_gdiff): (HEAD) End of step.