step.14.diff.txt -- narrative retro for step.14 (U3.1: State machine + engine-state persistence + the RATIFIED->EXECUTING execution-DAG compiler) Executed: 2026-07-21, same session as the skeleton (no separate skeleton-then-execute split -- app/orchestrator/'s three real modules, the three new test files, docs/orchestrator_state_machine.md, the design.md/end_to_end.md edits, and this step file were all built together, per the operator's explicit "draft AND execute (single pass)" direction, matching step.7/9/10/11/13's rhythm). Companion artifact: step.14.gdiff.txt (automated, scripts/step_gdiff). This is P3's FIRST real orchestrator code -- P2 (U2.1-U2.4) closed at step.13; this step opens P3 and, per the operator's own explicit instruction, stops there: U3.2 is not started, even though nothing structurally prevents continuing (this same operator note names the precedent directly -- a prior agent, in the U2.2 session, autonomously continued through U2.3/U2.4 unprompted, and that turned out fine on review, but was explicitly not asked for; this step honors a deliberate one-UoW-at-a- time checkpoint instead of repeating that pattern). ## Plan vs actual ``` +---+----------------------------------+------------------------------------------+ | # | Planned (§) | Actual | +---+----------------------------------+------------------------------------------+ | 1 | Ground the state machine in | Traced the diagram's box-drawing | | | design.md §4's ASCII diagram, | characters character-by-character to | | | including the ambiguous | resolve the "amendments" edge's | | | "amendments" edge direction | direction with confidence (RATIFIED -> | | | | SKELETON, not the reverse): the `^` | | | | arrowhead sits under SKELETON and the | | | | "+--amendments--+" line's right end | | | | joins the vertical bar falling FROM | | | | RATIFIED. Documented the reading | | | | explicitly in state_machine.py's own | | | | docstring since the diagram never states | | | | it in prose -- a genuine ambiguity | | | | resolved by close reading, not assumed. | | 2 | Investigate (not assume) the | Resolved with three concrete citations: | | | ledger-table-vs-design-§4 | end_to_end.md §0's own "skeleton -> user | | | state-machine relationship, | ratification -> execute -> retro" gloss | | | the brief's own central | (names exactly the SKELETON-through- | | | investigative ask (point 6) | RETRO span), §12's "step.N is assigned | | | | when a UoW goes active", and | | | | table_transitions.py's own already- | | | | resolved rule set. Conclusion: `active` | | | | is the superstate covering SKELETON | | | | through RETRO; `done`=CLOSED; | | | | `dropped`=DISCARDED; `todo`=INTENT. | | | | Documented in state_machine.py's module | | | | docstring, mirrored in design.md §4, and | | | | written up again in docs/orchestrator_ | | | | state_machine.md -- the brief's own | | | | "document this directly in the module's | | | | docstring, not just the step file" | | | | instruction, honored literally. | | 3 | Design persistence file-based | Built exactly this: EngineStateRecord | | | under ~/.b3ubot, mirroring | (uow/step/state/created_at/updated_at/ | | | app/egress.py's data_home() | history), start_state/load_state/ | | | exactly, one record per | advance_state, /orchestrator/ | | | in-flight UoW | .json, data_home() IMPORTED | | | | DIRECTLY from app/egress.py (not | | | | reimplemented). Justified JSON over | | | | app/config's key="value" shape | | | | explicitly in the module docstring | | | | (dynamic per-record data with nested, | | | | append-only history vs a static flat | | | | namespace) rather than defaulting to | | | | JSON just because it's easy, per the | | | | brief's own instruction. | | 4 | Compile a fixed-template | Built compile_execution_dag(): the SAME | | | RATIFIED->EXECUTING DAG | 4-node context_assembly/provider_run/ | | | compiler, confirmed against | human_gate/tool_call sequence | | | design.md's own v0-permissive | execution_example.dag hand-authored, | | | framing, +oracle_gate only | oracle_gate added only when a step's | | | when named | own Verification gates section names a | | | | T3/CCS/grammar-round-trip check -- | | | | confirmed live against all 13 real step | | | | files that NONE do today, so the branch | | | | is exercised only by a synthetic test | | | | fixture, named explicitly as such. | | 5 | (not in the original brief -- | A first-draft compiler run against | | | found live while testing | steps/step.9.txt produced a `dag_name`/ | | | against a real step file) | description corrupted with ~300 | | | | characters of unrelated prose. | | | | Investigated live: app/ledger/parser.py's| | | | `_parse_header()` joins wrapped | | | | continuation lines with a single space, | | | | and 12 of this repo's own 13 real step | | | | files (all but step.13.txt) wrap free | | | | prose onto `Step:`'s continuation lines | | | | -- confirmed by running parse_step_file()| | | | against every real file and printing | | | | header_fields["Step"]'s length (ranging | | | | 201-486 characters; only step.13.txt is | | | | the bare "13"). Fixed by extracting the | | | | LEADING integer via regex instead of | | | | trusting the field verbatim; re-verified | | | | clean against all 13 real files after. | +---+----------------------------------+------------------------------------------+ ``` Every other part of the plan held: `dag_model` import worked on the first try by mirroring `dag/dagModelPy/tests/test_smoke.py`'s own sys.path-insertion pattern (no pip install needed, no second import mechanism invented); the compiled DAG round-tripped through the real `bin/dagModel` binary on the first attempt once the header-field fix landed; `app/orchestrator/errors.py` mirroring `app/ledger/errors.py`'s shape needed no iteration. ## Gate evidence ``` +------+---------------------------------------------------------------------+ | G1 | pytest -q tests/test_orchestrator_state_machine.py -- 26/26 PASS | | | live: every one of the 9 real diagram edges (parametrized) | | | succeeds; every documented illegal case raises | | | IllegalStateTransitionError with the specific message substring | | | asserted, not just "raises something". | | G2 | pytest -q tests/test_orchestrator_compiler.py -- 12/12 PASS live, | | | including a REAL subprocess round-trip through dag_model's own | | | from_dag/to_dag (bin/dagModel actually invoked, not mocked) and a | | | direct shape comparison against dag/instances/execution_example.dag.| | G3 | pytest -q tests/test_orchestrator_persistence.py -- 12/12 PASS live,| | | including test_full_resume_after_simulated_process_restart: state | | | written through one call chain, read back through a completely | | | FRESH load_state() call with no shared in-memory object, confirmed | | | correct AND confirmed ACTIONABLE (the resumed process successfully | | | continues the arc to CLOSED). | | G4 | compile_execution_dag() run against all 13 real steps/step.N.txt | | | files (tests/test_orchestrator_compiler.py:: | | | test_compiled_dag_survives_the_whole_real_step_corpus) -- clean | | | `stepN` in every dag_name/description, 4-5 nodes each, verified | | | live post-fix. | | G5 | The oracle_gate branch: PASS on the paired positive (synthetic T3 | | | fixture -> 5 nodes) and negative (steps/step.9.txt, real gates | | | section, no T3/CCS wording -> 4 nodes) tests, both verified live. | | G6 | Full suite before this step: 277 passed/11 skipped (confirmed live, | | | matching step.13's own reported baseline). After: 329 passed/11 | | | skipped -- +52 (26+12+12=50 new orchestrator tests + 2 from the | | | step-file parser's/emitter's own self-inclusive corpus | | | parametrization picking up this step's own step.14.txt, the same | | | growth pattern steps 10-13 documented). One REAL live-data | | | collision found and fixed along the way: dogfooding U3.1's own | | | ledger row through `done` broke | | | tests/test_ledger_table_roundtrip.py:: | | | test_structural_round_trip_after_mutating_a_row (step.13's own fix, | | | which had repointed this exact test at U3.1 as "the next real todo | | | row" after U2.4 went done) -- confirmed live (suite went red | | | immediately after the dogfooding CLI commands ran), fixed by | | | repointing at U3.2 (the next real todo row as of this step) instead | | | of loosening or deleting the test. Final: 329 passed/11 skipped, 0 | | | regressions. | | G7 | end_to_end.md's own U3.1 bullet, LITERALLY -- state machine (G1), | | | engine-state persistence (G3), the compiler producing nodes=tasks/ | | | edges=deps/kv=state (G2) -- all three verified live. | | G8 | §11 ledger row for U3.1 updated THROUGH the existing `ledger` CLI | | | (transition -> active, close -> done/PENDING), against the REAL | | | end_to_end.md -- confirmed via `git diff --stat` before committing. | | G9 | Scope boundary: `grep -n "U3\.[2-8]" app/orchestrator/*.py` -> 15 | | | hits, every one read individually and confirmed a scope-boundary | | | ANNOTATION ("out of scope for U3.1", naming a future UoW), never an | | | implementation -- no tool-surface, PAL-drafting, ratify-UI, | | | scheduler, gate-tier, or repair-loop-budget code anywhere in this | | | step's new files. | | G10 | b3ubot porcelain clean before each commit; retro pair via the | | | SHARED v3_claude/scripts/step_gdiff (steps/step.14.gdiff.txt); | | | exactly 2 commits (work + gdiff-with-hash-backfill-folded-in), not | | | 3 -- step.7/9/10/11/13's own precedent. | +------+---------------------------------------------------------------------+ ``` ## Findings for later steps 1. **A step-file header field's `continuation-lines-joined-with-a-space` shape means `header_fields["Step"]` (and, by the same mechanism, `Parent`/`UoW`/`Date`/`Origin`) cannot be trusted to be JUST the labeled value -- 12 of this repo's own 13 real step files wrap free prose onto `Step:`'s continuation lines, and only the single most recent file (step.13.txt) happens to have a bare, unwrapped value.** Any future code that reads a header field expecting a short, structured value (not just this step's own `_step_number()`) should extract what it needs with a targeted parse (leading integer, first token, etc.), never trust the joined string verbatim. This is the same "verify, don't assume the shape of existing data" discipline step.10/12/13's own findings established for different data (corpus file counts, section numbering, stale cross-references) -- now confirmed a fourth time, for a different field of the SAME already- built parser. 2. **A live-data collision from ledger dogfooding, recurring exactly as step.13's own retro predicted.** step.13's retro named the general risk explicitly: "any test that names a specific real UoW id as 'the todo one' is one dogfooding step away from breaking." This step is that prediction landing -- `test_structural_round_trip_after_ mutating_a_row`, already once repointed (step.13, U2.4->U3.1), broke again the moment THIS step's own dogfooding moved U3.1 to `done`. Fixed the same way (repoint at the next real todo row, U3.2), but the underlying pattern will recur again the next time a UoW that happens to be some test's "todo" example gets promoted -- worth naming, again, for whoever hits it a third time: consider whether a future test should pick its "a real todo row" example by STATUS query (first row where status=="todo") rather than by hardcoded id, which would make this class of break impossible rather than merely fixable. Not changed here (out of scope -- this step's own dogfooding is what surfaced the break, fixing the test's fragility pattern itself belongs to whoever owns tests/test_ledger_table_ roundtrip.py next, not silently folded into an unrelated step). 3. **The design.md §4 diagram's "amendments" edge direction is genuinely ambiguous from prose alone -- it is only recoverable by tracing the ASCII box-drawing characters themselves.** design.md's own prose ("Human gates are load-bearing and few: ratify ... and escalation") never states which way the amendments arrow points; the diagram is the ONLY source of truth for it, and reading it correctly required literally following which arrowhead (`^`) sits under which state and which vertical bar the connecting line joins. Worth flagging for whoever next edits that diagram: if the ASCII art changes, the state machine's own `_LEGAL_TRANSITIONS` table (app/orchestrator/state_machine.py) needs a matching re-read, not just a glance, since nothing in the surrounding prose would catch a silently-reversed edge. ## Numbers ``` +---------------------------------------+----------------------+ | new files | 11 (app/orchestrator/ | | | __init__.py+errors.py| | | +state_machine.py+ | | | persistence.py+ | | | compiler.py; 3 test | | | files; docs/ | | | orchestrator_state_ | | | machine.md; steps/ | | | step.14.txt+step.14. | | | diff.txt) | | files edited | 3 (design.md, | | | end_to_end.md, tests/| | | test_ledger_table_ | | | roundtrip.py) | | app/orchestrator/ total LOC | ~660 across 5 modules| | new tests | 52 (26 state machine | | | + 12 persistence + | | | 12 compiler + 2 from | | | the parser/emitter's | | | own self-inclusive | | | corpus growth) | | header-field wrap bug found + fixed | 1 (12/13 real step | | (LIVE finding, not in original brief) | files affected) | | pre-existing test broken by this | 1 (found live, | | step's own dogfooding, then fixed | repointed U3.1->U3.2)| | suite before -> after | 277p/11s -> 329p/11s | | regressions | 0 | +---------------------------------------+----------------------+ ``` U3.1 done -- P3's first real orchestrator code lands. State machine, persistence, and the RATIFIED->EXECUTING compiler exist as concrete modules for the first time; the ledger-table-vs-design-§4 relationship that end_to_end.md and design.md left implicit is now investigated and documented, in two places (the module docstring and design.md §4 itself), with citations. Per the operator's explicit scope boundary, this session STOPS here -- U3.2 (tool surface + policy gates) is next, but is not started, and nothing in this step's own code implements any part of it (G9). Ledger updated in the work commit per the §12 execution contract, performed literally THROUGH this step's own existing `ledger` CLI (G8); commit hash of that same commit will be backfilled into both end_to_end.md §11 and this file's own steps/step.14.txt §7 in the immediately following gdiff commit -- the same 2-part, TWO-commit resolution step.7/9/10/11/13 used. End of retro.