step.10.txt -- b3ubot U2.1: step-file parser -- the canonical step-file shape as a schema, corpus-surveyed against the FULL b3u step corpus (not assumed from step.7-9's own shape), refuse-loudly on unknown sections and malformed input. Step: 10 -- promote UoW **U2.1** (end_to_end.md §4, P2) to execution depth: the FIRST P2 UoW and a genuinely new subsystem (not an incremental PAL/CLI extension like steps 7-9) -- a parser module that turns a `step.N.txt` file into a structured `StepFile` (header fields + numbered sections with raw body text), grounded in an empirical survey of the b3u corpus across its full age range, not an assumption that step.7/step.8/step.9's own shape is representative of the whole 50-step history. Parent: end_to_end.md §4 P2 entry/exit ("b3ubot parses and emits its own ledger byte-stably, with the b3u step corpus as the acceptance fixture") and the U2.1 bullet verbatim ("the canonical sections as a schema; the 34 b3u step files (read-only fixtures) must parse. Done when: corpus parses; unknown-section and malformed inputs refuse loudly" -- the "34" is corrected to 50 in this step, see (1)). design.md §2.3 (The ledger engine: "canonical sections (Status / ground truth / deliverables / gates / LOCKs / open questions / acceptance / hash backfill)") and §1.2 ("Honest interims: partial mechanisms are named at creation... never silently deferred") for the refuse-loudly framing. design.md §3 Data Model's `Step (UoW)` entity is a DIFFERENT, LATER Track B/P4 concern (SQLite/OO-API layers) -- explicitly NOT touched by this step, named to avoid the confusion the operator's own brief flagged up front. UoW: U2.1 -- "Step-file parser (b3u corpus)". First UoW of P2 (the ledger engine, Track A). Depends on nothing built so far (P2 has no prerequisite UoW inside itself); P2 as a phase depends on P1 (done, U1.1-U1.6 all closed). Unblocks U2.2 (emitter + byte round-trip oracle), which needs this step's `StepFile` shape to emit FROM. Date: 2026-07-20 (drafted and executed in the same pass, at the operator's explicit request -- the step.7/step.9 single-pass rhythm, not step.8's SKELETON-then-execute split.) Origin: Operator directive 2026-07-20: "go with U2.1, step file and its execution" -- single pass. ## (0) Status EXECUTED in this pass. Not a skeleton-then-later-execute split like step.8 -- the parser module, the schema doc, the tests, and this step file were built together per the operator's explicit direction. ## (1) Ground truth probed at drafting (outputs -> named writers) READ end_to_end.md §4 (P2 entry/exit + all four U2.x bullets) and §11 ledger (U2.1 row: `todo`, no step number -- confirmed before starting) and §12 maintenance rules (same-commit ledger update; step.7/step.9's PENDING-then-backfill resolution of the "this commit's own hash is unknown while writing it" problem, confirmed via `git show --stat 4f24b22` and `git show 4f24b22 -- end_to_end.md steps/step.7.txt` before repeating it here rather than assuming it). VERIFIED `ls b3u/steps/step.*.txt | grep -vE '\.(diff|cdiff|gdiff) \.txt$'` (relative to the v3_claude parent) -> 50 files, `step.1.txt` through `step.50.txt`, contiguous, no gaps (`sort -V` then `wc -l` = 50; range check against `seq 1 50` matched exactly). CORRECTS end_to_end.md's U2.1 bullet, which said "34" -- that count was accurate when P2 was originally scoped (2026-07-12, repo founding); the b3u corpus grew from 34 to 50 by 2026-07-20 (b3u's own P6 production-launch arc, steps 45-50, landed since). Fixed in end_to_end.md §4 in this same commit (small, real, worth recording per the operator's own framing). READ a SPREAD of the b3u corpus across its full age range, not just the steps this session already knew: EARLY -- b3u/steps/step.1.txt (founding, no UoW: field -- predates UoW numbering), step.5.txt (U3.3, OO API core), step.10.txt (U3.7, the b3ucli REPL -- b3u's OWN step.10, a DIFFERENT file in a DIFFERENT repo from this one); MID -- step.20.txt (U3.10), step.30.txt (UL.3), step.45.txt (U6.1); RECENT -- step.48.txt (U6.4), step.49.txt (U6.5), step.50.txt (U6.6, the FINAL P6 UoW -- b3u's corpus tops out at 50, not the 140s the task brief's own "recent examples" guess named; those numbers don't exist in b3u/steps/, confirmed by `ls` before reading anything, not assumed). This superseded an assumption carried in from steps 7-9 (that step.7.txt/ step.8.txt/step.9.txt's OWN shape, all drafted THIS repo, THIS week, is representative of the full 50-step b3u history) -- it mostly held, but two real deviations surfaced (next item). FOUND (grep-driven, across all 50 files, not eyeballed) two genuine era-variations the "read 3 files and assume" path would have missed: (a) section (0) is titled exactly "Status" in 50/50 files -- the one true universal invariant; (b) the gate-enumeration section is titled "Verification gates" in 49/50 files but "Oracle (gates G1-G3)" in b3u/steps/step.8.txt -- same role, different title; (c) the trailing-section run (Verification gates / LOCKs / Open questions / Acceptance / Hash backfill) is in that RELATIVE ORDER in 49/50 files, but b3u/steps/step.8.txt has Hash backfill BEFORE Acceptance; (d) header field `UoW:` is present in 49/50 (absent only from step.1.txt, the founding step, which predates UoW numbering) while `Step:`/`Parent:`/`Date:`/ `Origin:` are 50/50; (e) the `End of skeleton.`/`End of step.` trailer is present in 49/50 (b3u/steps/step.44.txt does not end that way). Full table in docs/step_file_schema.md. Design decision flowing from this: the parser enforces the MARKER SHAPE (`## (N) `, N strictly increasing) and section (0) == "Status" as hard requirements, but does NOT enforce a fixed closing-section title vocabulary, a fixed relative order beyond section 0, or the trailer -- the real corpus does not support that rigidity, and force-fitting it would make step.8.txt itself fail to parse, which would be absurd (it is a real, accepted, closed step in the source corpus). READ steps/step.7.txt + step.7.diff.txt and steps/step.9.txt + step.9.diff.txt (the two prior single-pass steps in THIS repo) for the retro-pair format and the exact 2-commit resolution of the ledger-row self-reference problem. READ steps/step.1.txt through step.9.txt (this repo's own step files) -- confirmed they mirror the SAME b3u convention (Status-first, numbered sections, LOCKs/Open questions/ Acceptance/Hash backfill trailing run), a second, smaller, already-familiar reference point -- NOT the graded fixture set (end_to_end.md's own wording names the b3u corpus), but worth parsing too as a bonus check (§2 below). READ design.md §3 (Data Model, Step (UoW) entity) in full and confirmed it is Track B/P4 (the SQLite/OO-API five-layer pipeline) -- a DIFFERENT, LATER, independent track from this step's Track A plaintext-file parsing. Not touched. READ design.md §1.2 ("Honest interims") and §2.7 (the tool surface's "refuse with a named next action, no --force" posture) for the refuse-loudly exception-message framing. READ app/pal/errors.py in full: the `AdapterError` base + `AdapterUnavailableError`/`AdapterRunError` subclass shape (each with a docstring naming the LOCK/precedent it enforces) -- the template this step's `app/ledger/errors.py` mirrors exactly (`StepFileError` base + `UnknownSectionError`/`MalformedStepFileError`). READ app/egress.py in full: the `$B3UBOT_HOME` env-var-with- default-path resolution pattern, and scripts/smoke.sh's `$CPPCCHOME` sibling-repo-path pattern (`HERE=$(cd "$(dirname ...)/.." && pwd)`, `CPPCCHOME="${CPPCCHOME:- $HERE/../cppcc}"`) -- the template tests/test_ledger_parser.py's `$B3U_SIBLING_HOME` (default `../b3u` relative to this repo's root) mirrors. READ app/tools.py + tests/test_tools.py for this repo's pytest style (plain functions, `pytest.raises`, `tmp_path`, docstring citing the UoW/LOCK it proves) and app/pal/__init__.py / app/__init__.py (both 0 bytes) for the empty-`__init__.py` convention `app/ledger/__init__.py` follows. CONFIRMED `git log --oneline -25` for exact commit-message phrasing: step.7/step.9's two-commit shape (`b3ubot step.N: UX.Y -- ...` then `b3ubot step.N: add step.N.gdiff.txt via scripts/step_gdiff`) is this step's template. ## (2) What U2.1 delivers 2.1 **app/ledger/parser.py** (new): `parse_step_file(path)` / `parse_step_text(text, source=...)` -> `StepFile` (`source`, `title_line`, `header_fields: dict[str,str]`, `sections: list[Section]`, `trailer: str | None`); `Section` (`number`, `title`, `body` -- raw text verbatim, outer blank lines only trimmed). `StepFile.section(n)` / `.section_titled(t)` convenience lookups. Header-field parsing recognizes exactly the five labels the corpus survey found (Step/Parent/UoW/Date/ Origin), continuation lines by indentation under the label, matching the observed shape byte-for-byte. Section parsing requires the `## (N) <title>` marker shape and strictly increasing N; does NOT validate title text past section (0) (per (1)'s findings). 2.2 **app/ledger/errors.py** (new): `StepFileError` base, `UnknownSectionError` (an unrecognized header-field label, or a `## ` line that isn't the numbered-marker shape), `MalformedStepFileError` (missing required header field, no section markers at all, section (0) not titled "Status", or section numbers not strictly increasing) -- mirrors app/pal/errors.py's `AdapterError` family shape exactly; no generic `ValueError` anywhere in this module. 2.3 **docs/step_file_schema.md** (new): the live schema doc -- corpus-surveyed shape (title line / INTERNAL banner / header fields / numbered sections / optional trailer), the header-field presence table, the closing-section title/order era-variation table, and what U2.2/U2.3/U2.4 will build on top of this return shape (named, not built). 2.4 **tests/test_ledger_parser.py** + **tests/fixtures/ledger/** (new): (a) all 50 real b3u corpus files parse without raising, parametrized per-file (`$B3U_SIBLING_HOME`-resolved, default `../b3u`, honest `pytest.skip` naming the override if the sibling isn't checked out); a live tripwire asserting the corpus is exactly `step.1.txt`..`step.50.txt` (50, contiguous) so a FUTURE corpus-size drift re-triggers a survey rather than going stale silently a second time; (b) this repo's own step files parse too (bonus, not graded -- self-inclusive: this collects whatever `steps/step.N.txt` files exist on disk at test-collect time, which includes step.10.txt itself once it's written, so the count is 10/10 this run, `>=9` asserted rather than a fixed number); (c) six synthetic malformed/ unknown-section fixtures (unknown header field, unknown section marker, missing header entirely, wrong section-0 title, out-of-order section numbers, no section markers at all), each asserted against the SPECIFIC exception type AND a message substring (not "it raised something"); (d) one known-good shape test against steps/step.9.txt asserting header-field/section/ trailer content, not just non-raising. 2.5 **end_to_end.md** same-commit edits: §4 U2.1 bullet -> DONE (step.10), 34 corrected to 50 with the verification method named, a `*Delivered:*` line (mirrors U1.5/U1.6's own style); §11 ledger row -> done/step.10/PENDING (backfilled at the gdiff commit, the step.7/step.9 resolution of the self-reference problem). EXPLICITLY OUT OF SCOPE (named here per the operator's brief, not silently built): - The EMITTER / byte-round-trip oracle (U2.2) -- `emit(parse(x)) == x`. This step's `StepFile` shape is designed to make that achievable (raw body text preserved verbatim) but no emitter exists yet. -> U2.2 - UoW ledger-table operations / status transitions (§9-table model: todo -> active -> done/dropped) (U2.3). -> U2.3 - Retro-pair machinery, cdiff generation, the `b3ubot ledger` CLI surface (U2.4). -> U2.4 - design.md §3's Data Model (`Step (UoW)` SQLite entity, the five- layer verbal->SGDL->SQLite->OO-API->CLI pipeline) -- Track B, P4, unrelated to this step's plaintext-file parsing. -> P4 - Fixing design.md §2.3's own stale "34 b3u step files" reference (line ~142) -- the operator's brief scoped the correction to end_to_end.md's U2.1 bullet specifically; design.md's instance is left as a named, deliberate follow-up, not silently inconsistent without comment. -> next touch of design.md §2.3 ## (3) Verification gates G1 All 50 real b3u corpus files (step.1.txt..step.50.txt) parse without raising, verified live via pytest parametrization, one assertion per file (verified live: 50/50 PASS). G2 This repo's own step files parse too (bonus check, not graded; self-inclusive of step.10.txt once written) (verified live: 10/10 PASS). G3 Each of the six synthetic malformed/unknown-section fixtures raises the SPECIFIC exception type named for it (UnknownSectionError vs MalformedStepFileError) with a message containing the concrete defect, asserted via `pytest.raises(..., match=...)` (verified live: 6/6 PASS, wrong-exception-type would fail the same as wrong-message-content). G4 Corpus-size tripwire: the b3u corpus is asserted to be exactly `step.1.txt`..`step.50.txt` (50 files, contiguous); this is a DELIBERATE early-warning assertion, not incidental (verified live: PASS at 50). G5 NO REGRESSION: the pre-existing 55p/11s baseline (post-step.9) is additive only -- no existing test file touched, no existing `app/`/`adapters/` file touched except the new `app/ledger/` package (verified live: `pytest -q` -> `125 passed, 11 skipped`, delta = +70 passed, 0 skipped delta, 0 failures). G6 end_to_end.md §4/§11 updated same commit as the work (§2.5); ledger row carries `PENDING` for the commit hash (the self- reference problem), backfilled at the gdiff commit. G7 Hygiene: b3ubot porcelain clean after each commit; retro via the SHARED `v3_claude/scripts/step_gdiff` (not a b3ubot-local copy -- the step.8/step.9 finding, confirmed again rather than rediscovered). ## (4) LOCKs C-10-A b3ubot porcelain clean after each commit. C-10-B READ-ONLY CORPUS: the b3u step files are read from `../b3u/steps/` IN PLACE via `$B3U_SIBLING_HOME` (default) -- never copied into this repo, never written to, never modified by any test (end_to_end.md's own "the 50 b3u step files (read-only fixtures)" wording, and this step's own tests/test_ledger_parser.py only ever calls `Path.read_text`, never `write_text`, against that tree). C-10-C NO SILENT SCOPE CREEP: U2.2 (emitter/round-trip oracle), U2.3 (ledger ops), U2.4 (retro-pair/CLI machinery), and design.md §3's Track B/P4 SQLite layer are named out of scope in §2 above, not silently begun. C-10-D NO GENERIC EXCEPTIONS: every refusal in app/ledger/ raises `UnknownSectionError` or `MalformedStepFileError` (never a bare `ValueError`/`Exception`), each with a message naming the concrete defect -- the "refuse with a named next action" posture (design.md §2.7) applied to parsing, not just to the tool surface / CLI it was originally written for. C-10-E SCHEMA NOT OVER-FIT: the parser's structural requirements (marker shape, section (0) == "Status", strictly- increasing section numbers, five known header-field labels) are exactly what the FULL 50-file corpus survey supports -- no requirement was added that b3u/steps/ step.8.txt (or any other real file) would fail. This is the load-bearing decision the whole step turns on; see (1)'s findings and docs/step_file_schema.md. C-10-F standing locks inherited: B-3 (no disclosure/no push); B-6 (this step's gates are T0/T1 deterministic pytest assertions; no AI review tier involved). B-1/B-2/B-4/B-5 are moot -- this step invokes no cppcc, no provider, no egress. ## (5) Open questions Q-10-A Should "unknown section" cover unrecognized BODY-section titles (e.g. a numbered section titled something the schema has never seen), not just header-field labels and non-numbered `## ` markers? LEAN (built this way): no -- the corpus survey (1) found real, legitimate title variation in the body/closing run (e.g. "Oracle (gates G1-G3)" instead of "Verification gates" in b3u/steps/step.8.txt); enforcing a title vocabulary there would reject a real, already-accepted, already-closed step in the source corpus, which would falsify the parser against its own graded fixture set. Revisit only if a FUTURE corpus survey (U2.2+ or beyond) finds title drift that actually breaks something downstream (e.g. U2.3's ledger ops needing to KNOW which section is "the gates section" reliably) -- at that point a `kind:` classifier with a small synonym table (not a hard reject) is the likely shape, not a schema tightening. Q-10-B Should the parser enforce the `End of skeleton.`/`End of step.` trailer, given it's 49/50? LEAN (built this way): no -- same reasoning as Q-10-A; captured as `.trailer` when present (informational), `None` otherwise, never a parse failure. b3u/steps/step.44.txt is a real, accepted step that doesn't end that way. Q-10-C `$B3U_SIBLING_HOME` vs mirroring `$B3UBOT_HOME`'s exact name pattern more closely (e.g. `$B3U_STEPS_DIR` pointing straight at the steps/ dir rather than the sibling repo root)? LEAN (built this way): sibling-repo-root env var (`$B3U_SIBLING_HOME`, `/steps` appended in code) -- mirrors `$CPPCCHOME`'s shape (points at a repo root, not a specific subdirectory inside it) more closely than `$B3UBOT_HOME`'s shape (points at a per-developer DATA home, a different kind of thing); a future U2.2+ that needs OTHER b3u paths (not just steps/) reuses the same var without a second env var being invented. ## (6) Acceptance - [x] app/ledger/parser.py + app/ledger/errors.py + app/ledger/ __init__.py (2.1, 2.2, G1-G4). - [x] docs/step_file_schema.md (2.3). - [x] tests/test_ledger_parser.py + tests/fixtures/ledger/ (6 fixtures) (2.4, G1-G4). - [x] All 50 b3u corpus files parse clean -- verified live, 50/50 (G1). - [x] This repo's own step files parse clean -- verified live, 10/10, self-inclusive of step.10.txt (G2, bonus). - [x] Six refuse-loudly fixtures raise the specific exception type + message content -- verified live, 6/6 (G3). - [x] Corpus-size tripwire asserted at exactly 50 -- verified live (G4). - [x] Full suite: 125 passed / 11 skipped (was 55p/11s) -- +70 passed, 0 skipped delta, 0 failures, 0 files outside app/ledger/, docs/, tests/ touched except end_to_end.md (G5). - [x] end_to_end.md §4 U2.1 bullet -> DONE (step.10), 34 -> 50 corrected with method named, *Delivered:* line added; §11 ledger row -> done/step.10/PENDING (2.5, G6). - [x] Retro pair: this file + step.10.diff.txt + step.10.gdiff.txt via the SHARED `v3_claude/scripts/step_gdiff` (G7). ## (7) Hash backfill WORK commit (parser + errors + docs + tests + this file + ledger): 424f7b2 GDIFF commit (step.10.gdiff.txt via scripts/step_gdiff): (HEAD) End of step.