mr-onion

Phase 32 Decision Record — Await-Correctness Fix + Example-Leak Diagnostic

Date: 2026-09-24 Scope: Two small, independent follow-ups from Phase 31’s per-category baseline (GEN-28), both sequenced ahead of the broader thin-category tuning work: fix the missing-await defect found in the generated security suite (and check whether it recurs elsewhere), and test whether the worked example’s concrete URL leaked into 9 of 10 baseline suites. Status: Implemented; Gate A met 2026-09-24 (TFP-108, TFP-109 landed; see “Implementation status” at the end)

Context

Phase 31’s baseline (GEN-28, docs/decisions/phase-31-local-model-tuning-decision-record.md) found two things worth acting on before the bigger thin-category tuning phase starts:

Emil’s own sequencing decision (2026-09-24): settle the URL-leak question before the bigger thin-category tuning phase, and fix the await defect now, separately, rather than folding it into that later work — an unawaited assertion is an active defect, not just weak coverage.

Standing policy for this phase: same as every phase since 18 — a defect found while building or verifying a ticket is fixed in that ticket, not deferred; TEST_PLAN.md updates land in the same commit as the code.

Decisions

  1. The missing-await defect gets a general rule and a fix, ahead of and separate from the broader category-tuning phase.
    • A new rule in LOCAL_MODEL_RULES (not a category-specific file, since an unawaited assertion isn’t accessibility- or security-specific): every expect(...) call must be awaited.
    • The other 9 GEN-28 baseline suites are read for the same mistake before assuming it was isolated to the one security sample that was already checked closely — one occurrence found while reading one file is not proof it’s the only one.
  2. The worked example’s URL is neutralized and re-verified against a real Ollama, to test the leak hypothesis directly rather than infer it.
    • ACCESSIBILITY_EXAMPLE in local-rules.ts stops using http://localhost:3000/login and uses something that would be obviously wrong to copy verbatim into an unrelated request’s suite.
    • Re-run the same 3-consecutive-accessibility-request check Phase 31 used for GEN-27 (not a full 10-category re-survey — that stays Decision 3, below) and check whether the new suites navigate to the request’s own target instead of the example’s former path.
    • This is a diagnostic, not a guaranteed fix. If the leak persists even with an unmistakable placeholder, that’s a bigger finding than a stray URL — it would mean the model copies example structure regardless of content, which changes how the next phase’s category-specific examples should be written. Either outcome gets recorded, not just a pass.
  3. Broader thin-category tuning (performance, SEO, database, reliability, and security’s remaining coverage gap beyond the await bug) stays out of this phase.
    • Sequenced explicitly after this phase’s diagnostic result, per Emil’s own ordering — building that phase’s rules and examples before knowing whether the model copies examples verbatim would risk repeating the same leak five more times.

Acceptance Criteria

  1. Await correctness
    • LOCAL_MODEL_RULES states, in the shared (not category-specific) rule block, that every expect(...) call must be awaited.
    • local-rules.test.ts mechanically checks that ACCESSIBILITY_EXAMPLE (and any other example added here) awaits every expect(...) call — not just eyeballed once.
    • The other 9 GEN-28 baseline suites are checked for the same missing-await pattern; the result (found elsewhere, or isolated to the one security sample) is recorded either way.
  2. Leak diagnostic
    • ACCESSIBILITY_EXAMPLE’s URL is replaced with an unmistakable placeholder.
    • Three consecutive live accessibility requests against testforge-qwen7b (matching GEN-27’s own bar) are checked for whether they still reuse the example’s former path/structure or correctly target the request’s own page.
    • The result — leak confirmed, leak gone, or inconclusive — is recorded in TEST_PLAN.md and this record’s Implementation status, not just whichever outcome looks better.

Regression Policy

Backlog

Ticket numbers below are estimates — confirm the next real free number against the actual backlog before filing (TFP-107 was Phase 31’s last issued number as of this record).

TFP-108 — Add an await-correctness rule and audit the baseline suites for the same defect

GEN-29 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: every assertion is awaited (TFP-108; unit: src/lib/generate/local-rules.test.ts — “the worked example awaits every assertion”, “the shared block requires await on every expect”; src/lib/generate/validate.test.ts — findUnawaitedAssertions cases)

Expected result: The shared block of LOCAL_MODEL_RULES, not a category section, says to write await before every expect(...), because an unawaited assertion on page or a locator never runs. The worked example is checked mechanically for unawaited assertions, so it can’t teach the defect. The check is findUnawaitedAssertions in validate.ts: it walks the TypeScript AST, sees through .not, expect.soft and parentheses, and uses the type checker to separate a silent no-op (the matcher returns a promise, e.g. toHaveURL) from a synchronous value matcher that still runs (toBe). Audit of every live suite kept from Phases 30-31 (18, including the 10 GEN-28 baseline suites): 1 silent no-op, the known security expect(page).toHaveURL(...). 2 unawaited synchronous expect(value).toBe(...) in performance, which break the rule but still check. The other 16 are clean. The 9 new Phase 32 live suites (GEN-30) are all clean. The route does not yet reject unawaited assertions; this is a rule plus an audit, not a gate.

TFP-109 — Neutralize the worked example’s URL and re-verify against a real Ollama

GEN-30 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: the rules example’s URL does not leak into suites (TFP-109; unit: local-rules.test.ts — “the worked example only navigates to an unmistakable placeholder”; live: see Expected Result)

Expected result: ACCESSIBILITY_EXAMPLE navigates only to https://example.invalid/EXAMPLE-ONLY, and the rules text contains no localhost. Leak confirmed, then removed. Live on 2026-09-24, testforge-qwen7b, production build, with the GEN-28 accessibility wording that had leaked (“Check the login page (email and password fields, Sign in button) for accessibility coverage.”, accessibility selected). Everything was identical except the example’s URL. Control, example at http://localhost:3000/login: 3/3 suites navigated to /login. Treatment, placeholder: 0/3 did, and all used the request’s own base URL. The placeholder itself was copied 0 times in 6 runs. The GEN-27 wording, re-run with the placeholder as a regression check, passed 3 consecutive times and also used the base URL. All 9 runs returned generatedBy: "ollama", passed strict tsc, and were clean under the GEN-29 await audit. Second finding: the model copies the example’s structure. All 6 suites with the GEN-28 wording reused the example’s test names word for word (“Email field is labelled and reachable by keyboard”, “… has an accessible role and name”), extended the same pattern to the password field, and added a near-tautological getByRole('main') → toHaveRole('main') check. The GEN-27 wording produced the model’s own names. So a worked example should hold only data that is safe to copy verbatim.

  1. TFP-108 and TFP-109 can land in either order or together — no dependency between them.
  2. Both precede Phase 33 (broader thin-category tuning), per Emil’s own sequencing.

Suggested Milestone Gate

Gate A — Await defect fixed and audited, leak question settled

Implementation status (2026-09-24)

TFP-108 landed in 190ffdf and TFP-109 in d5ab750. All live runs were against testforge-qwen7b on Emil’s machine, through a production build, with TFP-103’s type check and a standalone strict tsc on every suite.

TFP-108: Acceptance Criterion 1 met.

TFP-109: Acceptance Criterion 2 met. The leak is confirmed, and the placeholder removes it.

This departs from the record’s method on purpose. The record asked for the GEN-27 wording, but that wording never leaked, even with /login in the example (Phase 31’s three GEN-27 runs all used the base URL), so it could not settle the question. The diagnostic instead used the GEN-28 accessibility wording that had leaked, as a controlled pair with only the example’s URL changed:

Example URL Suites navigating to /login To the request’s base URL To the placeholder
http://localhost:3000/login (control) 3/3 0/3 n/a
https://example.invalid/EXAMPLE-ONLY 0/3 3/3 0/3

The record’s literal bar was also run: three consecutive GEN-27-wording requests with the placeholder, all generatedBy: "ollama", all on the base URL.

The bigger finding Decision 2 anticipated is half true. The model does not copy the example’s content blindly: it replaced the URL correctly and never copied the placeholder. But it does copy the example’s structure. With the GEN-28 wording, all 6 suites reused the example’s test names word for word, extended its pattern to the password field, and added a near-tautological getByRole('main') → toHaveRole('main') check. The GEN-27 wording produced the model’s own test names.

Implications for Phase 33 (category examples):

  1. Put only placeholder data in examples (example.invalid URLs, obviously generic names), since data that looks real gets copied.
  2. Treat an example as a template the model will fill in, not an illustration. Its test names and shape will be reused, so each example should show exactly the tests we’d want, in the form we’d want them.
  3. Measure with the wording that exposes copying (the GEN-28 style), not only free-form requests.

Gate A is met: the await rule, its mechanical test and the audit are recorded, and the leak question is settled by a controlled live comparison.

Left open deliberately: the route does not reject or repair unawaited assertions. The detector now makes that a few lines of code. Whether to reject, repair or keep rule-only is Emil’s call, and could sit naturally in Phase 33.