Date: 2026-09-24
Scope: Two small, independent follow-ups from Phase 31’s per-category
baseline (GEN-28), both sequenced ahead of the broader thin-category
tuning work: fix the missing-await defect found in the generated
security suite (and check whether it recurs elsewhere), and test whether
the worked example’s concrete URL leaked into 9 of 10 baseline suites.
Status: Implemented; Gate A met 2026-09-24 (TFP-108, TFP-109 landed; see
“Implementation status” at the end)
Phase 31’s baseline (GEN-28, docs/decisions/phase-31-local-model-tuning-decision-record.md)
found two things worth acting on before the bigger thin-category tuning
phase starts:
expect(page).toHaveURL('https://…'))
with no await — meaning the assertion never actually runs. That’s a
different, worse class of problem than a thin category: an unawaited
expect() silently reports nothing rather than failing loudly, so a
test built on it would look like coverage while checking nothing. Only
one sample surfaced this (by reading the generated file, not by
automated detection), so whether it’s a one-off or a pattern across the
other 9 baseline suites is genuinely unknown./login — the exact path used
in local-rules.ts’s ACCESSIBILITY_EXAMPLE — even though none of the
survey’s requests mentioned that path. Code flagged this as likely but
not proven, since the survey’s request wording also varied across
categories.Emil’s own sequencing decision (2026-09-24): settle the URL-leak question before the bigger thin-category tuning phase, and fix the await defect now, separately, rather than folding it into that later work — an unawaited assertion is an active defect, not just weak coverage.
Standing policy for this phase: same as every phase since 18 — a
defect found while building or verifying a ticket is fixed in that
ticket, not deferred; TEST_PLAN.md updates land in the same commit as
the code.
await defect gets a general rule and a fix, ahead of and
separate from the broader category-tuning phase.
LOCAL_MODEL_RULES (not a category-specific file, since
an unawaited assertion isn’t accessibility- or security-specific):
every expect(...) call must be awaited.ACCESSIBILITY_EXAMPLE in local-rules.ts stops using
http://localhost:3000/login and uses something that would be obviously
wrong to copy verbatim into an unrelated request’s suite.LOCAL_MODEL_RULES states, in the shared (not category-specific) rule
block, that every expect(...) call must be awaited.local-rules.test.ts mechanically checks that ACCESSIBILITY_EXAMPLE
(and any other example added here) awaits every expect(...) call —
not just eyeballed once.await pattern; the result (found elsewhere, or isolated to
the one security sample) is recorded either way.ACCESSIBILITY_EXAMPLE’s URL is replaced with an unmistakable
placeholder.testforge-qwen7b
(matching GEN-27’s own bar) are checked for whether they still reuse the
example’s former path/structure or correctly target the request’s own
page.TEST_PLAN.md and this record’s Implementation status, not just
whichever outcome looks better.TEST_PLAN.md gets a case for the await rule and an update to GEN-27’s
entry (or a new GEN case) recording the leak-diagnostic result, in the
same commit as the code, per standing policy.Ticket numbers below are estimates — confirm the next real free number against the actual backlog before filing (TFP-107 was Phase 31’s last issued number as of this record).
bug, generate, localexpect() — an assertion that silently never runs, which
looks like coverage while checking nothing. Not category-specific, so
it could be present in any locally-generated suite, not just security’s.expect()” rule to
LOCAL_MODEL_RULES; read the other 9 GEN-28 baseline suites (kept from
Phase 31, or regenerated if not retained) for the same pattern; record
findings even if none are found elsewhere.local-rules.ts; updated
local-rules.test.ts with a mechanical await check; TEST_PLAN.md
update recording the audit’s result.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: every assertion is awaited (TFP-108; unit: src/lib/generate/local-rules.test.ts — “the worked example awaits every assertion”, “the shared block requires await on every expect”; src/lib/generate/validate.test.ts — findUnawaitedAssertions cases)
Expected result: The shared block of LOCAL_MODEL_RULES, not a category section, says to write await before every expect(...), because an unawaited assertion on page or a locator never runs. The worked example is checked mechanically for unawaited assertions, so it can’t teach the defect. The check is findUnawaitedAssertions in validate.ts: it walks the TypeScript AST, sees through .not, expect.soft and parentheses, and uses the type checker to separate a silent no-op (the matcher returns a promise, e.g. toHaveURL) from a synchronous value matcher that still runs (toBe). Audit of every live suite kept from Phases 30-31 (18, including the 10 GEN-28 baseline suites): 1 silent no-op, the known security expect(page).toHaveURL(...). 2 unawaited synchronous expect(value).toBe(...) in performance, which break the rule but still check. The other 16 are clean. The 9 new Phase 32 live suites (GEN-30) are all clean. The route does not yet reject unawaited assertions; this is a rule plus an audit, not a gate.
test, generate, localACCESSIBILITY_EXAMPLE’s URL with an unmistakable
placeholder; re-run 3 consecutive live accessibility requests against
testforge-qwen7b and check whether the leak persists.local-rules.ts; TEST_PLAN.md and this
record’s Implementation status updated with the actual result.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: the rules example’s URL does not leak into suites (TFP-109; unit: local-rules.test.ts — “the worked example only navigates to an unmistakable placeholder”; live: see Expected Result)
Expected result: ACCESSIBILITY_EXAMPLE navigates only to https://example.invalid/EXAMPLE-ONLY, and the rules text contains no localhost. Leak confirmed, then removed. Live on 2026-09-24, testforge-qwen7b, production build, with the GEN-28 accessibility wording that had leaked (“Check the login page (email and password fields, Sign in button) for accessibility coverage.”, accessibility selected). Everything was identical except the example’s URL. Control, example at http://localhost:3000/login: 3/3 suites navigated to /login. Treatment, placeholder: 0/3 did, and all used the request’s own base URL. The placeholder itself was copied 0 times in 6 runs. The GEN-27 wording, re-run with the placeholder as a regression check, passed 3 consecutive times and also used the base URL. All 9 runs returned generatedBy: "ollama", passed strict tsc, and were clean under the GEN-29 await audit. Second finding: the model copies the example’s structure. All 6 suites with the GEN-28 wording reused the example’s test names word for word (“Email field is labelled and reachable by keyboard”, “… has an accessible role and name”), extended the same pattern to the password field, and added a near-tautological getByRole('main') → toHaveRole('main') check. The GEN-27 wording produced the model’s own names. So a worked example should hold only data that is safe to copy verbatim.
TFP-108 landed in 190ffdf and TFP-109 in d5ab750. All live runs were
against testforge-qwen7b on Emil’s machine, through a production build,
with TFP-103’s type check and a standalone strict tsc on every suite.
TFP-108: Acceptance Criterion 1 met.
LOCAL_MODEL_RULES requires await on every
expect(...), and a test asserts the rule sits in that block.findUnawaitedAssertions in validate.ts. It
walks the AST, and the type checker separates a silent no-op (the
matcher returns a promise) from an unawaited synchronous toBe, which
still runs. A test asserts the worked example is clean.toBe calls, which break the rule but still check. The
other 16 are clean, and so are all 9 suites generated in this phase.TFP-109: Acceptance Criterion 2 met. The leak is confirmed, and the placeholder removes it.
This departs from the record’s method on purpose. The record asked for
the GEN-27 wording, but that wording never leaked, even with /login in
the example (Phase 31’s three GEN-27 runs all used the base URL), so it
could not settle the question. The diagnostic instead used the GEN-28
accessibility wording that had leaked, as a controlled pair with only
the example’s URL changed:
| Example URL | Suites navigating to /login |
To the request’s base URL | To the placeholder |
|---|---|---|---|
http://localhost:3000/login (control) |
3/3 | 0/3 | n/a |
https://example.invalid/EXAMPLE-ONLY |
0/3 | 3/3 | 0/3 |
The record’s literal bar was also run: three consecutive GEN-27-wording
requests with the placeholder, all generatedBy: "ollama", all on the
base URL.
The bigger finding Decision 2 anticipated is half true. The model
does not copy the example’s content blindly: it replaced the URL
correctly and never copied the placeholder. But it does copy the
example’s structure. With the GEN-28 wording, all 6 suites reused the
example’s test names word for word, extended its pattern to the password
field, and added a near-tautological getByRole('main') →
toHaveRole('main') check. The GEN-27 wording produced the model’s own
test names.
Implications for Phase 33 (category examples):
example.invalid URLs,
obviously generic names), since data that looks real gets copied.Gate A is met: the await rule, its mechanical test and the audit are recorded, and the leak question is settled by a controlled live comparison.
Left open deliberately: the route does not reject or repair unawaited assertions. The detector now makes that a few lines of code. Whether to reject, repair or keep rule-only is Emil’s call, and could sit naturally in Phase 33.