mr-onion

Phase 33 Decision Record — Local Model Category Coverage

Date: 2026-09-24 Scope: Teach the local (Ollama) generation path real, category-specific coverage for the five best-practice areas Phase 31’s baseline found thin or broken (performance, SEO, database, reliability, security), and close Phase 32’s one open question — whether the route should reject or repair suites with unawaited checks. Status: Implemented; Gate A met 2026-09-24 with reliability recorded as short on quality (see “Implementation status” at the end)

Context

Phase 31’s baseline (GEN-28) generated one real suite per best-practice area against testforge-qwen7b and read each one, not just compiled it. Five categories fell short of what src/lib/generate/prompt.ts’s own BEST_PRACTICE_GUIDANCE already asks for — guidance already sent to the local model, same as the OpenAI path, so this isn’t a case of the model never being told:

Accessibility (Phase 31, TFP-106) proved the fix for “the model was told in prose and didn’t comply”: a concrete, copy-ready worked example, not more prose. Phase 32 (TFP-109/GEN-30) then found two things that change how those examples must be built:

  1. A worked example’s data leaks verbatim if it looks like real data — 9 of 10 GEN-28 suites reused the accessibility example’s exact /login path even though no request mentioned it.
  2. The model copies an example’s structure, not just its literals — even once the URL stopped leaking (GEN-30), suites built with the leak-prone wording reused the example’s test names word for word and extended its pattern, including one near-tautological check.

Phase 32’s record already states the resulting rule for every example this phase writes: use only placeholder data, write the example as exactly the tests that should be copied (because they will be), and test new examples with wording chosen to expose copying, not wording that happens to avoid it.

Phase 32 also left one question open: findUnawaitedAssertions (TFP-108) can detect a suite with an unawaited check, but the route doesn’t act on it yet — it’s a rule, not a gate.

Standing policy for this phase: same as every phase since 18 — a defect found while building or verifying a ticket is fixed in that ticket, not deferred; TEST_PLAN.md updates land in the same commit as the code.

Decisions

  1. An unawaited assertion is rejected the same way an invented API is — extend the existing gate, don’t build a repair step.
    • generateWithOllama calls findUnawaitedAssertions alongside findTypeErrors before labelling a suite generatedBy: "ollama"; any result from either fails the suite into the same fallback() path, with a reason that distinguishes which check failed (invalid_output vs. a new unawaited_assertion).
    • Rejecting, not auto-inserting await, because every gate this pipeline has built (TFP-103, TFP-108) detects and falls back; none of them rewrites the model’s output. Auto-repair would be new, riskier surface (where exactly to insert await inside .not/expect.soft/chained expressions is exactly the complexity isHandled already has to reason about) to fix something TFP-108’s audit found only once in 18 real suites. Not worth the new risk for a rare case a template fallback already handles safely.
  2. Each of the five categories gets its own worked example, built to Phase 32’s three rules — placeholder data only, written as exactly the tests that should be copied, measured with wording chosen to expose copying.
    • The category’s definition is the one already in BEST_PRACTICE_GUIDANCE / BEST_PRACTICE_OPTIONS — not reinvented here. The model was already told; a concrete example is the fix, the same lever that worked for accessibility, not new prose explaining the category again.
  3. Reliability is built first, on its own, and treated as the highest-risk category in this phase.
    • Unlike the other four, “graceful degradation when a dependency or local storage fails” is TestForge-specific meaning, not general testing knowledge, and the deterministic generator’s own implementation uses page.route() interception against a discovered dependency — a technique the model has to be taught, not reminded of.
    • This phase does not assume it will work as cleanly as accessibility did. If the model can’t reliably produce a working page.route()-based suite even with a worked example, that’s a real, recordable result — not a reason to lower the bar or fake a pass.
  4. Performance, SEO, and database are grouped into one ticket — each gets its own example, but they’re comparable in shape and difficulty (a single concrete pattern to demonstrate, using standard Playwright APIs the model already handles correctly elsewhere).

  5. Security is a refinement, not a rebuild.
    • The invalid-input check already works. Scope is fixing the specific HTTPS-protocol assumption (asserting https:// against whatever protocol the target’s base URL actually uses) and, within what BEST_PRACTICE_GUIDANCE already asks for, broadening beyond the one check that currently exists — not redesigning the category.

Acceptance Criteria

  1. Unawaited assertions are rejected, not just detected
    • A suite with an unawaited web-first assertion returns the labelled template with fallbackReason: "unawaited_assertion", proven against a fake local server the same way TFP-103/106 proved their own gates.
    • The existing invalid_output case is unaffected — an unawaited assertion and an invented API are distinct, both caught.
  2. Each category demonstrates genuine, category-specific coverage
    • Three consecutive real generations per category against testforge-qwen7b, checked by reading the suite (not just compiling it) for a check that actually belongs to that category, matching BEST_PRACTICE_GUIDANCE’s own description of it.
    • Reliability’s bar is specifically a working page.route()-based (or equivalent) failure-simulation test, not a generic assertion — the actual trait that defines the category. If three consecutive attempts don’t produce one, that’s recorded as the result, not smoothed over.
    • Every suite still passes findTypeErrors and findUnawaitedAssertions clean, and uses only the request’s own data — checked against Phase 32’s leak lesson, the same way GEN-30 did.
  3. TEST_PLAN.md and this record reflect the real per-category outcome
    • Each category’s entry states what it now actually checks, not “works” as a blanket claim — same standard GEN-28 already set.

Regression Policy

Backlog

Ticket numbers below are estimates — confirm the next real free number against the actual backlog before filing (TFP-109 was Phase 32’s last issued number as of this record).

TFP-110 — Reject suites with unawaited assertions

GEN-31 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: a suite with a never-run assertion is rejected (TFP-110; automated: route “a local suite with an unawaited web-first assertion falls back as unawaited_assertion”, “an unawaited synchronous value assertion is not rejected - it still checks”, “an invented API is still invalid_output even when an assertion is also unawaited”)

Expected result: After the type check passes, generateWithOllama runs findUnawaitedAssertions (GEN-29). Any unawaited assertion that returns a promise returns the template, labelled fallbackReason: "unawaited_assertion", with model reported and the offending lines logged. The Phase 31 security line expect(page).toHaveURL(...) is rejected. An unawaited synchronous expect(value).toBe(...) is not rejected, because it still runs, and TestForge’s own template generator writes that form. The type check runs first, so an invented API stays invalid_output and the two failures stay distinct. Detect and fall back, never rewrite (Phase 33 Decision 1). If the check itself can’t run, it fails closed.

TFP-111 — Reliability: a real dependency/storage failure-injection example

GEN-32 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: reliability tests simulate a real failure (TFP-111; unit: local-rules.test.ts [reliability] cases, “localRulesFor: …”; route: “the local path sends only the guides for the areas the template generator would cover”; live: see Expected Result)

Expected result: Reliability here means TestForge’s own definition: graceful degradation when a dependency or storage fails. The worked example mirrors the template generator. It blocks '**/api/**', blocks every other host, and makes localStorage throw, each before page.goto. It then asserts only that the page survived: status below 500, body visible, no uncaught pageerror. The unit tests check the example: failure set up before load, each test a different failure, no specific fallback text, placeholders only, all awaited, compiles. Guides are sent only for the areas getBestPracticeAreas puts in scope, because the model copies examples. Live (testforge-qwen7b, production build, the GEN-28 wording “Check the login page (email and password fields, Sign in button) for reliability coverage.”, reliability selected). Every attempt is recorded. Attempt 1: 0/3 genuine. Two failed the type check (response used but never declared). The one that compiled routed invented patterns like '**/email-input/**', which block nothing, and compared a promise with toBe. The cause was the rules’ own “replace placeholders with the request’s elements” instruction. Attempt 2, rules saying the three failures are generic and not placeholders, that page.route can’t block an element, to keep const response, and never to pass a promise to expect: 3/3 contain a real failure simulation ('**/api/**' aborted before load, survival asserted). But only 1/3 covered all three failure kinds with accurate names. 2/3 repeated the same API-block test three times under titles like “when email input is blocked”. Attempt 3 added a rule to name tests after what they block: 1/3 genuine. The other 2/3 kept the misleading titles and blocked a guessed '**/api/login' that only fires on submit and is never triggered, so they check nothing. That rule was reverted, and the committed rules are attempt 2’s. Result: the category’s defining trait now appears (it never did before), but the model maps the request’s inputs and button onto reliability tests in 4 of the 6 attempt-2 and attempt-3 suites. Those suites have misleading titles and duplicates, and sometimes block requests that never fire. None has been run against a real site. Phase 37 update (TFP-130, GEN-57): the duplication half is now gated. A local suite whose tests share one body, whatever their titles, is rejected as duplicate_tests and the template is kept; GEN-54’s reliability-2 is rejected. The naming half, titles that don’t match what the body does, needs judgement and is not gated.

TFP-112 — Performance, SEO, and database: real category-specific examples

GEN-33 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: performance tests measure time (TFP-112; unit: local-rules.test.ts [performance/load]; live: see Expected Result)

Expected result: The guide requires a timing measurement against a budget: Date.now() around page.goto, or Navigation Timing through page.evaluate. A status code or a visibility check alone doesn’t count. The example’s budgets are 3000 ms load, 800 ms first byte and 2000 ms DOM ready. Live, 3 consecutive (GEN-28 login wording, performance/load selected): 3/3 genuine. Each suite measures load time and first-byte and DOM-ready timing against budgets, and all compile, pass strict tsc and leak no placeholders. All three are the example almost verbatim, with the request’s URL and page name substituted. That is the copying Phase 32 predicted, and here it produces exactly the intended tests. The budgets are the example’s defaults, not ones tuned for any site. (Baseline, GEN-28: a 200 status check, no timing.)

GEN-34 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: SEO tests check what search engines read (TFP-112; unit: local-rules.test.ts [seo/content]; live)

Expected result: The guide asks for the title (toHaveTitle), the meta description (meta[name="description"] content) and a single level-1 heading, with the real text used if the request or profile gives it. Live, 3 consecutive: 3/3 genuine. 3/3 check title and meta description, and 2/3 also check the single h1. One suite also added two visibility checks for the request’s fields under the SEO heading, a minor spill-over. (Baseline: one visibility-only test.)

GEN-35 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: database tests check persistence through the page (TFP-112; unit: local-rules.test.ts [database]; live)

Expected result: The guide asks the suite to save a unique value (Date.now()), reload, and check the value is still shown. On a page with no saving form, it should write test.skip rather than fake it. Live. Since a sign-in page has nothing to persist, the 3 consecutive runs used a contact-form wording (“Check the contact page form (name and message fields, Send button) for database coverage.”). Attempt 1: 3/3 real save-reload-check flows, but all copied the example’s EXAMPLE-VALUE prefix into test data (EXAMPLE-NAME …). The unique value is copy-safe data, so the example now uses TestForge check …, and a test keeps placeholders out of it. Attempt 2: 3/3 genuine, no leaks. Separately, with the GEN-28 login wording (1 run), the skip rule did not work: the model invented a “create a user and sign in” flow asserting a guessed Welcome, text, which is not persistence. (Baseline: one visibility-only test.)

TFP-113 — Security: fix the protocol assumption, broaden within existing guidance

GEN-36 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: security tests don’t assume HTTPS (TFP-113; unit: local-rules.test.ts [security]; live)

Expected result: The guide covers three things. HTTPS is asserted only behind test.skip(pageUrl.startsWith('http://'), …). Invalid input must not be accepted (the page doesn’t move on). Script-like input must not open a dialog (page.on('dialog')). Live, GEN-28 login wording. Attempt 1, with a negated skip test.skip(!pageUrl.startsWith('https://')): 3/3 checked invalid input and script injection, but the HTTPS check was wrong in every suite. 2/3 replaced it with a trivial http: assertion. 1/3 dropped the !, which skips on HTTPS and asserts https: on the http site, the baseline’s false assertion again. The skip condition is now positive, so there’s no negation to lose, and a test enforces it. Attempt 2: 3/3 have category-specific checks, and 0/3 assert https on an http site. 2/3 have all three checks with the skip copied correctly. 1/3 dropped the HTTPS test and reduced the script check to “stays on the page” without a dialog listener. (Baseline: one real invalid-input check and an un-awaited, wrong https:// assertion.)

  1. TFP-110 can land any time — no dependency on the category work.
  2. TFP-111 (reliability) first among the category tickets, since it’s the highest-risk and its result may change how much confidence the phase has in the approach; TFP-112 and TFP-113 can follow in either order.

Suggested Milestone Gate

Gate A — Every category checked, the honest result recorded either way

Implementation status (2026-09-24)

Landed: TFP-110 aca8dfa, TFP-111 2d881ca, TFP-112 b77d696, TFP-113 4215d82, plus df47f8f, a test-file split. TFP-110 had pushed route.test.ts over the 500-line lint limit without lint being run. It was caught before pushing, and neither intermediate commit was pushed.

All live runs used testforge-qwen7b on Emil’s machine through a production build. Every returned suite also passed a standalone strict tsc and was read, not just compiled. Every attempt is recorded in TEST_PLAN.md, not only the last one.

TFP-110: Acceptance Criterion 1 met. Suites with an unawaited promise-returning assertion fall back as unawaited_assertion, proven against the fake server. An unawaited synchronous toBe is deliberately not rejected: it still runs, and the template generator writes that form itself. invalid_output stays distinct, because the type check runs first (GEN-31).

One design change beyond this record. Guides are sent only for the areas getBestPracticeAreas puts in scope, which is the template generator’s own notion of what a request covers. Sending all six examples every time would, given Phase 32’s structure-copying finding, invite unrelated tests. A free-form request still receives the default guides (accessibility, SEO, reliability). The regression runs showed no spill-over from them.

Per-category result (Acceptance Criterion 2), 3 consecutive live runs each:

Area Result Notes
Reliability (TFP-111) ⚠️ trait appears, quality short Attempt 1: 0/3 genuine. The rules told the model to replace placeholders with the request’s elements, so it routed '**/email-input/**'. Attempt 2, with the failures declared generic: 3/3 contain a real page.route failure simulation, but 2/3 repeat it under titles naming the email, password and button. Attempt 3, with a naming rule: 1/3 genuine, and 2/3 blocked a submit-only endpoint they never trigger. That rule was reverted and attempt 2’s rules shipped.
Performance (TFP-112) ✅ 3/3 The example near-verbatim with the URL substituted. The budgets are the example’s defaults.
SEO (TFP-112) ✅ 3/3 Title and meta description 3/3, h1 2/3, and one minor spill of field-visibility tests.
Database (TFP-112) ✅ 3/3 on attempt 2 Contact-form wording, since a sign-in page has nothing to persist. Attempt 1 copied the EXAMPLE- prefix into test data, so the unique value is now copy-safe. On a sign-in page (1 run), the test.skip rule was ignored in favour of an invented sign-up flow.
Security (TFP-113) ✅ 3/3 on attempt 2 Attempt 1’s negated skip lost its ! in 1/3, which asserted https: on http again. With a positive skip: 0/3 false HTTPS assertions, and 2/3 have all three checks.
Accessibility (regression) ✅ 3/3 GEN-27 wording, with the default guides also sent: real role, name and focus checks, and nothing from the other guides spilled in.
Functional (regression) ✅ 1/1 GEN-21 wording still passes.

prompt.ts has no diff, and the unit suite passes 240/240.

Gate A is met, as this record defines it. TFP-110 is proven. TFP-111, TFP-112 and TFP-113 each have three consecutive live results, read and recorded. Reliability falls short on quality and says so (GEN-32 ⚠️), rather than the bar being lowered.

What this phase learned about examples:

  1. Examples get copied as templates. When the example is exactly the test we want, as with performance, that works for us.
  2. Anything marked as a placeholder must really be a placeholder. Data meant to be copied, like a unique-value prefix, must look like real, copy-safe data.
  3. Negations and conditions get dropped when copying, so write them positively.
  4. A request’s UI elements pull the model toward element-shaped tests, even in categories where the elements are irrelevant (reliability, database on a sign-in page).

Open, for Emil to prioritise: