Date: 2026-09-24 Scope: Teach the local (Ollama) generation path real, category-specific coverage for the five best-practice areas Phase 31’s baseline found thin or broken (performance, SEO, database, reliability, security), and close Phase 32’s one open question — whether the route should reject or repair suites with unawaited checks. Status: Implemented; Gate A met 2026-09-24 with reliability recorded as short on quality (see “Implementation status” at the end)
Phase 31’s baseline (GEN-28) generated one real suite per best-practice
area against testforge-qwen7b and read each one, not just compiled it.
Five categories fell short of what src/lib/generate/prompt.ts’s own
BEST_PRACTICE_GUIDANCE already asks for — guidance already sent to the
local model, same as the OpenAI path, so this isn’t a case of the model
never being told:
https:// against an
http:// base URL, and (Phase 32, GEN-29) was missing its await, so
it never even ran.BEST_PRACTICE_GUIDANCE’s prose at all.
src/app/_lib/constants.ts’s BEST_PRACTICE_OPTIONS defines it as
“Graceful degradation when a dependency or local storage fails,” and
the deterministic generator (src/lib/generate/mock/dependency-failure-injection.ts)
implements it with page.route() interception against specific
dependencies discovered during introspection — a real, fairly advanced
Playwright technique, not standard testing vocabulary. The local
model’s suite had only a title check and a focus check — consistent
with a model that was never told what “reliability” means here, not
one that understood and skipped it.Accessibility (Phase 31, TFP-106) proved the fix for “the model was told in prose and didn’t comply”: a concrete, copy-ready worked example, not more prose. Phase 32 (TFP-109/GEN-30) then found two things that change how those examples must be built:
/login path even though no request mentioned it.Phase 32’s record already states the resulting rule for every example this phase writes: use only placeholder data, write the example as exactly the tests that should be copied (because they will be), and test new examples with wording chosen to expose copying, not wording that happens to avoid it.
Phase 32 also left one question open: findUnawaitedAssertions (TFP-108)
can detect a suite with an unawaited check, but the route doesn’t act on
it yet — it’s a rule, not a gate.
Standing policy for this phase: same as every phase since 18 — a
defect found while building or verifying a ticket is fixed in that
ticket, not deferred; TEST_PLAN.md updates land in the same commit as
the code.
generateWithOllama calls findUnawaitedAssertions alongside
findTypeErrors before labelling a suite generatedBy: "ollama"; any
result from either fails the suite into the same fallback() path,
with a reason that distinguishes which check failed
(invalid_output vs. a new unawaited_assertion).await, because every gate this pipeline
has built (TFP-103, TFP-108) detects and falls back; none of them
rewrites the model’s output. Auto-repair would be new, riskier surface
(where exactly to insert await inside .not/expect.soft/chained
expressions is exactly the complexity isHandled already has to
reason about) to fix something TFP-108’s audit found only once in 18
real suites. Not worth the new risk for a rare case a template fallback
already handles safely.BEST_PRACTICE_GUIDANCE
/ BEST_PRACTICE_OPTIONS — not reinvented here. The model was already
told; a concrete example is the fix, the same lever that worked for
accessibility, not new prose explaining the category again.page.route() interception against a discovered dependency — a
technique the model has to be taught, not reminded of.page.route()-based
suite even with a worked example, that’s a real, recordable result —
not a reason to lower the bar or fake a pass.Performance, SEO, and database are grouped into one ticket — each gets its own example, but they’re comparable in shape and difficulty (a single concrete pattern to demonstrate, using standard Playwright APIs the model already handles correctly elsewhere).
https:// against whatever
protocol the target’s base URL actually uses) and, within what
BEST_PRACTICE_GUIDANCE already asks for, broadening beyond the one
check that currently exists — not redesigning the category.fallbackReason: "unawaited_assertion", proven against a
fake local server the same way TFP-103/106 proved their own gates.invalid_output case is unaffected — an unawaited
assertion and an invented API are distinct, both caught.testforge-qwen7b, checked by reading the suite (not just compiling
it) for a check that actually belongs to that category, matching
BEST_PRACTICE_GUIDANCE’s own description of it.page.route()-based (or
equivalent) failure-simulation test, not a generic assertion — the
actual trait that defines the category. If three consecutive attempts
don’t produce one, that’s recorded as the result, not smoothed over.findTypeErrors and findUnawaitedAssertions
clean, and uses only the request’s own data — checked against Phase
32’s leak lesson, the same way GEN-30 did.TEST_PLAN.md and this record reflect the real per-category outcome
TEST_PLAN.md gets a case for the unawaited_assertion gate and an
entry per category recording its fixed (or not-yet-fixed) state, in the
same commit as the code, per standing policy.Ticket numbers below are estimates — confirm the next real free number against the actual backlog before filing (TFP-109 was Phase 32’s last issued number as of this record).
bug, generate, localfindUnawaitedAssertions (TFP-108) can detect
the defect but the route doesn’t act on it — a suite with a silent
no-op assertion can still be labelled generatedBy: "ollama".findUnawaitedAssertions in generateWithOllama
alongside findTypeErrors; route a non-empty result into fallback()
with a new unawaited_assertion reason. Regression test against a fake
local server.route.ts; regression test; TEST_PLAN.md
update.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: a suite with a never-run assertion is rejected (TFP-110; automated: route “a local suite with an unawaited web-first assertion falls back as unawaited_assertion”, “an unawaited synchronous value assertion is not rejected - it still checks”, “an invented API is still invalid_output even when an assertion is also unawaited”)
Expected result: After the type check passes, generateWithOllama runs findUnawaitedAssertions (GEN-29). Any unawaited assertion that returns a promise returns the template, labelled fallbackReason: "unawaited_assertion", with model reported and the offending lines logged. The Phase 31 security line expect(page).toHaveURL(...) is rejected. An unawaited synchronous expect(value).toBe(...) is not rejected, because it still runs, and TestForge’s own template generator writes that form. The type check runs first, so an invented API stays invalid_output and the two failures stay distinct. Detect and fall back, never rewrite (Phase 33 Decision 1). If the check itself can’t run, it fails closed.
enhancement, generate, localpage.route()-based (or
equivalent) failure simulation against a placeholder dependency, built
to Phase 32’s three rules; rule text explaining what “reliability”
means here, since it isn’t standard vocabulary.local-rules.ts (or a
category-specific file if that reads better once written); unit test
checking the example itself is a real, awaited, type-checking
page.route() pattern; TEST_PLAN.md update with the honest result.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: reliability tests simulate a real failure (TFP-111; unit: local-rules.test.ts [reliability] cases, “localRulesFor: …”; route: “the local path sends only the guides for the areas the template generator would cover”; live: see Expected Result)
Expected result: Reliability here means TestForge’s own definition: graceful degradation when a dependency or storage fails. The worked example mirrors the template generator. It blocks '**/api/**', blocks every other host, and makes localStorage throw, each before page.goto. It then asserts only that the page survived: status below 500, body visible, no uncaught pageerror. The unit tests check the example: failure set up before load, each test a different failure, no specific fallback text, placeholders only, all awaited, compiles. Guides are sent only for the areas getBestPracticeAreas puts in scope, because the model copies examples. Live (testforge-qwen7b, production build, the GEN-28 wording “Check the login page (email and password fields, Sign in button) for reliability coverage.”, reliability selected). Every attempt is recorded. Attempt 1: 0/3 genuine. Two failed the type check (response used but never declared). The one that compiled routed invented patterns like '**/email-input/**', which block nothing, and compared a promise with toBe. The cause was the rules’ own “replace placeholders with the request’s elements” instruction. Attempt 2, rules saying the three failures are generic and not placeholders, that page.route can’t block an element, to keep const response, and never to pass a promise to expect: 3/3 contain a real failure simulation ('**/api/**' aborted before load, survival asserted). But only 1/3 covered all three failure kinds with accurate names. 2/3 repeated the same API-block test three times under titles like “when email input is blocked”. Attempt 3 added a rule to name tests after what they block: 1/3 genuine. The other 2/3 kept the misleading titles and blocked a guessed '**/api/login' that only fires on submit and is never triggered, so they check nothing. That rule was reverted, and the committed rules are attempt 2’s. Result: the category’s defining trait now appears (it never did before), but the model maps the request’s inputs and button onto reliability tests in 4 of the 6 attempt-2 and attempt-3 suites. Those suites have misleading titles and duplicates, and sometimes block requests that never fire. None has been run against a real site. Phase 37 update (TFP-130, GEN-57): the duplication half is now gated. A local suite whose tests share one body, whatever their titles, is rejected as duplicate_tests and the template is kept; GEN-54’s reliability-2 is rejected. The naming half, titles that don’t match what the body does, needs judgement and is not gated.
enhancement, generate, localBEST_PRACTICE_GUIDANCE already describing
what each should check.TEST_PLAN.md update per category.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: performance tests measure time (TFP-112; unit: local-rules.test.ts [performance/load]; live: see Expected Result)
Expected result: The guide requires a timing measurement against a budget: Date.now() around page.goto, or Navigation Timing through page.evaluate. A status code or a visibility check alone doesn’t count. The example’s budgets are 3000 ms load, 800 ms first byte and 2000 ms DOM ready. Live, 3 consecutive (GEN-28 login wording, performance/load selected): 3/3 genuine. Each suite measures load time and first-byte and DOM-ready timing against budgets, and all compile, pass strict tsc and leak no placeholders. All three are the example almost verbatim, with the request’s URL and page name substituted. That is the copying Phase 32 predicted, and here it produces exactly the intended tests. The budgets are the example’s defaults, not ones tuned for any site. (Baseline, GEN-28: a 200 status check, no timing.)
Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: SEO tests check what search engines read (TFP-112; unit: local-rules.test.ts [seo/content]; live)
Expected result: The guide asks for the title (toHaveTitle), the meta description (meta[name="description"] content) and a single level-1 heading, with the real text used if the request or profile gives it. Live, 3 consecutive: 3/3 genuine. 3/3 check title and meta description, and 2/3 also check the single h1. One suite also added two visibility checks for the request’s fields under the SEO heading, a minor spill-over. (Baseline: one visibility-only test.)
Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: database tests check persistence through the page (TFP-112; unit: local-rules.test.ts [database]; live)
Expected result: The guide asks the suite to save a unique value (Date.now()), reload, and check the value is still shown. On a page with no saving form, it should write test.skip rather than fake it. Live. Since a sign-in page has nothing to persist, the 3 consecutive runs used a contact-form wording (“Check the contact page form (name and message fields, Send button) for database coverage.”). Attempt 1: 3/3 real save-reload-check flows, but all copied the example’s EXAMPLE-VALUE prefix into test data (EXAMPLE-NAME …). The unique value is copy-safe data, so the example now uses TestForge check …, and a test keeps placeholders out of it. Attempt 2: 3/3 genuine, no leaks. Separately, with the GEN-28 login wording (1 run), the skip rule did not work: the model invented a “create a user and sign in” flow asserting a guessed Welcome, text, which is not persistence. (Baseline: one visibility-only test.)
bug, enhancement, generate, localBEST_PRACTICE_GUIDANCE’s existing security description
(auth/session behavior, visible security errors) — not a new category
definition.TEST_PLAN.md
update.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: security tests don’t assume HTTPS (TFP-113; unit: local-rules.test.ts [security]; live)
Expected result: The guide covers three things. HTTPS is asserted only behind test.skip(pageUrl.startsWith('http://'), …). Invalid input must not be accepted (the page doesn’t move on). Script-like input must not open a dialog (page.on('dialog')). Live, GEN-28 login wording. Attempt 1, with a negated skip test.skip(!pageUrl.startsWith('https://')): 3/3 checked invalid input and script injection, but the HTTPS check was wrong in every suite. 2/3 replaced it with a trivial http: assertion. 1/3 dropped the !, which skips on HTTPS and asserts https: on the http site, the baseline’s false assertion again. The skip condition is now positive, so there’s no negation to lose, and a test enforces it. Attempt 2: 3/3 have category-specific checks, and 0/3 assert https on an http site. 2/3 have all three checks with the skip copied correctly. 1/3 dropped the HTTPS test and reduced the script check to “stays on the page” without a dialog listener. (Baseline: one real invalid-input check and an un-awaited, wrong https:// assertion.)
TEST_PLAN.md, rather than the bar being lowered to close
the gate.Landed: TFP-110 aca8dfa, TFP-111 2d881ca, TFP-112 b77d696,
TFP-113 4215d82, plus df47f8f, a test-file split. TFP-110 had
pushed route.test.ts over the 500-line lint limit without lint being
run. It was caught before pushing, and neither intermediate commit was
pushed.
All live runs used testforge-qwen7b on Emil’s machine through a
production build. Every returned suite also passed a standalone strict
tsc and was read, not just compiled. Every attempt is recorded in
TEST_PLAN.md, not only the last one.
TFP-110: Acceptance Criterion 1 met. Suites with an unawaited
promise-returning assertion fall back as unawaited_assertion, proven
against the fake server. An unawaited synchronous toBe is deliberately
not rejected: it still runs, and the template generator writes that
form itself. invalid_output stays distinct, because the type check
runs first (GEN-31).
One design change beyond this record. Guides are sent only for the
areas getBestPracticeAreas puts in scope, which is the template
generator’s own notion of what a request covers. Sending all six
examples every time would, given Phase 32’s structure-copying finding,
invite unrelated tests. A free-form request still receives the default
guides (accessibility, SEO, reliability). The regression runs showed no
spill-over from them.
Per-category result (Acceptance Criterion 2), 3 consecutive live runs each:
| Area | Result | Notes |
|---|---|---|
| Reliability (TFP-111) | ⚠️ trait appears, quality short | Attempt 1: 0/3 genuine. The rules told the model to replace placeholders with the request’s elements, so it routed '**/email-input/**'. Attempt 2, with the failures declared generic: 3/3 contain a real page.route failure simulation, but 2/3 repeat it under titles naming the email, password and button. Attempt 3, with a naming rule: 1/3 genuine, and 2/3 blocked a submit-only endpoint they never trigger. That rule was reverted and attempt 2’s rules shipped. |
| Performance (TFP-112) | ✅ 3/3 | The example near-verbatim with the URL substituted. The budgets are the example’s defaults. |
| SEO (TFP-112) | ✅ 3/3 | Title and meta description 3/3, h1 2/3, and one minor spill of field-visibility tests. |
| Database (TFP-112) | ✅ 3/3 on attempt 2 | Contact-form wording, since a sign-in page has nothing to persist. Attempt 1 copied the EXAMPLE- prefix into test data, so the unique value is now copy-safe. On a sign-in page (1 run), the test.skip rule was ignored in favour of an invented sign-up flow. |
| Security (TFP-113) | ✅ 3/3 on attempt 2 | Attempt 1’s negated skip lost its ! in 1/3, which asserted https: on http again. With a positive skip: 0/3 false HTTPS assertions, and 2/3 have all three checks. |
| Accessibility (regression) | ✅ 3/3 | GEN-27 wording, with the default guides also sent: real role, name and focus checks, and nothing from the other guides spilled in. |
| Functional (regression) | ✅ 1/1 | GEN-21 wording still passes. |
prompt.ts has no diff, and the unit suite passes 240/240.
Gate A is met, as this record defines it. TFP-110 is proven. TFP-111, TFP-112 and TFP-113 each have three consecutive live results, read and recorded. Reliability falls short on quality and says so (GEN-32 ⚠️), rather than the bar being lowered.
What this phase learned about examples:
Open, for Emil to prioritise:
test.skip behaviour on pages that save nothing.