Date: 2026-09-24 Scope: Fix the local (Ollama) generation path’s diagnosed accessibility-matcher defect through a local-only prompt/rules layer, and establish a real baseline across every best-practice category — without touching the OpenAI path. Status: Implemented; Gate A met 2026-09-24 (TFP-106, TFP-107 landed; see “Implementation status” at the end)
Phase 30 built the local generation path and closed Gate A with a
type-check-then-fallback safety net (TFP-103’s findTypeErrors): a suite
that doesn’t compile against @playwright/test is never labelled model
output. That gate is a safety net, not a fix — it makes the failure safe,
not less frequent. Phase 30 explicitly deferred two follow-ups for later:
trying lighter/alternative local models, and further tuning for speed or
accuracy. This phase picks up the accuracy half, at Emil’s request
(“let’s do the tuning first”).
Real evidence already collected, not assumed:
testforge-qwen7b all produced expect(locator).toBeAccessible(), which
is not a real Playwright matcher. All three were caught by TFP-103’s
validator and returned as the labelled template
(fallbackReason: "invalid_output").generatedBy: "ollama", fence-free, and it also passes a standalone
strict tsc.Emil’s own framing across this session (“give it rules or update skills for it”, “if we can get all the skills and rules into the local model”) points at prompt/rules-level tuning as the mechanism — not fine-tuning the model itself or a retrieval layer. That’s the working approach for this phase: teach the model, through what it’s told at request time, to use real Playwright APIs instead of inventing them.
Standing policy for this phase: same as every phase since 18 — a
defect found while building or verifying a ticket is fixed in that ticket,
not deferred; TEST_PLAN.md updates land in the same commit as the code.
SYSTEM_PROMPT.
toHaveRole, toHaveAccessibleName on a
locator) — is appended only when generating through the Ollama path.src/lib/generate/prompt.ts’s SYSTEM_PROMPT and
BEST_PRACTICE_GUIDANCE are not edited by this phase. Phase 28’s
already-hardened, already-tested OpenAI prompt stays exactly as it is —
a defect specific to one local model is not a reason to touch a path
that isn’t even funded right now.qwen2.5-coder:7b; lighter or alternative
models are not revisited here.
qwen2.5-coder:3b and any other candidate remain explicitly out of
scope, per Phase 30’s own deferral. 7b has one known defect class
(invented accessibility APIs); 3b had worse ones (a hallucinated API
and duplicate-named tests) in the original spike, before any rules
layer existed — reopening it is a separate follow-up once 7b’s accuracy
is established, not this phase’s job.toBeAccessible()-style APIs and use a real Playwright assertion
instead.testforge-qwen7b (matching the sample size that already established
the defect) must return generatedBy: "ollama", not
fallbackReason: "invalid_output", after the rules layer lands. One
lucky pass doesn’t count as fixed.testforge-qwen7b return generatedBy: "ollama".findTypeErrors and a
standalone strict tsc check — the same standard Phase 30 used for
GEN-21.TEST_PLAN.md — a documented baseline, not a guess.git diff on src/lib/generate/prompt.ts between this phase’s start
and end is empty — the rules layer lives in its own file, referenced
only from generateWithOllama.TEST_PLAN.md gets a new case (or amends GEN-21/26) recording the fixed
accessibility behavior and the per-category baseline, in the same commit
as the code, per standing policy.TFP-099 is currently scoped and
blocked on TFP-098’s funded-OpenAI requirement. Running that
comparison against the local path instead of OpenAI doesn’t have that
money blocker — but it means rescoping TFP-098/TFP-099 (or opening a
new ticket under Phase 21 or a new phase) to target testforge-qwen7b
rather than OpenAI’s candidate list. That’s a real decision, not assumed
here, and it’s better made once this phase gives the local path a known,
stable baseline — so the comparison measures the model at its best, not
against a defect this phase is about to fix.Ticket numbers below are estimates — confirm the next real free number against the actual backlog before filing (TFP-105 was Phase 30’s last issued number as of this record).
enhancement, generate, localtestforge-qwen7b invents
expect(locator).toBeAccessible(), a Playwright API that doesn’t exist,
on every accessibility-worded request tested so far. TFP-103’s validator
catches it safely, but every such request currently falls back to the
template instead of generating real coverage.generateWithOllama’s message construction) naming
the real Playwright accessibility assertions available. Does not touch
src/lib/generate/prompt.ts’s shared SYSTEM_PROMPT or
BEST_PRACTICE_GUIDANCE. Regression test: force an accessibility-worded
prompt against a fake local server and assert the rules text is actually
included in the outgoing request (a unit test can’t force real model
behavior, but it can prove the rules are wired in).src/app/api/generate/route.ts; regression test; TEST_PLAN.md update.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: accessibility requests generate real checks, not the template (TFP-106; automated: route “the local path sends the shared prompt plus the local rules”, “the OpenAI path never sees the local rules”; unit: src/lib/generate/local-rules.test.ts; live: see Expected Result)
Expected result: Only on the local path, src/lib/generate/local-rules.ts is appended to the system message. It names the accessibility matchers that exist, forbids the invented ones, says expect(page) only takes page matchers, and includes a worked example. The OpenAI path’s system message is exactly SYSTEM_PROMPT, and prompt.ts is unchanged. The unit tests type-check every matcher the rules recommend, forbid, or call a page matcher against the installed @playwright/test, so a Playwright rename fails the test instead of quietly teaching the model a wrong API. Live against testforge-qwen7b through a production build on 2026-09-24, with the exact wording that failed 3/3 before: the first attempt passed 2 of 3. The failure was a new mistake, expect(page).toHaveRole(...), so the page-vs-locator rule was added. The next three consecutive requests all returned generatedBy: "ollama" (74-83s), and each also passes a standalone strict tsc. Each suite asserts toHaveRole and toHaveAccessibleName per control, and two of the three also assert toBeFocused. They use the request’s base URL, not the example’s.
test, generate, localtestforge-qwen7b through a
running Ollama, run each through TFP-103’s validator, record pass/fail
per category.TEST_PLAN.md baseline entry; this record’s
Implementation status section updated with the actual per-category
results.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: per-category baseline (TFP-107; live, 2026-09-24)
Expected result: One real generation per best-practice area, all 10 in BEST_PRACTICE_OPTIONS (the Phase 31 record listed 9 and missed reliability), against testforge-qwen7b through a production build with the TFP-106 rules. Each request used the same wording (“Check the login page (email and password fields, Sign in button) for <area> coverage.”) with only that area selected. Compiles (route type check and standalone strict tsc): 10/10, 29-96s each, so no invented APIs appeared in any category. Has a check specific to its area, judged by reading each suite: functional ✅ (enabled state and attributes); responsive/mobile ✅ (3 viewports); accessibility ✅ (role, accessible name, focus); usability ✅ (keyboard reachability); cookie/session ✅ (context().cookies() before and after sign-in); security ⚠️ (invalid-input error is real, but its HTTPS check is a missing-await expect(page).toHaveURL('https://…') against an http base URL); reliability ⚠️ (title and focus only); performance/load ❌ (asserts status 200, measures no time); seo/content ❌ and database ❌ (one visibility-only test each). 9 of 10 suites navigated to /login, which the rules example uses but the requests don’t; earlier suites used the bare base URL. That pointed at the example leaking, but wasn’t proven here; GEN-30 later confirmed it with a controlled re-run, and removed it. One sample per area: a baseline, not a rate.
testforge-qwen7b, TFP-107’s per-category baseline is recorded in
TEST_PLAN.md, and src/lib/generate/prompt.ts shows no diff from
where Phase 28 left it.TFP-106 landed in 04ba706; TFP-107’s baseline in d311dad. All live runs
were against testforge-qwen7b on Emil’s machine (RTX 2050, 4GB VRAM),
through a production build (next build + next start), using TFP-103’s
type check plus a standalone strict tsc on every suite returned as
model output.
TFP-106: Acceptance Criterion 1 met, on the second attempt. The exact wording that failed 3/3 in Phase 30 was used throughout, and every run is recorded:
expect(page).toHaveRole(...), the right matcher on the wrong object,
which the rules hadn’t ruled out. It was caught as invalid_output.expect(page) takes only page matchers, and
restarted the count. Three consecutive runs all returned
generatedBy: "ollama" (74-83s), and all passed strict tsc. Each
asserts toHaveRole and toHaveAccessibleName per control, and two of
the three also assert toBeFocused. They are real accessibility checks,
not visibility checks relabelled.Acceptance Criterion 3 met. prompt.ts has no diff. The rules live in
src/lib/generate/local-rules.ts, referenced only from
generateWithOllama. A route test asserts the OpenAI path’s system
message is exactly SYSTEM_PROMPT, and TFP-083’s OpenAI tests pass
unchanged (unit suite 198/198).
TFP-107: Acceptance Criterion 2 met. TEST_PLAN.md GEN-28 has the
per-area results. Correction to this record: BEST_PRACTICE_OPTIONS has
ten areas, not nine. reliability was missing above, and all ten were
surveyed. All 10 compile, so accessibility was the only invented-API
defect. The functional request still passes (GEN-21’s behaviour held).
What the survey found beyond compiling, recorded and deliberately not fixed in this phase:
toHaveURL that can’t match. Reliability is title and focus
only. This is a content-quality problem, a different class from the
invented APIs this phase targeted./login, the rules example’s path, which no request mentioned. Every
earlier suite used the bare base URL. The survey’s wording also
differed, so this is likely but not proven.Gate A is met: three consecutive real accessibility passes, the
baseline recorded, and prompt.ts untouched.
Candidate follow-ups, for Emil to prioritise (none started):
/login
leakage. Cheap.toHaveTitle and meta checks for SEO), the same
mechanism as TFP-106, measured the same way.