mr-onion

Phase 31 Decision Record — Local Model Tuning (Rules Fix + Baseline)

Date: 2026-09-24 Scope: Fix the local (Ollama) generation path’s diagnosed accessibility-matcher defect through a local-only prompt/rules layer, and establish a real baseline across every best-practice category — without touching the OpenAI path. Status: Implemented; Gate A met 2026-09-24 (TFP-106, TFP-107 landed; see “Implementation status” at the end)

Context

Phase 30 built the local generation path and closed Gate A with a type-check-then-fallback safety net (TFP-103’s findTypeErrors): a suite that doesn’t compile against @playwright/test is never labelled model output. That gate is a safety net, not a fix — it makes the failure safe, not less frequent. Phase 30 explicitly deferred two follow-ups for later: trying lighter/alternative local models, and further tuning for speed or accuracy. This phase picks up the accuracy half, at Emil’s request (“let’s do the tuning first”).

Real evidence already collected, not assumed:

Emil’s own framing across this session (“give it rules or update skills for it”, “if we can get all the skills and rules into the local model”) points at prompt/rules-level tuning as the mechanism — not fine-tuning the model itself or a retrieval layer. That’s the working approach for this phase: teach the model, through what it’s told at request time, to use real Playwright APIs instead of inventing them.

Standing policy for this phase: same as every phase since 18 — a defect found while building or verifying a ticket is fixed in that ticket, not deferred; TEST_PLAN.md updates land in the same commit as the code.

Decisions

  1. Tuning happens through a local-only rules layer, not the shared SYSTEM_PROMPT.
    • Additional guidance — naming the real Playwright accessibility assertions available (e.g. toHaveRole, toHaveAccessibleName on a locator) — is appended only when generating through the Ollama path.
    • src/lib/generate/prompt.ts’s SYSTEM_PROMPT and BEST_PRACTICE_GUIDANCE are not edited by this phase. Phase 28’s already-hardened, already-tested OpenAI prompt stays exactly as it is — a defect specific to one local model is not a reason to touch a path that isn’t even funded right now.
  2. This phase stays scoped to qwen2.5-coder:7b; lighter or alternative models are not revisited here.
    • qwen2.5-coder:3b and any other candidate remain explicitly out of scope, per Phase 30’s own deferral. 7b has one known defect class (invented accessibility APIs); 3b had worse ones (a hallucinated API and duplicate-named tests) in the original spike, before any rules layer existed — reopening it is a separate follow-up once 7b’s accuracy is established, not this phase’s job.
  3. The fix targets the diagnosed accessibility-matcher defect specifically, verified against a real baseline across every best-practice category.
    • The rules layer’s job is narrow: stop the model inventing toBeAccessible()-style APIs and use a real Playwright assertion instead.
    • Acceptance requires generating one real sample per best-practice category (functional, responsive, performance, security, accessibility, usability, SEO, cookie, database) against the real local model, through TFP-103’s validator, to find out whether accessibility is the only affected category or one of several — cheap to do now that the validator exists, and this phase is the right time to find out rather than guess.
  4. Success is measured against repeated real Ollama runs, not a single passing sample.
    • Three consecutive real accessibility-worded requests against testforge-qwen7b (matching the sample size that already established the defect) must return generatedBy: "ollama", not fallbackReason: "invalid_output", after the rules layer lands. One lucky pass doesn’t count as fixed.

Acceptance Criteria

  1. Accessibility requests generate real code, not the fallback template
    • Three consecutive real accessibility-worded requests against a running Ollama with testforge-qwen7b return generatedBy: "ollama".
    • Each returned suite passes both TFP-103’s findTypeErrors and a standalone strict tsc check — the same standard Phase 30 used for GEN-21.
  2. No category regresses, and the baseline is known
    • One real generation per best-practice category, run against the real local model and checked with the validator, with pass/fail recorded per category in TEST_PLAN.md — a documented baseline, not a guess.
    • The functional-request result Phase 30 already proved (GEN-21) still passes after the rules layer is added.
  3. The OpenAI path is provably unaffected
    • git diff on src/lib/generate/prompt.ts between this phase’s start and end is empty — the rules layer lives in its own file, referenced only from generateWithOllama.
    • TFP-083’s existing OpenAI-path tests still pass unchanged.

Regression Policy

Backlog

Ticket numbers below are estimates — confirm the next real free number against the actual backlog before filing (TFP-105 was Phase 30’s last issued number as of this record).

TFP-106 — Add a local-only rules layer fixing the accessibility-matcher defect

GEN-27 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: accessibility requests generate real checks, not the template (TFP-106; automated: route “the local path sends the shared prompt plus the local rules”, “the OpenAI path never sees the local rules”; unit: src/lib/generate/local-rules.test.ts; live: see Expected Result)

Expected result: Only on the local path, src/lib/generate/local-rules.ts is appended to the system message. It names the accessibility matchers that exist, forbids the invented ones, says expect(page) only takes page matchers, and includes a worked example. The OpenAI path’s system message is exactly SYSTEM_PROMPT, and prompt.ts is unchanged. The unit tests type-check every matcher the rules recommend, forbid, or call a page matcher against the installed @playwright/test, so a Playwright rename fails the test instead of quietly teaching the model a wrong API. Live against testforge-qwen7b through a production build on 2026-09-24, with the exact wording that failed 3/3 before: the first attempt passed 2 of 3. The failure was a new mistake, expect(page).toHaveRole(...), so the page-vs-locator rule was added. The next three consecutive requests all returned generatedBy: "ollama" (74-83s), and each also passes a standalone strict tsc. Each suite asserts toHaveRole and toHaveAccessibleName per control, and two of the three also assert toBeFocused. They use the request’s base URL, not the example’s.

TFP-107 — Baseline per-category defect survey against the real local model

GEN-28 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: per-category baseline (TFP-107; live, 2026-09-24)

Expected result: One real generation per best-practice area, all 10 in BEST_PRACTICE_OPTIONS (the Phase 31 record listed 9 and missed reliability), against testforge-qwen7b through a production build with the TFP-106 rules. Each request used the same wording (“Check the login page (email and password fields, Sign in button) for <area> coverage.”) with only that area selected. Compiles (route type check and standalone strict tsc): 10/10, 29-96s each, so no invented APIs appeared in any category. Has a check specific to its area, judged by reading each suite: functional ✅ (enabled state and attributes); responsive/mobile ✅ (3 viewports); accessibility ✅ (role, accessible name, focus); usability ✅ (keyboard reachability); cookie/session ✅ (context().cookies() before and after sign-in); security ⚠️ (invalid-input error is real, but its HTTPS check is a missing-await expect(page).toHaveURL('https://…') against an http base URL); reliability ⚠️ (title and focus only); performance/load ❌ (asserts status 200, measures no time); seo/content ❌ and database ❌ (one visibility-only test each). 9 of 10 suites navigated to /login, which the rules example uses but the requests don’t; earlier suites used the bare base URL. That pointed at the example leaking, but wasn’t proven here; GEN-30 later confirmed it with a controlled re-run, and removed it. One sample per area: a baseline, not a rate.

  1. TFP-106 → TFP-107.

Suggested Milestone Gate

Gate A — Accessibility fixed, baseline known, OpenAI path untouched

Implementation status (2026-09-24)

TFP-106 landed in 04ba706; TFP-107’s baseline in d311dad. All live runs were against testforge-qwen7b on Emil’s machine (RTX 2050, 4GB VRAM), through a production build (next build + next start), using TFP-103’s type check plus a standalone strict tsc on every suite returned as model output.

TFP-106: Acceptance Criterion 1 met, on the second attempt. The exact wording that failed 3/3 in Phase 30 was used throughout, and every run is recorded:

Acceptance Criterion 3 met. prompt.ts has no diff. The rules live in src/lib/generate/local-rules.ts, referenced only from generateWithOllama. A route test asserts the OpenAI path’s system message is exactly SYSTEM_PROMPT, and TFP-083’s OpenAI tests pass unchanged (unit suite 198/198).

TFP-107: Acceptance Criterion 2 met. TEST_PLAN.md GEN-28 has the per-area results. Correction to this record: BEST_PRACTICE_OPTIONS has ten areas, not nine. reliability was missing above, and all ten were surveyed. All 10 compile, so accessibility was the only invented-API defect. The functional request still passes (GEN-21’s behaviour held).

What the survey found beyond compiling, recorded and deliberately not fixed in this phase:

Gate A is met: three consecutive real accessibility passes, the baseline recorded, and prompt.ts untouched.

Candidate follow-ups, for Emil to prioritise (none started):

  1. Give the rules example a neutral URL built from the request’s base URL, then re-run the survey to confirm or rule out the /login leakage. Cheap.
  2. Per-area rules for the thin categories (e.g. real timing for performance, toHaveTitle and meta checks for SEO), the same mechanism as TFP-106, measured the same way.
  3. The Phase 21 rescoping this record defers: compare the local path, now with a known baseline, against the deterministic generator.