mr-onion

Phase 21 Decision Record — AI-Assisted Generation as a Measured Alternative (Locked)

Date: 2026-09-24 Scope: Bring the existing OpenAI-backed generation path into active development — real CI coverage, and evaluated against the deterministic generator on the same sites using Phase 17’s run persistence and diffing. The product decision (default generator, opt-in, or user’s choice per run) is made from that comparison, not in advance of it. Status: Complete. Retargeted 2026-09-24 (see Amendment, below) at the local (Ollama) path — TFP-098-100 compare the deterministic generator against testforge-qwen7b, not OpenAI. TFP-098 and TFP-099 met 2026-09-24; TFP-100 decided 2026-09-28: Option B (see “TFP-100 — Decision recorded”). Option B is built in Phase 34.

Context

docs/decisions/product-roadmap.md’s Phase 21 paragraph already locks this phase’s shape in one sentence. Two things ground it further, confirmed against the real repo on 2026-09-24 rather than assumed:

All of this record’s placeholders were resolved by Emil on 2026-09-24 — see Decisions below. No ticket in this phase is blocked on an open product decision anymore; Phase 28’s live-check is the only remaining gate, and it’s an engineering step, not a decision.

Standing policy for this phase: same as Phases 18–20 (formalized repo-wide since 2026-09-23) — a defect found while building or verifying a ticket is fixed in that ticket, not deferred; TEST_PLAN.md and doc updates land in the same commit as the code.

Decisions

  1. The deterministic generator remains the default; this phase evaluates, it doesn’t replace.
    • Every capability built in Phases 16–20 keeps working unchanged with no OPENAI_API_KEY set. Nothing in this phase touches that default path.
  2. Comparison reuses Phase 17’s run diffing, not a new engine.
    • Both paths generate against the same target under the same crawl/ introspect data; the existing run-diff machinery (built for comparing two runs of a target over time) is pointed at “deterministic run vs. OpenAI-backed run.”

3a. Resolved 2026-09-24 — scheduled cadence (not per-PR), plus a manual on-demand trigger for the same job. Bounds routine spend by bounding frequency regardless of PR/contributor volume, without depending on anyone remembering to run it by hand (the failure mode that ruled out pure-manual) — and the manual trigger means Emil isn’t stuck waiting for the next scheduled slot when he actually wants a comparison right now. Concretely: a CI job with both a schedule trigger and a workflow_dispatch (or equivalent on-demand) trigger, sharing one capped-budget secret — the schedule is the routine cadence, the manual trigger is the same job invoked early, not a second code path. Exact cadence (nightly vs. weekly) is an implementation detail for whoever builds TFP-098, not a product decision — pick one, state it in the ticket’s own doc update.

3b. Resolved 2026-09-24 — product-level cost model, once this ships to other site owners: bring-your-own-key now, TestForge-hosted-key-by- default later.

  1. Resolved 2026-09-24 — the comparison scores all four dimensions together: discovered-surface coverage delta, pass/fail agreement between the two paths, defect-detection difference (one path catches something the other misses), and cost & latency per run. Where a dimension genuinely can’t be computed for a given target — e.g. defect-detection needs a known, injected defect to test against, which not every comparison run will have — the report states which dimension was skipped and why, rather than omitting it silently or forcing a number that isn’t real.

  2. Resolved 2026-09-24 — the exposure mechanism (UI flag vs. API-only) is not decided now. TFP-100 decides it from TFP-099’s actual comparison results, not in advance — consistent with the roadmap’s own framing that the product decision comes from the comparison.

Acceptance Criteria

  1. CI coverage for the real OpenAI path (TFP-098)
    • Live-check half: a real, currently-resolving model is confirmed via a models-list call or a real minimal request, recorded alongside Phase 28’s own live-check (same finding, don’t duplicate the work).
    • CI-wiring half: a CI job with both a schedule trigger and a manual workflow_dispatch (or equivalent) trigger exists, shares one capped-budget secret, and a manually-triggered run actually produces a real comparison result end to end.
  2. Comparison report (TFP-099 — gated on TFP-098; Decision 4 answered)
    • Against a real target, a completed comparison run scores discovered- surface coverage delta, pass/fail agreement, defect-detection difference, and cost & latency — or, for any dimension that can’t be computed for that target, states which one and why, rather than a silent omission or a guessed number.
  3. Product decision + exposure (TFP-100 — gated on TFP-099’s real output)
    • Not written in advance — the roadmap is explicit that the decision comes from the comparison, not before it (Decision 5 confirms this is deliberate, not an oversight).

Regression Policy

Backlog

Ticket numbers start at TFP-098 (the highest number in use across docs/decisions/ as of 2026-09-24 is TFP-097) — confirm against the live backlog before filing, since other work may have claimed numbers since.

TFP-098 — Get real CI coverage for the OpenAI-backed path

GEN-37 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Generator comparison, template vs. local model, against real targets (TFP-098/099; manual, on-demand: npm run compare:generators, see docs/GENERATOR_COMPARISON.md; report: docs/decisions/phase-21-comparison-report.md)

Expected result: One command generates both suites from the same request and introspection data through the real /api/generate. It runs both through /api/run, where they are persisted like any run, and diffs the clean pair with /api/runs/diff. On the comparison fixture it also re-runs each suite with that case’s injected defect switched on. Run live on 2026-09-24 against the fixture (7 areas, 6 injected defects) and https://the-internet.herokuapp.com (5 cases). Real diffs and real runs; nothing mocked. Findings: the template writes far more tests where it writes any (functional 31 vs. 1) in 1-11 ms, against a local median of 67 s, with 2/16 local generations timing out. For these requests it wrote no tests for performance, database or security. Defect detection on the fixture: local 6/6, template 2/6. The fixture’s caveat applies (designed alongside the local model’s examples). On the real site the template found 5 real SEO gaps to the local model’s 2, and each generator had 2-3 false failures. Every dimension is scored or explicitly skipped: coverage is shown both as the run report’s figure (title-matched, which scores model-written tests ~0% for functional, accessibility and reliability) and as a title-independent element-reference proxy; pass/fail is compared per area; detection is skipped on the real site because there is no injected defect. Defects fixed on the way: RPT-09 (model-written tests filed as Functional), a fixture defect hidden by form re-submission, and a skipped test counted as a failure in the report.

RPT-09 evidence

Moved word for word from docs/test-plan/RPT.md by TFP-253; the case’s row there links here.

Test: Model-written tests are reported under their area (TFP-098; unit: src/lib/report/execution.test.ts)

Expected result: A test’s area comes from its test.describe title. The template generator’s suite names (“Accessibility Coverage”, “SEO and Content Coverage”, …) map exactly as before. A describe titled exactly an area’s key or label, case-insensitive (“Accessibility”, “Performance/Load”, “SEO”, “Cookies”), now maps to that area too. This is what SYSTEM_PROMPT asks models to write. Before this, every model-written test was filed under Functional, which made per-area reports and the Phase 21 comparison meaningless. Only whole titles match: “Login security notes” stays Functional. Found while building the generator comparison.

TFP-099 — Build the deterministic-vs-OpenAI comparison report

TFP-100 — Record the default/opt-in/user’s-choice product decision

  1. Phase 28 (TFP-083) lands, or at minimum its guard/timeout/fallback and live-check — not a hard gate on this record, but real comparison data through an unguarded or non-resolving call is a false start
  2. TFP-098 → TFP-099 → TFP-100, in sequence (TFP-098’s live-check half can start immediately, in parallel with Phase 28’s own)

Suggested Milestone Gate

Gate — Phase 21 complete


Amendment (2026-09-24) — Retargeted at the local (Ollama) path

Why. Emil’s OpenAI account remains unfunded, with no near-term plan to fund it (Phase 28’s live-check found the account rejects any real completion at $0 balance). Separately, Phases 30-33 built and tuned a local generation path (testforge-qwen7b via Ollama) that now has a real, honestly-measured baseline across nine of the ten best-practice categories — only reliability remains partial (Phase 33, GEN-32, recorded ⚠️, not fixed, not hidden). Emil’s own steer at the end of that work: retarget this phase’s comparison at the local model now that it has a fair baseline to be measured from, rather than continue waiting on a funded OpenAI account to do the comparison this record originally described.

A technical fact, not a preference, forces one more change. Ollama runs only on Emil’s own machine and is unreachable from this session’s tools or any hosted CI runner — confirmed repeatedly across Phases 30-33, where every live check had to run inside Emil’s own environment (via the device bridge or Code directly). Decision 3a’s CI-wiring plan (a GitHub Actions job with a schedule and a manual workflow_dispatch trigger) cannot be built for the local path at all, not just deprioritized — there is no network path from a hosted runner to Emil’s GPU. This is recorded and routed around, not re-decided.

Decisions (retargeted)

Decisions 1, 2, 4, and 5 above are unchanged and apply to the local path exactly as written. Decisions 3a and 3b are superseded, for this comparison specifically, as follows:

3a (retargeted). No CI job exists for this comparison, and none is planned. It runs on demand, on Emil’s own machine, the same way every Phase 30-33 live check already has — triggered manually by Emil or Code, not on a schedule. TFP-098’s deliverable is a repeatable script/command, not a workflow file.

3b (retargeted). The bring-your-own-key cost model is moot for the local path — every run costs $0 by construction, no key, no billing surface. Cost stays one of the four Decision-4 dimensions the report scores (for contrast against what the same comparison would show if OpenAI is ever funded), but needs no BYOK or hosted-key design work to get there.

Decision 4’s four dimensions still apply, reweighted in practice: latency is the dimension that actually differentiates local generation (29-96s per suite, Phase 33) from the deterministic generator’s near-instant output — OpenAI would have been the fast comparison point instead. Coverage delta and pass/fail agreement matter most given how much category-by-category tuning Phase 33 just did. Defect-detection is scored exactly as originally decided: stated as skipped when there’s no injected defect to test against, never guessed.

New note on reliability. TFP-099’s report states reliability’s Phase 33 weakness plainly as a caveat on that category’s results, rather than waiting for it to be fixed first. An honest, current baseline for nine categories is more useful now than a complete one later.

Backlog (retargeted; same ticket numbers, updated scope)

TFP-098 (retargeted) — Build a repeatable local-generation comparison run

TFP-099 (retargeted) — Build the deterministic-vs-local comparison report

TFP-100 (retargeted) — Record the default/opt-in/user’s-choice product decision

Implementation status (2026-09-24)

A correction to the Amendment’s premise. Ollama is not unreachable from Claude Code’s session: every Phase 30-33 live check, and every run in this phase, ran from Claude Code against Emil’s Ollama on localhost. That changes nothing decided here. A hosted CI runner still can’t reach it, so 3a (retargeted) stands, but “needs Emil’s machine” means this machine, reachable by Code, not “only Emil can run it”.

TFP-098: met. npm run compare:generators -- --config <file> (scripts/compare-generators.mjs, docs/GENERATOR_COMPARISON.md). One command per target. Both generators get the same request and introspection data and go through the real /api/generate. Both suites run through /api/run, and the clean pair is diffed with /api/runs/diff (Decision 2). Run end to end against two real targets, nothing mocked. There is no workflow file, per 3a (retargeted).

TFP-099: met. The report is docs/decisions/phase-21-comparison-report.md, with TEST_PLAN.md GEN-37. All four Decision-4 dimensions are scored for every case, or explicitly skipped with a reason:

Two defects were found and fixed along the way:

TFP-100: proposed, not recorded as decided. See the next section. The product decision is Emil’s, so it is not yet written into product-roadmap.md.

TFP-100 — Proposed decision (drawn from TFP-099’s results; decided below)

Decision 5 says this is decided from the comparison, so the options are set against what the comparison actually showed (phase-21-comparison-report.md), not argued in the abstract.

What the data rules out. Making the local model the default, or keeping today’s all-or-nothing switch as the main way to use it:

What the data supports. The local model filled exactly the areas where the template wrote no executable tests (performance, database, security) and caught the injected defect in each. That points to it adding to the template rather than replacing it.

Option Shape For Against
A. Keep as is Env var switches all generation to the local model No work The data above: breadth lost in every area the template covers
B. Additive, gap-filling (recommended) The template always generates. The local model is asked only for the selected areas where the template produced no tests, and its describe blocks are appended. The local tests keep their own label, so a run shows which tests came from where. Keeps the template’s breadth and speed. Adds real coverage where there was none. Latency is only paid when such an area is selected. New work: merging two suites, and deciding which areas count as gaps. Local tests are still one sample per area and carry Phase 33’s reliability caveat, so reliability stays template-only in this shape.
C. User’s choice per run A UI toggle picks one generator for the whole suite Simple mental model Reproduces A’s trade-off per run, and the user must know which areas the template skips

Recommendation: B.

Not decided here, deliberately:

Once Emil confirms or changes this, the chosen option is recorded in product-roadmap.md’s Phase 21 text, as the Gate requires.

TFP-100 — Decision recorded (2026-09-28)

Emil chose Option B: additive, gap-filling. Recorded as proposed above, unchanged:

Options A and C are not taken, for the reasons in the table above.

Follow-up found while deciding. Reading src/lib/generate/mock/ on 2026-09-28 answered the comparison report’s caveat 2 (whether other wording changes “template: no tests”):

Emil decided the template should cover these three areas itself. That work, and building Option B, is Phase 34 (phase-34-template-coverage-and-gap-filling-decision-record.md). Because the template work changes which areas are gaps, Phase 34 sequences it before Option B.

Gate — Phase 21 complete: met 2026-09-28. The decision is recorded here and in product-roadmap.md’s Phase 21 text.