Date: 2026-09-24 Scope: Bring the existing OpenAI-backed generation path into active development — real CI coverage, and evaluated against the deterministic generator on the same sites using Phase 17’s run persistence and diffing. The product decision (default generator, opt-in, or user’s choice per run) is made from that comparison, not in advance of it. Status: Complete. Retargeted 2026-09-24 (see Amendment, below) at the local (Ollama) path — TFP-098-100 compare the deterministic generator against testforge-qwen7b, not OpenAI. TFP-098 and TFP-099 met 2026-09-24; TFP-100 decided 2026-09-28: Option B (see “TFP-100 — Decision recorded”). Option B is built in Phase 34.
docs/decisions/product-roadmap.md’s Phase 21 paragraph already locks this
phase’s shape in one sentence. Two things ground it further, confirmed
against the real repo on 2026-09-24 rather than assumed:
POST
/api/generate (src/app/api/generate/route.ts) calls gpt-4o today
when OPENAI_API_KEY is set. This is not a build-from-scratch phase.docs/decisions/phase-28-openai-path-hardening-backlog.md (TFP-083)
already closes the reliability gap in that same code path — try/catch,
timeout, fallback to mockGenerate, a pinned model string — and
explicitly, deliberately does not attempt this phase’s CI-coverage-and-
comparison work. Phase 28 is a prerequisite in practice, not just
principle: real comparison data through a path that 500s on a bad key or
hangs on a timeout is noise, not a measurement. Updated 2026-09-24:
this is now stronger than a reliability concern. Phase 28 itself was
updated the same day with a finding that the hardcoded gpt-4o model
may not resolve via the OpenAI API at all anymore (public sources
disagree on the exact date, but agree it’s being retired through 2026)
— meaning the path this phase evaluates may currently be fully
non-functional, not just unguarded. TFP-098 cannot produce real
comparison data until Phase 28’s live-check step confirms a working
model.All of this record’s placeholders were resolved by Emil on 2026-09-24 — see Decisions below. No ticket in this phase is blocked on an open product decision anymore; Phase 28’s live-check is the only remaining gate, and it’s an engineering step, not a decision.
Standing policy for this phase: same as Phases 18–20 (formalized
repo-wide since 2026-09-23) — a defect found while building or verifying a
ticket is fixed in that ticket, not deferred; TEST_PLAN.md and doc
updates land in the same commit as the code.
OPENAI_API_KEY set. Nothing in this phase touches that default path.3a. Resolved 2026-09-24 — scheduled cadence (not per-PR), plus a
manual on-demand trigger for the same job. Bounds routine spend by
bounding frequency regardless of PR/contributor volume, without
depending on anyone remembering to run it by hand (the failure mode
that ruled out pure-manual) — and the manual trigger means Emil isn’t
stuck waiting for the next scheduled slot when he actually wants a
comparison right now. Concretely: a CI job with both a schedule
trigger and a workflow_dispatch (or equivalent on-demand) trigger,
sharing one capped-budget secret — the schedule is the routine
cadence, the manual trigger is the same job invoked early, not a
second code path. Exact cadence (nightly vs. weekly) is an
implementation detail for whoever builds TFP-098, not a product
decision — pick one, state it in the ticket’s own doc update.
3b. Resolved 2026-09-24 — product-level cost model, once this ships to other site owners: bring-your-own-key now, TestForge-hosted-key-by- default later.
OPENAI_API_KEY — the same pattern Phase 20 already uses for login
credentials. TestForge stays local-only (Phase 16’s own locked posture)
and this needs no new infrastructure.docs/decisions/saas-prerequisites.md already names as a prerequisite
with “no implementation scheduled” (TestForge is local-only today:
“whoever can reach the process already has whatever access the process
has — there is no second party to authenticate against”). Hosted-key-
by-default is a SaaS-era feature, not a Phase 21 one — recorded here as
the stated long-term direction and cross-referenced from
saas-prerequisites.md, not scoped into TFP-098–100 below.Resolved 2026-09-24 — the comparison scores all four dimensions together: discovered-surface coverage delta, pass/fail agreement between the two paths, defect-detection difference (one path catches something the other misses), and cost & latency per run. Where a dimension genuinely can’t be computed for a given target — e.g. defect-detection needs a known, injected defect to test against, which not every comparison run will have — the report states which dimension was skipped and why, rather than omitting it silently or forcing a number that isn’t real.
Resolved 2026-09-24 — the exposure mechanism (UI flag vs. API-only) is not decided now. TFP-100 decides it from TFP-099’s actual comparison results, not in advance — consistent with the roadmap’s own framing that the product decision comes from the comparison.
schedule trigger and a manual
workflow_dispatch (or equivalent) trigger exists, shares one
capped-budget secret, and a manually-triggered run actually produces a
real comparison result end to end.TEST_PLAN.md gets a case for the comparison report matching a real
target’s actual diff as soon as TFP-099 lands. No cases stubbed in now
against placeholders that don’t have answers.Ticket numbers start at TFP-098 (the highest number in use across
docs/decisions/ as of 2026-09-24 is TFP-097) — confirm against the live
backlog before filing, since other work may have claimed numbers since.
feature, ciproduct-roadmap.md’s own Current Functionality section
already names this. Nothing changes until there’s a real key and a real
place for it to run.schedule
trigger (routine cadence — pick nightly or weekly, state it in the
doc update) and a manual workflow_dispatch trigger (same job, same
capped-budget secret, invoked on demand), per Decision 3a.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Generator comparison, template vs. local model, against real targets (TFP-098/099; manual, on-demand: npm run compare:generators, see docs/GENERATOR_COMPARISON.md; report: docs/decisions/phase-21-comparison-report.md)
Expected result: One command generates both suites from the same request and introspection data through the real /api/generate. It runs both through /api/run, where they are persisted like any run, and diffs the clean pair with /api/runs/diff. On the comparison fixture it also re-runs each suite with that case’s injected defect switched on. Run live on 2026-09-24 against the fixture (7 areas, 6 injected defects) and https://the-internet.herokuapp.com (5 cases). Real diffs and real runs; nothing mocked. Findings: the template writes far more tests where it writes any (functional 31 vs. 1) in 1-11 ms, against a local median of 67 s, with 2/16 local generations timing out. For these requests it wrote no tests for performance, database or security. Defect detection on the fixture: local 6/6, template 2/6. The fixture’s caveat applies (designed alongside the local model’s examples). On the real site the template found 5 real SEO gaps to the local model’s 2, and each generator had 2-3 false failures. Every dimension is scored or explicitly skipped: coverage is shown both as the run report’s figure (title-matched, which scores model-written tests ~0% for functional, accessibility and reliability) and as a title-independent element-reference proxy; pass/fail is compared per area; detection is skipped on the real site because there is no injected defect. Defects fixed on the way: RPT-09 (model-written tests filed as Functional), a fixture defect hidden by form re-submission, and a skipped test counted as a failure in the report.
Moved word for word from docs/test-plan/RPT.md by TFP-253; the case’s row there links here.
Test: Model-written tests are reported under their area (TFP-098; unit: src/lib/report/execution.test.ts)
Expected result: A test’s area comes from its test.describe title. The template generator’s suite names (“Accessibility Coverage”, “SEO and Content Coverage”, …) map exactly as before. A describe titled exactly an area’s key or label, case-insensitive (“Accessibility”, “Performance/Load”, “SEO”, “Cookies”), now maps to that area too. This is what SYSTEM_PROMPT asks models to write. Before this, every model-written test was filed under Functional, which made per-area reports and the Phase 21 comparison meaningless. Only whole titles match: “Login security notes” stays Functional. Found while building the generator comparison.
feature, reportingsrc/lib/generate/models.ts, TFP-083) — not
just whichever one happens to be pinned — an OpenAI-backed run of the
same target, so the report says which candidate is actually best for
this task, not just whether AI beats deterministic. Added 2026-09-24
(Cowork, at Emil’s request): runs on the same scheduled cadence
Decision 3a already set up, sharing its trigger and capped-budget
secret rather than adding a second schedule.TEST_PLAN.md case proving it against a real
target with a real diff, not a synthetic one; a documented process for
promoting a different candidate into Phase 28’s config from the
report’s results (a manual review of real data, then an ordinary code
change through Claude Code — not automatic switching).productTEST_PLAN.md case
proving the comparison reflects real (not assumed) behavior on both
paths, a manually-triggered run proven to work on Emil’s machine (see
Amendment: no CI job exists for this comparison), and the product
decision recorded in product-roadmap.md’s own text.Why. Emil’s OpenAI account remains unfunded, with no near-term plan to
fund it (Phase 28’s live-check found the account rejects any real
completion at $0 balance). Separately, Phases 30-33 built and tuned a
local generation path (testforge-qwen7b via Ollama) that now has a real,
honestly-measured baseline across nine of the ten best-practice
categories — only reliability remains partial (Phase 33, GEN-32, recorded
⚠️, not fixed, not hidden). Emil’s own steer at the end of that work:
retarget this phase’s comparison at the local model now that it has a
fair baseline to be measured from, rather than continue waiting on a
funded OpenAI account to do the comparison this record originally
described.
A technical fact, not a preference, forces one more change. Ollama
runs only on Emil’s own machine and is unreachable from this session’s
tools or any hosted CI runner — confirmed repeatedly across Phases 30-33,
where every live check had to run inside Emil’s own environment (via the
device bridge or Code directly). Decision 3a’s CI-wiring plan (a GitHub
Actions job with a schedule and a manual workflow_dispatch trigger)
cannot be built for the local path at all, not just deprioritized —
there is no network path from a hosted runner to Emil’s GPU. This is
recorded and routed around, not re-decided.
Decisions 1, 2, 4, and 5 above are unchanged and apply to the local path exactly as written. Decisions 3a and 3b are superseded, for this comparison specifically, as follows:
3a (retargeted). No CI job exists for this comparison, and none is planned. It runs on demand, on Emil’s own machine, the same way every Phase 30-33 live check already has — triggered manually by Emil or Code, not on a schedule. TFP-098’s deliverable is a repeatable script/command, not a workflow file.
3b (retargeted). The bring-your-own-key cost model is moot for the local path — every run costs $0 by construction, no key, no billing surface. Cost stays one of the four Decision-4 dimensions the report scores (for contrast against what the same comparison would show if OpenAI is ever funded), but needs no BYOK or hosted-key design work to get there.
Decision 4’s four dimensions still apply, reweighted in practice: latency is the dimension that actually differentiates local generation (29-96s per suite, Phase 33) from the deterministic generator’s near-instant output — OpenAI would have been the fast comparison point instead. Coverage delta and pass/fail agreement matter most given how much category-by-category tuning Phase 33 just did. Defect-detection is scored exactly as originally decided: stated as skipped when there’s no injected defect to test against, never guessed.
New note on reliability. TFP-099’s report states reliability’s Phase 33 weakness plainly as a caveat on that category’s results, rather than waiting for it to be fixed first. An honest, current baseline for nine categories is more useful now than a complete one later.
testforge-qwen7b against it and produces a comparable pair of runs,
reusing Phase 17’s run-diffing (Decision 2, unchanged).docs/LOCAL_MODEL_SETUP.md already set).testforge-qwen7b, reusing Phase 17’s diffing. State
reliability’s known weakness as a caveat on its own results rather than
fixing or hiding it.TEST_PLAN.md case proving
it against a real target with a real diff.A correction to the Amendment’s premise. Ollama is not unreachable from Claude Code’s session: every Phase 30-33 live check, and every run in this phase, ran from Claude Code against Emil’s Ollama on localhost. That changes nothing decided here. A hosted CI runner still can’t reach it, so 3a (retargeted) stands, but “needs Emil’s machine” means this machine, reachable by Code, not “only Emil can run it”.
TFP-098: met. npm run compare:generators -- --config <file>
(scripts/compare-generators.mjs, docs/GENERATOR_COMPARISON.md). One
command per target. Both generators get the same request and introspection
data and go through the real /api/generate. Both suites run through
/api/run, and the clean pair is diffed with /api/runs/diff (Decision 2).
Run end to end against two real targets, nothing mocked. There is no
workflow file, per 3a (retargeted).
TFP-099: met. The report is
docs/decisions/phase-21-comparison-report.md, with TEST_PLAN.md GEN-37.
All four Decision-4 dimensions are scored for every case, or explicitly
skipped with a reason:
Two defects were found and fixed along the way:
fix(report): model-written tests had all been filed under Functional
(TEST_PLAN RPT-09);TFP-100: proposed, not recorded as decided. See the next section. The
product decision is Emil’s, so it is not yet written into
product-roadmap.md.
Decision 5 says this is decided from the comparison, so the options are
set against what the comparison actually showed
(phase-21-comparison-report.md), not argued in the abstract.
What the data rules out. Making the local model the default, or keeping today’s all-or-nothing switch as the main way to use it:
TESTFORGE_OLLAMA_MODEL set means every suite comes from the local
model, which trades the template’s breadth (31 functional tests, 19 SEO
checks) for 1-3 tests per area;What the data supports. The local model filled exactly the areas where the template wrote no executable tests (performance, database, security) and caught the injected defect in each. That points to it adding to the template rather than replacing it.
| Option | Shape | For | Against |
|---|---|---|---|
| A. Keep as is | Env var switches all generation to the local model | No work | The data above: breadth lost in every area the template covers |
| B. Additive, gap-filling (recommended) | The template always generates. The local model is asked only for the selected areas where the template produced no tests, and its describe blocks are appended. The local tests keep their own label, so a run shows which tests came from where. | Keeps the template’s breadth and speed. Adds real coverage where there was none. Latency is only paid when such an area is selected. | New work: merging two suites, and deciding which areas count as gaps. Local tests are still one sample per area and carry Phase 33’s reliability caveat, so reliability stays template-only in this shape. |
| C. User’s choice per run | A UI toggle picks one generator for the whole suite | Simple mental model | Reproduces A’s trade-off per run, and the user must know which areas the template skips |
Recommendation: B.
TESTFORGE_OLLAMA_MODEL)
and API only, with no UI toggle, until B is built and measured with the
same comparison script. A UI control is worth adding once the merged
suites prove out, because a minute-plus wait needs visible progress, which
the UI doesn’t have for generation today.Not decided here, deliberately:
Once Emil confirms or changes this, the chosen option is recorded in
product-roadmap.md’s Phase 21 text, as the Gate requires.
Emil chose Option B: additive, gap-filling. Recorded as proposed above, unchanged:
TESTFORGE_OLLAMA_MODEL opt-in, API only, no
UI toggle until Option B is built and measured.Options A and C are not taken, for the reasons in the table above.
Follow-up found while deciding. Reading src/lib/generate/mock/ on
2026-09-28 answered the comparison report’s caveat 2 (whether other wording
changes “template: no tests”):
generic-suites.ts skips the
performance test whenever the request mentions performance;http://
targets;Emil decided the template should cover these three areas itself. That work,
and building Option B, is Phase 34
(phase-34-template-coverage-and-gap-filling-decision-record.md). Because
the template work changes which areas are gaps, Phase 34 sequences it
before Option B.
Gate — Phase 21 complete: met 2026-09-28. The decision is recorded
here and in product-roadmap.md’s Phase 21 text.