Date: 2026-09-24 Scope: Add an Ollama-backed local generation path alongside the existing deterministic and OpenAI paths, using a real-tested model and configuration rather than an assumed one. Status: Implemented; Gate A met 2026-09-24 (TFP-103, TFP-104, TFP-105 landed; see “Implementation status” at the end)
Phase 28 hardened the OpenAI path; Phase 21 evaluates it against the deterministic generator. Both assume a funded OpenAI account. Emil’s does not have one and can’t fund it right now, so this phase adds a third path that costs nothing per request and runs entirely on his machine.
This isn’t a guess at feasibility — it was spiked for real this session,
against Emil’s actual hardware (RTX 2050, 4GB VRAM) and actual downloaded
models, using TestForge’s real SYSTEM_PROMPT (imported from the repo, not
retyped) against a realistic-sized fake page profile:
llama3.1:8b (general model, already on the machine): 54.5s, valid
Playwright code, wrapped in a markdown fence despite the prompt saying not to.qwen2.5-coder:3b (1.9GB, fits fully in 4GB VRAM): 53.5s. Output used
expect(page).toHaveAccessibleRole(...), which does not appear to be a
real Playwright assertion (the real one is toHaveRole, and it’s a
locator assertion, not a page-level one) — and produced four tests, two
of them byte-for-byte duplicates, all four sharing the name “Login page
is accessible.”qwen2.5-coder:7b (4.7GB, same VRAM-ceiling class as llama3.1:8b): 73.9s.
No invented APIs, no duplicate names, correctly split functional and
accessibility into separate test.describe blocks per the
SYSTEM_PROMPT’s own instruction. Also fence-wrapped.Emil chose qwen2.5-coder:7b on this evidence, explicitly over the faster
3B, and explicitly deferred exploring lighter/alternative models until a
working local path exists — that’s future work (see “Deferred” below), not
this phase’s job.
Separately, two mechanical facts came out of the spike, confirmed against Ollama’s own documentation, not just observed once:
/v1/chat/completions) silently
discards num_ctx from the request body — the endpoint’s request
struct has no such field, so it’s dropped during JSON decoding and an
automatic, VRAM-based context size is used instead. The only way to
guarantee TestForge’s real prompt (which can carry a large page-inspection
JSON profile) isn’t silently truncated is to bake the context size into
a custom Modelfile-defined model, not pass it per request.Standing policy for this phase: same as every phase since 18 — a defect
found while building or verifying a ticket is fixed in that ticket, not
deferred; TEST_PLAN.md updates land in the same commit as the code they
describe.
qwen2.5-coder:7b, reached through Ollama’s
OpenAI-compatible endpoint.
http://localhost:11434/v1, overridable via
TESTFORGE_OLLAMA_BASE_URL. (Amended during TFP-103: this line
originally deferred the name to Decision 5, which never named one.)openai SDK client construction TFP-083 already
built (new OpenAI({ apiKey, baseURL, maxRetries: 0 })) — a different
baseURL and a placeholder API key, not a new HTTP client.num_ctx and num_predict are set in a Modelfile (e.g.
FROM qwen2.5-coder:7b / PARAMETER num_ctx 16384 /
PARAMETER num_predict 4096), built with ollama create, because the
OpenAI-compatible endpoint cannot receive these per request.TESTFORGE_OLLAMA_TIMEOUT_MS, same override pattern as
TESTFORGE_OPENAI_TIMEOUT_MS, defaulting to 180000ms.fallback() helper and generatedBy/fallbackReason
shape are reused.SYSTEM_PROMPT explicitly saying not to.<lang>?`
and trailing ` ` if present) — not model-specific parsing.TESTFORGE_OLLAMA_MODEL set → local path; else
OPENAI_API_KEY set → OpenAI path; else → template (mockGenerate).TESTFORGE_OLLAMA_MODEL holds the custom Modelfile-defined model name
(Decision 2), not the raw pulled name — its presence is both the opt-in
signal and the model selector.ollama create-d) reuses TFP-083’s existing fallback() shape with a
distinct fallbackReason.
generatedBy: "fallback" response shape as an
OpenAI rejection."ollama_unreachable" (connection refused/no
response) vs "ollama_model_missing" (reachable, but the custom model
name isn’t present); existing timeout/empty-completion reasons are
reused as-is for those cases."ollama_error": Ollama answered, but with an error
(e.g. HTTP 500 when the model fails to load). Labelling that
ollama_unreachable would send someone to restart a server that is
running; reusing TFP-083’s "openai_error" would name the wrong
service.:latest entry.
Ollama’s /v1/models lists testforge-qwen7b:latest, never the bare
testforge-qwen7b, so TFP-083’s exact-id rule alone would have
reported a model that is present as missing. That is the only rule
added; there is no prefix or pattern matching./api/generate call (not a mocked one) against a running Ollama
with testforge-qwen7b created produces valid Playwright TypeScript,
with no markdown fence in the stored code field, proven by an actual
run, not by reading the diff.generatedBy and the model name used, matching the
pattern already established for the OpenAI and fallback cases.code 200.AUTOMATED_ACCOUNT_SETUP.md precedent)
covers: which model to pull, the exact Modelfile content, the exact
ollama create command and custom model name TestForge’s code expects,
and how to point TestForge at it.TEST_PLAN.md’s Feature Status table and Known Limitations both
accurately describe the local path once it lands — including that model
selection/tuning beyond qwen2.5-coder:7b is explicitly out of scope for
this phase (see Deferred, below).TEST_PLAN.md gets real regression cases for the local-path success case
and its three failure modes (unreachable, missing model, timeout), added
in the same commit as the code they describe — same standing policy every
phase since 18 has followed.Ticket numbers below are estimates — confirm the next real free number against the actual backlog before filing (TFP-102 was Phase 29’s last issued number as of this record).
feature, generate, local/api/generate at Ollama’s
OpenAI-compatible endpoint using the existing openai SDK client
pattern; strip a markdown fence from the response before treating it as
code; apply the local-specific timeout from Decision 3; fall back per
Decision 6 on any failure; opt-in mechanism per Decision 5.src/app/api/generate/route.ts; the
Modelfile(s) and setup doc from Decision 2/Acceptance Criterion 3;
regression tests for the success case and the three failure modes;
TEST_PLAN.md updates.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local model generates a suite (TFP-103; automated against a fake Ollama: src/app/api/generate/route.test.ts — “a local model generates, fence stripped, labelled with the catalog id”, “a configured local model wins over an OpenAI key”; unit: src/lib/generate/ollama.test.ts; live: see Expected Result)
Expected result: With TESTFORGE_OLLAMA_MODEL=testforge-qwen7b and the model created per docs/LOCAL_MODEL_SETUP.md, /api/generate returns 200, generatedBy: "ollama", model: "testforge-qwen7b:latest", and code with no markdown fence — even when OPENAI_API_KEY is also set. A suite is only returned as generatedBy: "ollama" if it type-checks against @playwright/test (GEN-26). Live against a real Ollama (RTX 2050, 4GB VRAM) through a production build (next build + next start) on 2026-09-24: a functional request (“the email and password fields accept input and the Sign in button submits the form”) returned generatedBy: "ollama" in 29s, fence-free, and the suite also passes a standalone strict tsc. An accessibility-worded request in the same session was rejected as invalid_output (see Known Limitations item 9).
Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: Ollama returns an error or an empty answer (TFP-103; automated: “Ollama answering with an error falls back as ollama_error, not unreachable”, “a local completion that is only a fence falls back as empty_completion”)
Expected result: HTTP 500 from Ollama → fallbackReason: "ollama_error", not ollama_unreachable. A completion that is only a fence → empty_completion, never 200 with empty code.
Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Local path: a suite that doesn’t type-check is never labelled model output (TFP-103 follow-up; automated: route “a local suite that does not type-check falls back as invalid_output”; unit: src/lib/generate/validate.test.ts)
Expected result: The local model’s suite is type-checked in-process against the installed @playwright/test (TypeScript compiler API, strict: false, only errors in the generated file count). The real toBeAccessible() output → template suite, fallbackReason: "invalid_output", model still reported, errors logged on the server. The same suite using toHaveAccessibleName passes; a syntax error and an uninstalled import are rejected; the template generator’s own suite for every coverage area passes (no false positives on known-good output).
docsollama pull/ollama create
sequence and exact model name, this only works on the machine that
spiked it.AUTOMATED_ACCOUNT_SETUP.md
precedent — exact commands, exact Modelfile content, how to verify it’s
working.docs/.test, generateroute.test.ts file rather than a new one.src/app/api/generate/route.test.ts.TEST_PLAN.md
accurately separates “a working local path exists” (this phase) from
“compared against the other two paths” (still Phase 21) and “tuned for
speed/accuracy beyond the 7B baseline” (explicitly deferred, above).TFP-103, TFP-104 and TFP-105 landed together in 7cfc2ce.
Verified live, against a real Ollama with testforge-qwen7b (RTX 2050,
4GB VRAM, two /api/generate calls through the real route): 200,
generatedBy: "ollama", model: "testforge-qwen7b:latest", no markdown
fence in code, in 91s and 78s.
Verified against a fake server only (route.test.ts, the same method
TFP-083 used): unreachable, model missing, Ollama error, timeout, empty
completion, and the local model winning over an OpenAI key. None of these
failure modes was forced against the real instance.
Gate A was not met at this point (resolved below). Acceptance Criterion 1 requires valid Playwright
TypeScript. Both live suites called expect(locator).toBeAccessible(),
which Playwright does not have; tsc against @playwright/test rejects
both, three errors each, and nothing else in either suite. That
contradicts the spike’s “no invented APIs” result for this model, on the
same kind of prompt (“…and the page is accessible”).
Resolving it needs a decision this record doesn’t contain, so it was not guessed at in code. The options, roughly in order of cost:
SYSTEM_PROMPT
(e.g. toHaveAccessibleName/toHaveRole on locators). Cheapest, but it
changes the OpenAI path’s prompt too, and it is the “tuning” this phase
deferred.fallbackReason when it fails.
Honest, but a user of the local path would then mostly get the
template.docs/LOCAL_MODEL_SETUP.md and TEST_PLAN.md (GEN-21 ❌, Known
Limitations item 9) and close Gate A with the criterion amended.Emil chose option 2. It serves Acceptance Criterion 1 as written, leaves the OpenAI path’s prompt alone, and keeps the rule the project has followed since Phase 24: return a working suite or a clearly labelled fallback, never a broken suite labelled as success.
How it validates (src/lib/generate/validate.ts): the TypeScript
compiler API, in-process, against the installed @playwright/test type
definitions, with module resolution from the project root. No temp file
and no child process. Settings are deliberately lenient (strict: false,
skipLibCheck), and only error diagnostics in the generated file count,
so the check asks “does this call real APIs and load”, not “is it tidy”.
Library typings are parsed once and cached. Measured: about 0.8s for the
first check in a process, about 0.1s after that, against a 29-91s
generation. If the check itself cannot run, it fails closed.
typescript moved from devDependencies to dependencies, because the
route now needs it at runtime.
New fallbackReason: "invalid_output", with model still reported
and the first five errors logged on the server. It applies to the local
path only; the OpenAI path is unchanged.
Guarding against false positives: the template generator’s own suite
for every coverage area passes the check (validate.test.ts), as does the
invented-matcher suite once rewritten with a real matcher
(toHaveAccessibleName).
Verified live, through a production build (next build +
next start, real Ollama, testforge-qwen7b). That is where bundling a
compiler import into a route would break, and it didn’t:
toBeAccessible() again (a
third time). It was caught (the same three TS2339 errors standalone
tsc reports) and returned as fallbackReason: "invalid_output" in
74s.generatedBy: "ollama"
in 29s, fence-free. That suite also passes a standalone strict tsc.Option 2’s predicted cost (“would mostly get the template”) turned out narrower than feared: so far it is accessibility requests that fall back, while functional ones pass. That rests on three accessibility samples and one functional one, so it is not a measured rate.
Gate A is met:
invalid_output included, is proven
against a fake server, and invalid_output also live.TEST_PLAN.md separates “works”
(GEN-21 to GEN-26) from “accessibility requests currently fall back”
(Known Limitations item 9) and from the deferred tuning and comparison.Getting accessibility coverage out of the local model is the deferred model/prompt tuning follow-up, not a gap in this gate.