mr-onion

Phase 30 Decision Record — Local (Ollama) Test Generation Path

Date: 2026-09-24 Scope: Add an Ollama-backed local generation path alongside the existing deterministic and OpenAI paths, using a real-tested model and configuration rather than an assumed one. Status: Implemented; Gate A met 2026-09-24 (TFP-103, TFP-104, TFP-105 landed; see “Implementation status” at the end)

Context

Phase 28 hardened the OpenAI path; Phase 21 evaluates it against the deterministic generator. Both assume a funded OpenAI account. Emil’s does not have one and can’t fund it right now, so this phase adds a third path that costs nothing per request and runs entirely on his machine.

This isn’t a guess at feasibility — it was spiked for real this session, against Emil’s actual hardware (RTX 2050, 4GB VRAM) and actual downloaded models, using TestForge’s real SYSTEM_PROMPT (imported from the repo, not retyped) against a realistic-sized fake page profile:

Emil chose qwen2.5-coder:7b on this evidence, explicitly over the faster 3B, and explicitly deferred exploring lighter/alternative models until a working local path exists — that’s future work (see “Deferred” below), not this phase’s job.

Separately, two mechanical facts came out of the spike, confirmed against Ollama’s own documentation, not just observed once:

Standing policy for this phase: same as every phase since 18 — a defect found while building or verifying a ticket is fixed in that ticket, not deferred; TEST_PLAN.md updates land in the same commit as the code they describe.

Decisions

  1. The local path targets qwen2.5-coder:7b, reached through Ollama’s OpenAI-compatible endpoint.
    • Default endpoint is http://localhost:11434/v1, overridable via TESTFORGE_OLLAMA_BASE_URL. (Amended during TFP-103: this line originally deferred the name to Decision 5, which never named one.)
    • Reuses the existing openai SDK client construction TFP-083 already built (new OpenAI({ apiKey, baseURL, maxRetries: 0 })) — a different baseURL and a placeholder API key, not a new HTTP client.
  2. The model is never called by its raw pulled name — only through a custom Modelfile-defined variant that pins its context and output size.
    • num_ctx and num_predict are set in a Modelfile (e.g. FROM qwen2.5-coder:7b / PARAMETER num_ctx 16384 / PARAMETER num_predict 4096), built with ollama create, because the OpenAI-compatible endpoint cannot receive these per request.
    • The exact custom model name TestForge’s code expects is fixed and documented in the setup doc (Decision 6’s deliverable) — not left as “whatever you happened to name it when spiking.”
  3. The local path times out at 180 seconds, independent of the OpenAI path’s.
    • Chosen with margin over the spike’s measured 53-74s range: local calls cost nothing, so a premature timeout on a generation that would have succeeded is worse than the wait — but it stays bounded so nothing hangs indefinitely. A real TestForge prompt (full best-practice coverage, longer descriptions) could run longer than the spike’s single-category test, which is the other reason for the margin.
    • Env var: TESTFORGE_OLLAMA_TIMEOUT_MS, same override pattern as TESTFORGE_OPENAI_TIMEOUT_MS, defaulting to 180000ms.
    • On timeout or any other failure, the same fallback-to-template pattern TFP-083 already built applies here too — no new failure-handling code path, the existing fallback() helper and generatedBy/fallbackReason shape are reused.
  4. The route strips a leading/trailing markdown code fence from the local model’s response before using it as the generated code.
    • Confirmed necessary, not speculative: every model tested this session (llama3.1:8b, qwen2.5-coder:3b, qwen2.5-coder:7b) wrapped its response in a fence despite the SYSTEM_PROMPT explicitly saying not to.
    • This is a small, generic string operation (strip a leading ` <lang>?` and trailing ` ` if present) — not model-specific parsing.
  5. A run opts into the local path by environment-variable presence, the same mechanism the OpenAI path already uses — no new UI in this phase.
    • Priority order: TESTFORGE_OLLAMA_MODEL set → local path; else OPENAI_API_KEY set → OpenAI path; else → template (mockGenerate).
    • TESTFORGE_OLLAMA_MODEL holds the custom Modelfile-defined model name (Decision 2), not the raw pulled name — its presence is both the opt-in signal and the model selector.
    • A visible generation-mode selector in the run-config UI, if wanted later, is separate scope, not this phase’s job.
  6. Ollama being unreachable (not running, wrong port, custom model never ollama create-d) reuses TFP-083’s existing fallback() shape with a distinct fallbackReason.
    • No new code path — same generatedBy: "fallback" response shape as an OpenAI rejection.
    • Distinct reason strings so the failure class is visible in logs/UI without guessing: "ollama_unreachable" (connection refused/no response) vs "ollama_model_missing" (reachable, but the custom model name isn’t present); existing timeout/empty-completion reasons are reused as-is for those cases.
    • Amended during TFP-103, two gaps this decision didn’t cover:
      • A third reason, "ollama_error": Ollama answered, but with an error (e.g. HTTP 500 when the model fails to load). Labelling that ollama_unreachable would send someone to restart a server that is running; reusing TFP-083’s "openai_error" would name the wrong service.
      • “Model missing” matches an untagged name against its :latest entry. Ollama’s /v1/models lists testforge-qwen7b:latest, never the bare testforge-qwen7b, so TFP-083’s exact-id rule alone would have reported a model that is present as missing. That is the only rule added; there is no prefix or pattern matching.

Acceptance Criteria

  1. Local generation works end-to-end against a real Ollama instance
    • A real /api/generate call (not a mocked one) against a running Ollama with testforge-qwen7b created produces valid Playwright TypeScript, with no markdown fence in the stored code field, proven by an actual run, not by reading the diff.
    • The response records generatedBy and the model name used, matching the pattern already established for the OpenAI and fallback cases.
  2. Failure modes degrade cleanly, matching TFP-083’s standard
    • Ollama unreachable, the custom model missing, and a timeout each result in the same clean fallback shape TFP-083 already proved for the OpenAI path — no raw 500, no silent empty-code 200.
    • Regression tests force each of these three cases against a fake local server, the same way TFP-083’s tests do for the OpenAI path.
  3. Setup is documented, not assumed
    • A setup doc (matching Phase 20’s AUTOMATED_ACCOUNT_SETUP.md precedent) covers: which model to pull, the exact Modelfile content, the exact ollama create command and custom model name TestForge’s code expects, and how to point TestForge at it.
    • TEST_PLAN.md’s Feature Status table and Known Limitations both accurately describe the local path once it lands — including that model selection/tuning beyond qwen2.5-coder:7b is explicitly out of scope for this phase (see Deferred, below).

Regression Policy

Deferred (explicitly, not silently)

Backlog

Ticket numbers below are estimates — confirm the next real free number against the actual backlog before filing (TFP-102 was Phase 29’s last issued number as of this record).

TFP-103 — Build the Ollama-backed local generation path

GEN-21 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local model generates a suite (TFP-103; automated against a fake Ollama: src/app/api/generate/route.test.ts — “a local model generates, fence stripped, labelled with the catalog id”, “a configured local model wins over an OpenAI key”; unit: src/lib/generate/ollama.test.ts; live: see Expected Result)

Expected result: With TESTFORGE_OLLAMA_MODEL=testforge-qwen7b and the model created per docs/LOCAL_MODEL_SETUP.md, /api/generate returns 200, generatedBy: "ollama", model: "testforge-qwen7b:latest", and code with no markdown fence — even when OPENAI_API_KEY is also set. A suite is only returned as generatedBy: "ollama" if it type-checks against @playwright/test (GEN-26). Live against a real Ollama (RTX 2050, 4GB VRAM) through a production build (next build + next start) on 2026-09-24: a functional request (“the email and password fields accept input and the Sign in button submits the form”) returned generatedBy: "ollama" in 29s, fence-free, and the suite also passes a standalone strict tsc. An accessibility-worded request in the same session was rejected as invalid_output (see Known Limitations item 9).

GEN-25 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: Ollama returns an error or an empty answer (TFP-103; automated: “Ollama answering with an error falls back as ollama_error, not unreachable”, “a local completion that is only a fence falls back as empty_completion”)

Expected result: HTTP 500 from Ollama → fallbackReason: "ollama_error", not ollama_unreachable. A completion that is only a fence → empty_completion, never 200 with empty code.

GEN-26 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Local path: a suite that doesn’t type-check is never labelled model output (TFP-103 follow-up; automated: route “a local suite that does not type-check falls back as invalid_output”; unit: src/lib/generate/validate.test.ts)

Expected result: The local model’s suite is type-checked in-process against the installed @playwright/test (TypeScript compiler API, strict: false, only errors in the generated file count). The real toBeAccessible() output → template suite, fallbackReason: "invalid_output", model still reported, errors logged on the server. The same suite using toHaveAccessibleName passes; a syntax error and an uninstalled import are rejected; the template generator’s own suite for every coverage area passes (no false positives on known-good output).

TFP-104 — Setup documentation for the local path

TFP-105 — Regression coverage for the local path’s failure modes

  1. TFP-103 → TFP-105 (regression can land alongside 103, not strictly after) → TFP-104.

Suggested Milestone Gate

Gate A — Local generation path works and degrades safely

Implementation status (2026-09-24)

TFP-103, TFP-104 and TFP-105 landed together in 7cfc2ce.

Verified live, against a real Ollama with testforge-qwen7b (RTX 2050, 4GB VRAM, two /api/generate calls through the real route): 200, generatedBy: "ollama", model: "testforge-qwen7b:latest", no markdown fence in code, in 91s and 78s.

Verified against a fake server only (route.test.ts, the same method TFP-083 used): unreachable, model missing, Ollama error, timeout, empty completion, and the local model winning over an OpenAI key. None of these failure modes was forced against the real instance.

Gate A was not met at this point (resolved below). Acceptance Criterion 1 requires valid Playwright TypeScript. Both live suites called expect(locator).toBeAccessible(), which Playwright does not have; tsc against @playwright/test rejects both, three errors each, and nothing else in either suite. That contradicts the spike’s “no invented APIs” result for this model, on the same kind of prompt (“…and the page is accessible”).

Resolving it needs a decision this record doesn’t contain, so it was not guessed at in code. The options, roughly in order of cost:

  1. Prompt: name the accessibility API to use in SYSTEM_PROMPT (e.g. toHaveAccessibleName/toHaveRole on locators). Cheapest, but it changes the OpenAI path’s prompt too, and it is the “tuning” this phase deferred.
  2. Validate, then fall back: type-check or matcher-check the generated code and return the template with a new fallbackReason when it fails. Honest, but a user of the local path would then mostly get the template.
  3. Accept and document: keep the current warning in docs/LOCAL_MODEL_SETUP.md and TEST_PLAN.md (GEN-21 ❌, Known Limitations item 9) and close Gate A with the criterion amended.

Resolution: option 2, validate then fall back (decided 2026-09-24)

Emil chose option 2. It serves Acceptance Criterion 1 as written, leaves the OpenAI path’s prompt alone, and keeps the rule the project has followed since Phase 24: return a working suite or a clearly labelled fallback, never a broken suite labelled as success.

How it validates (src/lib/generate/validate.ts): the TypeScript compiler API, in-process, against the installed @playwright/test type definitions, with module resolution from the project root. No temp file and no child process. Settings are deliberately lenient (strict: false, skipLibCheck), and only error diagnostics in the generated file count, so the check asks “does this call real APIs and load”, not “is it tidy”. Library typings are parsed once and cached. Measured: about 0.8s for the first check in a process, about 0.1s after that, against a 29-91s generation. If the check itself cannot run, it fails closed. typescript moved from devDependencies to dependencies, because the route now needs it at runtime.

New fallbackReason: "invalid_output", with model still reported and the first five errors logged on the server. It applies to the local path only; the OpenAI path is unchanged.

Guarding against false positives: the template generator’s own suite for every coverage area passes the check (validate.test.ts), as does the invented-matcher suite once rewritten with a real matcher (toHaveAccessibleName).

Verified live, through a production build (next build + next start, real Ollama, testforge-qwen7b). That is where bundling a compiler import into a route would break, and it didn’t:

Option 2’s predicted cost (“would mostly get the template”) turned out narrower than feared: so far it is accessibility requests that fall back, while functional ones pass. That rests on three accessibility samples and one functional one, so it is not a measured rate.

Gate A is met:

Getting accessibility coverage out of the local model is the deferred model/prompt tuning follow-up, not a gap in this gate.