mr-onion

Phase 28 — Harden the OpenAI-Backed Generation Path

Version: 1.0 Date: 2026-09-23 Status: TFP-083 landed (8308b0b); Gate A open — the live catalog check passed, but no live completion has succeeded yet (see “Live-check result” below)

Context

TEST_PLAN.md’s Known Limitations item 5 and docs/decisions/product-roadmap.md’s Phase 21 both describe the same untested surface: when OPENAI_API_KEY is set, POST /api/generate (src/app/api/generate/route.ts) calls gpt-4o instead of mockGenerate, and that path has never run in CI — only the template-based generator has real automated coverage.

Phase 21’s own scope, as written, is bigger than a reliability fix: “given CI coverage, and evaluated against the deterministic generator on the same sites using Phase 17’s run persistence and diffing” — i.e. bring the path into active development, build the comparison tooling, and make a product decision (default / opt-in / user’s choice) from the result. Checked directly against the actual route on 2026-09-23: the code-level problem sitting underneath that untested status is smaller and more urgent than the full evaluation Phase 21 describes.

What’s actually there right now (src/app/api/generate/route.ts:49-66): the req.json() parse two lines above it is already guarded (.catch(() => ({})), the same TFP-067 pattern applied to the other five POST handlers in Phase 24) — but the client.chat.completions.create(...) call immediately below has no try/catch, no timeout, and no fallback to mockGenerate. A thrown error (bad key, rate limit, OpenAI outage, network failure) is an unhandled rejection in the route handler, which Next.js turns into a raw 500 — the exact failure shape TFP-067 fixed for malformed request bodies, just one call deeper in the same file. There is also no timeout, so a hung request blocks the response indefinitely. And if the response comes back malformed (chat.choices[0]?.message?.content missing), the route currently falls through to ?? "" and returns 200 with an empty code string — silently “successful” with nothing usable in it, worse than a clean error would be.

Emil’s steer (2026-09-23): concentrate on a stable working model for TestForge rather than opening new scope. This phase is scoped to match that — it closes the reliability gap in the path that already exists and already ships (anyone with an OPENAI_API_KEY set hits this code today), the same class of fix as Phase 24’s TFP-065/066/067. It deliberately does not attempt Phase 21’s CI-coverage-and-comparison work, which stays exactly where product-roadmap.md already has it: a separate, larger, not-yet-started phase, unblocked by this one but not advanced by it either.

Added 2026-09-23, from a narrow sre-code trial (skills/proposed/): run read-only against this same file per the skill’s own “language is code” section, on Emil’s steer to try a cheap trial rather than promote the set wholesale. One more real, concrete finding, same shape as the three above: route.ts:52 calls model: "gpt-4o" — a floating alias, not a pinned snapshot (e.g. gpt-4o-2024-08-06). OpenAI can move what gpt-4o resolves to with no diff anywhere in this repo, which is exactly the “build is not hermetic” failure sre-code names — generated-suite quality could drift silently, with nothing to bisect against. Folded into this ticket rather than filed separately since it’s the same file, same call, same fix motion.

Added 2026-09-24, from a Cowork research pass (not a repo read — this needed live/current information, not code): a second, more urgent finding on the exact same call. Checking OpenAI’s own current pricing and deprecation pages, gpt-4o no longer appears in the current pricing table at all, and multiple sources (OpenAI’s own retirement announcements, third-party coverage) describe GPT-4o API access being wound down through

  1. The exact retirement date and the specific recommended replacement model string were inconsistent across every source checked — this space is moving too fast for a web search alone to be the source of truth — so this finding does not hand down a replacement model string to hardcode. What it does establish: pinning gpt-4o to a dated snapshot (as originally scoped below) may be pinning something that no longer resolves at all. The first concrete step of TFP-083, before any pinning decision, is a live check against the real OpenAI API (a models-list call, or one minimal real request with a real key) to find out what gpt-4o currently does and what the actual available model catalog looks like today — not another round of public-doc research.

Scope (this phase)

In scope

  1. Live-check the real OpenAI model catalog, then define a short ordered candidate list from it — a models-list call (or one minimal real request) with a real key, to see what’s actually callable today. Don’t assume any specific model name going in — 2026-09-24 research found today’s names unreliable to guess from public docs alone (see the note above). From the real live output, define 2-3 candidates in an explicit, version-controlled config, ordered cheapest-adequate tier first with a known-working fallback second — not a single hardcoded string, and not name-pattern matching (e.g. “anything with ‘mini’ in it”), which breaks silently on the next OpenAI rename, the exact failure this whole finding is about. Check the live catalog against the candidate list and pick the first one actually present; record which one was picked. Added 2026-09-24 (Cowork, at Emil’s request) — Phase 21’s TFP-099 later scores every candidate in this list against real comparison data and can reorder/prune it; this phase only needs the mechanism and a reasonable starting list.
  2. Guard the OpenAI call in src/app/api/generate/route.ts against failure, timeout, and malformed responses, without changing behavior for the already-working no-key (mockGenerate) path.
  3. A real regression test that forces each failure mode and asserts the route degrades cleanly.
  4. Correct TEST_PLAN.md’s Known Limitations item 5 and the Feature Status table’s “Generate” row to describe what this phase actually closes — not the full Phase 21 evaluation.

Out of scope

Backlog

TFP-083 - Guard, time out, and fall back the OpenAI generation call

Single ticket.

Suggested Milestone Gates

Gate A - The existing OpenAI path fails safely

Live-check result (2026-09-24)

npm run check:openai-models, run with a real key from .env.local:

Gate A stays open until the check is re-run on a funded account and the completion succeeds. That’s a re-run, not new work: if it passes, the list needs no change; if gpt-4.1-mini rejects the request, reorder models.ts from the output. TFP-098 should reuse that same re-run, not duplicate it.