Date: 2026-09-28
Scope: Give the deterministic template generator real tests for the three
best-practice areas where it writes none today (performance/load, security,
database), then build Phase 21’s chosen Option B: the template always
generates, and the local model fills only the selected areas the template
still leaves empty.
Status: Complete 2026-09-29. Gates A, B and C met and independently
verified; everything through 3c4fe40 pushed. Option B confirmed by Emil
2026-09-28, with performance budgets, security-finding handling, Option C
skips, GEN-43 and Decisions 10-11 decided the same day. Next: Phase 35,
comprehensive testing (roadmap outline, not yet locked).
Phase 21’s comparison (phase-21-comparison-report.md) found the template
wrote zero executable tests for performance, database and security on the
requests tried, and its caveat 2 left open whether other wording or inputs
would change that. Reading src/lib/generate/mock/ on 2026-09-28 settles
it. The three gaps have three different causes:
generic-suites.ts generates its one
load-time test only when the description does not contain
“performance”, “load time”, “speed” or “response time”. Asking for
performance explicitly yields zero tests on every target except DemoQA,
whose own builder writes one exactly when the description names
performance. The check was a stand-in for that, present since the first
commit (75c20df); see Decision 1’s correction. The comparison’s request
(“…for performance/load coverage”) hit it exactly.http://127.0.0.1) it is
replaced by a note, so nothing checks input handling, which is what the
injected defect (security-unescaped-note) needed.areas.ts.For reference, the template’s full category set (BEST_PRACTICE_OPTIONS):
functional, usability, responsive/mobile, accessibility, seo/content and
reliability run by default; performance/load, security, cookie/session and
database run only when selected or keyword-triggered (getBestPracticeAreas).
Phase 21’s TFP-100 recorded Option B. This phase builds it. The template work comes first because it changes which areas are gaps: once the template writes performance, security and database tests for a target, the local model is no longer asked for those areas on that target.
generic-suites.ts generates the performance suite when performance/load
is selected or triggered, unless a target-specific builder already wrote
one.Content-Security-Policy, X-Content-Type-Options, frame protection via
X-Frame-Options or CSP frame-ancestors).Secure (HTTPS targets only),
HttpOnly and SameSite.HttpOnly
matters) from an analytics or preference cookie (where it is often
deliberate), so failing would turn most runs red on hardening gaps rather
than defects.BEST_PRACTICE_GUIDANCE
already describes for database, and the flow that caught
database-notes-not-saved in Phase 21.src/lib/run/interactive.ts), exactly as Phase 19’s journey tests are.getBestPracticeAreas adds
database for common words such as “save” and “update”, so without this
rule an ordinary request could start writing data to a live site.test(...) calls, as TFP-100 defined it.TESTFORGE_OLLAMA_MODEL is set.findTypeErrors,
findUnawaitedAssertions). A local suite that fails a gate is dropped;
the template suite is returned unchanged.test.skip(...)
with a stated reason, so under Decision 7 its tests report as “Not run”
with that reason.
fallbackReason).src/lib/report/discovered-coverage.ts); whether it extends that
structure or adds an area-level equivalent is decided when TFP-119 is
built.Blocked, which counts on the failed side of the pass/fail ratio
and turns a clean run into “Ready with caution”. Every run with
interactive tests off, the safe default, therefore could never read
“Ready for release”.
skip annotation, read since
915b69c) gets its own status. It is left out of the pass/fail ratio
and does not downgrade the release recommendation.Blocked and still
means “Ready with caution”: nobody chose it.generatedBy, fallbackReason) is returned by
/api/generate and then lost: it is not saved with the run or shown in
the report, and the evidence behind a rejection (which type errors, which
unawaited lines) is discarded. Nobody can later see why local output was
kept or dropped.findTypeErrors, findUnawaitedAssertions, and
the timeout), the run records: the gate’s name, its stated bar (e.g.
“0 type errors”), the measured result (e.g. “2 type errors”), the verdict
(kept or rejected), and the evidence (the error messages or source lines,
capped to a readable length).npm run compare:generators) is re-run
with template-only vs. template-plus-gap-fill.phase-34-option-b-measurement-report.md):
no UI control for gap-filling. With page data, gap-filling called the
model on neither target: the gap sweep found 0 gaps across every area on
both, and 15 of 17 comparison cases were the template’s suite byte for
byte. The only remaining gap is database without page data. There the
model’s test caught the fixture’s defect (biased, see above) and failed
falsely on the-internet (an invented login-and-reload flow), in both of
two full runs. Gap-filling stays opt-in through TESTFORGE_OLLAMA_MODEL
only. Revisit if a target’s sweep shows gaps with page data, or if Phase
35 gives the local model a depth role. Recorded by Claude Code from the
measurement; confirmed by Emil 2026-09-28.a11y-unlabelled-field defect on. Weak coverage inside a
covered area is not a gap (Decision 6), so the fix belongs in the
template, not in gap-filling, and it does not wait for Phase 35.select, textarea; not hidden, submit, button or image inputs) must
have an accessible name. A field with none fails the test, and the
message names each field (its id, name or type, and page)./api/generate receives no content profile, it runs the same
introspection the app’s own flow runs, for the base URL, before
generating, using the introspection code directly rather than an HTTP
call to /api/introspect.@playwright/test types
(the same findTypeErrors check the local path uses).http:// target, the security suite still contains the non-HTTPS
checks (input handling, headers) rather than only a note.TESTFORGE_OLLAMA_MODEL unset, generation output is identical to
template-only.a11y-unlabelled-field defect on, the template’s
accessibility suite fails and names the field; with it off, it passes. On
the-internet it produces no false failure, or the result is recorded as it
is rather than smoothed over.TEST_PLAN.md gets GEN cases (next free number after GEN-37) for: the
performance trigger fix; each new template category; the interactive-skip
rule for form-submitting template tests; gap-fill merging and labelling;
gap-fill fallback; and empty-area reporting. Each lands in the same
commit as its code, per the standing execution policy.product-roadmap.md as Phase 35; it gets
its own decision record when picked up.:visited link styling) was recorded
as a real but unclear finding (examples/testforge-app/test-assist/notes.md).
That is effectively two distinct cases, and the first was since fixed
by a rule (TFP-053/054), so it is not enough to calibrate a threshold
alone.phase-21-comparison-report.md), e.g. blocking /login’s network
route also blocking the page’s own navigation to /login (false) and
the SEO gaps (real), plus labelled failures from later runs.testforge-qwen7b on 4 GB VRAM) or
an LLM prompt is the judge is decided when this is picked up.page.route() setup and flagging
identical ones) is tried first, since duplicates under different titles
are structurally identical.Ticket numbers below are estimates — confirm the next real free number against the actual backlog before filing (TFP-113 was Phase 33’s last issued number as of this record).
bug, generate, template836d17f (GEN-38), independently verified
2026-09-28.generic-suites.ts; unit test; TEST_PLAN.md
GEN case.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Naming performance in the request produces a performance test on every target (TFP-114; unit: src/lib/generate/mock/performance-suite.test.ts)
Expected result: The shared template builder used to skip its performance test whenever the description said “performance”, “load time”, “speed” or “response time”. That check stood in for “the DemoQA builder already wrote one”, but no other target has a builder that does, so on every non-DemoQA target asking for performance by name produced zero performance tests. This is the Phase 21 comparison’s “template: no performance tests” finding. The DemoQA builder now passes the areas it covered, and the shared builder skips only those. Each trigger word, selected or keyword-triggered, now produces a performance suite on a generic target, and DemoQA still gets exactly one suite, never two. (At TFP-114 that suite was one test; TFP-115, GEN-40, made it four.) With the fix reverted, 8 of the 14 cases fail. Output measured, not diff-read: template suites for 84 fixed inputs (every area alone, the defaults, each trigger word selected and triggered, on a generic and a DemoQA target, fast and deep) were generated before and after. 66 are byte-identical, including all 42 DemoQA suites and every default-area suite. The 18 that changed are generic-target performance requests naming a trigger word, each gaining exactly one test with no line removed.
enhancement, generate, templategeneric-suites.ts and demoqa-suites.ts where they apply.TEST_PLAN.md.a2b0c57, independently verified 2026-09-28.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: The template’s performance checks measure real load behaviour and name the culprit (TFP-115; unit: src/lib/generate/mock/performance-checks.test.ts, performance-suite.test.ts; live: comparison fixture and the-internet.herokuapp.com)
Expected result: A performance suite has four read-only checks with Phase 34 Decision 2’s loose budgets. It runs whenever the area is selected or triggered, needs no interactive option, and is the same on generic and DemoQA targets. (1) Navigation under 5000 ms, measured after the load event. The old test loaded with domcontentloaded and read duration at once, which is 0 ms on a page whose load finishes later: measured 0 ms old vs. 3010 ms new on a page with a 3 s image. That test passed on any such page, and DemoQA’s own performance test had the same bug and now uses these checks. (2) Largest contentful paint under 4.0 s, the Core Web Vitals “poor” boundary, naming the largest element. No LCP at all is recorded as a finding, not a failure. (3) Page weight under 5 MB, from browser-reported request sizes (Resource Timing reports 0 bytes cross-origin), listing the 5 largest requests. (4) No same-origin request fails during load, listing each failing URL with its status; requests the page cancelled itself (ERR_ABORTED) don’t count. Live, 2026-09-28: on the fixture, clean 4/4. With performance-slow (4 s responses) 3/4: LCP failed with “LCP 4.04 s, budget 4.0 s: Core Web Vitals "poor". Largest element: h1 "Acme Notes"” (shown to 1 decimal at the time; now 2, since 4.0 vs. 4.0 read like a pass). That detection is marginal by design: 4 s sits just over the LCP budget and under the 5 s navigation budget. On the-internet.herokuapp.com, 4/4 passed, with no false failures from the loose budgets. Output measured: of 84 fixed template inputs, all 44 without performance in scope are byte-identical. All 40 with it changed, each to a 4-check suite with unique test ids. Since Phase 38 (TFP-149, GEN-71) LCP and CLS are graded in Core Web Vitals’ three bands, TBT is recorded, and every performance finding and failure says it was measured once, in a lab.
enhancement, generate, templatehttp:// targets.TEST_PLAN.md.5231b64, d496c58, independently verified 2026-09-28; follow-up 9393f12 (the payload test no longer passes when validation blocks the submit).Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: The template checks security on every target, not only HTTPS ones (TFP-116; unit: src/lib/generate/mock/security-checks.test.ts; live: comparison fixture and the-internet.herokuapp.com)
Expected result: The template’s only security check was “the URL is HTTPS”, written only for https targets; an http target got a note and nothing else. The security suite is now built as follows. (1) HTTPS, unchanged and still only on https targets. (2) Landing-page headers: Content-Security-Policy, X-Content-Type-Options: nosniff, and frame protection via X-Frame-Options or CSP frame-ancestors. (3) The flags on cookies the site itself sets: HttpOnly; Secure on https only; SameSite other than None. (4) A script-shaped payload typed into each discovered form’s visible free-text fields (text, search, textarea; never passwords; up to 3 forms), with email, url and tel fields filled with valid dummy values, then submitted. The test fails if the payload executes, detected by its own marker or any dialog, with the message “the page renders user input as HTML (cross-site scripting)”. It skips, with the reason, when the browser’s validation would block the submit (naming the rejected fields) or when submitting sends no request, so it can’t pass having tested nothing. The first version typed the payload into email/url/tel fields, where validation blocked the submit and the test passed vacuously. Proven in a real browser: a form blocked by a required checkbox skipped as “browser validation rejects: agree”; a form whose script cancels the submit skipped as “sent no request”; a form rendering input raw failed as XSS. Headers and cookie flags are findings (RPT-11) and never fail a test (Phase 34 Decision 3). Most sites have no CSP, and a missing HttpOnly on an analytics cookie is often deliberate. The payload test submits a real form, so it is skipped unless interactive tests are on (Decision 5), with the skip reason reported (RPT-10). Since Phase 38 (TFP-146, Decision 6) it sends attack-shaped input, so it sits at the active gate and is Not run on every run, with the reason “attack-shaped input: needs the ownership confirmation, not built yet”, until the ownership confirmation exists (GEN-67); the live results below predate that. Live, 2026-09-28: on the fixture, clean 4/4. With security-unescaped-note 3/4: the /notes payload test failed as XSS. The /login one then passed, but only because validation blocked its submit, which is the vacuous pass since fixed: /login has no free-text field, so it no longer gets a payload test. On the-internet.herokuapp.com (interactive off), 3/3 passed with the payload test reported skipped. Findings: no CSP; the session cookie rack.session has no Secure; analytics cookies have no HttpOnly/Secure. The last is exactly why these don’t fail tests. Output measured: 80 of 84 fixed template inputs are byte-identical; the 4 that changed are the security-only requests. The form grouping moved into mock/forms.ts, shared with the database round trip; its tests pass unchanged. Since Phase 38 (TFP-147, Decision 4) items (2) and (3), the landing-page header and session-cookie checks, are gone: the suite runs the Comprehensive passive checks on every inspected page, plus one missing-page probe (GEN-69).
Moved word for word from docs/test-plan/RPT.md by TFP-253; the case’s row there links here.
Test: Findings a test recorded without failing appear in the report (TFP-116; unit: src/lib/report/execution.test.ts - “finding annotations are kept on the record…”)
Expected result: A test can record a finding with test.info().annotations.push({ type: 'finding', description }) and still pass. The run report keeps these as the test record’s findings list, de-duplicated, and appends them to its remark (“… Findings: a; b.”), so the app’s execution table, the HTML export and the saved run all show them with no view-specific change. A test without findings keeps its remark unchanged. Live, 2026-09-28, the-internet.herokuapp.com: the security header test passed with “Findings: no Content-Security-Policy header: nothing limits what an injected script could do”; the cookie test passed listing, per cookie, the missing flags.
enhancement, generate, templatemock/ if it reads
better, like storage-failure-injection.ts); unit tests; TEST_PLAN.md.3b0345d (GEN-39), independently verified 2026-09-28.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: The template writes a database persistence round trip (TFP-117; unit: src/lib/generate/mock/database-round-trip.test.ts; live: comparison fixture)
Expected result: For database, which previously had only a manual follow-up comment, the template finds forms worth a round trip in the inspected pages. A suitable form has a visible plain text field or textarea, no password field, and isn’t a search box or an email-only signup. Up to 3 are used. For each, the test fills in a unique TestForge check <timestamp> value, saves, checks the value is shown, reloads, and checks it is still shown. Each failure message says which step broke, e.g. “after a reload the value saved through "Note" (…) is gone: it was shown once but not persisted”. It submits a real form, so it is skipped unless the run enables interactive tests, even when a keyword such as “save” triggered the area (Phase 34 Decision 5). The skip is reported with its reason (RPT-10). With no inspection data, or no suitable form, the suite carries a note saying so instead of being silently empty. The generated code passes the type check and the unawaited-assertion check. Live, 2026-09-28, on the comparison fixture: with interactive tests on, 1/1 passed on the clean site and 0/1 with database-notes-not-saved switched on, so the defect was detected. With interactive tests off, the test was reported skipped for that reason. Output measured: 80 of the 84 fixed template inputs are byte-identical. The 4 that changed are the database-only requests, which gained the explanatory note.
Moved word for word from docs/test-plan/RPT.md by TFP-253; the case’s row there links here.
Test: A test that skips itself says why (TFP-117; unit: src/lib/report/execution.test.ts - “flattenSuites: a test that skipped itself reports its own reason…”)
Expected result: When a test calls test.skip(condition, reason), the run report’s remark reads “Skipped.
enhancement, generate, localTESTFORGE_OLLAMA_MODEL), losing its breadth./api/generate route and generation modules;
tests against a fake local server, the way TFP-103/106/110 proved their
gates; TEST_PLAN.md.ef736c5, independently verified 2026-09-28. Follow-up
for Decision 6’s two GEN-43 amendments (template-reported “nothing
applicable” is not a gap; gap-fill tests skip unless interactive) landed
in 17651f2 (TEST_PLAN GEN-44), with a564543 (the test fake’s
keep-alive race behind GEN-43’s unexplained failure) and 238f406 (a run
where nothing ran gives no recommendation, RPT-14). Not yet
independently verified.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Option B: the local model fills only the areas the template left empty (TFP-118; automated: src/app/api/generate/route.local.test.ts (17 at the time), src/lib/generate/gap-fill.test.ts, src/lib/report/execution.test.ts; live: see Expected Result)
Expected result: With TESTFORGE_OLLAMA_MODEL unset, the response is exactly the template-only one. With it set, the template always generates (mockGenerateDetailed reports tests per area; its output is byte-identical to before across 264 fixed inputs). A gap is a selected or triggered area with zero template tests, reliability never included (GEN-32 open). No gap: no local call, and the response is the template with generatedBy: "template". Gaps: one local request asking for those areas only, with only their guides. Every Phase 30-33 gate applies, plus a type check of the merged file. On success the local tests are appended in test.describe('Local model (<model>)'), with an inner area block when there is one gap area, and the response has generatedBy: "template+ollama". On any failure the template suite is returned unchanged with generatedBy: "fallback", the fallbackReason, and gapFill.outcome: "rejected". Every response carries each gate’s verdict: bar, measured result, verdict, evidence (Decision 8; persisted with the run in TFP-119). The run report labels each test generator: "template" or "local model", and local tests’ remarks say so. Live, 2026-09-28, real page data, testforge-qwen7b: the-internet.herokuapp.com with database selected (its only form is a login, so database is a gap) gave template+ollama in 55 s, all 4 gates kept. The report labelled the local test local model under Database. The fixture with database selected (the notes form gives the template a round trip) gave no gap, no local call, 4 ms. Open, recorded not fixed: (1) on the page with no saving form the local model invented a login-and-reload flow with a guessed username, which failed falsely. That is the GEN-35 weakness, and it is the main case where gap-filling calls the model now that the template covers the other areas. (2) Local tests are not gated by the interactive rule (Decision 5): the local test above submitted a real login form on a live site with interactive tests off. This was already true of the Phase 30-33 replace mode. One unexplained failure of the gap-filled test in the first of 19 full-suite runs did not reproduce (12 isolated runs, 2 under CPU load); its assertion now prints a diff. Both open items and the flake are resolved by GEN-44: (1) a page inspected and found to have no saving form is no longer a gap, so the-internet case above no longer calls the model; (2) the local block is skipped unless interactive tests are on. The flake was the test’s fake server, not the route (see GEN-44).
enhancement, report/api/generate returns, with its evidence.src/lib/report/summary.ts, execution.ts); and
each gate’s bar, measured result, verdict and evidence persisted with the
run and shown in the report.TEST_PLAN.md RPT cases.de26bf1 and 13fdffb, independently verified
2026-09-28.Moved word for word from docs/test-plan/RPT.md by TFP-253; the case’s row there links here.
Test: A test skipped by choice doesn’t count against the run (TFP-119, Phase 34 Decision 7 Option C; unit: src/lib/report/summary.test.ts, execution.test.ts)
Expected result: A test that skipped itself with a stated reason (Playwright’s skip annotation) gets the status Not run and keeps its reason (skipReason). It is not executed, is left out of the pass/fail ratio, and doesn’t downgrade the release recommendation. The summary carries notRun, notRunReasons and notRunNote (“2 tests not run:
Moved word for word from docs/test-plan/RPT.md by TFP-253; the case’s row there links here.
Test: Empty areas say why, and generation gates are recorded with the run (TFP-119, Phase 34 Decisions 7 and 8; unit: src/lib/report/empty-areas.test.ts, src/lib/run/generation-record.test.ts)
Expected result: Gate verdicts with the run (Decision 8). The app keeps the generate response’s generatedBy, model, fallbackReason, gapFill and gates and sends them with the run request as generation. /api/run accepts only the known shape, capped (untrusted input), and persists it in run.json and the report. The summary tab and the HTML export show “How this suite was generated”: each gate’s bar, measured result, verdict and evidence. Empty areas (Decision 7). Every area in scope (selected, or inferred as the template does) with no test record is listed in “Areas with no tests” with its reasons. The template’s own note for that area comes first, read back from the suite that ran. Then the local-model side: timed out, output rejected (and by which gate), unreachable, not installed, asked-but-wrote-nothing, reliability never sent (GEN-32), or no local model result recorded (model variable not set). Live, 2026-09-28, the-internet.herokuapp.com, real page data, testforge-qwen7b: a database gap filled in 54 s was saved to run.json with all 4 gates (“time budget: 54327 ms -> kept”, …). With the local budget cut to 1 s, the saved run and report show the rejected time gate (“no answer within 1000 ms”), and Database is listed empty with the template’s note (no form that saves visible text) and “the local model did not answer within its time budget”. Not exercised in a browser: the three-line React hand-off (store in useGeneration, pass in page.tsx, send in useRun); it is type-checked, and the live check sent exactly what it sends.
measurement, docsdocs/decisions/; decision
on whether a UI control is justified recorded in this record.phase-34-option-b-measurement-report.md; outcome under Decision 9,
confirmed by Emil. Independently verified 2026-09-28 (see Gate B).Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Option B measured: template vs. template + gap-fill, against the fixture and a real target (TFP-120, Phase 34 Decision 9; manual, on-demand: npm run compare:generators, see docs/GENERATOR_COMPARISON.md; report: docs/decisions/phase-34-option-b-measurement-report.md)
Expected result: The comparison now pits the template alone against the template with the local model’s gap-fill, both through the real /api/generate. It saves each run with its generation record (gap-fill outcome, every gate verdict), and checks that a gap-fill suite that added nothing is byte-identical to the template’s; such a suite is not run twice. A gap sweep first requests every area alone, with and without the target’s page data, through the real route with the local model pointed at a closed port, so the route’s own gap detection decides and no model is called. Live, 2026-09-28, testforge-qwen7b, two full runs of both targets: 0 gaps with page data on either target; without page data the only gap is database. 15 of 17 cases made no local call, and the suite was the template’s byte for byte. Of the 2 gap-fill calls (database without page data, built for it): the fixture’s test caught database-notes-not-saved, a result the fixture’s bias weakens; the-internet’s failed on the healthy site, an invented login-and-reload flow that all four gates passed. Template-only detection on the fixture is 5 of 6 (Phase 21: 2 of 6). The miss is a11y-unlabelled-field, which gap-filling can’t reach because accessibility has template tests. Outcome: no UI control for gap-filling (Decision 9). The report script reads Phase 21 results unchanged apart from an “interactive tests on” label on the two cases that set it.
bug, reportbuildSummary counts passed + flaky + failed +
blocked as tests that ran. Since Option C, Blocked means interrupted or
skipped without a reason, so those did not run either. A run where every
test was interrupted still reads “Ready with caution”. The status badge
also stays green beside “No recommendation”, because Playwright treats an
all-skipped run as passed.passed + flaky + failed as run. A mix of run and
Blocked tests still reads “Ready with caution”. Neutral badge when no test
ran.summary.ts and badge change; summary tests for
all-Blocked and all-Not-run; TEST_PLAN.md RPT-14 update.2c9ecea (TEST_PLAN RPT-14). Not yet independently
verified.enhancement, generate, templatea11y-unlabelled-field undetected).aria-label, aria-labelledby,
placeholder-only and unlabelled fields; TEST_PLAN.md GEN case; output
measured before and after on fixed inputs.99f90f3 (TEST_PLAN GEN-46). Live: fixture fails with
the defect on, naming #email on /login, and passes clean; the-internet
passes with no finding. 240 of 264 fixed inputs byte-identical, the 24
changed only gaining the new accessibility tests. Not yet independently
verified.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Every form field on the inspected pages has an accessible name (TFP-121, Phase 34 Decision 10; unit: src/lib/generate/mock/form-field-names.test.ts; live: comparison fixture and the-internet.herokuapp.com)
Expected result: The template’s accessibility tests only checked the landing page’s elements, so TFP-120 found all 5 passing with the fixture’s sign-in email field unlabelled. The accessibility suite now gets one test per inspected page with form fields (up to 10). At run time it checks every visible field there: every input except hidden, submit, button, image and reset, plus select and textarea. The name is the one the browser computes, read from the field’s aria snapshot, not a hand-rolled label heuristic. A field with no name fails the test, and the message names the page and each field by id, name or type. A field named only by its placeholder is a finding (RPT-11): the test removes the placeholder in the page, reads the name again and restores it, so this is still the browser’s computation and nothing is sent to the site. Read-only. Proven in a real browser: a label, aria-label, aria-labelledby, a wrapping label and title all give a name. An unlabelled input and an unnamed select fail and are listed. Two placeholder-only fields become findings. CSS-hidden, hidden and submit inputs are skipped. Per input type (Gate C condition, 2026-09-29): in a real browser, labelled and unlabelled checkbox, radio, password, number, date, range, color and file inputs, plus checked, wrapped-label, value-bearing, aria-labelledby and multiple variants, are each named correctly. The first snapshot line shows state or a value for some types (- checkbox [checked], - slider: "50"), and neither is read as a name. The case found that Chromium names an unlabelled file input by its button text (“Choose File”), so it passed as named. A file input whose name equals that of a throwaway unlabelled file input in the same page (so any locale) now counts as unnamed. Without that fix the case fails on exactly the two unlabelled file inputs. Output: all 264 fixed inputs are identical to 99f90f3’s outside the form-field tests themselves. Live, 2026-09-28: fixture clean 7/7. With a11y-unlabelled-field on, 6/7, failing with “form fields on /login with no accessible name (a screen reader announces them without a label): #email”. the-internet 2/2, with no false failure and no findings. Template-only detection on the fixture is now 6 of 6. Output measured, not diff-read: of 264 fixed inputs, 240 are byte-identical, including every suite without page data and every area other than accessibility. The 24 that changed are the with-page-data suites with accessibility in scope (alone or defaults). Each only gains the new tests (fixture /login and /notes, the-internet /login) with no line removed, appended last so existing test ids don’t shift.
enhancement, generate, introspection/api/generate change calling the introspection code
directly; the opt-out flag; response and report fields saying page data
was gathered automatically, or why it failed; comparison scripts updated
to use the opt-out for their no-page-data cases; route tests for all three
paths; TEST_PLAN.md GEN case.cc7a810 (TEST_PLAN GEN-47). Full route responses for
264 fixed inputs (supplied page data or opt-out) byte-identical. Live: the
inspected-without-page-data, failed-inspection and opt-out paths all
behave as Acceptance Criteria 5 states. Not yet independently verified.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: A generate request with no page data is inspected automatically (TFP-122, Phase 34 Decision 11; automated: src/app/api/generate/route.inspect.test.ts (real browser, local site), src/lib/run/generation-record.test.ts, src/lib/report/empty-areas.test.ts; live: comparison fixture and the-internet.herokuapp.com)
Expected result: When /api/generate gets no contentProfile, it runs the same introspection the app’s own flow runs, by calling inspectPages directly (not over HTTP). It inspects the request’s contentRules.paths, or the base URL’s own path, normalized as /api/introspect does (the normalization moved to src/lib/introspect/paths.ts, unchanged). The response adds pageInspection (inspected with paths, pages and warnings; or failed with a one-line reason, with terminal colour codes and the call log removed) and generatedBy: "template" if no other generator set it. The run persists it (capped, untrusted input), and the app’s report and the HTML export say where the page data came from, or why there is none. Never blocks: a failed inspection returns exactly the no-page-data output plus the reason. Opt-out: autoInspect: false gives exactly today’s response; the comparison script’s no-page-data cases and the route’s own generation tests send it (so no test opens a browser on :3000). Supplied page data is used as-is. Live, 2026-09-28, testforge-qwen7b configured: the-internet database with no page data was inspected in 7 s, found not applicable, and made no model call. The fixture database request was inspected in 1.4 s, and the template’s round trip was written and passed; the saved report carries the inspection. Opted out, the same request called the model as before (48 s). An unreachable host generated in 2.3 s with “Could not inspect http://127.0.0.1:4499/: page.goto: net::ERR_CONNECTION_REFUSED …”. With inspection on, gap-filling runs only when inspection is opted out of or fails (Decision 11’s stated consequence). Output measured, not diff-read: full route responses for 264 fixed inputs (supplied page data or autoInspect: false) are byte-identical before and after. Not changed: the app, which always inspects before generating and so never reaches this path; the gathered page data is not returned in the response.
Met 2026-09-28 (TFP-114 to 117 landed and independently verified; the six default areas’ output unchanged by measurement).
Met 2026-09-28. Built and measured, and independently verified: the
GEN-43 follow-up (17651f2), RPT-14 (238f406), the route-test fix
(a564543) and TFP-120 (6618501, f799b7a) were checked against the
repository (typecheck and lint clean, 316 unit tests, template output
unchanged on fixed inputs, the report consistent with its tables). The live
Ollama runs could not be reproduced from the verifying session and are
taken from Claude Code’s run. The Decision 9 outcome (no UI control) was
confirmed by Emil the same day.
product-roadmap.md records the
outcome.Built 2026-09-28 (2c9ecea, 99f90f3, cc7a810); independently verified
2026-09-29 with one condition before pushing. Typecheck and lint clean;
325 unit tests pass in the verifying session plus the 7 that need a real
browser there, for 332; template output unchanged on fixed inputs without
page data; the code matches Decisions 10 and 11. The live runs are taken
from Claude Code’s run.
select and
textarea only. A real-browser case must also show labelled and unlabelled
checkbox, radio, password, number, date, range, color and file inputs
parsed correctly, so no real site gets a false “no accessible name” from
an input type whose snapshot line looks different.
Met 2026-09-29 (TEST_PLAN GEN-46). The parse is correct for every
listed type, including checked, wrapped-label and value-bearing variants
(- checkbox [checked], - slider: "50" read as unnamed; - slider
"Volume": "50" as named). The case found the opposite error for file:
Chromium names an unlabelled file input by its button text (“Choose
File”), so it passed as named. Fixed, not dropped: a file input whose
name equals that of a throwaway unlabelled file input in the same page
(so any locale) counts as unnamed. The new case fails on exactly the two
unlabelled file inputs without the fix.TEST_PLAN.md case and independently verified; TFP-121’s and TFP-122’s
output changes shown by measurement to be only the intended ones.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Selectors built from page values survive quotes and backslashes (Phase 34; unit: src/lib/generate/mock/locators.test.ts, database-round-trip.test.ts - fieldLocator)
Expected result: Generated code locates elements with CSS selectors built from discovered values: element and drag-target ids, drag-target classes, link hrefs, input types, form actions and form fields. Each value was escaped for the CSS string but not for the single-quoted JS string the selector is written into. So a value containing ' produced generated code that didn’t compile, and every backslash was halved. quoteSelector (format.ts) now escapes the whole selector for the JS string, in locators.ts, dependency-failure-injection.ts and forms.ts. Proven in real Chromium: ids, classes and hrefs containing ', \, ., : and " compile and select exactly the right element. With the old code, it's gives a TypeScript syntax error. Plain values give exactly the same locator text as before, and on the real fixture and the-internet profiles every suite outside the security change is byte-identical.
Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Gap-filling skips areas the template found nothing to test in, and the local block runs only with interactive tests on (GEN-43 follow-up, amended Phase 34 Decision 6; automated: src/lib/generate/gap-fill.test.ts, src/app/api/generate/route.local.test.ts (18); live: see Expected Result)
Expected result: (1) Not applicable is not a gap. When the template inspected the pages and found nothing its area applies to (database: page data exists but no form saves visible text), the suite records that as notApplicable. That area is not a gap, so no local call is made, and the response lists it in gapFill.notApplicableAreas (in the no-gaps, added and rejected shapes alike). An area with no page data is still a gap: the template could not look, so that is not a finding. Template output is byte-identical to before across the 264 fixed inputs. (2) The local block skips unless interactive tests are on. Its first statement is test.skip(process.env.TESTFORGE_INTERACTIVE !== '1', '<reason>'), with the reason “gap-fill test written by the local model (
Moved word for word from docs/test-plan/RPT.md by TFP-253; the case’s row there links here.
Test: A run where nothing ran gives no release recommendation and a neutral badge (GEN-44, Phase 34 RPT-14 follow-up; unit: src/lib/report/summary.test.ts)
Expected result: A test ran only if it Passed, was Flaky or Failed. If none did, because every test was Not run (e.g. the suite is only the local model’s gap-fill block, with interactive tests off) or Blocked (interrupted, or skipped without a reason), the recommendation is “No recommendation: no test ran”, beside the not-run note. Before, RPT-12’s rule gave “Ready for release” with zero tests executed (found in GEN-44’s live check on the comparison fixture), and the first fix still counted Blocked tests as run, so an all-interrupted run read “Ready with caution”. A mix of run and Blocked tests still reads “Ready with caution”. Badge: Playwright calls an all-skipped run passed, so the app’s report badge and the HTML export’s recommendation banner were green (and the banner orange for all-Blocked). Both are now neutral grey when no test ran, through one shared noTestRan check; the export’s banner class is covered by a unit test (rec-none). Not changed: the raw execution panel shown when there is no structured report, which has no summary to consult.