mr-onion

Phase 34 Measurement Report — Option B as Built

Date: 2026-09-28 Ticket: TFP-120 (Phase 34, Decision 9) Compared: the template generator alone vs. the template plus the local model’s gap-fill (Option B, TFP-118 with its GEN-43 follow-up), using testforge-qwen7b (qwen2.5-coder:7b via Ollama) on Emil’s machine. How to reproduce: docs/GENERATOR_COMPARISON.md. Previous comparison (replace mode, before Phase 34): phase-21-comparison-report.md.

Bottom line

With page data, gap-filling never calls the model on either target. The Phase 34 templates cover every area on both sites, or report the area not applicable. As the record expected after the GEN-43 amendments, the template closed the gaps Option B was built to fill.

Method

Caveats (not smoothed over)

What the template now catches, and what gap-filling can’t reach

Defect detection on the fixture with page data, template only: 5 of 6 (Phase 21: 2 of 6): SEO, performance, database, security and reliability. The miss is a11y-unlabelled-field: the template’s 5 accessibility tests pass with the unlabelled field on (Phase 21’s local suite caught it). Option B can’t be called for it: accessibility has template tests, so it is not a gap, by the definition Decision 6 fixes. Weak coverage inside a covered area is exactly the “depth” role the record says needs a new decision (roadmap Phase 35, comprehensive testing). That’s where this miss belongs, or in a template fix, not in widening what counts as a gap.

Decision 9 outcome: no UI control for gap-filling

A UI control isn’t justified by this measurement:

  1. In the normal flow it does nothing. With page data there were no gaps on either target, so a toggle would change no suite.
  2. Where it acts, the evidence is 1 right on a biased fixture and 1 false failure on the real target. A control that invites users to switch on that trade isn’t worth the UI surface.
  3. The no-page-data gap has a cheaper answer: inspect the pages before generating. With page data, the-internet’s database request became “not applicable” with the reason reported (GEN-44, RPT-13), and the fixture’s became a template round trip, generated in about 1 ms, that caught the defect. Whether generation should inspect automatically when no profile is supplied is a separate product question, not decided here.

Gap-filling stays as built: opt-in via TESTFORGE_OLLAMA_MODEL, env var only. Revisit if a target’s sweep shows gaps with page data, or if Phase 35 gives the local model a depth role in covered areas.

Tooling changes in TFP-120


Results by case (generated)

Output of node scripts/comparison-report.mjs comparisons/2026-09-28T20-16-15-419Z-fixture comparisons/2026-09-28T20-19-37-582Z-the-internet, unedited. “Template only” runs with TESTFORGE_OLLAMA_MODEL cleared; “Template + gap-fill” with it set. When gap-fill added nothing, the “Runs” row gives the template run’s results, which are the case’s only results.

fixture: http://127.0.0.1:4410

Local model testforge-qwen7b, TestForge http://127.0.0.1:3100, finished 2026-09-28T20:19:36.335Z. Source: comparisons/2026-09-28T20-16-15-419Z-fixture. Distinct discovered elements with a usable name, by type and name across all pages (for the reference proxy): 11.

Gap sweep: each area requested alone, through the real route with the local model unreachable, so only the route’s gap detection decides (no model called).

Area With page data Without page data
functional covered covered
responsive/mobile covered covered
performance/load covered covered
security covered covered
accessibility covered covered
usability covered covered
seo/content covered covered
cookie/session covered covered
database covered gap: database
reliability never a gap: not sent to the local model (GEN-32) never a gap: not sent to the local model (GEN-32)

functional

Request: “Check the sign-in page: the email and password fields accept input and the Sign in button submits the form.” - areas: functional.

  Template only Template + gap-fill
Generation 2 ms 1 ms (template, gaps: none)
Runs 8.1 s, 31/31 passed not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

accessibility

Request: “Check the sign-in page (email and password fields, Sign in button) for accessibility coverage.” - areas: accessibility - injected defect: a11y-unlabelled-field.

  Template only Template + gap-fill
Generation 2 ms 1 ms (template, gaps: none)
Runs 4.0 s, 5/5 passed; defect a11y-unlabelled-field: not detected not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

seo

Request: “Check the Acme Notes home page for SEO/content coverage.” - areas: seo/content - injected defect: seo-missing-meta.

  Template only Template + gap-fill
Generation 4 ms 1 ms (template, gaps: none)
Runs 34.3 s, 10/19 passed; defect seo-missing-meta: detected by 1 test(s): “TFSE-002: keeps meta description stable for /” not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

performance

Request: “Check the Acme Notes home page for performance/load coverage.” - areas: performance/load - injected defect: performance-slow.

  Template only Template + gap-fill
Generation 2 ms 1 ms (template, gaps: none)
Runs 6.0 s, 4/4 passed; defect performance-slow: detected by 1 test(s): “TFPL-002: largest contentful paint stays under 4 s” not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

database

Request: “Check the notes page form (Note field, Save note button) for database coverage.” - areas: database - injected defect: database-notes-not-saved - interactive tests on.

  Template only Template + gap-fill
Generation 1 ms 1 ms (template, gaps: none)
Runs 2.0 s, 1/1 passed; defect database-notes-not-saved: detected by 1 test(s): “TFDB-001: persistence round trip: Note on /notes” not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

database-not-interactive

Request: “Check the notes page form (Note field, Save note button) for database coverage.” - areas: database.

  Template only Template + gap-fill
Generation 1 ms 1 ms (template, gaps: none)
Runs 2.0 s, 0/0 passed, 1 not run not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

security

Request: “Check the notes page form (Note field, Save note button) for security coverage.” - areas: security - injected defect: security-unescaped-note - interactive tests on.

  Template only Template + gap-fill
Generation 1 ms 1 ms (template, gaps: none)
Runs 4.0 s, 3/3 passed; defect security-unescaped-note: detected by 1 test(s): “TFSC-003: script-shaped input is not executed: Note on /notes” not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

reliability

Request: “Check the Acme Notes home page for reliability coverage.” - areas: reliability - injected defect: reliability-api-crash.

  Template only Template + gap-fill
Generation 2 ms 1 ms (template, gaps: none)
Runs 4.0 s, 1/2 passed; defect reliability-api-crash: detected by 1 test(s): “TFRL-001: dependency failure injection: 127.0.0.1:4410 [xhr-fetch]” not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

database-no-page-data

Request: “Check the notes page form (Note field, Save note button) for database coverage.” - areas: database - injected defect: database-notes-not-saved - generated without page data - interactive tests on.

  Template only Template + gap-fill
Generation 1 ms 25.4 s (template+ollama, gaps: database)
Gates none (no model) time budget: 24384 ms -> kept; type check: 0 type errors -> kept; unawaited assertions: 0 never-awaited web-first assertions -> kept; merged suite type check: 0 type errors -> kept
Clean run no report (run failed); the suite has 0 test(s) 4.0 s, 1/1 passed
Cost $0 (no model) $0 (local, no API)
Coverage as the run report computes it (functional, accessibility, reliability: discovered elements, credited only to tests titled the template generator’s way - so model-written tests score ~0% there by construction; other areas: pass rate) n/a Database 100%
Discovered elements referenced in the suite (title-independent proxy) 0/11 2/11
Tests written by the local model (clean run) 0 1 (1 Passed)
Pass/fail by area (clean) Database - Database 1/1
Pass/fail agreement Database: only one generator tested it  
Defect detection (database-notes-not-saved) nothing to detect with: the suite has no tests detected by 1 test(s): “a new note is still shown after reloading the page”

Phase 17 diff: 0 test(s) pass only in the template only run and 1 only in the template + gap-fill run; the suites share 0 test title(s), so this lists each passing suite whole rather than any disagreement.

the-internet: https://the-internet.herokuapp.com

Local model testforge-qwen7b, TestForge http://127.0.0.1:3100, finished 2026-09-28T20:25:50.355Z. Source: comparisons/2026-09-28T20-19-37-582Z-the-internet. Distinct discovered elements with a usable name, by type and name across all pages (for the reference proxy): 5.

Gap sweep: each area requested alone, through the real route with the local model unreachable, so only the route’s gap detection decides (no model called).

Area With page data Without page data
functional covered covered
responsive/mobile covered covered
performance/load covered covered
security covered covered
accessibility covered covered
usability covered covered
seo/content covered covered
cookie/session covered covered
database not applicable: database gap: database
reliability never a gap: not sent to the local model (GEN-32) never a gap: not sent to the local model (GEN-32)

functional

Request: “Check the login page: the username and password fields accept input and the Login button submits the form.” - areas: functional.

  Template only Template + gap-fill
Generation 1 ms 2 ms (template, gaps: none)
Runs 58.5 s, 10/13 passed not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

accessibility

Request: “Check the login page (username and password fields, Login button) for accessibility coverage.” - areas: accessibility.

  Template only Template + gap-fill
Generation 2 ms 2 ms (template, gaps: none)
Runs 6.1 s, 1/1 passed not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

seo

Request: “Check the login page for SEO/content coverage.” - areas: seo/content.

  Template only Template + gap-fill
Generation 1 ms 1 ms (template, gaps: none)
Runs 46.4 s, 4/9 passed not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

security

Request: “Check the login page (username and password fields, Login button) for security coverage.” - areas: security.

  Template only Template + gap-fill
Generation 2 ms 2 ms (template, gaps: none)
Runs 12.1 s, 3/3 passed, 1 not run not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

free-form

Request: “Check the login page: logging in with the wrong password shows an error, and the page is accessible.” - areas: inferred.

  Template only Template + gap-fill
Generation 4 ms 7 ms (template, gaps: none)
Runs 165.2 s, 23/32 passed not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

performance

Request: “Check the login page for performance/load coverage.” - areas: performance/load.

  Template only Template + gap-fill
Generation 2 ms 1 ms (template, gaps: none)
Runs 18.2 s, 4/4 passed not run: no gaps: the template covered every area in scope, so the model was not called; the suite is the template’s, byte for byte

database

Request: “Check that data entered on the login page is saved.” - areas: database.

  Template only Template + gap-fill
Generation 3 ms 2 ms (template, gaps: none, not applicable: database)
Runs no report (run failed); the suite has 0 test(s) not run: no gaps: the template covered every area in scope or found it not applicable (database), so the model was not called; the suite is the template’s, byte for byte

database-no-page-data

Request: “Check that data entered on the login page is saved.” - areas: database - generated without page data - interactive tests on.

  Template only Template + gap-fill
Generation 3 ms 46.1 s (template+ollama, gaps: database)
Gates none (no model) time budget: 44952 ms -> kept; type check: 0 type errors -> kept; unawaited assertions: 0 never-awaited web-first assertions -> kept; merged suite type check: 0 type errors -> kept
Clean run no report (run failed); the suite has 0 test(s) 12.1 s, 0/1 passed
Cost $0 (no model) $0 (local, no API)
Coverage as the run report computes it (functional, accessibility, reliability: discovered elements, credited only to tests titled the template generator’s way - so model-written tests score ~0% there by construction; other areas: pass rate) n/a Database 0%
Discovered elements referenced in the suite (title-independent proxy) 0/5 3/5
Tests written by the local model (clean run) 0 1 (1 Failed)
Pass/fail by area (clean) Database - Database 0/1
Pass/fail agreement Database: only one generator tested it  
Defect detection skipped: no injected defect for this case (not the comparison fixture) same

Template + gap-fill clean-run failures: “data entered on the login page is saved” (Error: expect(locator).toHaveValue(expected) failed Locator: getByLabel(‘Username’) Expected: “TestForge check 1790627139917” Received: “” T)

Phase 17 diff: 0 test(s) pass only in the template only run and 0 only in the template + gap-fill run; the suites share 0 test title(s), so this lists each passing suite whole rather than any disagreement.