Date: 2026-10-12
Scope: Keep each site’s Standard tests and reuse them on later runs; keep
Comprehensive tests in a library of tests written against component
patterns, applied to any site that has the pattern; map which element and
flow every test covers; and write new tests only for what the map shows is
uncovered, so analysis time goes to the outliers instead of the standard
things.
Status: Complete 2026-10-15 (locked 2026-10-12; every decision confirmed by Emil the same day).
Starts after Phase 41’s Gate B. Tickets TFP-190 to TFP-197, plus TFP-200 and TFP-201 from Gate A.
Complete 2026-10-15 (TFP-205 verified; TFP-206 moves to the post-Phase 42 review).
Corrections from the review’s claims audit (R3, 2026-10-16) are marked
Correction (R3, 2026-10-16) where they apply. The history above them is
left as written. Findings: post-phase-42-review-findings.md.
save-suite saves a suite under the run’s
description, not the site, so nothing is reused.run-evidence-sweep-backlog-2026-10-13.md): site suites
retire tests, which today’s sweep would read as defects fixed.Ticket prefix TFP-. Built in batches, ticket by ticket with targeted tests;
the full unit suite, the regression suite and check:template-unchanged
run once at the end of each batch, which stops for verification (Emil,
2026-10-07). Every new test names its ISTQB technique and its gate. Measured
on the local Toolshop copies (Phase 41 Decision 7).
run, storage%LOCALAPPDATA%\TestForge
on Windows, ~/Library/Application Support/TestForge on macOS,
$XDG_DATA_HOME/testforge (or ~/.local/share/testforge) elsewhere;
TESTFORGE_DATA_DIR overrides it, and every test sets it to a
temporary folder.src/lib/site-suites/ and src/lib/storage/data-dir.ts. A
test is matched across runs by its describe chain, its title without the
id, and the pages it visits; position alone would match a removed page’s
per-page checks to the next page’s. A saved test the generator no longer
writes is retired even when its page type is unchanged: the signature is
coarse, and such a test tests something that is gone (a link taken off a
page). Only a hand-edited test outlives that.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: A site’s suite is saved outside the repo and reused while its page types are unchanged (TFP-190, Phase 42 Decisions 1 and 4; unit: src/lib/site-suites/site-suites.test.ts (generates through POST /api/generate on the comparison fixture’s /types shop, pages inspected in a real browser), src/lib/run/generation-record.test.ts)
Expected result: Each template-written suite (Standard, and Comprehensive per area; owner decision, 2026-10-13) is saved per origin and suite identity in TestForge’s data folder (TESTFORGE_DATA_DIR, else the OS app-data folder), with each page type’s structural signatures and each test’s file and written hash. Proven (decision table: site unchanged / one page’s structure changed / a page removed x a test edited by hand): a second run of the unchanged site reuses all 143 tests and generates none, and the suite is the saved one; types-contact-phone (the contact form gains a tel field) regenerates only /types/account/:slug’s tests and reuses every other, its hand-edited test kept as edited and flagged as on a changed page; types-privacy-removed retires the privacy page’s tests and the other pages’ link tests of it. The repo’s git status is the same before and after; the data folder resolves per platform (equivalence partitioning). The report and the HTML export say, in one sentence, what was reused, generated, regenerated and retired, and the edits to check. Gate: Phase 42 Gate A.
runsrc/lib/site-suites/stale.ts, run from the executor once
Playwright closes and before the run is finalized. The page is read again
with the same inspection that saved its signature. A reused test is one
written before this generation (each saved test records when it was
written). Site-wide tests are not read again: their failures stand. The
re-run covers what the run ran of that type, plus the type’s new tests,
so a run of selected tests stays one; hand-edited tests keep their
result. The site suite saves its generation inputs and page data for
this.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: A reused test that fails on a page whose structure changed is regenerated and re-run before it counts (TFP-191, Phase 42 Decision 1’s stale-test rule; owner decision, 2026-10-13; unit: src/lib/site-suites/stale.test.ts (generates through POST /api/generate and runs through POST /api/run on the comparison fixture’s /types shop, in a real browser), src/lib/run/generation-record.test.ts)
Expected result: After the run, each failing test that was reused from the site suite (written before this generation), not edited by hand and of one page type has its page read again with the inspection that saved its signature. Proven (decision table: reused or written by this generation x page structure changed or not): reused, types-contact-subject-list (Subject becomes a list): /types/account/:slug is named as changed with the contact page’s signature before and after, its tests are regenerated from a fresh inspection and re-run, and the report has only the re-run’s results - the stale Subject check is gone and its regenerated check of the list passes; the sentence names the regeneration, the failing tests and the re-run’s counts. Reused, types-quantity-readonly (the field turns read-only, the signature doesn’t change): the failure stands and is said to be on an unchanged page. Written by this generation: not read again, the failure stands. The re-run’s other failures on the contact page (the WCAG scan’s target size, the tab order of its inline labels) are reported as the re-run found them. Gate: Phase 42 Gate A.
report, selectionsrc/lib/report/coverage-map.ts, in every run’s report and
its HTML export. What a test touches is read from its code: an element
is touched by a test on its page that uses the locator the generator
writes for it (a form without an id as the form check finds it), a link
likewise or by a site-wide link check that requests its URL, a flow by
its journey test. A whole-page check (the WCAG scan, headings) touches
the page, not each element, so it makes no element covered. The map is
static: an element counts as touched whether or not the test ran or
passed.Stop for verification after TFP-190 to TFP-192 (batch 1).
4841a7a),
TFP-191 (73604e0), TFP-192 (dd9d9b8), the Toolshop coverage map
(612b144, run on dd9d9b8). Site suites live in the OS app-data folder;
saved tests are matched by describe chain, title and the pages they visit.
End checks: unit 530/530, regression 57/57, check:template-unchanged
166/166.Moved word for word from docs/test-plan/RPT.md by TFP-253; the case’s row there links here.
Test: The coverage map: each element, link and flow the inspection found, the tests that touch it, and what no test touches, per page type (TFP-192, Phase 42 Decision 3; unit: src/lib/report/coverage-map.test.ts (generates through POST /api/generate on the comparison fixture’s /types shop, pages inspected in a real browser))
Expected result: Read from the suite’s code: an element is touched by a test that visits its page and uses its locator (a form without an id by its recorded action, method and occurrence, as the form check finds it); a link likewise, or by a site-wide link check that requests its URL; a flow by its journey test. A whole-page check touches no single element. Proven (equivalence partitioning over what touches an element): every element check maps to the element it checks (the Subject check to the Subject field alone; the contact form’s check to the form); with the Subject check taken out of the suite, the Subject field is named in /types/account/:slug’s “touched by none”; every unselected candidate journey is listed as touched by none; the index gives exactly the Subject check for the Subject field, a run of one, and for a link its on-page check and the site-wide link check. The report (app and HTML export) shows the headline and, per page type, covered of total and the untouched names. Gate: Phase 42 Gate A.
<select> naming bug is ticketed (TFP-200), with a
fixture case of a bare label-wrapped list marked as a known failure until
the fix, and the coverage map naming the pattern as a known blind spot.
The fix changes template output, so it is its own commit, the change to
check:template-unchanged accepted deliberately.report, selection612b144’s suite,
reports both tiers with the distinct-link count; the index and the gap
list use the in-context tier.src/lib/report/coverage-map.ts. A link’s site-wide request
is its reachability tier (reachedBy); only a check that finds it on a
page is in context (tests), which the index, the gap list and the
headline’s first count use. The inspection records no page region, so a
repeated link is the same text and target on several pages, whichever
region it is in. A repeated link counts once in the totals and in each
page type it appears on.Moved word for word from docs/test-plan/RPT.md by TFP-253; the case’s row there links here.
Test: Coverage in two tiers, a repeated link counted once (TFP-201, Phase 42 Gate A ruling 1; unit: src/lib/report/coverage-map.test.ts)
Expected result: In context: a test finds and checks the item on its page; this tier is the gap list and the index. Reachability: a link whose URL a site-wide check requests without finding it on a page; it stays on the gap list, saying so. A link with the same text and target on several pages is one item, covered in context when checked on any of them. Proven on the fixture’s /types shop (equivalence partitioning): /types/legal/privacy’s link, requested by the site-wide link check and found on no page, is reachability-only and on /types/help/terms’s gap list; /types/product/101’s link, on every page, with its on-page check kept on the terms page alone, is one item covered in context by that one test; distinct links are fewer than links found on pages; the index for a link gives its on-page checks, not the site-wide request. The headline names both tiers and the distinct-link count. Gate: Phase 42 batch 2.
bug, generate<select> inside its <label> the
way the browser’s accessibility tree does (the label’s own text, not the
options’ text), so its check finds it; the fixture’s known-failure case
turned into a passing case; the coverage map’s blind-spot note removed.check:template-unchanged difference is only that list’s name, accepted
in the commit.src/lib/introspect/dom-snapshot.ts): a label’s text is read without
the labelled control’s own content, as Chromium names it (measured:
“Subject”, “Size *”, “Message”). It applies to a label[for] too. The
template writes what the inspection records, so
check:template-unchanged, which replays recorded profiles, shows no
difference; the change reaches suites through new inspections. The
fixture’s case is /wrapped, and the TFP-191 defect’s list lost its
aria-label. The coverage map’s blind-spot note was never added, since
the fix came in the same batch.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: A control inside its label is named as the browser names it (TFP-200, Phase 42 Gate A ruling 2; unit: src/lib/introspect/label-names.test.ts (inspects the comparison fixture’s /wrapped page, then runs the generated checks in a real browser))
Expected result: The inspection reads a label’s text without the labelled control’s own content, as Chromium’s accessible name does. Proven (equivalence partitioning over what the label holds besides its text): a list (Subject), a list followed by text (Size *) and a textarea holding text (Message) are named so, and their generated element checks pass. On the code before the fix the names were “Subject OrderOther”, “Size ChooseM *” and “Message Hello”, and the test fails. RUN-30’s list now has no aria-label and its regenerated check still passes. Template output: unchanged on recorded profiles (check:template-unchanged); the change is in what new inspections record.
generate, librarycheck:template-unchanged and the local score); each entry lists
its caught bugs.src/lib/library/ - 15 entries, one per Phase 41 check
(TFP-182 to TFP-187; TFP-181’s guard is a helper, not a check). Each
entry records its pattern, binding (page, or site), ISTQB technique,
gate, where it runs, its builder, and the bugs it caught at Phase 41’s
Gate B. The controls’ page patterns (sort list, filter field, number
field, list) moved from the controls check into the library; the other
entries bind to every inspected page, or to the run, as their checks
already did. Each check’s code stays with its generator, which asks
the library which pages each entry binds to. Measured: the generators’
whole output on the fixture’s and the Toolshop’s page data, Standard
and Comprehensive, is byte-identical before and after
(benchmarks/2026-10-14-tfp193-library/).Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: The library: Phase 41’s checks as pattern entries, applied by the patterns inspection finds (TFP-193, Phase 42 Decision 2; unit: src/lib/library/library.test.ts; output: benchmarks/2026-10-14-tfp193-library/)
Expected result: 15 entries in src/lib/library/entries.ts, each with its pattern, binding, technique, gate, where it runs, its builder and the bugs it caught. Proven: every caught bug is in the answer key, and together they are exactly Phase 41’s Gate B finds (2, 4, 19, 26, 27, 29, 31, 33, 34, 40, 42, 45, 53, 58, 59, 60, 62, 63, 66, 67, 75, 80, 82, 94); the image check lists none of its own. Applying an entry (decision table): a page with the pattern gets the check, one without doesn’t (the fixture’s /store sort list against /wrapped’s lists; / with no list or field); a page known only by its path matches only an entry for any page; the generated Standard suite has each check on exactly the pages its entry binds to, and the cross-page layout check needs two. The generators’ whole output (Standard fast and deep, Comprehensive security and accessibility, on the fixture’s and the local Toolshop’s page data) is byte-identical before and after the move: 8 of 8.
generate, aisrc/lib/gap-writer/. The opt-in is a per-site setting in
the data folder (settings.json, set through /api/site-settings;
absent reads as off). The targets are TFP-201’s in-context gap list,
elements and links (flows are interactive and stay out). The writer
gives a test body; TestForge composes the test at the read-only gate
with the whole body in the request guard, type-checks it, refuses a
body that reaches for Node, and runs it. A failure goes back for a
revision in the same session, up to 3 attempts; a breach is rejected
at once. Saved gap tests are kept while their page type holds; a gap
tried without a saved test is recorded, so it isn’t asked for again. At
most 5 gaps per generation (TESTFORGE_GAP_LIMIT), so a first run’s
cost is bounded. Claude Code is called with the prompt on stdin, never
as an argument, since the prompt carries text from the site. Not built:
a control in the app for the setting (the route only).Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: Gap tests, once per gap, opt-in per site (TFP-194, Phase 42 Decisions 5 and 6; unit: src/lib/gap-writer/gap-writer.test.ts (generates through POST /api/generate on the comparison fixture’s /types pages in a real browser, with a stand-in for Claude Code reached by the real call path))
Expected result: A site’s gapWriter setting (off by default; claude-code; local-model) decides whether its uncovered elements and links get tests. Each written body runs at the read-only gate inside the request guard. A failure is revised in the same session (up to 3 attempts); a breach is rejected at once. A saved test joins the site suite and is reused; a gap tried without one isn’t asked for again. Proven (decision table: opt-in x what the written test does): off, the writer is never called and nothing is written; on, one call per gap, every target on the in-context gap list and no covered link asked for, all saved in “Gap tests”, and the report names the count and the cost; the same site again, no call, the gap tests reused; a failing test fixed on the second attempt, resuming the first call’s session; a test that POSTs rejected as a breach, never saved or revised, and the next generation asks for another gap; the local model chosen with none configured, nothing written, and the report says why. The settings route refuses an unknown writer and a request that names no site.
library, aisrc/lib/candidates/. After a gap test is saved, the writer
is asked in the same session to generalise it, leaving __PAGE__ and
__LOCATOR__ blanks. TestForge binds the candidate back to the same
element and runs it, and only one that passes joins the pool. A pattern
is the element’s kind, type and name (lower-cased), read from the
coverage map; one pattern is one file, so a second source is merged. On
a Comprehensive run, each in-context gap a candidate matches gets it,
bound to that site, in “Candidate tests”: filtered out of every count,
listed in the report’s own section, and recorded as evidence when the
run ends (site, page, run, status). Matching doesn’t wait for the
site’s opt-in, since nothing leaves the machine. Before writing, a gap a
candidate matches gets no new test, on either run type, so on Standard
such a gap stays uncovered.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: The candidate pool: gap tests generalised, run on other sites outside the verdict (TFP-195, Phase 42 Decision 8; unit: src/lib/candidates/candidates.test.ts (two comparison fixture sites, through POST /api/generate and POST /api/run, with TFP-194’s stand-in for Claude Code))
Expected result: After a gap test is saved, its writer generalises it (__PAGE__, __LOCATOR__). TestForge binds it back to the same element and runs it; only a pass joins the pool, one file per pattern (kind, type, name). Proven (decision table: site x run type x the candidate’s state): first site, the gap test generalised, checked and pooled with its source; second site, Comprehensive, the candidate bound to it in “Candidate tests” and run beside a template check, the report listing it in its own section and the Comprehensive summary counting the template check alone, and its evidence gaining the run (site, status, run id); second site, Standard, never run; second site opted in, no new gap test for the candidate’s pattern; one pattern from two sites merged into one candidate with both sources; a rejected candidate kept with its reason, never bound, never proposed again.
librarysrc/lib/candidates/promotion.ts, /api/candidates, and
npm run library:promote. The rule reads each candidate’s evidence:
library:promote, which writes each approved candidate with its
evidence and caught bugs under src/lib/library/promoted/ and marks it
promoted, and then commits the entries. Not built: running promoted
entries in generated suites (they are recorded, not yet applied), and a
screen in the app (the route only).Stop for verification after TFP-193 to TFP-196 (batch 2).
c516a1e,
map 16bbef7), TFP-200 (567d2c4; fixed in the inspection, so template
output is unchanged), TFP-193 (7eba131; output byte-identical), TFP-194
(2de9aeb), TFP-195 (ee19dae), TFP-196 (4dc8e8d), b17f097
(lint split), c15dd2c (gap-test files out of Playwright’s test folder,
which caused 15 unrelated failures in the first full run). End checks:
unit 551/551, regression 57/57, check:template-unchanged 166/166.
TFP-194 was tested with a stand-in; no real Claude Code call was made.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: Promotion: only a candidate meeting the rule is offered; approved ones become library entries, rejected ones stay rejected (TFP-196, Phase 42 Decision 8’s rule; unit: src/lib/candidates/promotion.test.ts (through /api/candidates))
Expected result: The rule: gates passed on every run (no breach), and a real defect caught (a run labelled real, or a benchmark bug) or 3 sites with no unlabelled failure, and the tester’s approval. Proven (decision table): no runs, 2 clean sites, 3 sites with an unlabelled failure, and 3 sites with a breach are not offered; 3 clean sites and 1 site with a run labelled real (bug 27) are; approving one below the rule is refused (409, naming why); labelling the failure “not a defect” makes the 3-site candidate offered, while a breach never is. Approved: library:promote writes it as an entry with its evidence, caught bug 27 and approver, marks it promoted, and doesn’t write it twice. Rejected: kept with its reason, refused approval afterwards, and its pattern is still in the pool, so TFP-195 never proposes it again.
TESTFORGE_GAP_LIMIT) is accepted as Emil’s decision.edfa507),
TFP-203 (c7ce21f), TFP-204 (9a9681c), then 97aac4d (the TFP-094
leak test), defcc40 (Standard runs count as evidence), fd48f77 and
7e0116b (the guard’s helper). Pushed; origin/main at 7e0116b. End
checks on the final code: unit 566/566, regression 57/57;
check:template-unchanged 166/166 after TFP-204 (nothing since touches
the template). Claude Code’s four no-tools flags checked against its CLI
reference. No real Claude Code call was made.entries.ts’s runsIn rule.defcc40). Binding already completes the site’s own suite: the
candidate is that gap’s test there, the gap counts covered and no
writer is asked again. Promotion only decides whether it is offered to
every site, and the rule counts distinct sites, so a run on another
site is the cross-site evidence it asks for. A failure there is an
unlabelled failure and blocks promotion until labelled.fd48f77, 7e0116b): on a Standard run with
control checks, every test under “Gap tests” failed with
tfReadOnlyGuarded is not defined, because the template declares the
helper inside the functional describe. It hit written gap tests since
TFP-194 and bound candidates; the gates before saving were unaffected. A
top-level copy is now added only where a call can’t reach one.--safe-mode would turn them
off); the run environment drops proxy settings (HTTPS_PROXY,
NODE_EXTRA_CA_CERTS), fine locally, not behind a corporate proxy.security, aiclaude -p is called with its normal tools, from
the TestForge folder, with page text in the prompt (prompt injection could
steer what it reads or writes); the written test runs as Node with the
full environment (API keys included); the code check blocks require,
process and fs by pattern only.fetch,
XMLHttpRequest, WebSocket, import(, eval, Function(,
globalThis, child_process, page.request and request. calls other
than GET, and the like), still with no imports allowed;src/lib/gap-writer/code-check.ts, src/lib/run/child-env.ts,
and the writer’s call and prompts in writers.ts.
--tools "",
--disallowedTools "*", --strict-mcp-config,
--disable-slash-commands. Not used: --restricted (needs 2.1.248),
--permission-prompts none (2.1.259), --bare (API key only, no
subscription sign-in) and --no-session-persistence (would stop the
--resume revisions need). One empty temporary folder per generation.arguments (the module wrapper’s second is require), this,
prototype access and context()/newPage (a page the guard isn’t
armed on). Bound candidates are checked again.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: The gap writer and the tests it writes, hardened (TFP-202, P0, Phase 42 batch 2 ruling 5; unit: src/lib/gap-writer/hardening.test.ts (TFP-194’s stand-in for Claude Code by the real call path; a gap-test spec and a POST /api/run run in a real browser))
Expected result: Claude Code runs with every tool off, without this machine’s CLAUDE.md, auto memory or hooks (TFP-209), and from an empty temporary folder; written tests may use only the page and expect; every Playwright run gets only the environment Playwright needs, plus the proxy settings when set (TFP-209); page text is untrusted data in the prompt. Proven (equivalence partitioning): each call carries --tools "" --disallowedTools "*" --strict-mcp-config --disable-slash-commands --safe-mode, from an empty folder under the OS temp folder, kept for the revision and removed afterwards; through a .cmd shim on Windows the empty --tools value still arrives; 27 forbidden constructs (Node, imports, fetch, XHR, WebSocket, sendBeacon, page.request.post, routes, context(), computed names, arguments, this, prototypes…) are each refused, a body using process is refused before Playwright starts, and 6 ordinary bodies pass; a gap-test spec and a main-run test both read no OPENAI_API_KEY, ANTHROPIC_API_KEY, GITHUB_TOKEN or TESTFORGE_* setting while the run’s own TESTFORGE_INTERACTIVE arrives; runEnv passes HTTP_PROXY, HTTPS_PROXY, NO_PROXY and NODE_EXTRA_CA_CERTS when set, in either case, and not ALL_PROXY; an injection string (with a forged end marker) is one JSON line inside the PAGE-DATA block, and a revision’s run output is sent the same way. Live (2026-11-02, TFP-209, one real call, benchmarks/2026-11-02-tfp209-safe-mode/): a gap test saved under --safe-mode, with -p and --resume, no hook registered and no project memory, CLAUDE.md or MEMORY.md in the log or transcript.
librarysrc/lib/candidates/standard.ts, called by /api/generate on
Standard runs after the site suite is applied and before the gap writer.
The bound candidate goes through runGapTest (code check, type check, a
run at the read-only gate) and is saved as a gap test with
writer: "candidate" and its id, so the existing reuse rules keep it
while its page type holds. It doesn’t wait for the opt-in (nothing leaves
the machine, as TFP-195’s matching). A failure is recorded as a tried gap
with the candidate’s id. Found on the way: targetOf gave a link’s
locator in double quotes, which the coverage map never matched, so no
gap test of a link counted it covered; it now uses the generator’s own
expression.candidate annotation) and each run’s
record carries it. Found on the way: gap tests and bound candidates on a
Standard functional run with control checks failed with
tfReadOnlyGuarded is not defined (the template’s helper sits inside the
functional describe); the suite now gets a top-level copy.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: On Standard, a matching pool candidate becomes the gap’s test (TFP-203, Phase 42 batch 2 ruling 2; unit: src/lib/candidates/candidates.test.ts (two comparison fixture sites, through POST /api/generate, in a real browser), src/lib/run/generation-record.test.ts)
Expected result: A candidate whose pattern matches a gap on a Standard run is bound to the site and run through the gap test’s gates; if it passes it joins the site suite under “Gap tests”, inside the verdict. Proven (decision table): the second site’s Standard run binds the first site’s candidate (recorded with its id, writer: "candidate"), asks no writer, writes nothing under “Candidate tests”, and the coverage map counts the gap covered; the next run reuses it. A candidate that breaches the read-only gate there, or that the code check refuses (fetch), is not bound, the gap is recorded as tried with the candidate’s id, and the next run doesn’t try it again. The record is kept, cleaned when malformed, and said in the report (“Pool candidates: 1 bound as its gap’s test (…), 1 not bound, failing the gates here.”). Verification fix (2026-10-15): the bound test names its candidate, and a Standard run of it on the second site (1 executed, 1 passed, in the verdict) adds that run to its evidence, so its verdict goes from 1 site to 2.
library, generatesrc/lib/library/promoted/
is bound and run on a site with its pattern, inside the verdict like any
library entry; with nothing promoted, check:template-unchanged is
166/166.src/lib/library/promoted.ts, called by /api/generate on
Standard runs with the functional area in scope (where the library’s
control checks run), before the site suite is applied, so promoted tests
are saved and reused like the template’s. Each entry is matched by its
pattern against the inspected pages: an element gets a test on each page
it is on, a repeated link one test on its first page; the blanks take the
page and the generator’s own locator, so the coverage map counts it
covered. The tests sit in the functional describe at the read-only gate,
inside the request guard, titled library promoted.<id>: .... An entry
the TFP-202 code check refuses is not applied, and an unreadable entry
file is skipped. The template generator itself is untouched (its arity is
fixed by check:template-unchanged), so with nothing promoted its output
is byte for byte the same. Not done: promoted entries on Comprehensive
runs, and in entries.ts’s catalogue.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: Generation applies the library’s promoted entries (TFP-204, Phase 42 batch 2 ruling 3; unit: src/lib/library/promoted.test.ts (TFP-193’s recorded fixture profile; and one case through POST /api/generate and POST /api/run on the comparison fixture’s /types pages, in a real browser))
Expected result: A promoted entry in src/lib/library/promoted/ is bound wherever inspection found its pattern and runs inside the verdict on Standard runs with the functional area. Proven (decision table): an element’s pattern gets a test on each page with it, bound to the page and the generator’s locator, in the functional describe at the read-only gate, the guard’s helper written once, and each element counted covered; a repeated link gets one test, on its first page; a pattern on no page, an entry the code check refuses (fetch), and nothing promoted each leave the code byte for byte as it came; the repo’s folder holds no entry yet, and an unreadable entry file is skipped. Through the routes: a promoted entry for one of the site’s uncovered links is generated on Standard functional (not on Standard accessibility), covers the gap, and runs: 1 executed, 1 passed.
measurement, benchmarkae88e0b to a12aee1; outcome in
benchmarks/2026-10-15-tfp197-gap-writer/outcome.md). Official score 0 of
4, expected by design: none of the 4 is in Chromium’s reach on the
selected pages, and a gap test is revised until it passes on its own
site. No unlabelled failure on clean; an unchanged clean re-run reused
468 of 468 tests with 0 Claude Code calls. 446 calls, $11.84 in all
(224 of them, $5.87, on first attempts cut short by 394a096 and
31c0afd). The second measure showed nothing: clean routes by path and
with-bugs by #/, so every page type changed and clean’s 19 gap tests
were retired; 9 of clean’s candidates were bound on with-bugs and passed.href, the href equals itself): inventory the
template could test for nothing. The code stays, off unless a site opts
in; the hardening, site suites, reuse, candidates, promotion, the coverage
map and the default limit of 5 all stay.generate, coverageR1 (TFP-207) shows the same elements still uncovered. “Every element in context” holds only for clean’s elements other than forms.
href is a valid URL;tfGoto
inside the request guard (reuse the existing site-wide link check where
it already loads that URL, rather than loading it twice); external
targets get the href check only, no request;runs/clean-crawl.json,
runs/with-bugs-crawl.json), offline generation leaves no link in the
in-context gap list;check:template-unchanged changes only by the added link tests, with
the baseline updated deliberately and the change described in the
commit;Built: src/lib/generate/mock/link-audits/inventory-tests.ts, at the
end of the functional suite, so no existing test’s id moves. A same-origin
target is already loaded by link health (internal and download links) or
the linked-routes check (hash routes), and the start page by every test,
so in today’s template these tests send no request of their own. The load
through the guard, with its cap of 20 and the named rest, is for a target
no other check loads, and is proven on the fixture with the reuse turned
off. A link inspection found hidden is checked by the hidden-link locator
(present, valid href), and the coverage map counts that locator too. The
committed TFP-197 crawls hold the selection but no link data, so the
offline check uses Gate B’s committed inspection of the same pages
(TFP-188; clean’s product id differs, its database re-seeded). The gap
writer’s and candidates’ tests now generate without the functional area,
where the fixture’s links are still gaps.
fda373f, 263db24
(expectations it changed: the read-only test count over the 166 inputs,
1374 to 1390, and a promoted-entry case), 63de663 (the template
baseline). One test per distinct link, at the end of the functional
suite so no existing test id moves: on its page by the generator’s
locator or, failing that, its own address; a visible link visible and
named, a hidden one present; a valid href; a same-origin target loaded
only when no existing check (link health, linked routes, the start page)
already loads it, up to 20 a run, the rest named. External links: href
only. check:template-unchanged against 78a64fb: 162 identical, 4
different, and those 4 identical once their 16 added link tests are
taken out. Offline on TFP-197’s selections (with Gate B’s committed
inspection for the link data): no link left on the gap list. Regression
57/57. Unit 569/572 on its one full run: the two TFP-205 expectations
fixed in 263db24, and one Comprehensive security case TFP-205 doesn’t
touch; the three files then passed alone, 14/14. The next full unit run
is the review’s.href (its wording is the spelling and
naming checks’ job); the page visit itself is a plain page.goto, the
target load goes through the request guard.comprehensive-scenarios.test.ts,
performance-bands.test.ts and now a Comprehensive security case, each
failing once in a full run and passing alone.read-only-guard.test.ts:168, TFP-199’s beacon case;cookie-scope.test.ts:90.TFP-208 takes all five, with TFP-169.
Standard on with-bugs went from 1,235 s at Gate B to 1,535 s in R1, of its 1,800 s limit. The rest of the rise is a run about 15% slower all round.
/#/category/hand-tools, whose product window changes
between loads. TFP-205 has no rule for content that changes between
loads. TFP-208 decides whether card links are skipped or found by
something stable, as a ruling for Emil.isIdSegment). The page gets one test,
cards linking to <pattern> on <page>: such cards are there, and each
card there at run time, by position, is visible, named and has a valid
href of the pattern. Targets load as for one link (the inspected
cards’ by link health). The coverage map counts each card covered by its
group’s test. 21933f7, GEN-98.
paginate(9) and no ORDER BY, so the order is
unspecified. But asked ten times in a row on a freshly seeded database,
each copy’s API returned the same 9 (benchmarks/2026-10-16-tfp208/logs/window-*.log).
What moved R1’s window between its inspection and its run is not
reproduced; the ruling doesn’t depend on it, since it holds for any
listing whose cards move.benchmarks/2026-10-16-tfp208/outcome.md.02ed0ef, GEN-99: every
card is checked before the test fails, and it fails once, listing each
failing card with its position and what it showed at run time (name, a
price if shown, href), e.g. card 4 of 9 on /category/hand-tools
("ECO Sledgehammer", $17.75, #/product/10). A passing test’s output is
unchanged. check:template-unchanged against 980346f: 166 of 166.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: The template checks every link on the pages it selects (TFP-205, Phase 42 Gate B ruling; unit: src/lib/generate/mock/link-audits/inventory-tests.test.ts (fixture pages served locally and inspected in a real browser; the committed Toolshop data offline), src/lib/report/coverage-map.test.ts)
Expected result: One read-only test per distinct link (text and target, once, on the first selected page it is on), at the end of the functional suite: the link is on its page by the generator’s own locator (or its address), visible and named when inspection found it visible, with a valid href; a same-origin target is loaded once a run (link health and the linked-routes check already load them; any other is loaded through the page inside the request guard, up to 20, the rest named as not loaded); an external target gets the href check only. Proven (decision table): on the fixture, 5 tests for 5 distinct links, the one on two pages once; all pass, with no request to the external link’s origin; link health fails on the broken one, naming it; with no other check loading the targets, the broken link’s own test fails (“leads to /missing, which answers 404”), the external one sends nothing, and one over a cap of 2 is named as not loaded; offline, on the pages TFP-197’s crawls selected for both Toolshop copies (with Gate B’s committed inspection of them), every distinct link is covered in context and none is on the gap list.
response.request().method()security, gap-writer<response>.request().method() (and nothing else
on request()); a test that the other refused forms stay refused.Passed 2026-10-14 (Emil), with the four rulings above.
Passed 2026-10-15 (Emil), with the gap writer not kept as a bug finder (Gate B ruling above). TFP-205 verified the same day; Phase 42 complete.