Version: 0.1 Date: 2026-10-06 Status: Locked. Decisions 1 to 4 were approved by Emil on 2026-10-06. TFP-153 is done (classification and spot-check), pending verification of the answer key. TFP-154 is not started; it waits for the template fixes TFP-159, TFP-160, TFP-151 and TFP-161.
TestForge’s real-site runs show what it reports, not what it misses: a site’s real defects are unknown, so a quiet run could mean a healthy site or blind spots. A site with a published list of planted bugs gives an answer key.
Testsmith’s “Toolshop” practice shop has a bug-free version
(https://practicesoftwaretesting.com/, Sprint 5) and a version with planted
bugs (https://with-bugs.practicesoftwaretesting.com/, “Sprint 5 (with
bugs)”). Testsmith’s own repository
(github.com/testsmith-io/practice-software-testing,
sprint5-with-bugs/testSessions/listOfBugs.md) lists 94 bugs, read
2026-10-06. Each one has a page, title, description, expected result,
optionally an OWASP type, and the exploratory-testing tour that finds it.
What the list holds, by reading it (not yet classified for TestForge):
Licence. Testsmith’s licence (all rights reserved) permits viewing, copying and running the software “for private, internal, and non-commercial purposes only”. It forbids publishing a copy or using it to provide services to third parties. So:
(Claude is not a lawyer; this is a reading of the licence text, not advice.)
A failure that also appears on the bug-free site is not a find. The two versions’ code has drifted apart (460 differing files in the repository), so the bug-free site is an approximate control, not an exact one.
docker compose setup on Emil’s machine. The licence
permits that, and a local copy is a host TestForge may own. Active also
waits for Phase 35 Decision 8’s ownership confirmation.measurement, benchmarkbenchmarks/<date>-toolshop-known-bugs/answer-key.md: for each bug, ID,
TestForge’s own one-line description, area, gate, Chromium reach, which
run could find it, and present or not present on the hosted site;Dependencies: Phase 38 Gate B.
benchmarks/2026-10-06-toolshop-known-bugs/answer-key.md, from commit
c752a70. All 94 are classified, in TestForge’s own wording.
#/ hash, while the clean
site routes by path. TFP-154 must first check what TestForge’s crawl
finds there. If it only finds /, that is a gap of its own and needs a
ticket before scoring.#/ links as pages on sites that route by hashenhancement, crawl, introspect, generatec1685b7,
step1-crawl-and-inspection.md) measured it. On the with-bugs Toolshop,
which routes by #/ links, TestForge’s crawl finds 1 page; on the clean
Toolshop, which routes by path, it finds 46. The crawl clears every URL’s
#… part (normalized.hash = "" in src/app/api/crawl/route.ts), so
each #/… link counts as /. Read from the code, not run:
#… link as an in-page anchor;/contact, a false
“Anchor target missing”.Many single-page apps route this way (Angular’s and Vue’s hash modes, React Router’s HashRouter), so this isn’t just a Toolshop gap.
# link is a page when what follows the # starts with / or
!/. Everything else stays an in-page anchor, including #top,
#filters and a bare #./#/contact,
everywhere: the crawl, inspection, generated tests and reports./#/testforge-missing-<random>)
and record what the app draws;This is the client-side twin of the passive missing-page probe, and it works whatever wording the site uses. It stays read-only: it is only page loads.
TEST_PLAN.md cases. ISTQB technique: equivalence
partitioning (route fragment, anchor fragment, bare #).#top still
passes the anchor check, and no route fragment is treated as an anchor.check:template-unchanged gives 166 of 166 identical: no fixed input
is hash-routed.8324b2b, verified 2026-10-06 (Cowork):
check:template-unchanged 2f435a6 8324b2b gives 166 of 166 identical.b187e25) finds 34 pages on with-bugs, up from 1,
and the clean site’s 46 match its first run exactly. Both sites have
the same 8 main routes. /#/category/undefined and /#/chainsaws
exist only on with-bugs, and are listed bugs.isRouteFragment follows Decision 1, and
pagePathOf Decision 2. The start route (#/, #!/) is never
compared with the missing-route view, so an app that sends unknown
routes home doesn’t fail its own home link./spa fixture case (453/453 in the unit suite)
fails on the old code.#… part, so it is
meaningless on a hash-routed site. That is TFP-166.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: A #/ link is a page on a hash-routed site: the crawl and inspection find each route, named as the address bar shows it (TFP-165 Decisions 1 to 3, owner decision, 2026-10-06; unit: src/lib/introspect/routes.test.ts (the # rule, then a real crawl through /api/crawl and inspection of the fixture’s /spa); fixture: /spa)
Expected result: Rule (src/lib/introspect/routes.ts, and isRouteHref in the page snapshot): a # link is a page when its fragment starts with / or !/; #top, #details, a bare # stay in-page anchors. Name: the path, then the route: /#/contact, /app#!/legacy; a path-routed page is named as before. Crawl: keeps a route fragment in its de-duplication key (still drops the query and an anchor). Introspection: a route link is internal, with its full URL. Wait: a route is loaded as a fresh document (about:blank first), then network idle, bounded at 10 s, as for paths: moving between routes of one document doesn’t reload it, and Playwright’s network-idle state has already fired. Proven (equivalence partitioning: route, anchor, bare #, none), 2026-10-06: the rule and names on each partition; the crawl of /spa finds /spa#!/legacy, /spa#/, /spa#/chainsaws, /spa#/contact, /spa#/products, /spa#/products/1, each drawn (the app draws after a request; none read as “Loading…”); inspection records the 6 route links as internal and #details, #top as anchor. On the previous code the crawl found only /spa and every # link was an anchor.
Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: A linked route that the app draws as “not found” fails link health, naming it (TFP-165 Decision 4; unit: src/lib/introspect/routes.test.ts (the generated functional suite, deep mode, through the Playwright CLI, one worker); fixture: /spa, #/chainsaws)
Expected result: Probe (generated, link-audits/routes.ts): once per run (per worker), the app’s view of a route that can’t exist (<path>#/testforge-missing-<random>, or #!/ for hashbang links), read as the page’s text after a fresh load settled to network idle (bounded); read-only, page loads only. Bulk check “link health: linked routes draw a page, not the app’s "not found" view”: each linked route once (the start route #/ excepted), failing with those that draw the same as the probe; it stops after 14 s and names any not checked as a finding. The request-based bulk check skips routes (the server answers every route the same). Per-link check for a route link: present (hidden links found as TFP-159 finds them, with its reach finding) with its href, and its view differs from the probe. The anchor check no longer sees routes; link types classify them as internal. Proven, 2026-10-06: on /spa#/ the bulk check fails naming only /spa#/chainsaws (“draws the same as /spa#/testforge-missing-…, a route that cannot exist”); the per-link check fails Chainsaws with the same reason and passes Contact, Products, Legacy and First product; the anchor check passes (#top, #details exist), as do link types and the request-based check. Output measured: check:template-unchanged against 2f435a6: 166 of 166 identical; no fixed input is hash-routed. Limits: a view that changes on every load (random content) can make a missing route look found (a miss, not a false failure); TFP-161’s orphan check still strips every # part, so on a hash-routed site its counts are meaningless.
bug, generate, seo#… part of
each link, so on a hash-routed site every route counts as /. Its result
there means nothing, and it would mix into TFP-154’s score.pagePathOf, as the crawl now does./spa fixture, a route no other route links to fails as an
orphan, and linked routes pass.check:template-unchanged gives 166 of 166 identical.26fed09, verified 2026-10-06 (Cowork):
check:template-unchanged dfabfaf 26fed09 gives 166 of 166 identical. The
orphan-page test’s non-browser case passes here. Measured by Code: the
hash-routed column fails falsely on the old code, as “/spa#/contact is not
linked”, and on the new code only the genuinely unlinked /spa#/offer
fails. Profiles without requestedPaths keep the old check.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: On a hash-routed site the orphan check counts pages by route (TFP-166; unit: src/lib/generate/mock/link-audits/orphan-pages.test.ts (introspects the comparison fixture, then runs the generated check through the Playwright CLI); fixture: /spa, its unlinked route #/offer)
Expected result: For a profile with requestedPaths (TFP-161’s branch), each page and each of its links is counted by its page key, origin + pagePathOf (TFP-165): a hash route keeps its #/..., an in-page anchor is dropped. Page names follow the same rule, so the start page /#/ stays /#/. Until now every # part was cleared, so each route of a hash-routed site counted as the same page, and with TFP-161’s self-link rule every route’s links looked like links to itself. A profile without requestedPaths keeps the previous check, unchanged. Proven (decision table, the hash-routed column added to TFP-161’s), 2026-10-06: /spa#/, /spa#/contact, /spa#/products: passes (each linked from the nav on the other routes); /spa#/, /spa#/contact, /spa#/offer: fails, “/spa#/offer is not linked from any other inspected page”. The previous check failed the first set too, “/spa#/contact is not linked from any other inspected page”. TFP-161’s path-routed columns pass unchanged. Output measured: check:template-unchanged against dfabfaf: 166 of 166 identical; the fixed inputs’ profiles have no requestedPaths, so none reaches this branch.
bug, generatedomcontentloaded. A single-page app (Angular, React,
Vue) draws its content after that, so checks read a header-and-footer
shell:
It affects path-routed apps as well as hash-routed ones. TFP-165 added the wait to the crawl and inspection only.
This mirrors the crawl’s wait, so tests see what inspection saw. Rejected: waiting for a known element, because generated tests can’t know each app’s markers in advance.
TEST_PLAN.md cases.check:template-unchanged shows only the page-visit lines changed.e812795 and 7577252, verified 2026-10-07 (Cowork):
page.goto( in a Standard or Comprehensive suite becomes
tfGoto(page, . check:template-unchanged bc62533: 12 of 166
identical; in the 154 that differ, only the helper and the 1,042 visit
lines change./late-form, and the hash route
/spa#/contact added in 7577252), the form checks pass, and they fail
on the previous code.7577252 makes the routing idempotent and tightens the /late-form
assertion.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Every page visit in a generated test waits for the page to draw (TFP-167, owner decision, 2026-10-06; unit: src/lib/generate/page-visits.test.ts (the rewrite, then the fixture’s /late-form inspected, generated and run through the Playwright CLI, one worker); fixture: /late-form, drawn after /api/slow, 1.5 s)
Expected result: Helper (src/lib/generate/page-visits.ts): every page.goto(...) in a Standard or Comprehensive suite becomes tfGoto(page, ...) with its arguments unchanged, and tfGoto is added once after test.use: a hash route loads as a fresh document (as TFP-165’s crawl does), the page loads as the test asked, then the network must go quiet for at most 5 s; the wait never fails a test. Rejected: waiting for a known element (generated tests can’t know each app’s markers). Proven (equivalence partitioning: content in the HTML, drawn after a slow request, drawn by a hash route), 2026-10-07: on /late-form the form check and the form’s name check pass; on the previous code both failed (“no form on /late-form with action "/notes", method get”, “(a form) is not on the page”), as T25 on the Toolshop; the button’s name check and the axe scan pass on both. On the fixture’s hash route /spa#/contact (drawn after the same 1.5 s request, with a form), the generated visit is tfGoto(page, '/spa#/contact') and the route’s form check passes; on the previous code it failed, “no form on /spa#/contact with action "/notes", method get”. routePageVisits is idempotent: a second pass leaves a suite unchanged (otherwise the helper’s own page.goto would become a call to itself). Fixture pages with content in their HTML give the same results as before (a Standard run on /, /login, /notes: 87 passed, 9 failed, 3 not run, before and after). Run-time cost (Standard, all areas, fast, one worker): the fixture 104 s to 136 s (+31%); the with-bugs Toolshop’s 5 pages 766 s (TFP-154) to 1,170 s (+53%), within the 30-minute budget. Output measured: check:template-unchanged against bc62533: 12 of 166 identical (no page visits); in the 154 that differ, the helper is added once and every other changed line (1,042) is page.goto( to tfGoto(page, .
tfGotobug, generatetfGoto. The local model’s gap-fill block (mergeLocalSuite) and the
Comprehensive scenario blocks still use raw page.goto, so on a
single-page app they read the undrawn page. They’re gated as interactive
and kept outside the verdict, so this doesn’t gate anything, but it’s a
gap against “every page visit”.page.goto outside tfGoto’s own body, and type-check.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Model and OpenAI page visits go through tfGoto (TFP-168; unit: src/lib/generate/page-visits.test.ts, src/lib/generate/gap-fill.test.ts)
Expected result: The local model’s gap-fill block, its Comprehensive scenarios and an OpenAI-written suite route every page.goto through the helper TFP-167 added, defined once per file: the template’s, or one added before the model’s block when the template has none (the template stays a byte-for-byte prefix). An OpenAI suite whose imports aren’t the template’s line gets the helper after its last import. Re-running the routing on a routed suite changes nothing. Proven 2026-10-07: a merged gap-fill suite, a Comprehensive scenario suite and an OpenAI suite with its own double-quoted import each have no raw page.goto outside the helper’s body, the helper once, and type-check; the case fails on the previous code (“gap-fill: no raw page.goto”).
bug, teststandard-conformance.test.ts)
failed with fetch failed: read ECONNRESET from the fixture, 7 to 10 s
in, on three full unit runs (during TFP-159, TFP-167 and TFP-167’s
follow-up). It passes alone each time. It’s a recurring signal in the
fixture’s start-up or control requests, not a one-off flake.measurement, benchmarkdocs/decisions/, with found, hinted and missed counts
for each area and gate, and the missed bugs grouped by what would catch
them.91a7141 (rule) and 3b5b774 (report), verified
2026-10-06 (Cowork):
domcontentloaded (222 page visits in the with-bugs
Standard suite, against 35 that wait for network idle). On a
single-page app the checks run before the app has drawn the page. That
hit both Toolshops: bug 91 was missed, bug 92 was only hinted, and there
were form-check false failures on both sites (T25). This first score
stands as recorded. TFP-167 fixes the wait, and the benchmark is re-run
after it.878a49d, verified 2026-10-07 (Cowork): after
TFP-167, 9 found, 4 hinted and 24 missed of the same 37, recorded beside
the first score, which is unchanged in the report.
scoring-rule.md is unchanged since 91a7141.7577252. Each committed suite visits pages only
through tfGoto.c1685b7, verified 2026-10-06 (Cowork): the committed
crawl output gives 1 page on with-bugs and 46 on the clean site.
Scoring waits for TFP-165.bug, introspect/#/product/6 as “not linked from any other inspected page”,
because inspection read home and the category before their product cards
drew, so it recorded no product links. Any product page would have failed
the same way. Generated page visits wait for the page to draw (TFP-167);
inspection’s reading of a page doesn’t wait reliably enough.tfGoto waits: network quiet,
bounded, never failing for the wait.TEST_PLAN.md cases./#/category/hand-tools records their
product links (a fresh inspection, read-only).check:template-unchanged: any change is explained (fixed inputs carry
their own page data, so none is expected).Moved word for word from docs/test-plan/INT.md by TFP-253; the case’s row there links here.
Test: Inspection reads a page only once it has drawn (TFP-177, T30; unit: src/lib/generate/mock/link-audits/orphan-pages.test.ts)
Expected result: After network idle, inspection also waits for 1 s of quiet - no request in flight and no element added, removed or re-texted - within the same 10 s bound, and is never failed for the wait. Network idle alone can come true in a gap before an app’s data requests start; on TFP-175’s with-bugs run inspection read home and the category drawn but without their product cards, and the product page failed the orphan check. The crawl keeps network idle alone (not in scope). Proven 2026-10-07: the fixture’s /late-links draws its link to /late-links/item only after a request started 800 ms late; inspection records the link and the generated orphan check passes for the item (both fail on the previous code). A fresh, read-only inspection of with-bugs /#/ and /#/category/hand-tools records 9 product links on each (7 s for both pages).
measurement, benchmarkruns-tfp167/) and TFP-175 (benchmarks/2026-10-07-tfp175-selection/)
by hand from the committed records, and label each in
labelled-failures.md (an existing label where it fits). No new runs.Batch verified 2026-10-07 (Cowork): TFP-177 (acc0684), TFP-162
(cefa24d), TFP-163 (be688d3, f4ed183), TFP-164 (837de4f), TFP-168
(82cc8d0), TFP-178 (6d15268); TFP-169’s diagnostics (3b44370), with
TFP-169 still open. End checks, run once: unit 494/494, tsc clean, ESLint no
errors, check:template-unchanged 162/166 identical (the 4 that differ
change only TFP-164’s download line), regression 54/57 (two stale tests and
the progress-estimate bug, TFP-189 in the Phase 41 record).
3b44370 makes the next failure name
whether the fixture exited, its exit code and its error output.Known limit of this benchmark (2026-10-07): the clean Toolshop resets its catalogue, at least hourly on 2026-10-07, so product ids and card text change between runs (U5). Scores compare bugs, not pages, so they hold; failure-by-failure diffs on the clean site across runs hours apart don’t.
8324b2b,
verified 2026-10-06. Then TFP-166 (the orphan check on hash routes). Done, 26fed09,
verified 2026-10-06.e812795 and 7577252 (TFP-167) and 878a49d (re-score), verified
2026-10-07: 9 found, 4 hinted, 24 missed of 37.Met 2026-10-07 (Cowork): both scores are recorded with the rule committed first, and every with-bugs failure is a scored bug or labelled.