Date: 2026-10-06
Ticket: TFP-154 (known-bug-benchmark-backlog-2026-10-06.md, Decisions 1 to 4)
Evidence: benchmarks/2026-10-06-toolshop-known-bugs/runs/, holding each
run’s suite, Playwright report and records, plus run-tfp154.ts,
diff-runs.mjs and which-run.mjs. The scoring rule and page set
(scoring-rule.md) were committed before the first run (91a7141).
Setup: code at 26fed09, Emil’s machine, Chromium, read-only gate
(TESTFORGE_INTERACTIVE=0), one worker.
| Site | Run | Pages | Tests | Passed | Failed | Not run | Run time |
|---|---|---|---|---|---|---|---|
| with-bugs | Standard, all 10 areas | 5 | 288 | 235 | 47 | 6 | 766 s |
| with-bugs | Comprehensive security (passive) | 9 | 37 | 37 | 0 | 0 | 36 s |
| with-bugs | Comprehensive accessibility | 9 | 18 | 7 | 11 | 0 | 45 s |
| clean | Standard, all 10 areas | 5 | 277 | 234 | 39 | 4 | 590 s |
| clean | Comprehensive security (passive) | 8 | 33 | 33 | 0 | 0 | 29 s |
| clean | Comprehensive accessibility | 8 | 16 | 16 | 0 | 0 | 21 s |
The pages: home, a category (hand tools), a product (Combination Pliers),
contact, login, register, forgot-password and rentals, as hash routes on
with-bugs (/#/...) and paths on clean, plus /#/chainsaws on with-bugs.
Standard took the first 5. Comprehensive covered all.
“Present” is the hosted spot-check: yes (checked by hand) or ? (never checked). A ? on a miss means the bug may not be on the hosted site.
| ID | Bug (TestForge’s wording) | Area | Score | Evidence (run: check: message) | Present |
|---|---|---|---|---|---|
| 2 | Category sort order reversed | functional | missed | no check reads sort results | ? |
| 3 | “Chainsaw” link goes to a missing page | functional | found | Standard: “link health: linked routes draw a page, not the app’s “not found” view”: “/#/chainsaws (draws the same as /#/testforge-missing-…, a route that cannot exist)”, on each of the 5 pages | yes |
| 4 | Product names right-aligned (category) | visual | missed | no layout check | ? |
| 19 | Login email padding | visual | missed | no layout check | ? |
| 22 | Forgot-password button font | visual | missed | no layout check | ? |
| 23 | Forgot-password email padding | visual | missed | no layout check | ? |
| 26 | Product images on home don’t show | visual | missed | nothing says an image doesn’t show; axe names .img-fluid (a different image) for alt text |
yes |
| 27 | Price filter doesn’t send its maximum | functional | missed | no check reads filter requests | ? |
| 28 | Header logo is a broken image | functional | found | Standard: “no same-origin request fails while the page loads”: “HTTP 404 …/broken.png” on /; Standard and Comprehensive: axe image-alt on .navbar-brand > img[src$="broken.png"] |
yes |
| 29 | “Home” link points to contact | functional | missed | link health checks the target draws, not that it’s the right page | yes |
| 30 | Search icon is a broken image | functional | found | Standard and Comprehensive: axe image-alt on .grid-title:nth-child(7) > img[src$="broken.png"], the second broken.png on / the spot-check read as this bug |
likely |
| 31 | “Contact” misspelt | content | missed | no spelling check | yes |
| 33 | “Sort” label misspelt | content | hinted | Standard and Comprehensive: axe select-name and “every form field has an accessible name”: the sort <select> has no name on /#/ and the category (with-bugs dropped its aria-label="sort"); nothing names the misspelling |
yes |
| 34 | “Search” button misspelt | content | missed | no spelling check | yes |
| 40 | Quantity +/- don’t change the quantity | functional | missed | no check clicks +/- and reads the result | ? |
| 42 | “Related products” misspelt | content | missed | no spelling check | yes |
| 45 | Product buttons overlap | visual | missed | no layout check | ? |
| 46 | Product badge text black | visual | missed | axe flags .text-reset contrast only once the product page has drawn; the run read it before |
? |
| 49 | Rentals title centred | visual | missed | no layout check | ? |
| 53 | Registration tab order | accessibility | missed | no check follows form tab order | ? |
| 54 | Registration phone placeholder misspelt | content | missed | no spelling check; register is outside Standard’s 5 | ? |
| 58 | Registration button width | visual | missed | no layout check | ? |
| 59 | Registration dropdown fonts | visual | missed | no layout check | ? |
| 60 | Contact subject list shows error entries | functional | hinted | Standard: the contact form’s check (T27) quotes the options, “Error 101: Subject not found”; Comprehensive: axe lists #subject for manual contrast review; neither names the entries as the fault |
yes |
| 62 | Contact message placeholder misspelt | content | missed | no spelling check | yes |
| 63 | Contact attachment label font | visual | missed | no layout check | ? |
| 66 | Contact button width | visual | missed | no layout check | ? |
| 67 | Contact dropdown fonts | visual | missed | no layout check | ? |
| 75 | Products API exposes stock internals | security | missed | the passive checks don’t read API responses | ? |
| 80 | Log file reachable from the web | security | missed | the passive checks don’t probe known paths | ? |
| 82 | Sequential, guessable IDs | security | missed | no ID-pattern check | ? |
| 85 | Link contrast below 4.5:1 | accessibility | found | Standard and Comprehensive: axe color-contrast (serious) on the footer links p > a[target="_blank"], on every with-bugs page; the clean site passes |
? (found here) |
| 87 | Login fields have no labels | accessibility | found | Comprehensive: “every form field has an accessible name” on /#/auth/login: “form field named only by its placeholder …: #email (“Your E-mail *”)”, and #password (axe accepts a placeholder as a name) |
yes |
| 91 | Quantity field has no label | accessibility | missed | the field (<input type="number" data-test="quantity">, no label) wasn’t drawn when the checks ran |
yes |
| 92 | Product images have no alt | accessibility | hinted | axe image-alt on /#/product/1 names only img, the header logo; the product images are flagged only on a settled page |
yes |
| 93 | Home search form has no label | accessibility | found | Standard and Comprehensive: axe label on .form-control and “form fields on /#/ with no accessible name: … input[type=”text”]” |
yes |
| 94 | Home filters not in a fieldset | accessibility | missed | axe has no rule for it | yes |
| Area | Scored | Found | Hinted | Missed |
|---|---|---|---|---|
| functional | 8 | 3 | 1 | 4 |
| visual | 13 | 0 | 0 | 13 |
| content | 6 | 0 | 1 | 5 |
| accessibility | 7 | 3 | 1 | 3 |
| security | 3 | 0 | 0 | 3 |
| all (read-only gate) | 37 | 6 | 3 | 28 |
By run:
/#/chainsaws, a page the clean site
doesn’t have.Against the prediction:
| Would catch them | Bugs | Standard area it points to |
|---|---|---|
| A visual or layout check (alignment, padding, width, fonts, images that don’t render) | 4, 19, 22, 23, 26, 45, 46, 49, 58, 59, 63, 66, 67 | none today: usability is the nearest |
| A spelling and wording check on visible text, placeholders and labels | 31, 34, 42, 54, 62 (and 33’s cause) | seo/content |
| Checks that a control did the right thing: a sort’s order, a filter’s request, a link’s destination, +/- | 2, 27, 29, 40 | functional |
| Generated checks that wait for a hash route to draw, as the crawl does since TFP-165 | 91, 92’s images, 46 | every area (cross-cutting) |
| Keyboard order through a form, and grouping that axe doesn’t judge | 53, 94 | accessibility |
| Read-only security probes: API response contents, known paths, ID patterns | 75, 80, 82 | security |
Each is in docs/decisions/data/labelled-failures.md.
| Failure | Count | Verdict |
|---|---|---|
| Form lookups before the form is drawn (“no form on /#/auth/login”) | 1 (clean: 7) | T25, template error |
Quantity input located by its value, getByText('1') |
1 | T26, template error |
Contact form classed api, located by its joined labels |
1 | T27, template error |
| “password field in the URL” on a form whose password has no name | 1 | T28, template error |
<a href="#" role="button"> “Categories” looked up as a link |
5 | T29, template error |
| Meta description, canonical, robots.txt, sitemap | 12 | R13, real (not a listed bug) |
#filters link with no target |
2 | R14, real (not a listed bug) |
Show-password toggle with no name (axe button-name, login and register) |
2 pages | R15, real: the same as R10 on the clean site |
Password field check lands on a duplicate id="password" |
1 | U4, unclear |
:visited styling (also on clean) |
5 | U2 |
| “Links are distinguishable from surrounding text” | 4 | U3 |
The axe failures on every with-bugs page (color-contrast on the footer
links, image-alt on the logo) are bugs 85 and 28. The select-name and
field-name failures on home and the category are bugs 33 and 93.
The first score above stays as it was. This re-score used the same page
set, the same committed rule (scoring-rule.md), the same driver and both
Toolshops, read-only, one worker, Chromium. The code was 7577252: every
generated page visit now waits for the network to go quiet, for at most
5 s. Evidence: benchmarks/2026-10-06-toolshop-known-bugs/runs-tfp167/.
First (26fed09) |
Re-score (7577252) |
|
|---|---|---|
| Found | 6 | 9 |
| Hinted | 3 | 4 |
| Missed | 28 | 24 |
| Area | Scored | Found | Hinted | Missed |
|---|---|---|---|---|
| functional | 8 | 3 | 2 | 3 |
| visual | 13 | 1 | 1 | 11 |
| content | 6 | 0 | 1 | 5 |
| accessibility | 7 | 5 | 0 | 2 |
| security | 3 | 0 | 0 | 3 |
| all | 37 | 9 | 4 | 24 |
What moved, and why. Each change is on the product page or home, which the first run read before it drew:
| ID | First | Re-score | Evidence now |
|---|---|---|---|
| 91 | missed | found | axe label (critical) on input, and “form fields on /#/product/1 with no accessible name: input[type=”number”]” (Standard and Comprehensive); the clean product page passes |
| 92 | hinted | found | axe image-alt on the product page’s product images (a[href$="#/product/2"] > .card-img-wrapper > .card-img-top, …); the clean product page passes |
| 46 | missed | found | axe color-contrast (serious) on .text-reset on /#/product/1. Read by hand, that is the category badge “Pliers”, dark text rgb(33, 37, 41) on grey: the black badge text. The clean page passes |
| 26 | missed | hinted | axe image-alt on home’s product card images (a[data-test="product-1"] > .card-img-top), the bug’s elements. The message is about alt text, not images that don’t show |
| 40 | missed | hinted | axe button-name on #btn-decrease-quantity and #btn-increase-quantity, the bug’s buttons. It names their missing names, not that they don’t change the quantity |
The other scores are unchanged:
Against the prediction: all 7 “likely” bugs are now found (28, 30, 85, 87, 91, 92, 93).
| Site | Run | Tests | Passed | Failed | Not run | Run time (first) |
|---|---|---|---|---|---|---|
| with-bugs | Standard | 288 | 234 | 48 | 6 | 1,058 s (766 s) |
| with-bugs | Comprehensive security | 37 | 35 | 2 | 0 | 95 s (36 s) |
| with-bugs | Comprehensive accessibility | 18 | 6 | 12 | 0 | 78 s (45 s) |
| clean | Standard | 272 | 225 | 41 | 6 | 916 s (590 s) |
| clean | Comprehensive security | 33 | 31 | 2 | 0 | 85 s (29 s) |
| clean | Comprehensive accessibility | 16 | 12 | 4 | 0 | 61 s (21 s) |
Every run is within the 1,800 s budget. The longest, with-bugs Standard, leaves 742 s. (TFP-167’s own measurement of the same run took 1,170 s: two runs from this machine, about 10 % apart.)
The clean Standard suite was regenerated. The with-bugs Standard suite is the first run’s, test for test, apart from its page visits. The clean one comes from a fresh inspection of the live clean site, whose element ids changed and whose brand filters went from 8 inputs to 2 plus “Eco-Friendly”: 272 tests against 277. Bugs are scored on with-bugs, so the score isn’t affected.
Performance checks (navigation, LCP, CLS, TBT, page weight) now read
the page after up to 5 s more settling. Any change in them between the two
scores may come from that, not from found bugs. Measured here, none
changed: on both sites every performance check has the same outcome in both
scores. The only failure is the broken broken.png request (bug 28). There
is no LCP or CLS finding, and TBT is recorded as 0 ms.
The first score’s clean runs read some pages before they drew. Now they see what the with-bugs runs see:
button-name on the clean login
and register pages, as R15 does on with-bugs;list violation shows on both sites;So shared defects now cancel out in the diff, as the control intends.
All are in docs/decisions/data/labelled-failures.md:
<ul> (axe list);href="#"
links fail target-size;color-contrast (.float-start).Phase 40 Decision 9: the same answer key, scored with the same committed
rule (scoring-rule.md), on pages chosen by the app instead of by hand.
The first score and the re-score above stay as they were. Evidence:
benchmarks/2026-10-07-tfp175-selection/.
7233751 (TFP-172 to TFP-174 and their follow-ups)./#/product/6 became
C:/Program Files/Git/#/product/6), so the runs opened local files, not
the sites. The runner now turns that off, and the driver refuses any page
that isn’t a path.| Site | Standard’s 8 pages | Skipped | Comprehensive’s 20 |
|---|---|---|---|
| with-bugs (34 pages, 9 types) | home, /#/product/6, /#/category/hand-tools, register, contact, login, forgot-password, /#/chainsaws |
/#/rentals (content, by crawl order) |
the 9 types: 10 products (6 to 12, 1 to 3), 3 categories (with undefined), the rest one each |
| clean (46 pages, 9 types) | home, one product, /category/hand-tools, register, contact, login, forgot-password, /rentals |
/privacy (content, by crawl order) |
the 9 types: 9 products, 4 categories, the rest one each |
Against the hand-picked list:
/#/product/6), not /#/product/1.First (26fed09) |
Re-score (7577252) |
Automatic (7233751) |
|
|---|---|---|---|
| Found | 6 | 9 | 9 |
| Hinted | 3 | 4 | 4 |
| Missed | 28 | 24 | 24 |
Bug for bug, the same as the re-score:
The area counts are the re-score’s.
Where the evidence changed:
/#/product/6:
color-contrast on .text-reset (46);label on the quantity input (91);image-alt on the related products’ images (92);button-name on the +/- buttons (40)..text-reset is the same template element the re-score read by hand on
/#/product/1 (the category badge); it wasn’t re-read on product 6.
/#/chainsaws itself.| Site | Run | Tests | Passed | Failed | Not run | Run time (re-score) |
|---|---|---|---|---|---|---|
| with-bugs | Standard, 8 pages | 312 | 227 | 77 | 8 | 1,232 s (1,058 s, 5 pages) |
| with-bugs | Comprehensive security, 20 pages | 81 | 79 | 2 | 0 | 198 s (95 s, 9 pages) |
| with-bugs | Comprehensive accessibility, 20 pages | 40 | 6 | 34 | 0 | 173 s (78 s) |
| clean | Standard, 8 pages | 384 | 319 | 58 | 7 | 1,276 s (916 s, 5 pages) |
| clean | Comprehensive security, 20 pages | 81 | 79 | 2 | 0 | 225 s (85 s) |
| clean | Comprehensive accessibility, 20 pages | 40 | 25 | 15 | 0 | 168 s (61 s) |
Every failure was compared with the re-score’s on the same site, by check
and cause, with the page taken out (new-failures.mjs). A failure the
re-score already had on another page isn’t new.
| Site | New kind | Verdict |
|---|---|---|
| with-bugs | orphan check: “/#/product/6 is not linked from any other inspected page” (8) | T30, template error. Inspection read home and the category before their product cards drew, so they showed no product links. Not the selection: product 1 would have failed the same way |
| with-bugs | “form element check: Your password * [Password]” on register | U4, the duplicate id="password", now also on register |
| clean | axe target-size on the product pages’ related-product links (10) |
R17, real: now seen on both sites, so shared behaviour, not a planted bug |
| clean | axe button-name and list on register |
R10 and R16, already labelled; new only as a combination |
| clean | “link types”: a card link “Hammer More information” not found | U5, unclear. The clean catalogue was reset within the hour, so it can’t be read back. The re-score failed the same way on another card |
| clean | the footer’s GitHub link answered HTTP 500, once | U6, unclear: GitHub’s error, not the Toolshop’s |
Not failures, recorded:
#password) and forgot-password
(#email), as on login (bug 87). They’re real, but not listed bugs.| Condition | Measured | Holds |
|---|---|---|
| finds at least 9 of the 37 | 9 | yes |
| found plus hinted at least 13 | 13 | yes |
| every new failure on with-bugs is a scored bug or a labelled failure | T30 and U4 labelled; the rest are scored bugs or labelled before | yes |
| no unlabelled failure appears on the clean site | R17, R10/R16, U5 and U6 labelled | yes |
| Standard stays within 1,800 s on one worker | 1,232 s and 1,276 s | yes |
Outcome: kept. Every condition holds.
Read with it:
Today’s TestForge (4f636b1), scored on local copies of both Toolshop
versions with TFP-154’s committed rule and Phase 40’s selection. Evidence:
benchmarks/2026-10-07-phase41-local/.
practice-software-testing at
5a6ccc26, run under Docker on this machine, one version at a time.http://localhost:4200./#/product/6
for products), /#/rentals skipped;/privacy skipped.with-bugs routes by hash and clean by path, as the hosted sites do.
ng serve), where the hosted
sites are production builds. That shows as label U8 (page weight, no
robots.txt or sitemap.xml) and in axe’s selectors (Angular’s
_ngcontent attributes). Neither touches a scored bug.compose.local.yml).
On the mounted Windows folder, Laravel couldn’t write its file cache.
with-bugs ran with the file cache, which worked there. Either way the API
returns the same data.| Hosted re-score (hand-picked) | Hosted, automatic (TFP-175) | Local, automatic (TFP-180) | |
|---|---|---|---|
| Found | 9 | 9 | 9 |
| Hinted | 4 | 4 | 4 |
| Missed | 24 | 24 | 24 |
/#/product/6;So Phase 41’s checks aim at 21 missed and 4 hinted bugs a Chromium run can reach.
| Copy | Run | Tests | Passed | Failed | Not run | Run time |
|---|---|---|---|---|---|---|
| with-bugs | Standard, 8 pages | 402 | 330 | 64 | 8 | 1,118 s |
| with-bugs | Comprehensive security, 20 pages | 81 | 79 | 2 | 0 | 138 s |
| with-bugs | Comprehensive accessibility, 20 pages | 40 | 6 | 34 | 0 | 150 s |
| clean | Standard, 8 pages | 392 | 340 | 45 | 7 | 884 s |
| clean | Comprehensive security, 20 pages | 81 | 79 | 2 | 0 | 132 s |
| clean | Comprehensive accessibility, 20 pages | 40 | 25 | 15 | 0 | 122 s |
More recorded elements mean more checks.
Each local copy’s failures were compared with the same version’s hosted
TFP-175 run by check and cause, the page and the site’s address taken out
(new-failures.mjs).
| Copy | New kind | Verdict |
|---|---|---|
| with-bugs | WCAG scans: color-contrast on the footer links and .float-start |
bug 85 and R18; new only in the development build’s selectors |
| with-bugs | “every form field has an accessible name” on the product page: input[type="number"] |
bug 91, now also through Standard’s field-name check |
| with-bugs | “form element check: 1 [Ergonomic claw hammer …]” | T26, the quantity field found by its value |
| both | “page weight stays under 5 MB”; no robots.txt (and, clean, no sitemap.xml) |
U8, new: the development server, not the site |
| clean | “link types”: “Claw Hammer with Shock Reduction Grip ABCDE$13.41” not found | T31, a card’s name joined without spaces |
Every failure on either copy is a scored bug or labelled.
Phase 41’s fifteen checks and its request guard (TFP-181 to TFP-187), scored with TFP-154’s
committed rule and Phase 40’s selection, unchanged, first on the local copies
and then on the hosted sites. Code under test: f3fcf9a (src/ as at
9a9ac3b; the runs refuse uncommitted code). Read-only gate, one worker,
Chromium. Evidence: benchmarks/2026-10-13-tfp188/ (scorecards
score-local.md and score-hosted.md, the raw evidence, every run).
| Hosted re-score | Hosted, automatic (TFP-175) | Local baseline (TFP-180) | Local, Phase 41 (TFP-188) | Hosted, Phase 41 (TFP-188) | |
|---|---|---|---|---|---|
| Found | 9 | 9 | 9 | 33 | 32 |
| Hinted | 4 | 4 | 4 | 0 | 0 |
| Missed | 24 | 24 | 24 | 4 | 5 |
POST .../cdn-cgi/rum?, T33) before it
judged a link. The same breach fails the check on the hosted clean site.evidence.mjs marks shared items; Gate A’s diff finds 128 shared items
overall, none of them evidence).| Where | Run | Tests | Passed | Failed | Not run | Run time |
|---|---|---|---|---|---|---|
| local with-bugs | Standard, 8 pages | 461 | 357 | 84 | 20 | 1,235 s |
| local with-bugs | Comprehensive security, 20 pages | 122 | 119 | 3 | 0 | 209 s |
| local with-bugs | Comprehensive accessibility, 20 pages | 80 | 7 | 41 | 32 | 220 s |
| local clean | Standard, 8 pages | 451 | 387 | 46 | 18 | 1,049 s |
| local clean | Comprehensive security, 20 pages | 122 | 120 | 2 | 0 | 216 s |
| local clean | Comprehensive accessibility, 20 pages | 80 | 30 | 19 | 31 | 215 s |
| hosted with-bugs | Standard, 8 pages | 464 | 350 | 95 | 19 | 1,816 s |
| hosted with-bugs | Comprehensive security, 20 pages | 122 | 119 | 3 | 0 | 298 s |
| hosted with-bugs | Comprehensive accessibility, 20 pages | 80 | 7 | 41 | 32 | 264 s |
| hosted clean | Standard, 8 pages | 459 | 399 | 42 | 18 | 1,335 s |
| hosted clean | Comprehensive security, 20 pages | 122 | 120 | 2 | 0 | 330 s |
| hosted clean | Comprehensive accessibility, 20 pages | 80 | 30 | 19 | 31 | 272 s |
Every failure or finding new since Gate A (local) or TFP-175 (hosted) is a
scored bug or labelled (docs/decisions/data/labelled-failures.md):
| Where | New | Verdict |
|---|---|---|
| local and hosted, both versions | grouping on the category pages | R19, real, shared (now on every category page Comprehensive covered) |
| local and hosted with-bugs | positive tabindex on contact and login | R20, real, outside the answer key |
| both versions | “Unsplash” in the product photo credit | T32, a brand |
| hosted, both versions | the link-destination check’s guard breach on /cdn-cgi/rum |
T33, new: Cloudflare’s beacon |
| local and hosted with-bugs | the route check’s “routes not checked: stops after 14 s” | U9, new: a coverage note |
| local with-bugs | CLS 0.100 on home | U10, new: a lab reading at a band’s edge |
| hosted with-bugs | “More information” href="#" on the product page |
R21, new: placeholder links (R17’s elements) |
| hosted with-bugs | five older checks timed out | U11, new: host speed |
| hosted clean | “Search” 905 px down at desktop width | U12, new: 5 px past a threshold |
| with-bugs | broken.png in the image check on every page |
bugs 28 and 30 (already found) |