mr-onion

Known-Bug Benchmark Report: the Toolshop (TFP-154)

Date: 2026-10-06 Ticket: TFP-154 (known-bug-benchmark-backlog-2026-10-06.md, Decisions 1 to 4) Evidence: benchmarks/2026-10-06-toolshop-known-bugs/runs/, holding each run’s suite, Playwright report and records, plus run-tfp154.ts, diff-runs.mjs and which-run.mjs. The scoring rule and page set (scoring-rule.md) were committed before the first run (91a7141). Setup: code at 26fed09, Emil’s machine, Chromium, read-only gate (TESTFORGE_INTERACTIVE=0), one worker.

Bottom line

Runs

Site Run Pages Tests Passed Failed Not run Run time
with-bugs Standard, all 10 areas 5 288 235 47 6 766 s
with-bugs Comprehensive security (passive) 9 37 37 0 0 36 s
with-bugs Comprehensive accessibility 9 18 7 11 0 45 s
clean Standard, all 10 areas 5 277 234 39 4 590 s
clean Comprehensive security (passive) 8 33 33 0 0 29 s
clean Comprehensive accessibility 8 16 16 0 0 21 s

The pages: home, a category (hand tools), a product (Combination Pliers), contact, login, register, forgot-password and rentals, as hash routes on with-bugs (/#/...) and paths on clean, plus /#/chainsaws on with-bugs. Standard took the first 5. Comprehensive covered all.

Scores (Decision 3)

“Present” is the hosted spot-check: yes (checked by hand) or ? (never checked). A ? on a miss means the bug may not be on the hosted site.

ID Bug (TestForge’s wording) Area Score Evidence (run: check: message) Present
2 Category sort order reversed functional missed no check reads sort results ?
3 “Chainsaw” link goes to a missing page functional found Standard: “link health: linked routes draw a page, not the app’s “not found” view”: “/#/chainsaws (draws the same as /#/testforge-missing-…, a route that cannot exist)”, on each of the 5 pages yes
4 Product names right-aligned (category) visual missed no layout check ?
19 Login email padding visual missed no layout check ?
22 Forgot-password button font visual missed no layout check ?
23 Forgot-password email padding visual missed no layout check ?
26 Product images on home don’t show visual missed nothing says an image doesn’t show; axe names .img-fluid (a different image) for alt text yes
27 Price filter doesn’t send its maximum functional missed no check reads filter requests ?
28 Header logo is a broken image functional found Standard: “no same-origin request fails while the page loads”: “HTTP 404 …/broken.png” on /; Standard and Comprehensive: axe image-alt on .navbar-brand > img[src$="broken.png"] yes
29 “Home” link points to contact functional missed link health checks the target draws, not that it’s the right page yes
30 Search icon is a broken image functional found Standard and Comprehensive: axe image-alt on .grid-title:nth-child(7) > img[src$="broken.png"], the second broken.png on / the spot-check read as this bug likely
31 “Contact” misspelt content missed no spelling check yes
33 “Sort” label misspelt content hinted Standard and Comprehensive: axe select-name and “every form field has an accessible name”: the sort <select> has no name on /#/ and the category (with-bugs dropped its aria-label="sort"); nothing names the misspelling yes
34 “Search” button misspelt content missed no spelling check yes
40 Quantity +/- don’t change the quantity functional missed no check clicks +/- and reads the result ?
42 “Related products” misspelt content missed no spelling check yes
45 Product buttons overlap visual missed no layout check ?
46 Product badge text black visual missed axe flags .text-reset contrast only once the product page has drawn; the run read it before ?
49 Rentals title centred visual missed no layout check ?
53 Registration tab order accessibility missed no check follows form tab order ?
54 Registration phone placeholder misspelt content missed no spelling check; register is outside Standard’s 5 ?
58 Registration button width visual missed no layout check ?
59 Registration dropdown fonts visual missed no layout check ?
60 Contact subject list shows error entries functional hinted Standard: the contact form’s check (T27) quotes the options, “Error 101: Subject not found”; Comprehensive: axe lists #subject for manual contrast review; neither names the entries as the fault yes
62 Contact message placeholder misspelt content missed no spelling check yes
63 Contact attachment label font visual missed no layout check ?
66 Contact button width visual missed no layout check ?
67 Contact dropdown fonts visual missed no layout check ?
75 Products API exposes stock internals security missed the passive checks don’t read API responses ?
80 Log file reachable from the web security missed the passive checks don’t probe known paths ?
82 Sequential, guessable IDs security missed no ID-pattern check ?
85 Link contrast below 4.5:1 accessibility found Standard and Comprehensive: axe color-contrast (serious) on the footer links p > a[target="_blank"], on every with-bugs page; the clean site passes ? (found here)
87 Login fields have no labels accessibility found Comprehensive: “every form field has an accessible name” on /#/auth/login: “form field named only by its placeholder …: #email (“Your E-mail *”)”, and #password (axe accepts a placeholder as a name) yes
91 Quantity field has no label accessibility missed the field (<input type="number" data-test="quantity">, no label) wasn’t drawn when the checks ran yes
92 Product images have no alt accessibility hinted axe image-alt on /#/product/1 names only img, the header logo; the product images are flagged only on a settled page yes
93 Home search form has no label accessibility found Standard and Comprehensive: axe label on .form-control and “form fields on /#/ with no accessible name: … input[type=”text”]” yes
94 Home filters not in a fieldset accessibility missed axe has no rule for it yes

Counts

Area Scored Found Hinted Missed
functional 8 3 1 4
visual 13 0 0 13
content 6 0 1 5
accessibility 7 3 1 3
security 3 0 0 3
all (read-only gate) 37 6 3 28

By run:

Against the prediction:

Not scored (57)

What the misses point to

Would catch them Bugs Standard area it points to
A visual or layout check (alignment, padding, width, fonts, images that don’t render) 4, 19, 22, 23, 26, 45, 46, 49, 58, 59, 63, 66, 67 none today: usability is the nearest
A spelling and wording check on visible text, placeholders and labels 31, 34, 42, 54, 62 (and 33’s cause) seo/content
Checks that a control did the right thing: a sort’s order, a filter’s request, a link’s destination, +/- 2, 27, 29, 40 functional
Generated checks that wait for a hash route to draw, as the crawl does since TFP-165 91, 92’s images, 46 every area (cross-cutting)
Keyboard order through a form, and grouping that axe doesn’t judge 53, 94 accessibility
Read-only security probes: API response contents, known paths, ID patterns 75, 80, 82 security

Other with-bugs failures, read by hand

Each is in docs/decisions/data/labelled-failures.md.

Failure Count Verdict
Form lookups before the form is drawn (“no form on /#/auth/login”) 1 (clean: 7) T25, template error
Quantity input located by its value, getByText('1') 1 T26, template error
Contact form classed api, located by its joined labels 1 T27, template error
“password field in the URL” on a form whose password has no name 1 T28, template error
<a href="#" role="button"> “Categories” looked up as a link 5 T29, template error
Meta description, canonical, robots.txt, sitemap 12 R13, real (not a listed bug)
#filters link with no target 2 R14, real (not a listed bug)
Show-password toggle with no name (axe button-name, login and register) 2 pages R15, real: the same as R10 on the clean site
Password field check lands on a duplicate id="password" 1 U4, unclear
:visited styling (also on clean) 5 U2
“Links are distinguishable from surrounding text” 4 U3

The axe failures on every with-bugs page (color-contrast on the footer links, image-alt on the logo) are bugs 85 and 28. The select-name and field-name failures on home and the category are bugs 33 and 93.

Limits

Re-score after TFP-167 (2026-10-07)

The first score above stays as it was. This re-score used the same page set, the same committed rule (scoring-rule.md), the same driver and both Toolshops, read-only, one worker, Chromium. The code was 7577252: every generated page visit now waits for the network to go quiet, for at most 5 s. Evidence: benchmarks/2026-10-06-toolshop-known-bugs/runs-tfp167/.

The score, beside the first

  First (26fed09) Re-score (7577252)
Found 6 9
Hinted 3 4
Missed 28 24
Area Scored Found Hinted Missed
functional 8 3 2 3
visual 13 1 1 11
content 6 0 1 5
accessibility 7 5 0 2
security 3 0 0 3
all 37 9 4 24

What moved, and why. Each change is on the product page or home, which the first run read before it drew:

ID First Re-score Evidence now
91 missed found axe label (critical) on input, and “form fields on /#/product/1 with no accessible name: input[type=”number”]” (Standard and Comprehensive); the clean product page passes
92 hinted found axe image-alt on the product page’s product images (a[href$="#/product/2"] > .card-img-wrapper > .card-img-top, …); the clean product page passes
46 missed found axe color-contrast (serious) on .text-reset on /#/product/1. Read by hand, that is the category badge “Pliers”, dark text rgb(33, 37, 41) on grey: the black badge text. The clean page passes
26 missed hinted axe image-alt on home’s product card images (a[data-test="product-1"] > .card-img-top), the bug’s elements. The message is about alt text, not images that don’t show
40 missed hinted axe button-name on #btn-decrease-quantity and #btn-increase-quantity, the bug’s buttons. It names their missing names, not that they don’t change the quantity

The other scores are unchanged:

Against the prediction: all 7 “likely” bugs are now found (28, 30, 85, 87, 91, 92, 93).

Runs and the 30-minute budget

Site Run Tests Passed Failed Not run Run time (first)
with-bugs Standard 288 234 48 6 1,058 s (766 s)
with-bugs Comprehensive security 37 35 2 0 95 s (36 s)
with-bugs Comprehensive accessibility 18 6 12 0 78 s (45 s)
clean Standard 272 225 41 6 916 s (590 s)
clean Comprehensive security 33 31 2 0 85 s (29 s)
clean Comprehensive accessibility 16 12 4 0 61 s (21 s)

Every run is within the 1,800 s budget. The longest, with-bugs Standard, leaves 742 s. (TFP-167’s own measurement of the same run took 1,170 s: two runs from this machine, about 10 % apart.)

The clean Standard suite was regenerated. The with-bugs Standard suite is the first run’s, test for test, apart from its page visits. The clean one comes from a fresh inspection of the live clean site, whose element ids changed and whose brand filters went from 8 inputs to 2 plus “Eco-Friendly”: 272 tests against 277. Bugs are scored on with-bugs, so the score isn’t affected.

Performance checks (navigation, LCP, CLS, TBT, page weight) now read the page after up to 5 s more settling. Any change in them between the two scores may come from that, not from found bugs. Measured here, none changed: on both sites every performance check has the same outcome in both scores. The only failure is the broken broken.png request (bug 28). There is no LCP or CLS finding, and TBT is recorded as 0 ms.

The control is even now

The first score’s clean runs read some pages before they drew. Now they see what the with-bugs runs see:

So shared defects now cancel out in the diff, as the control intends.

With-bugs failures that match no bug

All are in docs/decisions/data/labelled-failures.md:

The automatic selection (TFP-175, 2026-10-07)

Phase 40 Decision 9: the same answer key, scored with the same committed rule (scoring-rule.md), on pages chosen by the app instead of by hand. The first score and the re-score above stay as they were. Evidence: benchmarks/2026-10-07-tfp175-selection/.

Runs

Site Standard’s 8 pages Skipped Comprehensive’s 20
with-bugs (34 pages, 9 types) home, /#/product/6, /#/category/hand-tools, register, contact, login, forgot-password, /#/chainsaws /#/rentals (content, by crawl order) the 9 types: 10 products (6 to 12, 1 to 3), 3 categories (with undefined), the rest one each
clean (46 pages, 9 types) home, one product, /category/hand-tools, register, contact, login, forgot-password, /rentals /privacy (content, by crawl order) the 9 types: 9 products, 4 categories, the rest one each

Against the hand-picked list:

The score, beside the first two

  First (26fed09) Re-score (7577252) Automatic (7233751)
Found 6 9 9
Hinted 3 4 4
Missed 28 24 24

Bug for bug, the same as the re-score:

The area counts are the re-score’s.

Where the evidence changed:

Run times

Site Run Tests Passed Failed Not run Run time (re-score)
with-bugs Standard, 8 pages 312 227 77 8 1,232 s (1,058 s, 5 pages)
with-bugs Comprehensive security, 20 pages 81 79 2 0 198 s (95 s, 9 pages)
with-bugs Comprehensive accessibility, 20 pages 40 6 34 0 173 s (78 s)
clean Standard, 8 pages 384 319 58 7 1,276 s (916 s, 5 pages)
clean Comprehensive security, 20 pages 81 79 2 0 225 s (85 s)
clean Comprehensive accessibility, 20 pages 40 25 15 0 168 s (61 s)

New failures, read by hand

Every failure was compared with the re-score’s on the same site, by check and cause, with the page taken out (new-failures.mjs). A failure the re-score already had on another page isn’t new.

Site New kind Verdict
with-bugs orphan check: “/#/product/6 is not linked from any other inspected page” (8) T30, template error. Inspection read home and the category before their product cards drew, so they showed no product links. Not the selection: product 1 would have failed the same way
with-bugs “form element check: Your password * [Password]” on register U4, the duplicate id="password", now also on register
clean axe target-size on the product pages’ related-product links (10) R17, real: now seen on both sites, so shared behaviour, not a planted bug
clean axe button-name and list on register R10 and R16, already labelled; new only as a combination
clean “link types”: a card link “Hammer More information” not found U5, unclear. The clean catalogue was reset within the hour, so it can’t be read back. The re-score failed the same way on another card
clean the footer’s GitHub link answered HTTP 500, once U6, unclear: GitHub’s error, not the Toolshop’s

Not failures, recorded:

Decision 9’s keep rule, applied as written

Condition Measured Holds
finds at least 9 of the 37 9 yes
found plus hinted at least 13 13 yes
every new failure on with-bugs is a scored bug or a labelled failure T30 and U4 labelled; the rest are scored bugs or labelled before yes
no unlabelled failure appears on the clean site R17, R10/R16, U5 and U6 labelled yes
Standard stays within 1,800 s on one worker 1,232 s and 1,276 s yes

Outcome: kept. Every condition holds.

Read with it:

Local baseline (TFP-180, Phase 41 Gate A, 2026-10-08)

Today’s TestForge (4f636b1), scored on local copies of both Toolshop versions with TFP-154’s committed rule and Phase 40’s selection. Evidence: benchmarks/2026-10-07-phase41-local/.

What the copies are

The score, beside the hosted ones

  Hosted re-score (hand-picked) Hosted, automatic (TFP-175) Local, automatic (TFP-180)
Found 9 9 9
Hinted 4 4 4
Missed 24 24 24

Run times

Copy Run Tests Passed Failed Not run Run time
with-bugs Standard, 8 pages 402 330 64 8 1,118 s
with-bugs Comprehensive security, 20 pages 81 79 2 0 138 s
with-bugs Comprehensive accessibility, 20 pages 40 6 34 0 150 s
clean Standard, 8 pages 392 340 45 7 884 s
clean Comprehensive security, 20 pages 81 79 2 0 132 s
clean Comprehensive accessibility, 20 pages 40 25 15 0 122 s

New failures, read by hand

Each local copy’s failures were compared with the same version’s hosted TFP-175 run by check and cause, the page and the site’s address taken out (new-failures.mjs).

Copy New kind Verdict
with-bugs WCAG scans: color-contrast on the footer links and .float-start bug 85 and R18; new only in the development build’s selectors
with-bugs “every form field has an accessible name” on the product page: input[type="number"] bug 91, now also through Standard’s field-name check
with-bugs “form element check: 1 [Ergonomic claw hammer …]” T26, the quantity field found by its value
both “page weight stays under 5 MB”; no robots.txt (and, clean, no sitemap.xml) U8, new: the development server, not the site
clean “link types”: “Claw Hammer with Shock Reduction Grip ABCDE$13.41” not found T31, a card’s name joined without spaces

Every failure on either copy is a scored bug or labelled.

Re-score with Phase 41’s checks (TFP-188, Gate B, 2026-10-13)

Phase 41’s fifteen checks and its request guard (TFP-181 to TFP-187), scored with TFP-154’s committed rule and Phase 40’s selection, unchanged, first on the local copies and then on the hosted sites. Code under test: f3fcf9a (src/ as at 9a9ac3b; the runs refuse uncommitted code). Read-only gate, one worker, Chromium. Evidence: benchmarks/2026-10-13-tfp188/ (scorecards score-local.md and score-hosted.md, the raw evidence, every run).

The score, beside the earlier ones

  Hosted re-score Hosted, automatic (TFP-175) Local baseline (TFP-180) Local, Phase 41 (TFP-188) Hosted, Phase 41 (TFP-188)
Found 9 9 9 33 32
Hinted 4 4 4 0 0
Missed 24 24 24 4 5

Run times

Where Run Tests Passed Failed Not run Run time
local with-bugs Standard, 8 pages 461 357 84 20 1,235 s
local with-bugs Comprehensive security, 20 pages 122 119 3 0 209 s
local with-bugs Comprehensive accessibility, 20 pages 80 7 41 32 220 s
local clean Standard, 8 pages 451 387 46 18 1,049 s
local clean Comprehensive security, 20 pages 122 120 2 0 216 s
local clean Comprehensive accessibility, 20 pages 80 30 19 31 215 s
hosted with-bugs Standard, 8 pages 464 350 95 19 1,816 s
hosted with-bugs Comprehensive security, 20 pages 122 119 3 0 298 s
hosted with-bugs Comprehensive accessibility, 20 pages 80 7 41 32 264 s
hosted clean Standard, 8 pages 459 399 42 18 1,335 s
hosted clean Comprehensive security, 20 pages 122 120 2 0 330 s
hosted clean Comprehensive accessibility, 20 pages 80 30 19 31 272 s

New failures and findings, read by hand

Every failure or finding new since Gate A (local) or TFP-175 (hosted) is a scored bug or labelled (docs/decisions/data/labelled-failures.md):

Where New Verdict
local and hosted, both versions grouping on the category pages R19, real, shared (now on every category page Comprehensive covered)
local and hosted with-bugs positive tabindex on contact and login R20, real, outside the answer key
both versions “Unsplash” in the product photo credit T32, a brand
hosted, both versions the link-destination check’s guard breach on /cdn-cgi/rum T33, new: Cloudflare’s beacon
local and hosted with-bugs the route check’s “routes not checked: stops after 14 s” U9, new: a coverage note
local with-bugs CLS 0.100 on home U10, new: a lab reading at a band’s edge
hosted with-bugs “More information” href="#" on the product page R21, new: placeholder links (R17’s elements)
hosted with-bugs five older checks timed out U11, new: host speed
hosted clean “Search” 905 px down at desktop width U12, new: 5 px past a threshold
with-bugs broken.png in the image check on every page bugs 28 and 30 (already found)