mr-onion

Phase 39 Measurement Report — Where the Run Time Goes (TFP-211)

Date: 2026-10-20 Ticket: TFP-211 (Phase 39 Decision 1, Acceptance Criteria 1) Evidence: benchmarks/2026-10-19-tfp211/. For each site: the crawl and selection, the three suites, and for each run its records and its raw timing. analysis.md has every number below, with its spread. Code: the timing record is 684dc0a (the reporter, RUN-38) and afa8d4f (naming a link check’s targets from the suite). The runs started on 160606e; afa8d4f changed only the summary, which is recomputed from the raw timing at that commit. Setup: Emil’s machine (Windows, i5-12450H, 12 logical cores, 40 GB). Chromium, one worker, --timeout=30000, the read-only gate. Machine load at the start: 70 Chrome processes, Ollama, Docker Desktop with Emil’s Plane stack (8 containers) running (logs/machine-at-start.txt). For Emil to lock: the targets in the last section, before TFP-212 starts.

Bottom line

Durations

“Run” is the Playwright process, timed by the driver. Each cell is the median of three runs, with min to max in brackets when they differ. Times are in seconds.

Site, suite Tests Run Loads (count) Loads (s) Settle (s) Failing tests (s) Failed steps (s) Outside tests (s)
Fixture, Standard 123 178 (177–178) 125 39 62 23 15 11
Fixture, security 20 22 19 7 10 0 0 2
Fixture, accessibility 12 16 12 4 6 0 0 2
with-bugs, Standard 483 1,486 (1,464–1,511) 563 (561–566) 202 (200–210) 608 (605–612) 468 (454–477) 137 136
with-bugs, security 122 240 (239–247) 121 43 124 (123–126) 12 0 9
with-bugs, accessibility 80 245 (237–247) 80 27 83 (78–86) 117 (113–118) 0 58
clean, Standard 472 1,231 (1,192–1,252) 474 147 (139–147) 550 (538–557) 222 (216–229) 96 97
clean, security 122 229 (223–233) 121 38 131 (124–133) 4 0 9
clean, accessibility 80 245 (228–283) 80 24 90 (84–101) 75 (70–86) 0 42

Where a Standard run’s visits go (run 1)

Loads by area, over the 8 selected pages:

Area with-bugs clean
Functional 329 244
Accessibility 72 72
Usability 55 47
SEO and content 50 50
Security (passive) 32 32
Reliability 17 18
Performance 6 6
Responsive 4 4
Cookie and session 1 1
All 566 474
Check with-bugs: tests, loads, requests, time Targets clean: tests, loads, requests, time Targets
Navigation links (navigation link health:) 24, 48, 8, 98 s 3 24, 24, 24, 44 s 2
Linked routes (TFP-165) 8, 66, 0, 115 s 12 none (path routing)  
TFP-205’s link tests and card tests 22, 22, 0, 47 s 0 loaded 21, 21, 0, 44 s 0 loaded
Link health (requests only) 8, 0, 34, 7 s 6 8, 0, 132, 8 s 30

Controls by Decision 3’s kinds (inspection, for Emil’s later look)

Inspection can’t tell a single-page app’s path routes from a multi-page site’s links, so same-origin path links are counted as their own kind, “path navigation”, not guessed.

Page with-bugs: in place / in-app / path / unclear clean: in place / in-app / path / unclear
Home 24 / 17 / 1 / 9 16 / 0 / 19 / 10
Product 3 / 14 / 1 / 12 4 / 0 / 16 / 13
Hand tools 19 / 17 / 1 / 6 17 / 0 / 19 / 6
Register 2 / 8 / 1 / 19 3 / 0 / 10 / 20
Contact 2 / 8 / 1 / 13 3 / 0 / 10 / 14
Login 2 / 10 / 1 / 10 3 / 0 / 12 / 11
Forgot password 2 / 8 / 1 / 8 3 / 0 / 10 / 9
Chainsaws (with-bugs) / Rentals (clean) 2 / 8 / 1 / 5 3 / 0 / 10 / 6

The Toolshop is one single-page app with two routings. Its in-app links are hash routes on with-bugs and path links on clean. “Unclear” is mostly form fields and plain buttons.

Proposed targets (for Emil to lock before TFP-212)

All are judged the same way. Before and after are measured in one session and interleaved, three runs each, and their medians compared. One worker, the local Toolshop copies, and the suites generated once per side, as here. Between sessions the same code drifted 14 to 19%, more than any target below, so a target can’t be judged against this report’s numbers.

  1. Gate B: shared read-only visits, the shared-page rules (TFP-212, TFP-213), and the settle and timeout tuning (TFP-215), together.
    • Standard is at least 20% faster on each copy (median).
      • Basis: Accessibility, usability, SEO, passive security and reliability load the pages 226 times on with-bugs and 219 on clean.
      • One visit per page for them is 8 visits. That saves about 218 visits at 1.45 s, about 316 s: 21% on with-bugs (1,486 s) and 25% on clean (1,231 s).
      • Functional’s read-only checks sharing the visit too would add to that. Some of these tests interact and keep their own page under Decisions 4 and 5, so 20% is a target the measurements support, not a guess.
    • Comprehensive security and accessibility are each at least 30% faster (median, each copy).
      • Basis: security loads its 20 pages 121 times, about 101 visits over one per page, at 1.39 s: about 140 s of 240 s.
      • Accessibility: 60 visits over one per page, about 84 s of 245 s.
    • Decision 7’s proof holds. The fixture gives every test the same result shared and one-page-per-test. On both copies the failure set is this report’s, test by test, apart from differences explained one by one (the sort check below included). The Toolshop score is unchanged.
    • Not met: the change is reverted, and the result recorded either way.
  2. TFP-215 alone: each value it changes must save at least 5% of the Standard median on with-bugs, or it is reverted. 5% is the within-session spread; less can’t be seen in three runs.
    • The settle cap can’t reach 5%: it was reached by at most 4 visits, 17 s.
    • The failed steps’ 5 s waits might: 137 s on with-bugs is 9%.
    • For Emil: narrow TFP-215 to the read-only assertion timeouts, and leave the settle cap at 5 s. Decision 6 already keeps what counts as settled.
  3. Gate C: functional checks by kind of control (TFP-214): at least a further 10% on Standard (median, each copy), measured against Gate B’s code.
    • Basis: functional is 244 to 329 of Standard’s loads.
    • Of those, the linked-routes check’s 66 fresh documents stay as they are by design. The navigation-link and link tests are about 46 to 70 visits.
    • The rest depends on the kinds Emil looks at (the control counts above), so this is the lowest of the three. Emil may set it after that look.
  4. Decision 13 (1 worker against the machine limit, 6 workers here): keep 40%. It stays measured on Comprehensive, every area, on the local Toolshop, after Gates B and C, and medians of three each side.
    • After Gate B a Comprehensive run should be about 100 to 160 s, so worker start-up weighs more than now. On the other side, the per-failure worker restarts (42 to 58 s of an accessibility run today) spread over the workers.
    • Nothing here argues for moving the 40%. The local copies’ API, in Docker on Windows, answered in 0.4 to 1 s while measuring. Six workers may queue on it, and Decision 13’s run will show whether they do.

Raised, not fixed