Date: 2026-10-15 Scope: the review the roadmap lists under “After Phase 42” (Emil, 2026-10-13), in the order it is done, with who does each step and what counts as done. Status: Agreed with Emil 2026-10-15. Code builds and runs only from a ticket below; planning steps happen in Cowork first.
| # | Step | Who | Ticket |
|---|---|---|---|
| R1 | Baseline: the full regression | Code | TFP-207 |
| R2 | Flakes, from R1’s evidence | Code | TFP-208 (written after R1) |
| R3 | Claims audit: every phase’s claims against code and measurements | Cowork, with Emil | tickets it finds |
| R4 | Small security items, and CI’s Node version | Code | TFP-206, TFP-209, TFP-216, TFP-217, TFP-222, TFP-242, TFP-243, TFP-258 |
| R5 | Three decisions: the Docker sandbox, app screens, scenario gaps | Cowork, with Emil | a decision record each |
| R6 | The skills relook | Cowork, with Emil | skill proposals; TFP-236 to TFP-239 |
R1 comes first so every later step works from today’s numbers. R3 needs R1’s numbers. R4 can run alongside R3. R5 builds nothing until a decision record is locked. R6 is last, so the skills describe what the review left.
Progress (2026-10-16):
post-phase-42-review-findings.md). TFP-210 is written there.decbfad to 139f8df,
02ed0ef; benchmarks/2026-10-16-tfp208/outcome.md. Its card-group
ruling is confirmed by Emil (2026-10-19), on the condition 02ed0ef meets
(Phase 42 record). Next for Code: TFP-210.4eff1dc, 23a1338;
benchmarks/2026-10-19-tfp210/. Both nightly jobs were green once on
GitHub, the unit step under pipefail.TFP-208 first, so Phase 39’s measurements and its same-results proof aren’t read through flakes and false failures. TFP-210 next, so a nightly unit run guards Phase 39 while it changes how every test reaches its page.
measurement, benchmarkcheck:template-unchanged once (against 18b47d8: expected 166/166);outcome.md committed under
benchmarks/2026-10-1x-tfp207-regression/;TFP-207 accepted (bfd900a..6d8f7e4), with three corrections made to its
outcome.md. Findings and the numbers used since:
post-phase-42-review-findings.md.
flaky, test, generate/#/category/hand-tools. The page drew a different window of
products from one load to the next: products 1 to 9 at Gate B, 6 to 14 at
R1’s inspection, and 10 to 14 gone by the time the tests ran.read-only-guard.test.ts:168, TFP-199’s beacon case (R1 run 1:
sendBeacon's POST, typed ping expected true, got false);cookie-scope.test.ts:90 (R1 run 3: a record missing from the run,
reading 'status' of undefined);comprehensive-scenarios.test.ts:85,
performance-bands.test.ts:67, api-fields.test.ts:65. Look for a
shared cause with the two above before closing them as passing;ECONNRESET at fixture start-up), probably
the same load flake as the Comprehensive security case.next build and start missed
the config’s 120 s once in R1. Either raise the timeout, or build in a
step before and only start in webServer. Pick one, name it in the
commit; TFP-210’s CI job uses the same.category.component source and say whether
the window moves by design, by data (the database or its pagination)
or by a planted bug./product/:id), not by its text, and the test passes when any
card’s link matches the pattern and loads.phase-39-run-concurrency-decision-record.md, locked 2026-10-16).check:template-unchanged: if 3 changes template output, the change is
described and the baseline is updated in its own commit, as TFP-205’s
was;TEST_PLAN.md cases for the changing-listing fixture, in the same
commit as the code.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: A listing’s repeated cards get one test, by position, not a test per card (TFP-208, ruling confirmed by the owner 2026-10-19; unit: src/lib/generate/mock/link-audits/inventory-tests.test.ts (fixture pages served locally and inspected in a real browser; the committed Toolshop data offline), src/lib/report/coverage-map.test.ts)
Expected result: A card group is 3 or more links on a page whose targets share an address but a last segment that is an identifier (Phase 40’s isIdSegment: #/product/1, /product/01M50R4WF...). Its links get no test each; the page gets one read-only test, cards linking to <pattern> on <page>: such cards are on the page, and each one there at run time, by position, is visible and named (when inspection found them visible) and has a valid href of the pattern. Their targets are loaded as one link’s would be (the inspected cards’ by link health, unchanged). The coverage map counts each card covered in context by its group’s test. Why: the Toolshop pages a category’s products with no order (paginate(9), no ORDER BY, both versions), so with-bugs’ /#/category/hand-tools showed other products from one load to the next, and R1 (TFP-207) had 5 false failures there. Proven (decision table): on a fixture listing that shows another 4 of 12 items on every load, no test per card and one cards test, which passes on the run’s other items; a link outside the cards keeps its own test; every card is covered in context; a listing whose card targets answer 404 passes its cards test and fails link health, naming the card; with no cards, the cards test fails (“a card linking to /item/:id is on /cards”). Boundary values: 2 links to one pattern are no group, 3 are; words (/category/hand-tools) are not ids. Offline, on both Toolshop copies’ selected pages: no link on the gap list, each listing’s cards (home, a product’s related products, hand-tools) have their test. check:template-unchanged against 980346f: 166 of 166 identical (the fixed inputs have no card group).
Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: A failing cards test names each failing card’s product, and every one of them (TFP-208, the owner’s condition on confirming the ruling, 2026-10-19; unit: src/lib/generate/mock/link-audits/inventory-tests.test.ts (fixture pages served locally and run in a real browser))
Expected result: The cards test checks every card before it fails, then fails once, with <n> of <count> cards on <page> failed: and a line per failing card: its position and what it showed at run time, its name (a heading in the card, else its first line of text), a price if it shows one, and its href, e.g. card 4 of 9 on /category/hand-tools ("ECO Sledgehammer", $17.75, #/product/10): does not link to #/product/:id. A passing cards test’s output is unchanged. Proven (decision table, against GEN-98’s fixture listing): cards all well-formed: passes, as before; two of the run’s 4 cards linking to no item: one failure, 2 of 4 cards on /cards failed, naming card 2 and card 4, each by its own product name and price and its href; on the code before the change the case fails (the test stopped at the first card, named only by its slot). check:template-unchanged against 980346f: 166 of 166 identical (the fixed inputs have no card group).
Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: An artifact asked for while its run is being saved is served, not refused (TFP-208; unit: src/app/api/run/route.test.ts)
Expected result: A run reads as finished once its result is set; its evidence is then moved from Playwright’s output into the run’s folder (persistRunJob). The job keeps that save as job.saving, set with no await between it and the result, and GET /api/run?runId=...&artifactId=... waits for it before resolving the artifact, so it never reads a path whose file has just been moved. Found by the regression suite on 2026-10-16 (“captures session evidence artifacts for failed generated runs”: 404 “Artifact file is unavailable”, once in three attempts). Proven (state transition, saving to saved): a job whose save moves the file at once and names its new place only later; a request in between waits and gets the file (200, its contents). Mutation check: without the wait, the same test gets 404.
Moved word for word from docs/test-plan/TST.md by TFP-253; the case’s row there links here.
Test: Real-browser unit tests run Playwright in a folder of their own (TFP-208; src/test-support/playwright-cli.ts)
Expected result: Every unit test that runs a generated spec through the Playwright CLI does it through runPlaywrightSpec: its own folder under .playwright-unit-runs/, with a config that is the repo’s with testDir set to that folder, so Playwright’s start-up walk of its test folder sees no other run’s files. R1’s (TFP-207) cookie-scope.test.ts:90 and an earlier api-fields.test.ts case failed with a report holding no test; the cause, named by the helper’s first run: the repo config’s testDir is the shared .playwright-runs, and a folder another run removed mid-walk failed it (ENOENT: no such file or directory, scandir '...'). A run whose report holds no test fails at once, naming its exit code, the report’s errors and its error output (no retry). A folder Windows still holds (EBUSY, a failed test’s video and trace) is retried and left, never failing a test; a later run removes it after an hour.
Moved word for word from docs/test-plan/TST.md by TFP-253; the case’s row there links here.
Test: A unit test that starts a run waits until the run is saved, not just finished (TFP-208; src/test-support/saved-run.ts)
Expected result: waitForSavedRun(runId, limit) returns once the job has its result and its run.json is written (no run.json.tmp left). A job’s result is set before run.json is written; a test that returned at the result left that write running into its file’s cleanup, which removes the runs folder, and the late rename failed the file as an unhandled rejection (ENOENT ... rename 'run.json.tmp' -> 'run.json', hardening.test.ts, once in TFP-208’s first full unit run, a flake no earlier run showed). All five such waits use it: hardening.test.ts, candidates.test.ts (two), promoted.test.ts, stale.test.ts.
Commits decbfad, 60aac09, 21933f7, 4c0aca0, 139f8df, then
02ed0ef (2026-10-19, Emil’s condition); runs in
benchmarks/2026-10-16-tfp208/ (recorded in a5a3e71), outcome.md for
every number.
hardening.test.ts, a test ending before its run was saved) found and
fixed. Three full unit runs on 4c0aca0: 574/574 each. On 139f8df:
575/575 three times.webServer only starts. Three
attempts on 4c0aca0 all started in time; attempt 3 found an app defect
(an artifact asked for while its run was saved: 404), fixed in 139f8df.
On 139f8df: 57/57 three times..playwright-runs
too (the race the unit tests had). Decided 2026-10-19: its own ticket, TFP-222 (R4),
before Phase 39’s TFP-155.02ed0ef: each failing card is listed with its
position, name, price if shown and href, and the test fails once naming
them all (GEN-99). Its checks: check:template-unchanged against
980346f, 166 of 166. Regression 57/57. Unit 574/575: the one failure was
performance-bands.test.ts:69 (the 3 s LCP fixture measured over 4 s, in
a run that took 1,035 s against 659 to 731 s before, on a loaded machine).
It passed alone afterwards (2 of 2). It is one of the three TFP-208 closed
as passing, so it is a timing flake under load, not yet ticketed. For
Emil.Verified (Cowork, 2026-10-19): accepted.
4c0aca0, then 57 ×3;02ed0ef’s generated test, and a TEST_PLAN case with every code change
(60aac09 is configuration only).02ed0ef’s own check runs (unit, regression, template) are recorded in
its commit body only, not as logs under benchmarks/. Accepted for a
change this small.try still waits out its own
timeout when it fails, so the speed reason given for rejecting
expect.soft applies to both. One list is the real gain.performance-bands.test.ts:69: recorded as D23 in
deferred-work-register.md, a known flake. TFP-210’s nightly run will
show its rate.Ticket: post-phase-42-review-findings.md, “Tickets”. Commit 4eff1dc;
logs and times in benchmarks/2026-10-19-tfp210/.
.github/workflows/nightly.yml: jobs unit and template-unchanged,
at 02:00 UTC and on manual dispatch, never on push. Node 24, because both
commands use --experimental-strip-types.TEMPLATE_BASELINE (980346f),
in the workflow, or a base named at dispatch. A deliberate change of
template output updates the baseline in its own commit. Rejected: the
commit from 24 hours before, which goes red for one night and then green
again with nothing recorded.npm run build is gone, so it
builds once, as 60aac09 set up: build first, webServer only starts.4eff1dc):
unit: 575 tests, 574 passed, 0 failed, 1 skipped (Windows only, with
its reason). Suite 1,123 s, job 1,181 s; 659 to 731 s locally.template-unchanged: 166 of 166 identical against 980346f, job 22 s.23a1338 (2026-10-19, Cowork’s verification): the unit step ran
under shell: /usr/bin/bash -e {0} with no pipefail, so
npm run test:unit | tee took tee’s exit code and couldn’t fail.
shell: bash on that step. Re-dispatched once (run 39317809076); the step
header reads
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}. unit green:
575 tests, 574 passed, 0 failed, 1 skipped (Windows only); suite 1,036 s,
job 1,089 s. template-unchanged green: 166 of 166 against 980346f, job
31 s. Logs in benchmarks/2026-10-19-tfp210/23a1338/.Moved word for word from docs/test-plan/TST.md by TFP-253; the case’s row there links here.
Test: CI runs the unit suite and the template check nightly and on manual dispatch, not on every push (TFP-210; .github/workflows/nightly.yml)
Expected result: Six jobs, on a 02:00 UTC schedule and on workflow_dispatch: audit (TFP-266), dead-code (TFP-259: npx knip, knip.jsonc), issue (TFP-252, TST-07), notify (TFP-217, TST-06) and these two. unit: Node 24, Chromium installed, npm run test:unit, its log kept as an artifact; with Chromium there, no real-browser test skips (TST-01). template-unchanged: full history, npm run check:template-unchanged -- <base> HEAD, the base the recorded baseline (TEMPLATE_BASELINE, 06489fd) unless dispatch names another; a deliberate change of template output updates the baseline in its own commit. Neither runs on a push. The regression workflow builds once, through test:regression:allure (TFP-208’s build first, webServer start only), no separate npm run build. Proven (2026-10-19, 4eff1dc, dispatched by hand; benchmarks/2026-10-19-tfp210/): unit green: 575 tests, 574 passed, 0 failed, 1 skipped (Windows only, with its reason), suite 1,123 s, job 1,181 s. template-unchanged green: 166 of 166 identical against 980346f, job 22 s. Regression green: 57 passed, built once, job 403 s.
Each phase record (38 to 42), the README, TEST_PLAN.md and the handover
checked against the code and R1’s numbers: claims that are false, stale or
undersold. Done when the findings are listed in a review findings doc, the
doc fixes are made from Cowork, and anything needing code is a ticket.
response.request().method() in the code check.--safe-mode if Claude Code 2.1.238 has it (check its CLI
reference), otherwise the nearest documented equivalent, named in the
commit; a stand-in test shows the flag is passed.HTTP_PROXY,
HTTPS_PROXY, NO_PROXY, NODE_EXTRA_CA_CERTS) when they are set,
and nothing else new; a test of runEnv shows it.--safe-mode, checked in
the CLI reference and --help of Claude Code 2.1.311 (this machine’s
claude, 2.1.238 no longer). The reference gives no first version, and
doesn’t say the flag works with -p and --resume; the ticket rules
out a live call, so that stays unproven, and a claude without the flag
would fail every gap-writer call. The proxy names are passed on; nothing
sets Playwright’s proxy option, so carrying a run through a proxy
isn’t shown.2036c1d): --safe-mode is in
Claude Code’s CLI reference (“CLAUDE.md, skills, plugins, hooks, MCP
servers … and auto memory do not load. Authentication, model selection,
built-in tools, and permissions work normally”); hardening.test.ts 7
passed, 1 skipped (the Windows-only .cmd case); the commit’s
seen-failing evidence meets QUALITY_GATES.md §1, its two mutants
included. Rulings (Emil, 2026-11-02):
--safe-mode. Done when it returns a gap test
and Claude Code’s log shows no CLAUDE.md, memory or hook loaded.proxy option) and the
lowercase names (http_proxy, https_proxy, no_proxy).SECURITY.md says; the changed runEnv test is accepted.2036c1d
(benchmarks/2026-11-02-tfp209-safe-mode/): 1 gap test saved at the
first attempt ($0.066), its writing and generalising processes both under
--safe-mode, so -p and --resume work with it. The debug log has
Registered 0 hooks and project memory is off, and the transcript
none of CLAUDE.md, MEMORY.md or this machine’s instructions, which an
earlier session’s transcript without the flag has. Claude Code’s built-in
hook modules still load, and user settings are still read for
permissions.ci, maintenancepr-validation.yml,
regression-test.yml and self-test.yml set node-version: 20.
node-version: 24 in all four workflows..nvmrc (or engines in package.json, named in the commit);setup-node step reading it (node-version-file).setup-node step shows Node 24.Verified 2026-11-02 (Cowork): 7671c6e’s notify job needs all four
jobs and runs if: always(); the SMTP password reaches only the send step,
in a job installed with --ignore-scripts. Nightly 41514922513’s run page
shows the summary (read in the owner’s browser): “TestForge nightly: PASS
· 7671c6e · 2026-11-02”, the run link, trigger, commit, template base
24c4449, unit 687 tests / 686 pass / 1 skipped, template 166 of 166,
audit and dead-code success. Accepted departures: needs lists all four
jobs, TST-06, nodemailer pinned at 10.1.7. Small fix for the next
commit: duration_ms prints as 1009532.209072; show it as whole
seconds or minutes. Received (owner, 2026-11-02 22:27): from
testforgeapp@gmail.com to testforgeapp+ci at 22:17, the body as on the
run page (run link, trigger, commit 7671c6e, template base 24c4449,
unit 687/686/0 failed/1 skipped, template 166 of 166, audit and dead-code
success), and its link opened run 41514922513. TFP-217 is done; TST-06
closes. A FAIL email and a cancelled run are left to the first real ones;
not forced.
needs lists all four jobs, not
the two named below: audit and dead-code came later, and a night with
either red mustn’t email PASS. The case is TST-06 (TST-05 was taken). The
send is scripts/nightly-email/send.mjs with nodemailer pinned at 10.1.7.
Sent 2026-11-02, awaiting the owner’s receipt: manual nightly
41514922513 on 7671c6e, all five jobs green; Gmail answered
250 2.0.0 OK, 1 accepted, 0 rejected; the password appears in the run’s
log only as *** (benchmarks/2026-11-02-tfp217-email/). This session
can’t read the inbox, so the received subject and body, and that the link
opens the run, are still to record.ci, enhancementnotify job;$GITHUB_STEP_SUMMARY),
so each run shows it at the top on GitHub.No secret is needed. Until Part B, GitHub’s own failure emails cover a red night (see Dependencies).
testforgeapp@gmail.com; the repository secret
TESTFORGE_SMTP_PASSWORD and the variable CI_REPORT_TO
(testforgeapp+ci@gmail.com). Part B can be built.notify job in .github/workflows/nightly.yml.
needs: [unit, template-unchanged] and if: always(), so it
runs whether the jobs passed, failed or were cancelled.testforgeapp@gmail.com, through Gmail’s SMTP server
(smtp.gmail.com, port 465, TLS).
TESTFORGE_SMTP_PASSWORD.CI_REPORT_TO, set to
testforgeapp+ci@gmail.com.
ci, not a
site’s name.TestForge nightly: PASS or FAIL, then the short commit
and the date, e.g. TestForge nightly: FAIL · 4eff1dc · 2026-10-20.
$/$/actions/runs/$;unit: the job’s result, and the counts from node:test’s summary
(tests, pass, fail, skipped, duration_ms), passed on as job
outputs;template-unchanged: the job’s result and the check’s line
(<base> -> <head>: N inputs, N identical, N different);scripts/ using a pinned nodemailer
devDependency.Rejected: an action referenced by a movable tag, which runs whatever that tag points to next, with the SMTP secret in reach.
notify job, with the reason. A missing
email then shows on GitHub as a failed send, not as silence.testforgeapp+ci@gmail.com,
and its link opens that run.
benchmarks/, as
TFP-210’s logs were.TEST_PLAN.md case (TST-05) for the workflow, in the same commit as
the workflow change.shell: bash fix: the email reports the job
results, so they must be able to fail.Moved word for word from docs/test-plan/TST.md by TFP-253; the case’s row there links here.
Test: Every nightly says how it went, on its own page and by email (TFP-217; unit: scripts/nightly-email/summary.test.mjs; .github/workflows/nightly.yml, job notify)
Expected result: After all four jobs and issue (TST-07), whatever their results, notify writes the summary to the run’s page and sends it from testforgeapp@gmail.com through Gmail (port 465, TLS, the TESTFORGE_SMTP_PASSWORD secret, on the send step only) to the CI_REPORT_TO variable. Subject TestForge nightly: PASS or FAIL, the short commit and the date; FAIL when any job didn’t succeed or didn’t report. Plain-text body: the run’s link, trigger, commit, template base; unit’s result, counts and failing tests (at most 20, then “and N more”); the template check’s line; audit and dead-code. Counts that can’t be read are said to be, never left out. A failed send fails the job. Proven (equivalence partitioning, boundary at 20 names; each assertion seen failing by mutant, 15 of 15): PASS; FAIL for each of the four jobs failed, cancelled or skipped, and for a job not reported; 20 and 23 failing tests; a cancelled job with no counts; a green job whose counts weren’t found; the log’s parser on node:test’s output, colour codes included; a send with no password or recipient fails before connecting; a refused send fails with the server’s reason and without the password; a rejected recipient fails. Live (2026-11-02, nightly 41514922513, benchmarks/2026-11-02-tfp217-email/): sent, Gmail 250 2.0.0 OK, the password only as *** in the log; the owner received it at 22:17 with the body as on the run page, and its link opened the run. A FAIL email and a cancelled run are left to the first real ones (owner, 2026-11-02).
flaky, runbenchmarks/2026-10-16-tfp208/outcome.md, “Raised, not
fixed”).decbfad, TST-02). The app’s own runs still share it.
testDir is the shared
.playwright-runs:
gap-run.ts;stale-recheck.ts.ENOENT ... scandir, and the run
executes nothing..playwright-runs/, with a config whose
testDir is that folder. This is TST-02’s pattern, through the same
helper or one shared with it.TEST_PLAN.md case, in the same commit as the code.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: Each app run in a folder of its own (TFP-222, R4; unit: src/lib/run/run-folders.test.ts (the executor, in a real browser))
Expected result: TFP-208 found that a Playwright run whose config’s testDir is the shared .playwright-runs walks all of it as it starts, and a folder another run removes mid-walk fails it (ENOENT ... scandir): the run executes nothing. TST-02 fixed it for the unit tests; the app’s runs (the executor, a gap run, a stale re-check) still used that config. Each now runs in its own folder under .playwright-runs/ (run-<id>, gap-..., recheck-<id>-...) with a config that is the repo’s with testDir set to that folder (src/lib/run/run-folders.ts, which TST-02’s helper now shares). A run’s folder holds its spec, reporters, report, timing file and login state; its artifacts stay where they were (artifacts-<id>), so persistence, saved runs and diffing read what they read before. The folder is removed when the run is finalized (a gap run’s and a re-check’s when they end; a run whose login fails, at once), retried while Windows holds it and then left, never failing a run. At each run’s start, a login state no active run owns is removed at once (Decision 5, as before, now also inside run folders), and a run, gap or re-check folder over an hour old whose run isn’t active is removed. Proven (state transition, real browser): two app runs started at once (startRunJob) while a third run’s folder (144 subfolders) is made and removed over and over in .playwright-runs: each executes and passes its 3 tests, with no ENOENT or scandir in its output; each held its spec and config in run-<id>, and once finalized its record is saved and its folder gone. Proven (the stale re-check, real browser; unit: src/lib/site-suites/stale.test.ts, added at verification, Cowork 2026-10-21): a run through /api/run whose reused contact test fails on a changed page: its re-check ran in one folder of its own, recheck-<runId>-... under .playwright-runs/, holding test-<runId>-regenerated.spec.ts and a config whose testDir is the folder; its tests executed (every contact result reported is the re-run’s, some passing); and the folder is gone after. On the code before, 3 of 3 attempts failed: a run executed 0 of 3, with ENOENT: no such file or directory, scandir '.playwright-runs/artifacts-...'. check:template-unchanged against ae4a1ff: 166 of 166 identical. Run time (one Standard run on local with-bugs within TFP-211’s spread) is measured in Gate C’s session (C2 against F), as the record decides.
2a3e234): removeOrphanedArtifactFolders
in persistence.ts, called by the executor after the run- sweep. Each
candidate is checked against its own run’s record, as a folder is only ever
its own run’s. Beyond the ticket: a folder whose saved run’s record can’t be
read or has a malformed entry is kept, as is one whose run is saved in the
real runs/ while a test points RUNS_ROOT elsewhere, and names that aren’t
artifacts-<a run id> are left alone.
TEST_PLAN.md RUN-48.run, maintenance.playwright-runs (505 artifacts-<id> folders and about 40 run files,
2026-08-28 to 2026-10-16).runs/<id>/
and removes its artifacts-<id> folder (TFP-198). But a run that is
interrupted before it’s saved leaves artifacts-<id> behind. TFP-222’s
sweep removes only run-, gap- and recheck- folders, so the clutter
builds up again, slowly.artifacts-<id> folders older than
an hour that belong to no active run and that no saved run points to.artifacts-<id> is removed.TEST_PLAN.md case; unit and regression suites pass.runs/ (D29, the retention policy).2a3e234,
run-folders.test.ts gave 14 passed, 0 failed, 0 skipped. Read the keep
rules in the diff: age, active run, run-id name, the real runs/ check,
an unreadable record (kept when its run folder exists) and a malformed
entry (kept) all match the commit’s claims, and the red-first failure is
quoted in its body. Claude Code’s other figures (unit 701, regression 72,
template 166, 13 mutants killed) were not re-run here.runJobs keeps finished runs
until then; and a run.json that parses but isn’t a record’s shape counts
as pointing nowhere, so its folder can be removed. Only a hand edit makes
that shape, and keeping such folders would also keep real orphans of runs
without artifacts. Either can be reopened as a small ticket on Emil’s word.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: The start-of-run sweep removes orphaned artifacts-<id> folders (TFP-242; unit: src/lib/run/run-folders.test.ts, directly and through the executor in a real browser)
Expected result: Removed: a folder older than the hour of a run never saved, of a saved run whose evidence moved out, and of a saved run pointing only at a folder whose name starts the same or at another drive. Kept: an active run’s folder however old; one under the hour; one a saved run’s record points into, as persistRunJob leaves it when a move and its clean-up both fail, including a path differing only in the drive letter’s case; one whose saved run’s record can’t be read or has a malformed entry; one whose run is saved in the real runs/ while a test points RUNS_ROOT elsewhere; and anything not named artifacts-<a run id>. Starting a run sweeps a planted orphan on the real .playwright-runs. Thirteen mutants of the rule, each killed. Not proven: the retry-then-leave for a folder Windows holds, which is removeRunFolder’s, unchanged and shared with the run- sweep. Known limits: a run active in another server process sharing .playwright-runs isn’t in this one’s runJobs, and a folder’s time changes only with its direct children: the run- sweep has the same two. A record’s pointer to a file already gone (TFP-243’s duplicate link) still keeps its folder. runJobs also holds finished runs until the server restarts, so a finished run’s leftover waits for that. A run.json that parses but isn’t a record’s shape, which only a hand edit makes, counts as pointing nowhere.
afterAll hook gets its error-context.md attached twice, as
the trace fixture is set up again for the hook, and every shared visit ends
with one. The hook stays, as TFP-241 needs it, so the template is unchanged.
createExecutionArtifacts lists a path once, and persistRunJob points a
repeated path’s links at the one moved copy. TEST_PLAN.md RUN-49.
evidence/ folder, after a dry run.
Claude Code measured 35 records written, 35 run.json.bak-tfp243 backups
(kept for now), every link pointing at a file that exists, and each record
otherwise identical to its backup. Every evidence link opened in two
reports: 9577f3c5 (TFUS-004, 9 of 9) and fb75f1ac (50 of 50). Saved
reports still show the file twice; that’s D29’s.25d1d12
on a Linux clone: artifacts.test.ts and persistence.test.ts gave 19
passed, 0 failed, 0 skipped, and tsc exit 0. With the two source changes
removed, the listing test and the save test fail (17 passed, 2 failed) and
the same-name test passes, as the report said; its proof against a wrong fix
wasn’t re-run here, nor the real-browser shared-visit test, the full unit
and regression suites, or the template check, which are Claude Code’s.
The diff matches the claims: resolved paths de-duplicated in
createExecutionArtifacts, one move per old path in persistRunJob.
The two unit failures on the earlier full run (a 2.68 s page-speed reading,
a 6,782 ms wait against 5,000 ms) passed alone and on the final run:
load, not proven.bug, report.playwright-runs clean-up.error-context.md twice.
runs/<id>/evidence/.TEST_PLAN.md case.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: A failed test’s evidence is listed once, and a file listed twice keeps both links (TFP-243; unit: src/lib/report/artifacts.test.ts, src/lib/run/persistence.test.ts; in a real browser, through a shared visit: src/lib/generate/shared-visits.test.ts)
Expected result: Playwright 1.59.1 attaches a failed test’s error-context.md a second time when an afterAll hook follows it, and every shared visit ends with one. The report lists a file once, by its full path, and two files of one name in different folders stay two. Saving moves a file listed twice once and points both links at the moved copy.
*_HELPER(S) constants and build*Helper(s) functions under
src/lib/generate). Of their top-level declarations, the 166 inputs left 5
unreached: the journey helpers, the route helpers and tfSameSite, which
only Comprehensive writes and the check never ran. 6 inputs come after the
166, whose keys and outputs are unchanged: selected journeys and a
hash-route link (both audit modes), Comprehensive security and
accessibility. helper-coverage.test.mjs fails on any declaration no
output carries, and on any tf* helper the source declares that no output
does (which catches tfLeakSignature, outside the naming convention).
172 of 172 identical against HEAD and the nightly’s 24c4449, so the
baseline doesn’t move. TEST_PLAN.md GEN-107 (⚠️). A ref from before
134b183 can no longer be compared (exit 2, named).
getText() and removes them from
getFullStart(), so the comments before each one are dropped: 20
distinct lines (controls.ts 8, READ_ONLY_GUARD_HELPER 7, also part of
CONTROL_HELPERS, SPELLING_HELPERS 3, LAYOUT_HELPERS 2) are in no
fixed output. controls.ts:245, above
tfWords, is one: it is in no suite the template generator writes, so
TFP-250’s premise (“changed generated output”) held for :282, not
:245. But READ_ONLY_GUARD_HELPER’s 7 reach saved suites in full
through site-suite reuse (withGuardHelper) and the gap writer, which
the check doesn’t run. Decided (Emil, 2026-11-03): keep those
comments through the move, as TFP-269 (workflow backlog, Part 4).normalizeContentProfile drops every
journey whose path revisits a page, as its path list is de-duplicated
(normalizeStringList). All 10 in the fixed profile end in a form that
posts to its own page, so all 10 are dropped; a user selecting such a
journey gets no journey test. The journeys input adds one journey that
doesn’t revisit, so the helpers are reached either way. Now TFP-268
(P1, workflow backlog, Part 4; Emil, 2026-11-03).24c4449 (TFP-250): no name is left
in controls.ts.4c4d560, on a
Linux clone: the template-check tests 8 of 8 passed, 0 skipped;
check:template-unchanged 24c4449 gave 172 inputs, 172 identical, 0
different. With the coverage test limited to the first 166 inputs, both
its tests fail, naming the route helpers and tfSameSite among the
unreached declarations, as reported. Not re-run here: the full unit
suite, tsc, ESLint and the planted-helper proofs, which are Claude Code’s.test, generatecontrols.ts’s generated helpers changed
generated output.check:template-unchanged still reported every fixed input identical,
because none of its 166 inputs reaches those helpers.tf* helpers and their kin), and which fixed input reaches each.controls.ts’s
helper makes check:template-unchanged report the input as different.TEST_PLAN.md case.controls.ts, “(Emil, 2026-10-12)” and “Emil’s look at the
control counts”) are reworded to the decision vocabulary (“owner
decision, Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: Every helper the generator can write into a suite is reached by a fixed input of the template check (TFP-258; unit: scripts/template-unchanged/helper-coverage.test.mjs, scripts/template-unchanged/compare.test.mjs)
Expected result: A helper is any export const *_HELPER(S) or export function build*Helper(s) under src/lib/generate, 13 today. Each is parsed into its top-level declarations, and each declaration, comments inside it included, must appear in at least one of the 172 fixed outputs, ignoring indentation and with or without its visits routed through tfGoto. The 166 left 5 declarations unreached: the journey helpers, the route helpers, and tfSameSite. 6 inputs were added after the 166, whose keys and outputs are unchanged: selected journeys and a hash-route link, in both audit modes, and Comprehensive security and accessibility. A second test catches a tf* helper outside that convention by name (tfLeakSignature, written inline in a Comprehensive test): every one the generator’s source declares must be declared in some output. Proven: with the 166 the first test fails naming the 5; a tf* declaration in a template no input writes fails the second by name; a reworded comment in the route helpers is reported by the check in the 2 hash-route inputs, and one inside controls.ts’ tfCheckNavLinks in 8 (4 of them among the 166). Not covered: comments between a helper’s declarations: 20 distinct lines, in controls.ts (8), READ_ONLY_GUARD_HELPER (7, also part of CONTROL_HELPERS), SPELLING_HELPERS (3) and LAYOUT_HELPERS (2), are in no fixed output, as the shared-visit pass moves declarations without the comments before them. They are in no suite the template generator writes, controls.ts’ line above tfWords included; but READ_ONLY_GUARD_HELPER’s 7 are written in full by site-suite reuse (withGuardHelper) and the gap writer, which the check doesn’t run.
Done 2026-10-20 (Emil): all three are locked. Their build order is in
deferred-work-register.md.
phase-44-scenario-gaps-decision-record.md, tickets
TFP-223 to TFP-227. It also answers D6.phase-45-docker-sandbox-decision-record.md,
tickets TFP-228 to TFP-231.phase-46-app-screens-decision-record.md, tickets
TFP-232 to TFP-235. It covers the site lookup, the confirmation form,
promotion and the active switch.Each gets a decision record locked with Emil before any code:
Done when each record is locked; its tickets then join the backlog.
Done 2026-10-20 (Emil): r6-skills-relook-2026-10-20.md.
write-playwright-test, and crawl-and-classify-pages,
which settles D12);sre-testing, sre-agents,
sre-overload, sre-release);docs/CLAUDE_SKILLS.md: resynced.docs/CLAUDE_SKILLS.md, Martin’s draft QA/SRE skills and the Cowork skills
read against what Phases 38 to 42 and the review changed. Done when each
skill is kept, updated or retired, and new skills for repeatable procedures
seen in this work are proposed for Emil to save.