Status: Draft, proposed in a planning session 2026-10-26. Nothing here is locked. The CI tickets (TFP-244 to TFP-249) need their own decision record before they are built; TFP-244 is that record. The rest are small and can be picked up between Phase 39’s tickets, after TFP-157’s verification.
Ticket numbers: TFP-244 to TFP-257, allocated against main at
8e93998 (highest then: TFP-243); TFP-259 and TFP-260 added 2026-10-27
(TFP-258 was taken meanwhile); TFP-261 and TFP-262 added 2026-10-27; TFP-268 to TFP-270
added 2026-11-03 (highest then: TFP-267); TFP-271 added 2026-11-04; TFP-272 added 2026-11-08. Re-check before committing; renumber if
another session took any of them.
Why this exists:
docs/WORKING_AGREEMENTS.md) and
some of what it adopts needs code, config or a larger edit.security, bug/api/run
executes the code it is given. Two things contradict it today, both
measured 2026-10-27 on 6bd7fc0:
npm run dev and npm start run
next dev / next start with no hostname, and Next.js defaults to
0.0.0.0; the dev server announces a Network address. Anyone on the
same network who can reach port 3000 can call the API.Host or
Origin, and POST /api/run parses its body with req.json()
whatever the Content-Type. A POST carrying a foreign Origin and
Content-Type: text/plain — the kind of request any web page can send
without a CORS preflight — was accepted (202) and the code in it ran.
Whether a given browser will send it to localhost depends on the
browser (current Chrome asks the user first for public sites reaching
local addresses; not every browser does). A missing Host check also
leaves DNS rebinding open.-H 127.0.0.1 on dev, dev:development and
start (the self-test scripts already do);/api/* route (Next’s proxy/middleware):
refuse when Host isn’t localhost, 127.0.0.1 or [::1] on the
server’s port; refuse when Origin is present and isn’t the same
origin; refuse a POST, PUT or DELETE that isn’t
Content-Type: application/json. Refusals are 403 with a message that
says why;Origin,
text/plain) gets 403 and runs nothing (checked with a marker the code
would have written); a foreign Host gets 403; a same-origin JSON
request from the UI still gets 202;SECURITY.md (TFP-254) describes the server as
it is after this ticket, so it is committed with or after this fix,
never before.6d1e854 (the fix, RUN-43), then SECURITY.md
and CONTRIBUTING.md (c284e82). The startup criterion reworded to
“no LAN address” (owner decision, 2026-10-27).Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: The API answers only the tester’s own pages on this machine (TFP-261, P0; unit: src/proxy.test.ts; regression: tests/regression/testforge.request-guard.spec.ts, the built server)
Expected result: npm run dev, dev:development and start bind to 127.0.0.1, so the port isn’t reachable on the machine’s LAN address. A request guard (src/proxy.ts, Next’s proxy, on every /api/* route) refuses with 403 and the reason: a Host that isn’t localhost, 127.0.0.1 or [::1] (DNS rebinding); an Origin that isn’t the same origin, port included (another site’s page in the tester’s browser); a POST, PUT, PATCH or DELETE that isn’t Content-Type: application/json, which a page can’t send cross-site without a preflight. Measured 2026-10-27 before it: a POST /api/run with a foreign Origin and text/plain was accepted (202) and its code ran. The port isn’t checked: a loopback Host comes only from this machine, and the Origin check compares the whole origin. Proven (decision table): each loopback name with and without a port goes on, other names or none are refused; the same origin goes on, another site, another port or null is refused; every write method needs JSON (a charset parameter allowed), a read needs no type. Proven (regression, the built server, with a positive control): in one test, the 2026-10-27 request (another site’s Origin, text/plain) and a same-origin text/plain one each get 403 with their reason, then the same suite sent as the UI sends it (same Origin, JSON) gets 202 and writes its marker file; once that control run has finished, and 5 s more, neither refused request’s marker exists (a run either had started would have been queued before the control’s); a request naming another host gets 403. Observed: npm run dev on port 3105 listened on 127.0.0.1:3105 only, answered on 127.0.0.1 and localhost, and got no connection on the LAN address. Its startup still prints “Network: http://127.0.0.1:3105”: Next.js names the bound address, loopback, and no LAN address appears, which meets the criterion as reworded (owner decision, 2026-10-27: “no LAN address”, not “no Network line”). Next’s Proxy bypasses (TFP-266, Next 16.4.3): the regression spec sends the shapes that can apply to an API route from another host (GHSA-267c-6grr-h53f and GHSA-26hh-7cqf-hhc6’s .rsc and segment-prefetch variants, a route parameter in the query, spellings of the path) and a POST to /api/run.rsc from another site, each pinned to the status measured: 403 from the guard for 8, Next’s 308 to /api/runs for 3 (which the guard then refuses), the 404 page for /API/runs. GHSA-492v-c6pp-mqqv can’t apply: src/app/api has no dynamic route. GHSA-6gpp-xcg3-4w24 can’t apply: next.config.ts has no i18n. These cases pin the guard; they don’t show the upgrade fixed it: every shape gets the same status on 16.2.3 and 16.4.3 (2026-10-31), as the advisories describe pages, not route handlers. Without the guard (16.2.3): /api/runs?_rsc=1 200 with the runs, the .segment.rsc path 204, the diff path its own 400, and 5 others 404 instead of 403.
5a93ecc, 1a9c2f5 (2026-10-31). Independently
verified 2026-10-31 (Cowork, Linux, fresh clone at 1a9c2f5): next
16.4.3; npm audit --omit=dev: 0 vulnerabilities; the 2026-10-27 attack
replayed against npm run dev on 16.4.3 — a foreign-origin text/plain
POST, a same-origin text/plain POST, a rebinding Host, a null
Origin, /api/runs?_rsc=1 from a foreign origin, and an RSC-prefetch
request with a rebinding Host — all 403, the run’s marker file never
written, the LAN address not answering; src/proxy.test.ts 5 of 5; no
generator file changed. Rulings (Emil, 2026-10-31):
dompurify), and ESLint’s (minimatch, glob). npm audit on
1a9c2f5 here shows 27 by package: 7 critical, 1 high, 19 moderate —
the builder’s “8” counted critical and high only. Re-checked when Allure
or ESLint publish fixes; no overrides.70a0ad3, before the upgrade, and several pushes
since haven’t triggered a check. If it hasn’t rescanned by the next
session, turning Dependabot alerts off and on in the repo’s settings
forces one (owner’s go-ahead first).
2026-10-31, 06:40 (owner’s go-ahead): Dependabot alerts turned off
and on in the owner’s browser (security updates left off). Half a
minute later “Dependency files checked” still read 2026-10-28; the
dependency graph itself had re-read package-lock.json on 2026-10-30.
Re-checked after v0.2.0; if it is still stale a day later, ask GitHub
support.useRun.ts effect removed in
5a93ecc): a unit test of the derived selection — the first group
when none or a vanished one is selected, the chosen one otherwise —
lands with TFP-267’s first commit.security, dependenciesnpm audit on 211a4a7 (2026-10-30) gives 23 advisories: 2 critical, 17
high, 3 moderate, 1 low; 7 of them in packages the app runs
(--omit=dev). The one that matters is next 16.2.3, with 33
advisories fixed by 16.4.3 (not a major version):
src/proxy.ts), so a bypass could undo
it;next/image, next/og,
use cache, Server Actions): no file in src/ uses them. Loopback
binding (TFP-261) limits who can reach the server, but not a page in the
tester’s own browser if the guard is bypassed.
The rest are transitive: through next (sharp, postcss, zod, nanoid,
source-map-js, baseline-browser-mapping) and through dev tooling
(Allure’s handlebars and adm-zip, axios, markdown-it, and others),
each with a non-major fix.next and eslint-config-next to 16.4.3 or later (read
node_modules/next/dist/docs/ for changes, per AGENTS.md);npm audit fix for the rest, without --force; anything it can’t fix
without a major version is listed, not forced;npm audit --omit=dev --audit-level=high fails the
nightly (with shell: bash), so the app’s own dependencies can’t drift
back unnoticed;npm audit --omit=dev reports no critical or high advisory;src/proxy.test.ts pass on the
new Next, and a request through each bypass’s documented shape that
applies to an /api/* route is still refused (or the ticket says why a
shape can’t apply);npm run dev start, bound to 127.0.0.1;cad75ad, 1551810, 24c4449, b65020a, e8a52c7
(2026-10-31); CodeQL moved to advanced setup in 31eae23 (5,000 alerts to
16; none under benchmarks/ or examples/). Verified 2026-10-31
(Cowork, Linux, e8a52c7): check:template-unchanged 1a9c2f5 e8a52c7:
4 of 166 differ, and every changed line read: the link selectors’
.replace(/["\\]/g, '\\$&') (escapes " and \; the old one escaped
nothing) and the helper comment’s reworded provenance — nothing else;
link-audit, persistence, runs-route and SEO-selection tests 77 of 77,
the stray-folder skip included. Rulings (Emil, 2026-10-31):
persistence.ts, path injection) is fixed, not dismissed: add
a containment check CodeQL recognises — the resolved run path must stay
inside the runs root — beside the UUID check, so the scanner keeps
guarding that file; it closes on the next scan.skills/ ships in the
plugin; the pattern is built from caseIdCore, a deliberate caller
option); #4, #5, #9, #11 used in tests; #5891, #5892 used in tests
(the local comparison fixture, dev only).e8a52c7 stands (its
“8” was npm’s total on 30 Oct; the 19 moderates are newer advisories).js/bad-code-sanitization, medium, opened
2026-11-08 with the CLI (TFP-245). Each points at one expression of
the tfShared.rerun(...) wrapper in
src/lib/generate/shared-visits.ts:351: wrapped, indent and
lineIndent(close).JSON.parse(JSON.stringify(...)) request copy
(src/cli/run.ts:40). The only value carried is Comprehensive’s
pageCap, which is written into the suite’s header comments
(src/lib/generate/comprehensive.ts:139, :148). From there the
whole suite counts as tainted.Number(scope.pageCap) at the two comment lines. It
guards nothing real, and only CodeQL’s next scan could show whether
it clears the alerts.security, generate, cibenchmarks/ (frozen copies of generated suites). GitHub’s own
production-code filter (autofilter:true, excluding tests and generated
files) leaves 13. Triage, by reading each one’s code on 211a4a7:
src/lib/generate/mock/link-audits/detail-tests.ts (lines 23, 41, 61,
94) and experience-tests.ts (213). Inside the template literal,
.replace(/"/g, '\\"') writes .replace(/"/g, '\"') into the suite,
which in the suite is " replaced by ": nothing is escaped, and
backslashes aren’t either. A link whose href contains " or \
produces an invalid selector, so its link test fails on TestForge’s own
error, not the site’s. This is where most of the benchmark alerts come
from (thousands of copies).src/lib/generate/mock/index.ts:117 (alert #10): hostname.includes("demoqa.com")
also matches demoqa.com.example; use an exact host or a .demoqa.com
suffix;src/lib/run/persistence.ts:181 (#1, path injection): every API
caller already checks isValidRunId (UUID) first; also check inside
runJsonPath, so a future caller can’t skip it;src/lib/generate/mock/journey-tests.ts:26 (#3): the attribute name
escapes : only; escape it properly for CSS. It is then quoted as a
JS string, so this is selector correctness, not code injection.src/lib/introspect/dom-snapshot.ts 517, 650, 680 and
src/app/api/crawl/route.ts:188 (“incomplete URL scheme check”): the
crawl follows same-origin http(s) links only, and these lists skip
inert links; nothing is rendered or executed from them;src/lib/generate/benchmark-quality.ts:28 (“bad HTML filtering
regexp”): a heuristic that scores generated code, not a filter;scripts/comparison-fixture.mjs 412, 435: the local test fixture,
dev only;skills/ helper scripts (#4, #5, #7, #8, #9,
#11): test-only code.benchmarks/ and examples/ are evidence, never edited.href containing ", and one containing \, are found and checked);
the three hardening items; then — owner decision — move CodeQL from
default to advanced setup with a committed config that ignores
benchmarks/** and examples/** (default setup can’t exclude paths), and
dismiss the by-design alerts with their reasons." and \ (unit and real-browser);check:template-unchanged shows only the escaping lines changed;
TEMPLATE_BASELINE re-recorded in its own commit;autofilter:true) shows
no open alert that isn’t dismissed with a reason;Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: A link with no text is found by its href, whatever the href holds, and generated selectors and hosts are matched exactly (TFP-267; unit: src/lib/generate/mock/link-audits/href-escaping.test.ts, in a real browser; demoqa-host.test.ts; journey-tests.test.ts)
Expected result: The link checks build a[href="..."] inside the generated suite. Inside the generator’s template literals '\\"' wrote '\"' into the suite, which there is just ", so the replace did nothing (CodeQL: “replacement of a substring with itself”). An href holding " broke the selector, one holding \ matched nothing, and the check failed on TestForge’s own error, not the site’s. Fixed on 11 lines in four files: the ticket’s five, and coverage-tests.ts (4), experience-tests.ts:214 and reveal.ts (1 each), which it didn’t list; format.ts’s escapeForDoubleQuotedSelector was already right. Proven (real browser): textless links with a ", a \, both, and a \ at the end are found; on the old code those 4 fail and an href with neither passes. Also: DemoQA’s suites go only to demoqa.com and hosts under it, not demoqa.com.example; a journey step’s attribute name is escaped for CSS whatever it holds, not just :. Template: 4 of 166 inputs change, 54 selector lines and, with TFP-250’s helper wording, 4 comment lines, and nothing else.
Locked 2026-10-27 as Phase 47 (
phase-47-ci-integration-decision-record.md). The record’s Decisions and Acceptance Criteria govern; TFP-244 is done (the record itself). The tickets below stand with the record’s changes: a new P0, TFP-263, moves the crawl and generate logic intosrc/lib/pipeline/first; TFP-245 calls it and opens no server; TFP-246 usesdiffRuns; TFP-247 publishes to GitHub’s container registry. Order: after Phase 39, before Phase 43.
.github/workflows/./api/crawl, /api/generate, then POST /api/run, which
returns a 202 job that the UI polls). Nothing sets a process exit code;
the report is JSON and HTML only; site suites and run history live in a
local data folder that a CI runner wipes; ownership confirmation (Phase 43)
is designed as a local, interactive step.@playwright/test). Save it, commit it to the
target’s repo and run npx playwright test in that repo’s CI. This gives
pass/fail on a frozen suite, and loses everything that makes TestForge
TestForge: honest coverage, the gap list, suite reuse and regeneration on
changed pages, run diffing, and the gates. It is worth saying when asked;
it is not the product answer.decision, ci/api/run’s code execution accepted under that model. Running on a
target’s CI runner is neither local nor the deferred SaaS, and it changes
where state lives and what “failed” means. Those are owner decisions and
need recording before anything is built.docs/decisions/phase-47-ci-integration-decision-record.md, locked;2c4b6f3, verified (Cowork, 2026-11-08) after
the fixes below. First pass: on a Linux clone (2
cores), the uncommitted files on 6b617b2: an unreachable URL exits 2;
a config with an unknown area exits 3, naming practices[0]; the CLI,
junit, config, run and pipeline tests pass, except four server tests
that fail on every Linux run (an interrupt’s stop, stopAllServers, a
port already in use, and, intermittently, “once stopped nothing
listens”). Measured: stopServers returns after the shell exits and
before the killed process group has gone; the server’s port still
answers at +0 ms and is free by +100 ms. CI runners are 2-core Linux, so
this is the platform that matters. A fifth, the IPv6 loopback test,
fails where ::1 can’t be bound (EAFNOSUPPORT). Fixes: stopServers
waits, bounded, until each process group has gone; the IPv6 test skips,
with its reason, where ::1 isn’t available. Read: Not run never fails
the build and becomes <skipped> in JUnit; Blocked fails it and
becomes <error>; --report-only turns test failures into exit 0.
Claude Code’s Toolshop runs, the UI-versus-CLI byte comparison, the port
sampling and the full suites weren’t re-run here (no Docker here).
Rulings (Emil, 2026-11-08): (1) criterion 2’s “clean: exit 0”
contradicts Decision 5 (no baseline: any failing test fails) on a site
with 46 real failures; it becomes “clean: exit 1, JUnit counts equal
the report’s; with --report-only, exit 0” (the record’s criterion 2,
amended), and “clean exits 0” moves to TFP-246 (clean, with its own
baseline); (2) Blocked fails the build, Not run doesn’t; (3) the
default practices are all ten areas; (4) the stop order (stdin, SIGTERM,
then the tree after 30 s); (5) page-data.json in --out, redacted as
every other file (Decision 7), and serve-toolshop.mjs --ready-port; (6)
Claude Code’s CLAUDE.md line on jobs past 1.5x their projection.
Fixed after verification (2026-11-08): stopServers waits, up to
5 s, until each server’s process group has gone (Windows: its shell);
the IPv6 test skips, with its reason, where ::1 can’t be bound;
page-data.json’s redaction has its own test. Re-verified (Cowork,
2026-11-08) on the same 2-core Linux clone, the fixed files on
6b617b2: the server and exit tests 5 runs in a row, each 6 passed, 0
failed, 1 skipped (# SKIP IPv6 loopback ::1 can't be bound on this
machine (EAFNOSUPPORT)); all CLI, pipeline and introspect tests 40
passed, 0 failed, 1 skipped; check:template-unchanged 6b617b2: 172
inputs, 172 identical; tsc exit 0. Claude Code’s clean --report-only
run (exit 0, 470 tests, 46 failures, 0 errors, 18 skipped, matching the
report) wasn’t re-run here.ci, run, reportnpx testforge run --url <base> --mode standard
--out <dir>), plus a config file for anything longer than a flag
(login config, page cap, selected practices);src/lib modules the API routes use — no second
implementation of generation or running;0 passed (no new failures); 1 test failures; 2
couldn’t run (target unreachable, crawl found nothing, browser failed);
3 invalid configuration;--out: report.json (the existing report), the HTML export,
a JUnit XML file, the executed suite.spec.ts, and evidence for
failures;README.md section.report.json’s; with --report-only, exit 0 (amended 2026-11-08,
Emil: clean exits 0 with its baseline in TFP-246);check:template-unchanged’s inputs).9dca554, verified (Cowork, 2026-11-08)
after the rulings below. Re-verified on a Linux clone, the
updated files on 653aaf2: the CLI, diff, persistence and runs-route
tests 78 passed, 0 failed, 1 skipped (IPv6, its reason given);
check:template-unchanged 653aaf2: 172 identical; tsc exit 0. Read:
pageTypeTitles replaces a whole path only (not inside /en/product/1
or /product/1.html) and keeps exact paths for a type any one run tested
twice; latest refuses a verdict with exit 2 or new failures, and
takes exit 0, a first run, or a UI run. Claude Code’s clean-twice runs
across two Toolshop starts (exit 1, then 0 with 0 new and 46 known)
weren’t re-run here. First pass: --baseline <run.json | latest> compares
through diffRuns, which gains failures (known, new, fixed); only new
failures fail the build; --out gains run.json and baseline.json.
Verified on a Linux clone, the uncommitted files on 653aaf2: the
CLI, diff and diff-route tests 52 passed, 0 failed, 1 skipped (the IPv6
test, its reason given); check:template-unchanged 653aaf2: 172
identical; tsc exit 0. Read: diffFailures keys on (browser, title);
Failed and Blocked count as failing; a failing duplicate title wins over
a passing one. Claude Code’s Toolshop cases (with-bugs twice 0, one fixed
0, one planted 1 naming only TFSE-040, a page type changed 0, clean
twice 0 on one start) and its unit run (781 of 781) weren’t re-run here.
Rulings (Emil, 2026-11-08, as recommended): (1) before the commit, a test is
matched by its page type, not its exact path: /product/<id> gets a new
id on every Toolshop start, so a fresh start in each CI job (Decision 2)
shows 4 “new” failures that aren’t; titles are compared with each page’s
path replaced by its page-type pattern, as site-suite reuse groups
pages, and the clean-twice case is re-run across two starts; (2)
latest skips a run whose verdict was failing (each run.json records
its verdict), so a failure can’t become known by being saved; (3)
criterion 3 is met: the changed type was regenerated at generation
(RUN-29) and only it, and the stale recheck ran with nothing to do
(CI-17 says so); (4) run.json and baseline.json join Decision 7’s
outputs, and /api/runs/diff gains failures (an addition); (5) the
current run’s titles are redacted the same way before comparing, so a
title quoting a login value matches its redacted baseline. Built
(Claude Code, 2026-11-08): (1) pageTypeTitles in src/lib/run/diff.ts,
over both runs’ pages, used for failures only (the diff’s other lists are
unchanged); (2) verdict (exitCode, baselineRunId, newFailures) on
the saved run and in --out/run.json (with firstRun); “failing” read
as: failed on a new failure against its baseline, couldn’t run, or
failed without --baseline; a first run (--baseline latest with none
yet), which fails on its own failures, stays a baseline (Decision 5), and
a run with no verdict (the UI’s) is accepted; (5) diffRuns(..., {
redact }). A second fresh-context review’s fixes: a path is matched
whole (not after /en, nor before .html); a type of which either run
tested two pages keeps exact paths (merged, a failure moving between the
two would read as known); a verdict that can’t be saved is said and the
reports are still written. Left for Emil: once a real new failure
lands, latest stays on the last accepted run until the failure is
fixed, or accepted by giving that run’s run.json as --baseline; two
processes sharing one data folder can race on a record’s write (TFP-198’s
one-process assumption).ci, site-suites--data-dir (or TESTFORGE_DATA_DIR) pointed at a workspace folder
that the pipeline restores and saves (cache or artifact), per the
TFP-244 decision;--baseline <run.json>: compare with the run diff and fail only on new
failures; report fixed, still-failing and new separately;7b65bc4 (its record 3adaccb),
verified in a planning session (2026-11-12), on main by fast-forward
(PR #30). First run on GitHub’s runner (Release Image, pull
request, 3adaccb): build 46 s, the image check 1 min 46 s, every
check held (the fixture 123 planned, 113 executed, 104 passed, 9
failed, 10 not run, as locally; the verdict run exit 1; the
unreachable URL), 1237 MB by docker image inspect, the publish step
skipped. PR Validation and CodeQL green on 3adaccb. Nothing is
published yet: the first push to ghcr.io waits for v0.4.0 at Phase
47’s completion (owner decision, 2026-11-12). Two notes from the
verification go to TFP-248 (its Notes).
Built 2026-11-12 (Claude Code), on the branch tfp-247-docker-image.
Unit 830 of 830, 0 skipped (821 plus the 9 new).
Dockerfile on Ubuntu 24.04 (CI’s runner pin, so TFP-284 moves both),
base images pinned by digest: Node 24.14.1 copied from the official
image, Playwright 1.59.1 from the lockfile (npm ci --omit=dev), its
Chromium headless shell 147.0.7727.15 (every launch in src/ is
headless); scripts/image/check-pins.mjs fails the build if any of
them drifts (CI-24). .dockerignore lets in only what mr-onion run
loads. .github/workflows/release-image.yml: a v* tag builds, runs
scripts/image/verify-image.mjs on the built image (CI-25), then pushes
that same image as ghcr.io/emil3663/mr-onion:<version>; a pull request
touching the image, or a manual run, builds and verifies without
pushing. Measured locally (Windows 11, Docker Desktop 29.4.0):
build 2 min 41 s, 1.74 GB unpacked; verify-image.mjs every check
held, and failed 4 of 5 on an image with its browsers moved; Acceptance
Criteria 4’s Toolshop comparison (CI-26) identical on clean
(exit 1, 470 / 46 / 0 / 18, 17ec8e9) and with-bugs (exit 1,
481 / 84 / 0 / 20, 5ca6b7a). A fresh-context review of the diff
found the new test missing from test:unit (a lost space, which also
dropped src/cli/config.test.ts) and a manual run from a tag able to
publish; both fixed, with the pull-request paths widened to everything
the image holds; the image rebuilt and re-verified. Left as found: the
system libraries playwright install --with-deps takes from Ubuntu are
unpinned (the Dockerfile says so); a pull request’s run stamps its
merge commit. Not shown at build time: the workflow on
GitHub’s runner (shown above, on PR #30), and any push to ghcr.io,
which makes the image public (left for v0.4.0, above).ci, releaseverified-release-pipeline skill), so a green release could have gone
red.9b5e573 and 04f9e08, on main by
fast-forward (PRs #31 and #32). Gate A was verified by a planning session
and passed (owner ruling, 2026-11-13; the Phase 47 record, under Gate A).
a388eaf adds the gate’s weekly schedule and its dispatch before each
version tag, which the ruling asked for.
js/incomplete-sanitization) was in the summary’s escaping of test
titles: a title’s backslash was escaped and its pipe wasn’t. 04f9e08
fixes it, shown red first. PR #32 merged once every check passed,
that one included. CLAUDE.md now says a pull request is green only
when gh pr checks passes every check on its head commit.9b5e573,
e406694, a388eaf and 04f9e08.04f9e08.action/) is a composite action around the image. It
runs the image as the job’s own user, attaches the reports, writes a
job summary, and ends the step with the run’s exit code. The image is
an input, defaulting to ghcr.io/emil3663/mr-onion at the action’s
own version; nothing is published before v0.4.0..github/workflows/action-gate.yml) builds the image in
the job and starts the local Toolshop there (toolshop.sh start,
start-toolshop.ps1’s steps for a runner without WSL).9b5e573): clean report-only exit 0;
clean against it exit 0 (0 new, 46 known); with-bugs exit 1; a URL
nothing answers on exit 2. The reports and artifacts were there,
and every file was the job user’s (CI-29). Release Image’s
verification held on the same commit, its new checks included.verify-image.mjs requires the fixture’s verdict run to fail a test
and exit 1 (CI-25). Each was shown red live in WSL.docs/CI_INTEGRATION.md and
action/examples/deploy-preview.yml.mode default overrode the config file’s; a signal-killed run gave a
misleading message; and a Gate A check could end without its
summary. It also found three limits, now in the guide.scripts/image/verify-image.mjs should assert that the fixture run
fails at least one test and exits 1.ci, docsuses: emil3663/testforge/action@<version>)
wrapping the image: inputs for URL, mode, baseline and config;
uploads the HTML report and JUnit results; writes a job summary;docs/CI_INTEGRATION.md: setup, secrets, what fails the build, what
doesn’t run in CI and why, and the stopgap described above;ci, ownership1a4f18b, 6d82c2e, 18d1e1c).
Independently verified 2026-10-27 (Cowork, read-only, on a fresh clone
at 82421c9):
src, README.md, docs/QUALITY_GATES.md,
CLAUDE.md and TEST_PLAN.md, only the seven test-fixture values and
the two helper-template lines are left, each listed in a commit body —
met;skills/ are credit lines —
met;.ts files compile to identical output
with comments stripped (TypeScript transpileModule,
removeComments), 1906d52 → 18d1e1c. Template strings aren’t
comments, so this also covers helper code no fixed input reaches
(TFP-258’s blind spot) — met;check:template-unchanged 1906d52 18d1e1c: 166
inputs, 166 identical, 0 different — met;plugin.json parses and all 12 SKILL.md frontmatters
are valid here; the load itself was shown by the builder on CLI
2.1.311 (12 skills listed, two invoked) — met, load checked by the
builder only;benchmarks/ or docs/decisions/ changed.docs, choreWORKING_AGREEMENTS.md now defines owner decision,
confirmation required, independent review and, in product code,
the tester.8e93998, excluding “Saint Martin” fixtures):
src/: 81 lines in 56 files, all comments and test descriptions; no
user-visible string names anyone. The candidate API already takes the
approver as data (approve’s by). Rewrite authority comments
(“offered for Emil’s approval” → “offered for the tester’s approval”)
and provenance (“decided by Emil 2026-10-06” → the decision reference);
test fixtures may keep a name as a value or use "tester".README.md (1 line), docs/QUALITY_GATES.md (2: “Emil’s machine” →
“the local dev machine, with a real browser”), TEST_PLAN.md (27:
provenance → decision references). CLAUDE.md’s one line was done in
the planning pass that wrote this backlog.skills/: 31 lines in 16 files. “How Emil invokes this skill” →
“How to invoke this skill” in every SKILL.md and in the template note in
the READMEs. Credit lines (Martin’s proposals and branch, the plugin
author) stay.benchmarks/, git history, decision records’ own ruling
lines, docs/decisions/ citations (planning sessions update those as
they touch each record), already-committed promoted library entries’
approved.by (a record of who approved).git grep -nE "\b(Emil|Martin)\b" -- src README.md docs/QUALITY_GATES.md
CLAUDE.md TEST_PLAN.md returns only test-fixture values and credit lines, each
listed in the commit body;SKILL.md heading contains a person’s name;npm run test:unit and check:template-unchanged unchanged (comments
only; template output byte-identical);toolingCLAUDE.md. A Claude Code Stop hook makes the cheap checks unskippable.tsc --noEmit and eslint on changed files
on this machine. If the pair takes under 30 s, add a project Stop hook in
.claude/settings.json that runs eslint on changed files and tsc,
and blocks the turn from ending on failure. If it’s slower, eslint only,
and say so.SessionStart lock-file hook is unchanged..claude/settings.json affects every
session on the repo: confirmation required.scripts/hooks/stop-check.sh (818936f, the type-error case shown
once), wired in as a project Stop hook (owner decision the same day),
the SessionStart hook unchanged.c44f1e8
(feat(ci): open, update and close a nightly failure issue (TFP-252)).
An issue job in nightly.yml (only issues: write and
actions: read; skipped when a run is cancelled; notify waits for it)
runs scripts/nightly-email/issue.mjs, no model: a red night opens
“Nightly failing” or comments on the open one, the next green run comments
and closes it, and the email links it. A branch’s runs use “Nightly failing
on cinightly.yml, a step on failure that opens or updates one GitHub
issue (“Nightly failing”) with the failing tests, the run link and the
commit range since the last green run — deterministic, no model;Moved word for word from docs/test-plan/TST.md by TFP-253; the case’s row there links here.
Test: A red nightly opens or updates one failure issue, and the next green one closes it (TFP-252; unit: scripts/nightly-email/issue.test.mjs; .github/workflows/nightly.yml, job issue)
Expected result: After the four check jobs, whatever their results unless the run was cancelled (then it is skipped, and the email says so), issue (its own job, with issues: write and actions: read; never the SMTP password) runs notify.mjs issue, no model. A red night (the email’s verdict, nightlyPassed) with no open issue titled Nightly failing opens one; with one open, comments on it; a green night closes it with a comment naming the run; a quiet green night only reads. The report: the run’s link, the short commit, each job’s result, the failing tests (as the email names them) and the commits since the last run of nightly.yml that concluded green on the branch, with GitHub’s compare link (at most 20 listed, then “and N more” from GitHub’s total); no green run, a green run that can’t be looked up, or a range that can’t be listed, is said with gh’s own reason, and the issue is opened all the same. “Green” is the run’s own conclusion, which counts issue and notify too, so a range can be longer than it need be, never shorter. Test names and commit subjects are in a text fence longer than any run of ~ inside it, so an @name in one notifies nobody. A branch’s runs use Nightly failing on <branch>, so a test dispatch can’t touch main’s. The open issue is found by exact title through the list API, not search, so one opened a minute earlier is found. The email links the issue (opened, updated, closed), or says the issue job couldn’t do its part. Proven (unit, offline; gh played by fake-gh.mjs, a JSON state, not GitHub): the plan’s four cases; red, red, green, green through notify.mjs: one issue opened, then commented on, then closed, then left alone, an issue with another title untouched; a branch’s red night opens its own issue and leaves main’s; a failed green-run lookup still opens the issue; a ~~~~ name, 300 commits, gh’s 404 and 502 reasons; the email’s line for each action and for an issue job that failed or was cancelled. Against HEAD’s notify.mjs the command tests fail (unknown command: issue); with the email’s line removed, its two tests fail; each review fix undone (a fixed fence, the listed count, stderr not captured, a lookup that throws) fails its test. Proven live (2026-11-04, branch tfp-252-check, four manual dispatches, branch deleted after): run 41741173103 green, issue job none, no email line; an empty commit, then run 41743846846 with template-base=435d78f (template check 170 of 172, so red) opened #17 “Nightly failing on tfp-252-check” with the run, each job’s result and the 1-commit range from the green run’s bc54e7d, email “Failure issue: opened …/issues/17”; run 41746797189, the same, commented on #17 and opened none, “updated”; run 41749704644 green closed #17 with “Green again: …”, “closed”. One issue opened in all, closed at the end; none for main. Email lines read from bodies rebuilt from each notify job’s logged inputs; each sent subject matched the log. Not proven: the cancelled-run skip (workflow only). Gap seen live, closed (Emil, 2026-11-04): a failure outside the unit job was named by its result alone. A red template check now carries its own line (**Template check:** N inputs, N identical, N different, or that it can’t be read), as the email does; audit and dead-code have no line of their own. Unit-proven (fails without the change: 2 tests), not re-run live.
TEST_PLAN.md by area and slim its rows0bf35c5: nine area files under
docs/test-plan/ (HLT, GEN, CRW, INT, RUN, RPT, EXP, SAV, TST), each its
old §3.x text under a heading and a link back; TEST_PLAN.md keeps
everything else, with §3 as an index (335,601 to 19,365 bytes); living
docs and five code comments point at the area files; dated records keep
their historical mentions. Verified from the uncommitted files against
c44f1e8’s TEST_PLAN.md, with Cowork’s own script: 205 case rows and
205 IDs before, 205 and 205 after, none left in TEST_PLAN.md; none
missing, extra or duplicated; every row byte-identical; LF only; all nine
index links resolve and each area file links back. Claude Code’s
repo-wide link check (199 files, 19 links into the plan, no new broken
link) wasn’t re-run here. Step 2, slimming the rows: done and verified
2026-11-04, committed with this status (docs(test-plan): slim case rows
and move their evidence to the records (TFP-253)): 166 rows over 400
characters now read ID | case | short expected result with an [Evidence]
link | status; each one’s original Test and Expected text moved word for
word under a <ID> evidence heading in its ticket’s record (8 without a
ticket at the end of the Phase 17 or Phase 34 record); the 39 short rows
untouched. Verified from the uncommitted files against 0bf35c5, with
Cowork’s own script (its own cell splitting, aware of code spans): 205
case IDs before and after, none missing, extra or duplicated; longest row
399 characters, none over 400; every status cell unchanged; all 166
evidence links land on exactly one heading of that name, and both original
cells are word for word in the section under it; no line deleted in any
decision record. Claude Code’s 20-row spot-check, repo-wide link check and
the reviewer’s two reads of the 98 hand-written short results weren’t
re-run here; three slim rows read (GEN-95, GEN-51, INT-09) say what their
evidence says. The test-plan-doc skill is a user-level skill: updating it to the
new layout is the owner’s (proposed from a planning session).docs, contextTEST_PLAN.md is 298 KB in one file; 163 rows
exceed 400 characters and the longest (RUN-42) is 5,923. A session that
needs one case pays for all of them, and most of each row is a
measurement narrative that already lives in a decision record or under
benchmarks/. The information is already filed; the plan should point
at it, not repeat it.TEST_PLAN.md keeps the feature-status table, the
known limitations and an index; each case prefix moves to its own file
under docs/test-plan/ (GEN.md, RUN.md, RPT.md, HLT.md, …).
A ticket names the area it touches, so a session reads one file.CLAUDE.md gets one line:
read the handover, then only the test-plan area and record section the
ticket names.TEST_PLAN.md#… anchors still
resolve, or are updated;test-plan-doc skill updated to the split layout.TEST_PLAN.md.Status: Built a1b4b46; verified 2026-11-02 (Cowork, Linux, fresh clone):
npx knip exit 0 (one harmless .css hint); tsc --noEmit clean;
check:template-unchanged 769afb4 a1b4b46 166 of 166 identical; one
playwright-core (1.59.1, deduped under @playwright/test and
@axe-core/playwright); .gitignore keeps /runs/ and the commit holds
nothing under runs/; the nightly’s dead-code job runs npx knip with
no pipe, so it can fail. Rulings (Emil, 2026-11-02): dist/ stays out
(nothing produces it); knip.jsonc accepted (its comments explain each
exception); PAGE_ANNOTATION left as over-exported; the .venv
patterns go from eslint’s ignore list in the next commit; knip as a
dev dependency accepted; the slower unit run (22 min) is one data point.
Done 2026-11-02: a manual nightly (run 41506182823, on a1b4b46)
shows the dead-code job green on GitHub.
The config is knip.jsonc, not knip.json, as the false positives are explained in
comments. playwright-core is declared at ^1.59.1, the range of
@playwright/test, so the tree keeps one copy. knip 6 found two things
the 2026-10-27 scan hadn’t: TOP10_SOURCE (unused, removed; the header
comment of standards.ts has the same provenance) and unused re-exports,
removed with their five env-var exports in request-cap.ts. scripts/
isn’t a blanket entry: only package.json’s scripts and the three named in
knip.jsonc are, so an unused script fails the check too.
chore, toolingknip 5, scoped to
src/, scripts/, tests/, with the unit tests as entry points) found
the codebase lean, with a few real leftovers:
getDefaultInspectionPath is defined twice, identically:
src/app/_lib/derive.ts:220 and src/lib/introspect/paths.ts:14;websiteSchema is defined in src/app/_lib/constants.ts:118 and again
in src/app/page.tsx:36; only the second is used;playwright-core is imported directly (src/lib/run/redact.ts) but
isn’t in package.json (it arrives through @playwright/test);.env.example still describes TESTFORGE_ENV as a FastAPI overlay
(it now only labels /api/health’s output) and says that without an
OpenAI key TestForge “falls back to a sample test template” (the
template is the default generator); .gitignore keeps a Python
section for a toolchain Phase 16 removed.
Not findings: knip also lists @axe-core/playwright, nspell and the
dictionaries as unused (they’re used by generated suites and by
src/lib/generate, which knip can’t see through code strings) and
request-cap-fixture.ts (copied into run folders by path); 79 exports
are only used inside their own file — over-exported, not dead.playwright-core or import through @playwright/test,
fix the .env.example and .gitignore text; commit a knip.json with
the false positives explained, and add npx knip to nightly.yml.npx knip exits 0 on the committed config, and fails when an unused
file is added (shown once);check:template-unchanged byte-identical; unit suite unchanged.769afb4 (2026-10-31). Verified (Cowork, 2026-10-31):
docker-compose.yml binds MinIO’s 9000/9090 and Plane’s 80 to
127.0.0.1; scripts/toolshop-local/compose.loopback.yml replaces
Testsmith’s port lists with !override (4200, 8091, 8000, 3306 on
127.0.0.1), and toolshop.sh loads it for both versions; SECURITY.md
describes it. The builder measured every port bound to 127.0.0.1 and no
answer on the LAN address from the same machine (MinIO had answered there
before). Other-machine check done (owner, 2026-11-03): with the stacks
up, a phone on the same Wi-Fi got ERR_CONNECTION_TIMED_OUT on
http://192.168.1.17:9090 and :4200. RUN-47 is ✅.v0.2.0 tagged on b19f314 and released 2026-10-31 04:32 UTC (checked
via the API: published, not a draft). Dependabot re-checked at 06:55:
still “Dependency files checked: 2026-10-28 07:31”, 114 open.
Owner decision, 2026-10-28: Plane stays, secured. It may be used by
other people or setups, and the next sites under test will come in the
same way. Plane’s 80 and MinIO’s 9000/9090 are published on 127.0.0.1
in docker-compose.yml; the Toolshop’s ports through TestForge’s own
override file. The committed credentials stay as development
placeholders; .env.example and SECURITY.md say to change them before
Plane runs anywhere else. Removal was considered and rejected.
security, configdocker-compose.yml publishes Plane on port 80
and MinIO on 9000 and 9090 on every network interface (measured
2026-10-27: docker ps shows MinIO on 0.0.0.0:9000 and
0.0.0.0:9090), and the local Toolshop copies publish 4200 (the app),
8091 (the API), 8000 and 3306 (MariaDB, root password root) the same
way, from Testsmith’s own docker-compose.yml in WSL; all with credentials
that are defaults in a public repo (plane-access-key /
plane-secret-key, change-me-before-production). On a shared network
anyone who can reach the machine can open the MinIO console with them.
TestForge itself is loopback-only by rule; its test targets should be too.127.0.0.1:<port>:<port> in docker-compose.yml
and for the Toolshop (D26). Testsmith’s file isn’t TestForge’s to edit
(scripts/toolshop-local/README.md), so the Toolshop’s go in a
TestForge override beside compose.local.yml, which replaces the
ports of angular-ui, web and mariadb (compose merges ports
lists, so the override needs !override or a reset, to be checked);
then remove the known gap from SECURITY.md; say in
.env.development and the README that the values are local placeholders,
and how to override them. Owner decision first: is Plane still used as a
target? If not, remove it (compose file, env vars, README sections)
instead.netstat shows the ports bound to 127.0.0.1
only; http://localhost still serves Plane, and localhost:4200 and
:8091 the Toolshop, with its start script’s bug-80 check passing;Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: The local Docker test targets answer on this machine only (TFP-260; docker-compose.yml, scripts/toolshop-local/compose.loopback.yml)
Expected result: Plane’s 80 and MinIO’s 9000 and 9090 are published on 127.0.0.1, and the Toolshop’s 4200, 8091, 8000 and 3306 through an override with ports: !override, as compose merges ports lists and Testsmith’s file isn’t TestForge’s to edit. MariaDB’s root password is root and MinIO’s keys are defaults in a public repo, so their reach matters. Proven (2026-10-31, Windows, Plane and with-bugs running): netstat lists all seven on 127.0.0.1 only; on this machine’s LAN address none answers, where MinIO’s two did before; through localhost, Plane, the app, the API and the MinIO console answer 200 in 9 to 151 ms; start-toolshop.ps1 passes its bug-80 check; the clean version’s merged config has the same four. Other machine (owner, 2026-11-03): with Plane, MinIO and the Toolshop running, a phone on the same Wi-Fi opened http://192.168.1.17:9090 and :4200: both ended in ERR_CONNECTION_TIMED_OUT. From another machine a timeout can’t tell a loopback bind from Windows Firewall dropping the connection; with netstat showing all seven on 127.0.0.1 only, together they meet the acceptance.
Found 2026-10-26. A reviewer evaluating TestForge for a team would open, in roughly this order: README, a security policy, an architecture overview, a decisions index, the contribution rules, the version and release notes.
SECURITY.md and CONTRIBUTING.mddocs, securityCONTRIBUTING.md that never existed here.SECURITY.md: how to report, what’s supported, the security model
(from README), and a short threat model: generated-code execution,
model-written tests (restrictions, not a sandbox; Phase 45), the gap
writer’s tool-less Claude Code call, credentials (env only, redacted,
storageState never persisted), local data (site details, contact email);CONTRIBUTING.md: setup pointer to README, the loop and vocabulary
from WORKING_AGREEMENTS.md, Conventional Commits, the test commands,
and that outside changes come as issues or PRs, not direct pushes;SECURITY.md that .env.development’s committed values are
local development placeholders for the optional Plane stack, not
secrets.main supported until versioning starts; issues first, pull requests by
invitation.docs, processCloses #67 to #82) and for #2 and #3, then
work moved entirely to TFP- tickets in docs. An outside reader can’t
see what is known, what was found or what was done about it without
digging. Mirroring every TFP- ticket would be noise; nothing at all
hides a well-run project..github/ISSUE_TEMPLATE/): bug report (the fields
in CONTRIBUTING.md), idea, and a config.yml that disables blank
issues and links security reports to private reporting;TEST_PLAN.md’s set, plus known-limitation and
phase;TEST_PLAN.md’s known-limitations list, linking the case and the
record that would close it;TFP- number in its title (as WORKING_AGREEMENTS.md says).WORKING_AGREEMENTS.md and
CONTRIBUTING.md describe the same rules.config.yml (97e3d2f); 8 labels added
(technical-debt, refactor, testing, investigation, skills,
chore, known-limitation, phase; bug and documentation already
existed); #5 to #13, one per known limitation, each linked from its
item in TEST_PLAN.md; #14, Phase 39’s tracking issue; the same rules
in WORKING_AGREEMENTS.md and CONTRIBUTING.md.docsdocs/decisions/ has 60+ files and no index; the
roadmap’s header still says “Version 1.0, 2026-08-28, Draft” with a
pre-Phase 16 baseline, and its Phase 39 heading says “not started” though
Gates A to D are measured. docs/consolidation/ (April 2026) describes
the FastAPI backend and proxy cutover that Phase 16 deleted, including a
CI plan for that architecture, with nothing saying it’s historical.
docs/TA-230.md is test-assist material kept as skill source.docs/README.md: an index of the docs and what each is for;product-roadmap.md or in docs/decisions/README.md;docs/consolidation/ file: historical, superseded by
Phase 16; or move the folder under docs/history/;docs/TA-230.md beside the skill that cites it.docs/README.md,
docs/decisions/README.md (the index), the roadmap header and Phase 39’s
heading, a historical banner on every docs/consolidation/ file and a
consolidation/README.md. Done 2026-10-27 (Claude Code): the move
of docs/TA-230.md to skills/testing/find-the-hidden-platform/, with
its links, and the three status lines, which the index no longer flags.
Was left: moving docs/TA-230.md beside its
skill (a git mv), and correcting three records’ own status lines the
index flags: Phase 42 (“Locked”, complete 2026-10-15), Phases 35 and 36
(“Approved”).docs/ is reachable from
docs/README.md; no phase heading’s status contradicts its record’s
status line.repo-claim-audit), against
main at f8333b6. 25 claims checked: 6 false, 4 stale, 2 understated,
0 unverifiable. All fixed in README.md:
.nvmrc
is 24 — the README’s own tool-stack table said so); “Comprehensive goes
deep on one area, or all of them” (COMPREHENSIVE_AREAS is security and
accessibility; all-areas is D5); “active … only behind a typed ownership
confirmation” (not built: active tests are always Not run until
Phase 43); self-test and regression “keep the latest 5” artifacts (each
upload has retention-days: 30); the setup step “add your OpenAI API
key” as the one configuration step (the path is parked, no completion
has succeeded); the tagline “you tell it what you want to test in plain
English … the AI handles the syntax” (generation is the deterministic
template from site inspection; the description field only steers area
selection).127.0.0.1 and guards the
API); “Open http://localhost:3000” (the dev server prints
127.0.0.1); “572 tests at TFP-207”; GitHub Copilot as the AI tool
(Claude Code is, and the gap writer calls its CLI).npm run script in the
table exists and every package script is documented; the example suite
file exists; save-suite’s .playwright-runs/generated-suites and the
chromium project; runs/ as the run root (TESTFORGE_RUNS_ROOT);
the gap writer’s four Claude Code flags; the gap limit of 5; promotion
after 3 clean sites; the library has entries; the nightly’s 02:00 UTC
schedule, 30-day log and pipefail step.docsCOMPREHENSIVE_AREAS is security and
accessibility. The fix is to say both — the goal, and what’s built —
wherever the README describes coverage.repo-claim-audit skill on README and fix what it
finds; add a last-reviewed date.releasepackage.json says 0.1.0, unchanged since the
start, and the repo has no tags, so “which TestForge found this?” has no
answer, and a CI user has nothing to pin.0.2.0), the patch for fixes
between phases; 1.0 when the report and CLI are declared stable
(after Phase 47). Rejected: 1.0.0 now (every report-format change
would become a major version); date-based versions (say nothing about
compatibility to a CI user pinning one).package.json’s version./api/health, every run’s
run.json and the report footer carry the version and the commit,
so a saved report answers the question by itself.CHANGELOG.md, one entry per version, written from
the phase records (what changed for a user), not from commit
subjects. The first entry, 0.1.0, summarises Phases 16–38 in a few
lines and links the index.v0.1.0 on the commit that
starts versioning, and v0.2.0 when Phase 39 completes;
CHANGELOG.md drafted 2026-10-27 (planning pass): 0.1.0 covers
everything complete so far — Phases 16–38 and 40–42, the post-Phase 42
review and TFP-261 — not “16–38” as first proposed, because 40–42
finished before 39; Phase 39’s built work is listed under Unreleased; a GitHub
release per tag with the changelog entry. Tags are pushed by Claude
Code; creating the first one is confirmation required.v0.1.0 exists; /api/health and a fresh
run.json show the version and commit; the changelog’s entries each cite
their phase record.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: The product says its version: which TestForge ran a saved report (TFP-257; unit: src/lib/version.test.ts, offline and through the executor in a real browser; regression: /api/health in testforge.sections.spec.ts)
Expected result: package.json’s version is the one source of truth, and the commit comes from git. /api/health gives both. A built server gives what it was built from (next.config.ts, at next build), as a pull under a running next start changes HEAD, not the code running; a dev server reads them on each call. A run takes its stamp when it starts (producedBy), kept in run.json and on its report, or on the result where a cancelled or failed run has no report, so a saved report says what made it, not what reads it, and a pull during a run can’t change it. The line after the report comes from that stamp; the report’s HTML is also built in the browser, so it can’t ask git. A run from before this says its version wasn’t recorded. A copy with no package.json or no git says unknown and a null commit, and doesn’t fail, and one with no .git of its own doesn’t report the commit of a repo it sits inside. Proven (real browser): a fresh run’s run.json, its report and the saved report’s HTML carry the stamp. Proven (offline): both report exports end with the version line after </main> (src/lib/selection/coverage.test.ts), and a report-less result’s export carries it.
Emil agreed all three on 2026-11-03, and their place in the work order (the handover’s list): TFP-268 first, then TFP-252 and TFP-253, then TFP-269, then TFP-270, then Phase 47.
6211ae7; baseline moved in
8d6f408). Found by TFP-258’s build (2026-11-03). A journey’s path keeps its
order and repeats (normalizeStringSequence); other lists still
de-duplicate. Verified on a Linux clone, the uncommitted files on
435d78f: journeys.test.ts and journey-tests.test.ts 28 of 28
passed, 0 skipped; with the old normalize.ts, the two TFP-268 tests fail
(26 passed, 2 failed). check:template-unchanged 435d78f: 172 inputs, 170
identical, 2 different, the two “journeys selected” inputs; the fast one
read: +373 −1, ten journey tests added (TFFU-037 to -046, each entering,
submitting the form and checking the page still renders), the existing
journey renumbered to TFFU-047, no other line changed. Claude Code’s full
unit run (708 of 708), tsc and ESLint weren’t re-run here.
The scope’s “say so where the user sees it” isn’t built (route.ts
narrows the selection silently; GEN-108 ⚠️): split out as TFP-271 (Emil,
2026-11-04).bug, generate, introspectnormalizeJourneys (src/lib/generate/normalize.ts)
reads a journey’s path through normalizeStringList, which removes
repeats. A journey whose path visits a page twice (for example a form that
posts back to its own page) loses a path entry, its steps no longer line
up with the path, and the journey is dropped as not aligned. Measured in
TFP-258’s build: all 10 journeys in the fixed content profile end in a form
that posts to its own page, so all 10 are dropped. A user who selects such
a journey gets no journey test and no message saying why. It has been so
since 6a5b4c4, which added journeys.check:template-unchanged shows the change only in inputs with selected
journeys, and each changed line is read; the baseline moves in its own
commit if output changes.TEST_PLAN.md case.Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.
Test: A journey whose path visits a page twice survives the generate route’s normalising, so selecting it writes its test (TFP-268; unit: src/lib/introspect/journeys.test.ts — “a journey whose path revisits a page survives normalizeContentProfile”; src/lib/generate/mock/journey-tests.test.ts — “TFP-268: a selected journey through a form that posts back to its own page gets a test, after the route’s normalising”)
Expected result: normalizeJourneys read a journey’s path through the de-duplicating string-list normaliser, so a form posting back to its own page (/contact > /contact) lost an entry, its steps no longer lined up with the path, and the journey was dropped without a word: selected on the review screen, no test written. Since 6a5b4c4, which added journeys. The path now keeps its order and repeats (trimmed, empty entries dropped); every other string list is still de-duplicated. Ranking, selection (by id) and the journey test writer (step by step, from steps) don’t depend on a path’s pages being distinct. Proven: both unit tests fail on the old code (no journeys survive; no test written) and pass on the new one. On the fixed content profile (benchmarks/2026-09-29/content-profile.json), all 10 of whose journeys end in a form posting to its own page: 0 of 10 survived and 0 tests were written before, 10 and 10 after. Template: check:template-unchanged against 435d78f, 170 of 172 identical; the 2 “journeys selected” inputs each gain the 10 journey tests (+373 −1 lines: the journey that was already there moves from the 1st to the 11th id, TFFU-037 to -047 in fast mode, -044 to -054 in deep), every line read. Not covered: a selected journey dropped for another reason (steps that don’t match the graph) still gets no test and no message (route.ts narrows selectedJourneyIds to the survivors without a word). Introspect returns journeys unnormalised and generate normalises them, so this holds only while the two agree, and no test checks that they do: this bug was such a disagreement.
06489fd; baseline moved in
2d210e8. The shared-visit
pass carries each moved declaration’s leading comments
(withLeadingComments in shared-visits.ts); declarations are still
compared by their code alone. Verified on a Linux clone, the
uncommitted files on 7e2f5ef: helper-coverage.test.mjs and
shared-visits.test.ts 8 of 8 passed, 0 skipped; with the old
shared-visits.ts, the new coverage test fails (“20 distinct comment
lines reached by no fixed input”). check:template-unchanged 7e2f5ef: 172
inputs, 98 identical, 74 different; Cowork’s own line analysis of all 74:
0 lines removed, 0 added lines that aren’t comments (no blank lines
either), 29 distinct comment lines added (the 20, plus 9 from the
Comprehensive security and accessibility helpers that Standard suites had
been dropping). GEN-107 ✅ (row 381 characters). Claude Code’s full unit
run (724 of 724) and tsc/ESLint weren’t re-run here.generate, testgetText() and removes them from getFullStart(),
so the comment lines before each declaration are dropped: 20 distinct
lines (controls.ts 8, READ_ONLY_GUARD_HELPER 7, SPELLING_HELPERS 3,
LAYOUT_HELPERS 2). A change to them changes no output the template check
compares. READ_ONLY_GUARD_HELPER’s 7 do reach users’ suites in full,
through site-suite reuse (withGuardHelper) and the gap writer, which the
check doesn’t run. Options weighed (TFP-258’s entry): keep the comments
through the move; run the check on the site-suite path too; accept the
gap. Chosen (Emil, 2026-11-03): keep the comments through the move.helper-coverage.test.mjs to fail on a helper comment line no
output carries.TEMPLATE_BASELINE in nightly.yml in its own commit.check:template-unchanged shows only added comment lines (about 100
inputs, per TFP-258’s estimate), every kind read; no code line changes.804c191. After the push
Dependabot marked #142, #139, #138, #110 and #56 fixed; #144 was
dismissed as tolerable risk (2026-11-04, with the TFP-270 comment), so
none is open. Only package.json and package-lock.json: npm audit fix
without --force (brace-expansion 1.1.14 to 1.1.21, and 19 ESLint-tree
packages within their ranges) and an overrides entry pinning dompurify
3.4.17, its reason and removal check in a top-level "//" field.
Verified on a Linux clone against b091875’s lockfile: npm audit
(dev included) went from 27 (19 moderate, 1 high, 7 critical) to 21
moderate, only the glob and minimatch <11.0.1 sections left;
npm audit --omit=dev --audit-level=high 0; the lockfile has dompurify
3.4.17 and brace-expansion 1.1.21 (the other two copies 5.0.12). So #142,
#139, #138, #56 and #110 are gone; #144 (minimatch) stays: fixed only
in 11.0.1+, which no released ESLint plugin supports (minimatch 11 has no
default export, which eslint-plugin-import calls). Ruling (Emil,
2026-11-04): dismiss #144 in Dependabot as tolerable risk (dev-only lint
tooling, globs from our own config; revisit when ESLint’s plugins support
minimatch 11). The override is partial by
nature: Allure’s report pages still run a DOMPurify 3.4.16 compiled into
its prebuilt bundles, which only an Allure release replaces; Allure calls
it with none of the options the three advisories need. Claude Code’s
Allure smoke report (5 of 5, opened, no errors), lint (0 errors) and unit
run (725 of 725) weren’t re-run here.security, dependenciesnpm audit
--omit=dev doesn’t see them: DOMPurify 3.4.16 under Allure’s report
packages (#142 critical, #139 high, #138 low; Allure has no fixed release),
brace-expansion 1.1.14 under ESLint (#56 high, #110 moderate), minimatch
under ESLint’s plugins (#144 moderate).overrides entry pinning a patched DOMPurify, with a comment
saying why and when to remove it (when Allure ships a fixed version).npm audit fix where it works;
an override only if not).npm audit (dev included) shows none of the six; the count of what’s
left is stated.npm run lint passes; unit tests
pass.5a2673d, verified (Cowork, 2026-11-08);
the review’s two low items (the inspected-path reason, the untested
“steps don’t match” reason) were fixed in it. Split from TFP-268’s scope (Emil, 2026-11-04). The journey check
moved to src/lib/generate/journey-check.ts, used by both the normaliser
and explainDroppedJourneys; the route adds droppedJourneys (id and
reason) only when one is dropped; GeneratedSuiteSection shows them
under the suite. Verified on a Linux clone, the uncommitted files on
804c191: the route, dropped-journeys and journey tests 42 of 42 passed,
0 skipped; with the old route.ts, the new route test fails (10 passed,
1 failed); check:template-unchanged 804c191: 172 inputs, 172
identical; tsc exit 0; GEN-108 ✅ (394 characters). Read: every
respond(...) call comes after droppedJourneys is set (the closure
reads it), as the code comment says. Claude Code’s end-to-end test (fails
against HEAD’s component at toBeVisible, passes with the change), its
screenshot and its unit run (729 of 729) weren’t re-run here.generate, uisrc/app/api/generate/route.ts,
around lines 93–96) narrows the user’s selected journey ids to the
journeys that survive normalizeContentProfile, and says nothing about the
rest. TFP-268 fixed the known cause (a path that revisits a page), but any
other mismatch would again drop a selected journey silently: the user
selected it, and no test and no reason appear.check:template-unchanged: 172 of 172 identical (suites don’t change).ad498f8, verified (Cowork, 2026-11-08).
Found while building TFP-263 (2026-11-08); present before and
after it. Launch and the first page now sit in their own try in
src/lib/pipeline/crawl.ts; a throw closes any browser that started and
returns “failed” with “The browser for the crawl could not start: “ and
the cause’s first line. Verified on a Linux clone, the uncommitted
files on b493265: src/app/api/crawl/route.test.ts 3 of 3 passed, 0
skipped; with the old crawl.ts the new test fails (2 passed, 1 failed,
browserType.launch: Executable doesn't exist at /no/such/chrome).
CRW-04 ✅. Claude Code’s standalone status script, its 18 browser-crawl
unit tests and regression (73 of 73) weren’t re-run here. The CLI’s half
of the scope (report and exit non-zero) comes with TFP-245.bug, crawl/api/crawl starts the crawl with void
runCrawlJob(job). If launching Chromium throws (a missing browser, a bad
PLAYWRIGHT_BROWSERS_PATH), the throw comes before the crawl’s own
try, the promise rejects unseen, and the job’s status never leaves
“running”: the UI polls forever with no error.error).79c9466, on main by fast-forward
(PR #33, every check passed on its head bd1872c: CodeQL and its alert
check, PR Validation, Release Image). Verified by a planning session
(2026-11-14): only the generated suite’s 14 s budget line is replaced,
and that is asserted once; the unchecked-routes assertion runs before
the verdict.
src/lib/introspect/routes.test.ts.npm run test:unit runs in a row: each 840 of 840, 0
skipped. The route test took 98.9 s, 127.3 s and 116.6 s; Gate B’s
failing run was 96 s.check:template-unchanged main 79c9466, 172 of 172
identical, no map. tsc and eslint clean.test, flakysrc/lib/introspect/routes.test.ts) got Passed instead of Failed.
Beside the full suite’s other browser tests it took 96 s, and the
generated check spent its 14 s route budget (“2 linked routes not
checked: the check stops after 14 s (/spa#/chainsaws,
/spa#/products/1)”) before reaching the missing route. Alone it took
39 s and passed. The budget is a known coverage limit (U9 in
data/labelled-failures.md). The weakness is that the test passes
when the check never reached what it is meant to catch.npm run test:unit, three runs in a row.check:template-unchanged: identical with no map.dc38039 and bc3a416, on main by
fast-forward (PR #34, every check passed on its head bc3a416: CodeQL
and its alert check, PR Validation, Release Image). Verified by a
planning session (2026-11-14), with owner rulings the same day:
bc3a416 runs
npm run build (without .env.development’s values) after the smoke
tests and their artifact upload. It runs before the two PR comment
steps rather than literally last, so a failing build is never
commented as a pass. On PR #34 it built (“Compiled successfully in
2.4s”, from the smoke build’s cache; 12 of 12 static pages). Smoke
Test keeps no separate build: the ruling named PR Validation only.test:self and test:self:allure run build:self-test first, and
the webServer runs start:self-test only. PR Validation and Smoke
Test drop their separate npm run build before the tests, which would
build twice. That build was never the one tested (serve:self-test
rebuilt over it). serve:self-test stays, for use by hand..next/ removed,
today’s command built and started in about 51 s and passed 5 of 5
(200 s). So the window was forced to 20 s with a throwaway config:
taskset -c 0-1) was not run: it is a timing session, and the
handover asks first. For the verifier: whether the forced
window meets the criterion.LOCALAPPDATA (the real
data folder had 41 files before and after): cold, 5 of 5, 198 s
(before: 200 s); warm, 5 of 5, 171 s (before: 185 s).check:old-name passes;
check:template-unchanged main, 172 of 172 identical.playwright test -c directly), the
suite now starts whatever .next/ holds. A stale build passes
silently, where before it was rebuilt. A missing build fails (“Could
not find a production build”). playwright.regression.config.ts has
had the same exposure since TFP-208. A guard (a check of the build’s
age, or a flag only the scripts set) is outside this ticket’s scope.test, ciplaywright.self-test.config.ts’s webServer runs
npm run serve:self-test, which builds and then starts the app, within
timeout: 120_000. In Phase 48’s Gate B (2026-11-11), a fresh session
with no build cache took 3.3 min to compile, and the suite stopped with
“Timed out waiting 120000ms from config.webServer” before any test ran.
The rerun, with a warm cache, passed 5 of 5. Whether the suite runs
depends on the cache, not the code.test:regression already does), or the
timeout covers a measured cold build. CI’s self-test job gets the same
fix..next/ removed, today’s command times out;
after the fix it runs all 5 tests.904db9b, 71a208f, 0fe8ffe and
f5b8e5c, on main by fast-forward (PR #37; every check passed on its
head f5b8e5c: PR Validation, CodeQL and its alert check, Release
Image, Action Gate). Verified by a planning session (2026-11-14), with
owner rulings the same day:
fetchLater() too, closing #36. 0fe8ffe extends the stub with
keepalive fetch’s methods. Red first at the server, on 1.62.1 without
it: POST /api/store/cart?later-page and ?later-frame arrived. With
it, the guard’s tests passed 5 of 5, three runs in a row, and
npm run test:unit 842 of 842, 0 skipped. Known limitation 13 is
removed.0fe8ffe: 164 of 172 identical. The 8 are the guard’s
suites, each byte-identical to its new output with the helper swapped.
f5b8e5c moves TEMPLATE_BASELINE to 0fe8ffe in its own commit.f5b8e5c (43591682394),
passed every job: template “0fe8ffe -> f5b8e5c: 172 inputs, 172
identical, 0 different”, plus unit, audit and dead-code. It closed
#38.f5b8e5c (43591680332): the same counts as 0.4.0’s
and as the runs below.6799874.route.abort and add an init-script stub in every frame for
sendBeacon, and for fetch with keepalive on any method but GET,
HEAD and QUERY. Keep TFP-199’s classes (a beacon is noted, any other
such write is a breach). Prove it at the server for the existing case
and from an iframe. Name any path it can’t cover as a known limitation
with an issue. Compare Action Gate’s Toolshop counts with 0.4.0’s.904db9b): Playwright 1.62.1, the first release
supporting Ubuntu 26.04 (Chromium 147 → 151). Every job is pinned to
ubuntu-26.04, not ubuntu-latest, so verify-image.mjs’s check of
the image’s OS against CI’s runner can’t drift silently. The image’s
base is ubuntu:26.04 by digest. Trace scrubbing reads Playwright’s
zip library from lib/utilsBundle, since 1.62 removed lib/zipBundle.
The action’s example and docs/CI_INTEGRATION.md keep
ubuntu-24.04: they describe running v0.4.0’s image elsewhere.71a208f): from Chromium 151, a beacon or keepalive
fetch sent as a page is left reached the server although the route
aborted it, writes included. The stub stops them in the page in every
frame, and the route’s rule judges them. Red first on 1.62.1 without
the stub: TFP-199’s beacons reached the server, and so did the new
test’s iframe beacons and keepalive cart POSTs. With it, the guard’s
tests passed 4 of 4, four runs in a row. Measured with the stub:
a link’s ping and a Web Worker’s keepalive POST are still stopped by
the route. References taken before the guard armed, and fetchLater(),
reach the server: known limitations 12 and 13 (#35, #36), RUN-54.LOCALAPPDATA; the real data
folder had 41 files before and after): npm run test:unit 841 of
841, 0 skipped (840 plus the iframe test); test:self 5 of 5;
test:regression 73 passed. tsc and eslint clean.check:template-unchanged main: 164 of 172 identical, 8
different. The 8 are the 8 inputs whose suite contains the guard
helper. Each base output with the old helper’s text replaced by the
new one is byte-identical to its head output, so the helper is the
only change. For the owner to accept. Then TEMPLATE_BASELINE
in nightly.yml is re-recorded in its own commit, as TFP-205’s was.71a208f: every check passed: PR
Validation, CodeQL and its alert check, Release Image, Action Gate.
The image reports "os":"26.04", "playwright":"1.62.1",
"chromium":"151.0.7922.34". Dispatched on the branch:
ci, dependenciesubuntu-latest moved from ubuntu-24.04 to ubuntu-26.04.
Playwright 1.59.1 stops there: “ERROR: Playwright does not support
chromium on ubuntu26.04-x64” (PR Validation run 43048248301, both
attempts), before lint. Phase 48’s merge was unblocked by pinning every
job to ubuntu-24.04 (owner decision, 2026-11-11). That pin holds only
while GitHub keeps the 24.04 image.ubuntu-26.04
(1.65.0 is the latest on 2026-11-11; support not yet checked), then
return the jobs to ubuntu-latest or pin ubuntu-26.04. The browser
version moves with it, so the generated template is re-checked.check:template-unchanged: identical with no map, or each difference
named and owner-accepted.dependencies, testnpm run test:unit gave 833 of 840:
src/lib/run/timing.ts:70-77, and the
shared-visits load counts, which come from the same steps).npm run test:unit passes on the new release.check:template-unchanged: identical with no map, or each difference
named and owner-accepted.6756fb9, 7ba7399 and ecb9fd4, on
main by fast-forward (PR #39). Every check passed on its head
ecb9fd4: PR Validation, CodeQL and its alert check, and Release Image.
The nightly dispatched on the branch was green on ecb9fd4 (run
43668391841), with its template check against the new baseline
7ba7399. #35 was closed by 6756fb9 and known limitation 12 replaced.
Verified by a planning session (2026-11-15).tfGuardFrame wraps
sendBeacon, fetch and fetchLater once per frame, dormant; the
guard switches the wrappers on. Generated tests that navigate before the
guard call tfReadOnlyWrap(page) first: the guarded control checks, and
the link tests that load their target under the guard. Gap and candidate
tests navigate only inside the guard, whose own init script wraps each
new document already switched on. Red first at the server on TFP-284’s
guard: the three guarded POSTs arrived. With the wrapper, only the six
load-time POSTs arrive, with or without it. check:template-unchanged
against 0fe8ffe: 8 of 172 differ, each only by the helper and one
tfReadOnlyWrap line per guarded test that navigates first (8 of 8).
The baseline waits for the owner’s acceptance. Full unit suite 844 of
844, 0 skipped. The fresh-context review found two paths kept from
TFP-284: a Request made in another frame, and the page writing
__tfGuard.armed.about:blank iframe’s fetch under the guard, and if it gets
through, make it a known limitation with an issue, without widening the
ticket; list the load-queued fetchLater as a known limitation; show
the template check again, and stop for verification before the
baseline moves.__tfGuardArm, a one-way switch it can’t replace or remove.6756fb9: cross-fetch, cross-later,
unarmed-fetch and unarmed-later all arrived. With the fix, none
does, and both fetchLater POSTs are named as breaches.about:blank iframe’s fetch is the wrapper (Playwright runs
init scripts in that frame), and its POST doesn’t arrive. So there
is no limitation and no issue.fetchLater was measured reaching the server. It is
known limitation 12, pinned by the test.read-only-guard.saved.test.ts, for the
500-line limit.0fe8ffe is unchanged from before: 8 of
172 differ, each only by the helper and the wrap lines.tfReadOnlyWrap line
only. TEMPLATE_BASELINE moves to 7ba7399.__tfGuardRequest and __tfGuardBreach
bindings. A request the guard stops is still stopped, but it goes
unreported.__tfGuardArm first, the guard’s form listener
is never added in that frame. A GET form submission there goes
unreported. POSTs are still stopped and reported, by the route.bug, securitysendBeacon, fetch and fetchLater when the guard arms,
after the page has loaded. A page that saved its own reference to one of
them at load (as an analytics script can) calls the original. Measured on
1.62.1 (2026-11-14): such a request, sent as the page is left, reached
the server.sendBeacon,
fetch and fetchLater, installed by an init script before each
read-only test’s first navigation, so it is what the page saves at load.
It passes every call through until the guard switches it on, then
stops non-safe calls as TFP-284’s stub does (TFP-199’s classes). Loading
stays unjudged: the wrapper changes nothing until the guard arms.sendBeacon and to fetch saved at load, used as the
page is left under the guard), failing on TFP-284’s guard.npm run test:unit pass;
check:template-unchanged shows only the guard’s lines changed,
owner-accepted.To add to deferred-work-register.md once Claude Code’s D37 commit has
landed (the file is in its working set now):
v0.2.0 tagged under the current name. Superseded by the
register’s D39 entry. To settle before Phase 47’s TFP-247 publishes an image or
action, and before the first GitHub release, since outside pipelines will
pin the name. It touches the repo and package names, the
testforge-skills plugin, the local data folder
(%LOCALAPPDATA%\TestForge, which needs a one-time move), the
TESTFORGE_* settings, test-ID prefixes (TF…), the TFP- ticket
prefix and the testforgeapp@gmail.com addresses.Phase 39 and its tickets stay first. These slot in between gates:
Where Phase 47 sits relative to the deferred-work build order (after Phase 39; ownership confirmation; Standard sets…) is an owner decision, taken in TFP-244.