Date: 2026-09-17
Scope: Give runs durable identity and persist them to disk with a tiered
retention scheme; recompute coverage against the discovered surface from
crawl/introspect instead of the tests that happened to run; add a “not
tested, and why” report section; support diffing two persisted runs of the
same site.
Status: Complete. TFP-064 482454d (2026-09-17), TFP-071/072 8cd901f and TFP-073 6a0ca45 (2026-09-22); independently re-verified 2026-09-22. The hold on TFP-053-063 (Decision 6) was met first. Reconciled with git 2026-09-29.
docs/decisions/product-roadmap.md’s own Sequencing note called for a live
walkthrough of what TestForge actually produces before this phase’s decision
record was drafted. That walkthrough ran (TFP-040/041/042,
examples/comparison-test-assist.md) and its findings are scoped in
docs/decisions/phase-17-walkthrough-backlog.md. Phase 16 Gate D is closed
(TFP-005/030/031/032 landed) — this document is what that gate was blocking.
Confirmed directly against the current code before writing any decision below, not assumed from the roadmap outline:
runJobs (src/lib/run/jobs.ts) is a process-global, in-memory Map —
gone on restart. finalizeRunJob actively unlink()s the generated test
file, the JSON report file, and the reporter file once a run finishes;
nothing about a completed run survives except what’s sitting in that
in-memory job until the process recycles it.playwright.config.ts’s
outputDir and collectArtifactFiles() (src/lib/run/output.ts) reads
them back by path. They physically persist today; they’re just not tracked
anywhere durable, and a later run reusing the same outputDir can
overwrite them.buildCoverageMatrix (src/lib/report/coverage.ts) computes coverage
entirely from selectedPractices × the executed suite’s own
ExecutionRecord[] — one row per selected practice area, “covered” if
every test in that area passed. It never touches crawl or introspect
output. This is the exact, literal mechanism behind the roadmap’s “coverage
is computed from the tests that ran, not against a discovered surface”
problem statement — confirmed by reading the 59-line file, not inferred.package.json
(@playwright/test, next, openai, react are the only runtime deps).
The project’s stated posture throughout Phase 16 is deliberately minimal
and local-only.dom-snapshot.ts already uses to deduplicate them
(elementType + containerLabel + locator.id + locator.name, falling
back to text) are also embedded in every generated test’s own title via
buildActionableElementTests. Coverage matching against the discovered
surface can reuse this without inventing a new ID scheme.runs/<runId>/ at the repository root:
run.json (the config that produced it — baseUrl, description,
selectedPractices, auditMode, createdAt, finishedAt — plus the
full RunResultPayload/RunReport), suite.spec.ts (the exact code that
was executed, copied before finalizeRunJob’s existing cleanup deletes the
transient copy), and evidence/ (trace/video/screenshot files moved,
not copied, out of Playwright’s shared outputDir into this run-scoped,
stable location, so a later run cannot overwrite them).runs/ is gitignored, same pattern as artifacts/.runs/;
run.json is self-describing. This is a deliberate choice against a
database, consistent with the project’s local-first, single-developer
posture — not an oversight.runJobs (the in-memory Map) is not removed. It continues to hold
in-flight run state for live progress polling, which genuinely needs
low-latency in-process access; persistence happens once at
finalizeRunJob, when a run reaches a terminal status.buildCoverageMatrix (or a new function alongside it) additionally takes
the crawl/introspect output for the run’s baseUrl — pages, actionable
elements, link details.run.json’s report, not a
UI-only computation, so a stored run’s report is complete on its own.runIds sharing the same baseUrl, a diff shows: tests that
passed in one but not the other, the coverage-percentage delta per
practice area, and elements newly discovered or newly missing between the
two runs’ introspection snapshots.run.json gains three fields: pinned (boolean, default false),
retention ("full" | "archived", default "full" at creation), and
lastAccessedAt (timestamp, null until the run’s report is actually
read via the API/UI — this is the “has this even been looked at” signal,
surfaced in the run listing, and is informational only in this phase: not
being accessed is not by itself a reason to delete something).evidence/ (traces,
videos, screenshots) for a run — run.json and suite.spec.ts are kept
in full, so history, coverage numbers, and diffing all keep working for
an archived run. Nothing about a run’s pass/fail record or config is ever
deleted by archiving; only the heavy replay evidence is.baseUrl, i.e. it has been superseded; (c) none of its failed
tests, if any, are still failing in the newest run for that baseUrl —
in plain terms, it isn’t the current evidence for a bug that’s still
open. A same-day regression run that passed cleanly and was later
superseded is exactly what this prunes automatically; a run capturing a
reproducing, not-yet-fixed defect is exempted until a later run shows
that specific test passing.README.md documents the tiered scheme and the automatic rule in plain
language, the same way it already states the local-only security model
rather than leaving retention behavior implicit.phase-17-walkthrough-backlog.md
generator-hardening tickets (TFP-053–063) land, not in parallel with them.
phase-17-walkthrough-backlog.md’s own milestone gate is met.runs/<runId>/run.json.runs/<runId>/suite.spec.ts, byte-for-byte identical to what was executed.runs/<runId>/evidence/, including after a second,
unrelated run has completed.test-assist, per examples/comparison-test-assist.md),
proving the bucket isn’t vacuous by construction.baseUrl, the diff API returns a
structured result naming specific tests and specific coverage deltas, not
a percentage-only summary.runs/<runId>/evidence/
while run.json and suite.spec.ts remain readable and that run still
appears correctly in coverage history and diffing.evidence/ untouched, even when it would otherwise qualify.lastAccessedAt is null on a freshly persisted run and is set the
first time that run’s report is read back.README.md documents the tiered retention scheme (full/archived/pinned)
and the automatic rule, in the same section style as the existing
Security Model statement.TEST_PLAN.md cases for: a run surviving a server restart
(persistence), coverage dropping below 100% on a target with undiscovered-
but-untested elements (honest coverage), the “not tested, and why” section
populating all three reason buckets against a real target, and a two-run
diff against the same baseUrl with a known, deliberate difference between
them (e.g. a regenerated suite with one fewer test).buildCoverageMatrix’s current
category-only shape must be updated to the new discovered-surface shape as
part of this phase, not left passing against stale behavior.Both placeholders in the prior draft were confirmed directly by Emil rather than assumed:
phase-17-walkthrough-backlog.md’s
TFP-053–063 first — a product-completeness preference (“as complete a
product as possible before stepping into more backlog”), not a reversal
of the technical reasoning that persistence and generation-quality are
decoupled. This record’s decisions are locked now so implementation can
start immediately once that backlog’s own milestone gate is met, without
redrafting this document then.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: A finished run survives a server restart (automated: npm run test:unit — src/lib/run/persistence.test.ts, “round-trip: read back with no reference to the originating job models a restart”, plus “evidence move: files land under evidence/…” and “evidence move failure: falls back to the original path…”)
Expected result: runs/<runId>/run.json and suite.spec.ts are readable, and evidence resolves under runs/<runId>/evidence/, using only data on disk — no reference to the in-memory job that produced it, modeling exactly what a fresh process sees after a restart. A failed evidence rename() falls back to the artifact’s original path rather than losing the record.
Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: Retention: pin overrides automatic archiving; archiving keeps run.json/suite.spec.ts and deletes only evidence/; a fixed defect’s run becomes sweep-eligible once superseded by a passing re-test of the same suite (RUN-28) (automated: npm run test:unit — src/lib/run/persistence.test.ts, “pin overrides sweep…”, “archive: keeps run.json and suite.spec.ts…”, “still-open-defect exemption…”, “lastAccessedAt: null after persisting…”)
Expected result: A pinned run is never archived by the sweep even when superseded and its one failure is fixed elsewhere; an unpinned, superseded run whose failure is fixed becomes archived (evidence gone, report/suite intact); a run that is still the only evidence for a still-failing test is never auto-archived; lastAccessedAt is null on a fresh run, set on first read, unchanged on a second.
Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: Honest coverage: a target with more discovered elements than written tests shows sub-100% coverage even when every written test passes, and the uncovered list names specific elements (automated: npm run test:unit — src/lib/report/discovered-coverage.test.ts, “Group 2: more discovered elements than tests still shows <100% coverage…”)
Expected result: With 2 discovered elements and only 1 matching executed (passing) test, the Functional row is partial at 50%, not covered, and both the row’s and the top-level uncovered list name the specific untested element, not just “Functional”.
Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: “Not tested, and why”: every unmatched discovered element carries exactly one of not-selected / excluded-by-generator / generator-gap, and generator-gap is real, not vacuous (automated: npm run test:unit — src/lib/report/discovered-coverage.test.ts, “Group 3: not-selected, excluded-by-generator…” and “…an unselected practice area buckets all of its discovered elements as not-selected”)
Expected result: A link element with dedicated link-audit coverage and elements beyond the generator’s 50-element cap are excluded-by-generator; elements in an unselected practice area are not-selected; a selected, eligible element with no matching executed test is generator-gap — no element carries two reasons, and the generator-gap bucket is demonstrably non-empty.
Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: Diffing two runs of the same baseUrl: specific test names (not just a percentage), per-area coverage deltas, and newly discovered/missing elements, against a deliberately regenerated suite with one fewer test (automated: npm run test:unit — src/lib/run/diff.test.ts, “Group 4: two runs with a deliberately dropped test…”, plus src/app/api/runs/diff/route.test.ts’s real-request coverage of GET /api/runs/diff)
Expected result: Given run A (3 passing tests, 100% Functional coverage) and run B (a regenerated 2-test suite where one carried-over test now fails and the third is simply absent), the diff names both affected tests by title with their exact from/to status (the absent one reported as “not run”, not conflated with a failure), reports the Functional coverage delta as -50, and lists the specific elements newly discovered/missing between the two runs’ own introspection snapshots. Diffing two runs with different baseUrl values is rejected.