mr-onion

Backlog — Before Phase 42: the Evidence Sweep (GitHub issue #2) and the Guard’s Beacons

Date: 2026-10-13 Source: https://github.com/emil3663/testforge/issues/2, filed 2026-09-23 by hugo76 at Martin’s request (“sweep deletes evidence for defects the newer run never tested”). Read with Emil on 2026-10-13; every problem in it is still in src/lib/run/persistence.ts at e5b1add. Status: Done and verified 2026-10-13 (Cowork): TFP-198 15e0205, TFP-199 3a8f420, records and the hosted re-run 3955847. GitHub issue #2 can be closed once these are pushed.

Why before Phase 42

Phase 42 builds on run history: site suites retire tests on purpose, and a retired test looks exactly like “not run” to today’s sweep. Phase 42 would make the bug fire more often, not less.

Decisions (Emil, 2026-10-13)

  1. Only positive evidence clears a failure. An older run’s failing test counts as fixed only if a test with the same title ran and passed in the newer run. A title that is absent, skipped or Not run keeps the older run exempt. The issue’s option 1.
  2. Only compare like with like. Runs are grouped by base URL plus a suite-identity signal (the selected practices and run type), so an unrelated audit never supersedes another. Phase 42’s site suites supply a stronger identity later. The issue’s option 2.
  3. Run storage assumes one TestForge process at a time, written down where persistence is documented. No locks are added.

Backlog

TFP-198 — The sweep keeps evidence until a defect is shown fixed

RUN-28 evidence

Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.

Test: The sweep keeps a run’s evidence until its failures are shown fixed, compares only runs of the same suite, and writes records atomically (TFP-198, GitHub issue #2, run-evidence-sweep backlog Decisions 1 to 3; unit: src/lib/run/persistence.test.ts)

Expected result: Clearing (Decision 1): an older run’s failing test counts as fixed only if a test with the same title ran and passed in the newest run of the same suite; absent, Failed, Not run, Blocked or Flaky there keeps the older run, and its evidence/, as it is. Grouping (Decision 2): runs are compared only within one suite identity (suiteIdentity): the baseUrl, the selected practices in any order, and the run type (Standard, or Comprehensive for one area; records without runMode are Standard). Records: run.json is written to run.json.tmp and renamed into place, so a failed write leaves the previous record whole and the run in listRunRecords(). Artifacts: a file counts as under the job’s artifact root only below the root plus a separator (artifacts-run1 never claims artifacts-run10/...). Storage assumes one TestForge process at a time (Decision 3; README and persistence.ts). Proven (decision table: what the newest run says about the failing title x whether the runs are the same suite): passed, same suite - archived (also with the practices in another order); failed, absent, Not run, Blocked, Flaky - kept; passed but different practices, or Comprehensive against Standard - kept; the issue’s own drop-in test (an accessibility run, then a links run of the same site) keeps the accessibility run. A run.json.tmp that can’t be written makes the pin fail with run.json byte-identical and still listed. The four new tests fail on the previous code (the sweep, the table, the atomic write, the prefix) and the ten existing ones pass on both.

TFP-199 — The read-only guard notes site beacons instead of failing

Verification (2026-10-13, Cowork)