Date: 2026-10-13
Source: https://github.com/emil3663/testforge/issues/2, filed 2026-09-23 by
hugo76 at Martin’s request (“sweep deletes evidence for defects the newer run
never tested”). Read with Emil on 2026-10-13; every problem in it is still in
src/lib/run/persistence.ts at e5b1add.
Status: Done and verified 2026-10-13 (Cowork): TFP-198 15e0205, TFP-199
3a8f420, records and the hosted re-run 3955847. GitHub issue #2 can be
closed once these are pushed.
Phase 42 builds on run history: site suites retire tests on purpose, and a retired test looks exactly like “not run” to today’s sweep. Phase 42 would make the bug fire more often, not less.
bug, run, data-losssweepAutomaticArchiving() changed per Decisions 1 and 2;writeRunRecord made atomic: write a temporary file, then rename;persistRunJob’s artifact-root check compares against the root plus a
path separator (artifacts-run1 never matches artifacts-run10), with a
unit case;TEST_PLAN.md case.e5b1add’s logic and passes after;Refs #2); the issue is closed only
once Emil has seen the verification.15e0205 (Refs #2; the issue
is open). The issue’s test fails on the previous sweep and passes; with
the decision table, the interrupted-write case and the prefix case, the
four new tests fail on the previous code and the ten existing ones pass on
both. RUN-28.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: The sweep keeps a run’s evidence until its failures are shown fixed, compares only runs of the same suite, and writes records atomically (TFP-198, GitHub issue #2, run-evidence-sweep backlog Decisions 1 to 3; unit: src/lib/run/persistence.test.ts)
Expected result: Clearing (Decision 1): an older run’s failing test counts as fixed only if a test with the same title ran and passed in the newest run of the same suite; absent, Failed, Not run, Blocked or Flaky there keeps the older run, and its evidence/, as it is. Grouping (Decision 2): runs are compared only within one suite identity (suiteIdentity): the baseUrl, the selected practices in any order, and the run type (Standard, or Comprehensive for one area; records without runMode are Standard). Records: run.json is written to run.json.tmp and renamed into place, so a failed write leaves the previous record whole and the run in listRunRecords(). Artifacts: a file counts as under the job’s artifact root only below the root plus a separator (artifacts-run1 never claims artifacts-run10/...). Storage assumes one TestForge process at a time (Decision 3; README and persistence.ts). Proven (decision table: what the newest run says about the failing title x whether the runs are the same suite): passed, same suite - archived (also with the practices in another order); failed, absent, Not run, Blocked, Flaky - kept; passed but different practices, or Comprehensive against Standard - kept; the issue’s own drop-in test (an accessibility run, then a links run of the same site) keeps the accessibility run. A run.json.tmp that can’t be written makes the pin fail with run.json byte-identical and still listed. The four new tests fail on the previous code (the sweep, the table, the atomic write, the prefix) and the ten existing ones pass on both.
bug, run, gatesPOST /cdn-cgi/rum) fires as a page
navigates. The guard rightly aborts it but fails the test as a breach, so
the link check fails on the clean site too, and bug 29 is lost there. Many
real sites send beacons (Cloudflare, Google Analytics and the like).navigator.sendBeacon and a
link’s ping, whatever resource type Chromium gives them; confirm which
one Cloudflare’s is first);fetch (/cdn-cgi/rum first).
Every other blocked non-safe request, and every form submission, still
fails the test as now.sendBeacon POST: the link
check passes, the beacon is aborted and named in a note;3a8f420. Measured first: one
link followed on the hosted clean site sends two POSTs to /cdn-cgi/rum,
typed ping and xhr; both are now beacons (ping, and the named path).
Fixture: the beacon case passes with both named in a note; the POST
control and form submissions still fail. Hosted re-run on 3a8f420
(benchmarks/2026-10-13-tfp199/): with-bugs fails naming bug 29 ("Home"
on /#/ leads to /contact), 9 beacons noted; the clean site passes, 19
beacons noted.suiteIdentity (base URL, sorted
practices, run type) and keeps a candidate unless every one of its failing
titles is in the newest run’s Passed set, so absent, Not run, Blocked,
Flaky and failed all keep the evidence. run.json is written to
run.json.tmp and renamed; the artifact-root check adds the path
separator; the one-process assumption is in the code and the README. The
issue’s test is unchanged; the four new cases fail on the old code.ping, or a non-safe request to a
path in TF_BEACON_PATHS (/cdn-cgi/rum, measured), is aborted and
recorded as a note; every other non-safe request and every form
submission is still a breach. Hosted re-run: bug 29 found on with-bugs,
clean passes; T33 fixed.check:template-unchanged
162/166 identical (the 4 differ only by the guard’s beacon lines).