mr-onion

Test cases: Run (RUN)

Section 3.5 of the test plan, which keeps the app overview, the feature status, the known limitations and the index of areas. Since TFP-253 each area’s cases are in a file of their own; a ticket names the area it touches.

ID Test Expected Result Status
RUN-01 Run a generated suite and open the report popup (automated smoke: “runs a generated suite and opens report popup”) Run completes and the structured report popup opens automatically. ✅
RUN-02 Stopping a run preserves planned/interrupted/not-run state (automated: “preserves planned, interrupted, and not-run test states when a run is stopped”) A test already completed before Stop keeps its result; the test in flight is marked interrupted; tests that hadn’t started are marked not-run — not silently dropped. ✅
RUN-03 Rerun only selected failed tests via API (automated: “runs only the selected failed test titles when includeTestTitles is provided”) Passing includeTestTitles reruns exactly those tests; passing tests and untitled failed tests are excluded from the rerun. ✅
RUN-04 Rerun a selected failed test from the report UI (automated: “reruns the selected failed test from the execution report”) Selecting a failed test in the execution report and triggering rerun re-executes only that test. ✅
RUN-05 Session evidence capture on failure (automated: “captures session evidence artifacts for failed generated runs”) A failed generated test produces captured evidence artifacts (trace/video/screenshot per playwright.config.ts’s retain-on-failure/only-on-failure settings). ✅
RUN-06 Manual: cancel a run mid-execution from the UI Run stops promptly; already-completed test results remain visible. ⬜
RUN-07 Run executes with no external backend configured (automated: tests/regression/mr-onion.sections.spec.ts — “executes generate, run, crawl, introspect, and save-suite without an external backend configured”) POST /api/run starts a job that reaches runStatus: "passed", with no environment variable set beyond .env.development. ✅
RUN-08 A finished run survives a server restart runs/<runId>/run.json and suite.spec.ts are readable, and evidence resolves under runs/<runId>/evidence/, using only data on disk — no reference to the in-memory job that produced it, modeling exactly what a fresh process sees after a restart. Evidence. ✅
RUN-09 Retention: pin overrides automatic archiving; archiving keeps run.json/suite.spec.ts and deletes only evidence/; a fixed defect’s run becomes sweep-eligible once superseded by a passing re-test of the same suite Pinned runs are never archived; archiving keeps the report and suite. Evidence. ✅
RUN-10 Honest coverage: a target with more discovered elements than written tests shows sub-100% coverage even when every written test passes, and the uncovered list names specific elements 2 elements, 1 passing test: Functional is partial at 50%, naming the untested element. Evidence. ✅
RUN-11 “Not tested, and why”: every unmatched discovered element carries exactly one of not-selected / excluded-by-generator / generator-gap, and generator-gap is real, not vacuous Exactly one reason per unmatched element; the generator-gap bucket is non-empty. Evidence. ✅
RUN-12 Diffing two runs of the same baseUrl: specific test names (not just a percentage), per-area coverage deltas, and newly discovered/missing elements, against a deliberately regenerated suite with one fewer test Names each changed test with its from/to status, and a coverage delta. Evidence. ✅
RUN-13 Reliability: one page.route() failure-injection test per routable dependency, in the discovered-coverage denominator Blocking a real recorded dependency gives a named pass or a named failure, never a bare timeout. Evidence. ✅
RUN-14 Reliability: storage entry-point failure injection, from a real behavioural probe of which storage a page touches A page that touches no storage gets no test; one that does gets a test per kind detected, in the Reliability row. Evidence. ✅
RUN-15 Coverage matches real executed tests to discovered elements despite the TFxx-000: id prefix real titles carry, and a skipped-by-default journey is neither “covered” nor a generator gap Real titles’ id prefix no longer hides a match. Evidence. ✅
RUN-16 Config-driven login: a run can log in once and every test in it starts signed in /api/run takes a login block (environment-variable names, not values), logs in once, and every test reuses its storageState. Evidence. ✅
RUN-17 A failed login is a named run-level error, never a silent unauthenticated run The run does not start (no runId, no per-run state left behind). Evidence. ✅
RUN-18 An authenticated run hands back none of its credentials or its session — not in the API result, the HTML report, the persisted evidence, a trace, or a download No credential or session value in any of them; the trace’s storageState was found and removed. Evidence. ✅
RUN-19 A login references credentials by the NAME of an environment variable and never carries a value; the literal credentials appear in no generated code, persisted run config, or API response Neither credential value appears anywhere the run writes or returns. Evidence. ✅
RUN-20 The automation-account setup doc names all four exemption categories and matches the real login config, so it cannot drift A section each for lockout, rate limiting, MFA and sign-in alerts. Evidence. ✅
RUN-21 The login can be entered in the UI, and behaves exactly as the equivalent /api/run request An “Authenticated Run” section, labelled as the setup doc; credential fields take a variable name. Evidence. ✅
RUN-22 A run is Standard (unchanged) or Comprehensive for one chosen area runMode absent or { kind: "standard" } is a Standard run; { kind: "comprehensive", area } with security or accessibility is a Comprehensive one; anything else is refused. Evidence. ✅
RUN-23 A Comprehensive run covers every known page up to the cap, and names the rest Pages come from introspection first, then the crawl’s site graph (pages known only as a link target), same origin, no repeats; with no page data, /. Evidence. ✅
RUN-24 A Comprehensive run uses one worker, enforced by its own suite Every Comprehensive suite starts with test.describe.configure({ mode: 'default' }), keeping each file’s tests in order; its worker count now comes from the host (RUN-42). Evidence. ✅
RUN-25 Gates at run time: a Standard run sends only GETs with interactive tests off, and the active gate stays closed with them on The suite: every area, every journey selected, plus a local-model block. Evidence. ✅
RUN-26 The planned test count counts declared tests only test( and test.only( are counted; a property access (.test(, as in TFP-167’s tfGoto helper /#!?\//.test(url)), test.describe(, test.skip( and longer names (latest() are not. Evidence. ✅
RUN-27 A read-only test that interacts runs under the request guard tfReadOnlyGuarded aborts every request that isn’t GET, HEAD or QUERY, and stops every form submission. Evidence. ✅
RUN-28 The sweep keeps a run’s evidence until its failures are shown fixed, compares only runs of the same suite, and writes records atomically A failure counts as fixed only if the same test passed in the newest run of the same suite; until then the run and its evidence stay. Evidence. ✅
RUN-29 A site’s suite is saved outside the repo and reused while its page types are unchanged Saved per origin and suite in Mr Onion’s data folder, with each page type’s signatures and each test’s hash. Evidence. ✅
RUN-30 A reused test that fails on a page whose structure changed is regenerated and re-run before it counts Its page is read again; a changed structure gets the test regenerated and re-run. Evidence. ✅
RUN-31 Gap tests, once per gap, opt-in per site A site’s gapWriter setting (off by default; claude-code; local-model) decides whether its uncovered elements and links get tests. Evidence. ✅
RUN-32 The candidate pool: gap tests generalised, run on other sites outside the verdict After a gap test is saved, its writer generalises it (__PAGE__, __LOCATOR__). Evidence. ✅
RUN-33 Promotion: only a candidate meeting the rule is offered; approved ones become library entries, rejected ones stay rejected Gates passed on every run, a real defect caught or 3 clean sites, and the tester’s approval. Evidence. ✅
RUN-34 The gap writer and the tests it writes, hardened Claude Code runs with every tool off, no CLAUDE.md, memory or hooks, from an empty folder; page text is untrusted data. Evidence. ✅
RUN-35 On Standard, a matching pool candidate becomes the gap’s test A candidate whose pattern matches a gap on a Standard run is bound to the site and run through the gap test’s gates; if it passes it joins the site suite under “Gap tests”, inside the verdict. Evidence. ✅
RUN-36 Generation applies the library’s promoted entries A promoted entry in src/lib/library/promoted/ is bound wherever inspection found its pattern and runs inside the verdict on Standard runs with the functional area. Evidence. ✅
RUN-37 An artifact asked for while its run is being saved is served, not refused A run reads as finished once its result is set; its evidence is then moved from Playwright’s output into the run’s folder (persistRunJob). Evidence. ✅
RUN-38 The run record says where the run’s time went, measured beside the suite, not in it A second reporter beside the progress reporter (src/lib/run/timing.ts) keeps each test’s page loads, settle waits, requests and assertions as Playwright names them, without fixture set-up. Evidence. ✅
RUN-39 Each app run in a folder of its own Each app run (the executor, a gap run, a stale re-check) runs in its own folder under .playwright-runs/, removed when it ends. Evidence. ✅
RUN-40 A suite runs as units, one file each, across the run’s workers splitSuite makes each page’s group, each other top-level describe, and the top-level tests a unit of their own. Evidence. ✅
RUN-41 Data-changing tests run one at a time, after the read-only tests All data-changing tests and model-written blocks go in one serial unit, run on one worker after the rest. Evidence. ✅
RUN-42 Worker limits and a request cap by host; the ownership confirmation lifts both Local hosts get the machine’s limit (at most 3) and no cap; any other at most 3 workers sharing 5 requests per second (the confirmation is a stub until Phase 43). Evidence. ✅
RUN-43 The API answers only the tester’s own pages on this machine npm run dev, dev:development and start bind to 127.0.0.1, so the port isn’t reachable on the machine’s LAN address. Evidence. ✅
RUN-44 On a capped host, a failure the request cap starved is Not run, not Failed A test’s own waiting for request slots is recorded; a failure that waited is reported Not run. Evidence. ✅
RUN-45 The request cap, proven at the server; a test whose context closes while its requests wait isn’t failed by the cap The slot server counts requests given a slot and those that gave up waiting; a test whose context closes while its requests wait isn’t failed by the cap. Evidence. ✅
RUN-46 The product says its version: which Mr Onion ran a saved report package.json’s version is the one source of truth, and the commit comes from git. Evidence. ✅
RUN-47 The local Docker test targets answer on this machine only Plane’s 80 and MinIO’s 9000 and 9090 are published on 127.0.0.1, and the Toolshop’s 4200, 8091, 8000 and 3306 through an override with ports: !override, as compose merges ports lists and Testsmith’s file isn’t Mr Onion’s to edit. Evidence. ✅
RUN-48 The start-of-run sweep removes orphaned artifacts-<id> folders Removed: a folder older than the hour of a run never saved, of a saved run whose evidence moved out, and of a saved run pointing only at a folder whose name starts the same or at another drive. Evidence. ✅
RUN-49 A failed test’s evidence is listed once, and a file listed twice keeps both links A file is listed once, by its full path; a file listed twice is moved once and both links point at the copy. Evidence. ✅
RUN-50 The data folder’s one-time move (automated: src/lib/storage/data-dir.test.ts, a decision table on folders in memory, then real renames under a temporary folder) Only the old folder: renamed to the new one, one notice saying where it went. Both: the new one, nothing moved or merged, the notice names the old one as left in place. Only the new one, or neither: the new one, no notice. The rename fails: the old folder for that process, the notice gives the reason, no second try in that process; another process moved it first: the new one, no failure notice; a process left on the old folder follows it once another has moved it, rather than recreating it empty. A generation record made before the move finds its site suite under the new name. A folder set by MRONION_DATA_DIR, TESTFORGE_DATA_DIR or --data-dir is used where it is, and the default is not moved. Windows’ %LOCALAPPDATA%\TestForge → MrOnion and XDG’s testforge → mr-onion, files included; macOS’s paths by unit test only. Phase 48 Decision 6 (TFP-277). ✅
RUN-51 The report footer names the product by the run’s version (automated: src/lib/version.test.ts) Below 0.3.0, “TestForge "; from 0.3.0, "Mr Onion "; a version that can't be read, "version ", neither name; no stamp, "TestForge version not recorded". Phase 48 Decision 7 (TFP-277). ✅
RUN-52 Manual, Windows: a 0.2.0 data folder (made by 0.2.0’s code at cf501ad: a run, its site suite and a candidate) under a temporary LOCALAPPDATA, never the real one The first start moves it with one notice and loses nothing; the saved site suite is reused; its candidate is listed; a 0.2.0 run serves as --baseline latest with the same split as on 0.2.0. Measured 2026-11-09 against local Toolshop clean: the built app’s first request moved all 655 files, one notice over two requests, /api/candidates listed the 0.2.0 candidate both times; mr-onion run’s first start moved it, its stderr only the notice, the site suite firstRun: false, 470 reused, 0 generated; both folders present: the new one, the old one’s 655 files left; a file held open by another process: EPERM, the old folder used and the notice saying so, then the next start moved all 655; --data-dir left a 0.2.0 folder in the default place where it was; --baseline latest from 0.2.0’s folder (0.2.0’s code) and from the moved one (the new code): the same run 0a01b729, “0 new failure(s), 46 known, 0 fixed”, exit 0 both. The real %LOCALAPPDATA%\TestForge was 41 files, with no MrOnion, before and after. ✅
RUN-53 Manual, Linux (the WSL worktree of the branch): the same 0.2.0 folder under a temporary XDG_DATA_HOME Measured 2026-11-09 against local Toolshop clean (a later start: new product ids): mr-onion run’s first start moved testforge to mr-onion, all 654 files, its stderr only the notice; the site suite firstRun: false, 379 reused, 91 generated for the reseeded product pages; --baseline latest from the moved folder took run 0a01b729, “0 new failure(s), 46 known, 0 fixed”, exit 0; a 0.2.0 folder in the default place beside --data-dir stayed where it was. ~/.local/share held neither folder before or after. macOS: unit tests only (RUN-50). ✅
RUN-54 The request guard stops a beacon and a keepalive fetch in every frame, measured at the server (automated: src/lib/generate/read-only-guard.test.ts, TFP-199’s and TFP-284’s real-browser cases behind the recording proxy) From Playwright 1.62, a beacon or keepalive fetch sent as a page is left reached the server although the route aborted it. The guard also stops sendBeacon, and a keepalive fetch or a fetchLater() (any method but GET, HEAD and QUERY), in the page, in the main frame and in an iframe loaded before the guard or added under it. Nothing but GETs reaches the server, whether the page follows a link or stays. TFP-199’s classes hold: a beacon is noted, any other such write is a breach, named. A reference the page saved at load: RUN-55. Red first on 1.62.1 without the stub: both cases’ beacons, the iframes’ keepalive cart POSTs, and the page’s and an iframe’s fetchLater POSTs reached the server. ✅
RUN-55 The request guard covers references to sendBeacon, fetch and fetchLater a page saves at load, measured at the server (automated: src/lib/generate/read-only-guard.saved.test.ts, TFP-287’s real-browser case behind the recording proxy) A read-only test that navigates before the guard first calls tfReadOnlyWrap, an init script that wraps the three in every frame before the page’s own scripts run, dormant until the guard arms. A page script that saved its own references at load: under the guard, a beacon and a keepalive POST sent as the page is left and a fetchLater POST made while it is alive reach nothing; the fetchLater POST is named as a breach. The same references used at load, before the guard, with and without the wrapper: all six POSTs reach the server, so loading is unchanged. Red first on TFP-284’s guard: the three guarded POSTs reached the server. Owner rulings, 2026-11-15: a keepalive POST or fetchLater built from another frame’s Request, and a page that tries to switch the guard off (it writes the old flag and replaces the arm function), reach nothing and the fetchLater POSTs are named; red first on the previous build, where all four POSTs arrived. A fresh about:blank iframe’s fetch is the wrapper (Playwright runs init scripts there), and its keepalive POST reaches nothing. A fetchLater queued at load: known limitation 12. ✅
RUN-56 The ownership confirmation is saved with the site and read back whole (automated: src/lib/ownership/confirmation.test.ts, TFP-218) POST /api/site-ownership saves it to <data>/site-suites/<site>/ownership.json in the local data folder, never the repo, and GET returns every field of Phase 43 Decision 1: hosts (lower case), basis, the exact wording, name, contact email, the dates confirmed and expiring, and the active and load ticks, with its status valid. No confirmation reads as missing. So does a file that can’t be read, or one whose confirmation lacks a field or holds one of the wrong type (an empty one, an expiry that isn’t a date, no hosts, an unknown basis, non-boolean ticks): it fails closed, and a malformed history entry is dropped. GET also gives each basis’s wording, the written-permission one with the practice-site rule (Decision 12). ✅
RUN-57 A confirmation with a missing or malformed field is refused, with every reason (automated: as RUN-56) Each is a 400 naming its reason, and nothing is saved: no hosts; a host that isn’t text, or is empty; an address or a port in place of a host name; a wildcard; a host that neither the site’s address nor the page data of its saved Standard suite uses (that data counts a page’s host, and a request’s host, with or without a sample address); no or an unknown basis; no wording, or another basis’s; no name; no or an invalid contact email; an expiry that isn’t a date, is ambiguous (a date-time with no zone, free text such as Nov 20 2026), isn’t after the date confirmed, or is beyond 1 year. A request missing everything names all 5 missing fields, each by its reason. ✅
RUN-58 The confirmation expires after 90 days by default, or at a chosen date up to 1 year, and then counts as absent (automated: as RUN-56) With no expiry given it expires 90 days after it was confirmed. A date alone is midnight UTC. Exactly 1 year is accepted; a second more is refused. Confirmed on 29 February, the limit is 28 February a year on. Its status is valid up to its expiry and expired from it. ✅
RUN-59 Withdrawing or replacing a confirmation keeps the old one as evidence (automated: as RUN-56) POST /api/site-ownership/withdraw marks it withdrawn with the date, keeps it, and its status is withdrawn; withdrawing again, or on a site with none, is a 404. Replacing one moves the old one to the history, and confirming again after a withdrawal keeps the withdrawn one there. Five confirmations saved at the same moment all succeed, and the history keeps every one but the last, which is current. ✅
RUN-60 Nothing is sent to the contact email (automated: as RUN-56) The store and its two routes import only the file system, next/server and Mr Onion’s own modules, and contain no network or mail call. No mail code exists (D25 adds it). ✅