Section 3.10 of the test plan, which keeps the app overview, the feature status, the known limitations and the index of areas. Since TFP-253 each area’s cases are in a file of their own; a ticket names the area it touches. Phase 47’s cases (docs/decisions/phase-47-ci-integration-decision-record.md, Regression Policy); the command is mr-onion run (README, “Running Mr Onion in a pipeline”).
| ID | Test | Expected Result | Status |
|---|---|---|---|
| CI-01 | A malformed configuration (automated: src/cli/config.test.ts, 16 malformed inputs and the config file’s three failures) |
Exit 3; the message starts with the field it names (practices[1], servers[0].port, --port, a config file’s unknown field, …). A valid one gets the UI’s defaults: Standard, all ten areas, fast. |
✅ |
| CI-02 | An unreachable URL (automated: src/cli/run.test.ts — “an unreachable URL: the crawl finds no page, exit 2, never 0”) |
Exit 2, naming the URL and the crawl’s first warning. | ✅ |
| CI-03 | A browser that can’t start (automated: src/cli/run.test.ts — “TFP-272: a browser that can’t start is reported, and the run exits 2”) |
Exit 2; the crawl’s “The browser for the crawl could not start: …” message is printed (TFP-272’s CLI half). | ✅ |
| CI-04 | The verdict (automated: src/cli/run.test.ts — “the verdict: …”) |
0 with no failure (flaky and not run don’t fail); 1 with a failed or blocked test; --report-only 0; 2 with no report, a cancelled run, or no test executed, even with --report-only. |
✅ |
| CI-05 | JUnit against the report (automated: src/cli/junit.test.ts) |
Totals equal the report summary’s (tests planned, failures Failed, errors Blocked, skipped Not run); the failing cases are the report’s failing tests by name; messages escaped, without colour codes. | ✅ |
| CI-06 | The same suite as the UI (automated: src/cli/run.test.ts — “for the same page data the CLI’s request is the UI’s, and its suite the route’s, byte for byte”) |
For the same page data, the CLI’s generate request equals the UI’s body after the JSON round trip, and its suite equals POST /api/generate’s byte for byte. |
✅ |
| CI-07 | No model in CI (automated: src/lib/pipeline/generate.test.ts) |
With models off, a configured OpenAI key and local model are never asked: the template’s suite, no generatedBy, no gap fill. With them on (the route), the key is used as before. |
✅ |
| CI-08 | --server start and stop (automated: src/cli/servers.test.ts; manual: the local Toolshop with --server-timeout 20, shorter than its start) |
Each server answers before the run, over IPv4 or IPv6 loopback; afterwards no port answers and no process is left. One that never answers, exits, or finds its port taken ends with a named error and stops every server already started. Measured 2026-11-08: the Toolshop timed out at 20 s, exit 2; its start was ended and its stop run, and 90 s later no container, port or Toolshop process. | ✅ |
| CI-09 | Manual: local Toolshop clean, started by --server (scripts/toolshop-local/serve-toolshop.mjs sprint5 --ready-port 4299) |
Exit 1 (clean’s Standard suite fails 46 tests in every run, Phase 39 Gate C), and with --report-only exit 0; junit.xml’s counts equal report.json’s; afterwards no container runs and nothing listens on 4200, 8091 or 4299. Criterion 2 amended 2026-11-08 (Emil); clean exiting 0 comes with its baseline in TFP-246. Measured 2026-11-08: exit 1, counts equal (470 tests, 46 failures, 0 errors, 18 skipped), nothing left running; with --report-only, exit 0, the same counts and the same 46 failing names. |
✅ |
| CI-10 | Manual: local Toolshop with-bugs, the same way | Exit 1; the failing JUnit cases are the report’s failing tests; nothing left running afterwards. Measured 2026-11-08: exit 1, 84 failing cases by the same names in both (Gate C’s 84), nothing left running. | ✅ |
| CI-12 | An interrupt, a cancelled job or a crash (automated: src/cli/exit.test.ts; manual, on Linux: SIGINT to testforge run while its --server starts) |
Every server it started is stopped first, then exit 2, never 1; a second interrupt doesn’t cut the stop short. Measured 2026-11-08 in WSL: SIGINT 4 s in, exit 2, the server’s process group gone (0 processes left). | ✅ |
| CI-11 | Manual: ports while a run goes (the CLI’s process tree, sampled every 3 s) | No listening port of Mr Onion’s own, other than Phase 39’s slot server on a capped host; the only other listener is the server --server started. Measured 2026-11-08 on both Toolshop runs: the one listener was the wrapper’s ready port 4299. |
✅ |
| CI-13 | A run against its baseline (automated: src/lib/run/diff.test.ts — “TFP-246: failures against a baseline …” and “diffRuns carries the failures …”; src/cli/baseline.test.ts — “with a baseline, only new failures fail the build”) |
Per test and browser: new is failing now (Failed or Blocked) and passing, not run or absent in the baseline; known is failing in both; fixed is failing in the baseline and passing (or flaky) now. Only new failures fail the build (exit 1; --report-only 0); a run that executed nothing is still 2. The comparison is diffRuns’ (failures), the UI’s run diff, with each page’s path in a title read as its page type over both runs’ pages (/product/<id> -> /product/:id; a whole path only, not inside /product/12, /en/product/1 or /product/1.html; a page that is its own type, like /contact, unchanged; a type of which either run tested two pages kept exact, so a failure moving between them is new, not known), and both runs’ titles redacted first, so a redacted baseline matches. |
✅ |
| CI-14 | Finding the baseline (automated: src/cli/baseline.test.ts) |
--baseline latest takes the newest finished run (by time, not text) with a report of the same tests, none being a first run; never one whose saved verdict failed: on a new failure against its baseline (even --report-only’s exit 0), couldn’t run (a first run included), or failed without --baseline; a first run (--baseline latest with none yet, exit 1, firstRun) and a run with no verdict (the UI’s) are taken; each run’s verdict is saved with it and in --out/run.json, and one that can’t be saved is said, the reports still written; another site’s, other areas’, another audit mode’s, interactive’s or login’s, a cancelled, report-less or untimed run is passed over. --baseline <run.json> from outside the data folder is copied into it and compared there. A file that can’t be read, isn’t JSON or a run, is another site’s, has no report or no finish time, or has the id of a different stored run, is exit 3, naming baseline, before any server starts or any crawl; --out’s redacted copy of a stored run is that run. A run that can’t be compared still writes its reports, then exits 2. run.json, baseline.json and the comparison’s log lines are redacted like every other output. |
✅ |
| CI-15 | Manual: local Toolshop with-bugs twice, --data-dir kept, the second with --baseline latest |
The first exits 1; the second exits 0, every failure known. Measured 2026-11-08: first exit 1, 481 planned, 84 failed (Gate C’s 84); second exit 0, “0 new failure(s), 84 known, 0 fixed”. |
✅ |
| CI-16 | Manual: with-bugs, one bug fixed in the local copy, then one new bug planted (each against the second run of CI-15) | Fixed: exit 0, that test reported fixed. Planted: exit 1, naming only that test. Measured 2026-11-08: the quantity stepper’s empty plus()/minus() given clean’s bodies: exit 0, “0 new, 83 known, 1 fixed”, the fixed one TFFU-217: quantity steppers change the value by one on /#/product/6. That fix undone, and the forgot-password page given its own title: exit 1, “1 new, 84 known, 0 fixed”, the new one TFSE-040: keeps page title stable for /#/auth/forgot-password (Passed -> Failed). The local copy restored with git checkout after each. |
✅ |
| CI-17 | Manual: with-bugs with one page type’s structure changed (the forgot-password form’s e-mail field text -> email, its heading h3 -> h2), against the second run of CI-15 |
The changed type is regenerated, not failed, as in the UI path. Measured 2026-11-08: the site suite found /#/auth/:slug changed when the suite was generated: 121 tests regenerated, 1 retired (a Not run persistence test of the changed field), 359 reused; exit 0, “0 new, 84 known, 0 fixed”; the regenerated heading test passed. This is RUN-29’s path (the type’s signature compared when the suite is generated). RUN-30’s, the stale recheck after a run, ran too (44 failing reused tests, all on unchanged pages) but had nothing to regenerate: it fires only for a page that changes between the inspection and the run, which this didn’t, so it is shown here only by its own tests (Phase 42). |
✅ |
| CI-18 | Manual: local Toolshop clean twice, --data-dir kept, the second with --baseline latest (moved from TFP-245’s criterion 2, Emil, 2026-11-08) |
The second exits 0, its known failures listed as known. Measured 2026-11-08: first exit 1, 470 planned, 46 failed (Gate C’s 46); second exit 0, “0 new failure(s), 46 known, 0 fixed”, the same 46 failing names in both, junit.xml 470 tests, 46 failures, 0 errors, 18 skipped. Toolshop stopped afterwards: no container, nothing listening on 4200 or 8091. Across two starts (ruling 1, 2026-11-08): a later run after a Toolshop restart had exited 1, “4 new, 42 known”: the restart reseeds the products with new ids, so the four tests of /product/<id> had new titles. With tests matched by page type, clean twice on two separate starts, both with --baseline latest and one --data-dir (the final code, after the second review): first “no earlier run … a first run”, exit 1, 470 planned, 46 failed (/product/01M6XZCDA5KVEETNDRJWQJN558), verdict {exitCode 1, firstRun true}; second, after a restart, took the first as its baseline: exit 0, “0 new failure(s), 46 known, 0 fixed” (/product/01M6XZY5YJ6JBK2910715MBVA5), junit.xml 470 / 46 / 0 / 18, verdict {exitCode 0, newFailures 0, firstRun false}. |
✅ |
| CI-19 | Settings under either name (automated: src/lib/settings.test.ts, a decision table of the two names) |
MRONION_X alone is used with no notice; TESTFORGE_X alone is used with one notice naming both; both the same, no notice; both different, MRONION_X wins and the notice says so; MRONION_X blank (empty or spaces) with TESTFORGE_X set, the old value with a notice that the new name is blank (Decision 5, amended 2026-11-09), and a blank MRONION_X alone is returned blank. A real process reading two old names twice prints exactly two notices, one per setting. Phase 48 Decision 5 (TFP-276). |
✅ |
| CI-20 | Both commands (automated: src/cli/bin.test.ts, real processes) |
mr-onion run --help and testforge run --help exit 0 with the same usage; an invalid flag exits 3 from both with the same message. testforge adds one line on stderr, “testforge is now mr-onion; the old name stops working at 1.0.”; mr-onion adds nothing. Phase 48 Decision 5 (TFP-276). |
✅ |
| CI-21 | A run’s child environment (automated: src/lib/gap-writer/hardening.test.ts, src/lib/run/split.test.ts, tests that read their own environment in a real run) |
Every setting the run passes on (the interactive switch, the timing file, and the rest through runEnv) arrives under both names with the same value, in the read-only pass and in the serial group’s; neither name of the server’s own settings is inherited. Phase 48 Decision 8 (TFP-276). |
✅ |
| CI-22 | Manual: mr-onion run and testforge run against one start of local Toolshop, clean then with-bugs, each with its own new --data-dir |
The same exit code, the same summary, and the same JUnit cases with the same results; testforge adds only its stderr line. Measured 2026-11-09: clean, both exit 1, 470 planned, 406 passed, 46 failed, 18 not run, JUnit 470 / 46 / 0 / 18, every test’s title and status identical (SHA-1 of the list da92855); with-bugs, both exit 1, 481 planned, 377 passed, 84 failed, 20 not run, JUnit 481 / 84 / 0 / 20, identical (5288925); mr-onion’s stderr empty both times. Not identical text: the run’s random probe routes, the time-capped “routes not checked” lists, measured page weight and CLS, and the with-bugs site’s own rendering findings, which differ between any two runs. |
✅ |
| CI-23 | The CLI and JUnit say Mr Onion (automated: src/cli/*.test.ts; manual: RPT-22’s check) |
Every message starts mr-onion: or names Mr Onion; junit.xml’s <testsuites name> is “Mr Onion”, and a record with no suite falls back to it; the counts equal report.json’s. Phase 48 (TFP-279). Measured 2026-11-09 on the local comparison fixture: Standard <testsuites name="Mr Onion" tests="123" failures="9" errors="0" skipped="10">, Comprehensive security tests="20" failures="0", each equal to its report.json. |
✅ |
| CI-24 | The image’s pins (automated: scripts/image/verify-image.test.mjs; at every build, scripts/image/check-pins.mjs in the Dockerfile) |
The build fails unless Node’s major is .nvmrc’s, playwright-core, playwright and @playwright/test are the lockfile’s, Chromium’s headless shell for that Playwright is installed in PLAYWRIGHT_BROWSERS_PATH, and a given build version is package.json’s. The image is Ubuntu 24.04 (CI’s runner pin) with base images pinned by digest. Measured 2026-11-12: {"os":"24.04","node":"v24.14.1","playwright":"1.59.1","chromium":"147.0.7727.15","chromiumRevision":"1217","version":"0.3.0","commit":"3136c3e"}; with PLAYWRIGHT_BROWSERS_PATH=/nowhere, “Chromium’s headless shell 1217 isn’t installed in /nowhere”, exit 1. Phase 47 (TFP-247). |
✅ |
| CI-25 | The built image runs and is checked (automated: scripts/image/verify-image.mjs, the release workflow’s gate; its checks in scripts/image/verify-image.test.mjs) |
The pins as CI-24; run --help exits 0 with the usage; a run on the comparison fixture, started by the image itself with --server: --report-only exits 0, JUnit totals equal the report’s, at least one test executed, the report names the build’s version and commit; without --report-only, at least one failed test and exit 1, so the failing path runs on every release (TFP-248); an unreachable URL exits 2. Every run is the caller’s user (--user, as the action runs it), and on Linux every file the run wrote into the workspace is that user’s (TFP-248). Any check that fails exits 1, and nothing is published. Measured 2026-11-12: every check held (fixture: 123 planned, 113 executed, 104 passed, 9 failed, 10 not run, CI-23’s counts; 42 s and 43 s, and again after the review’s fixes, 42 s and 41 s); an image with its browsers moved failed 4 of 5 checks, exit 1. Measured 2026-11-13 (TFP-248, WSL, uid 1000): every check held, the ownership check included (fixture 123 / 113 / 104 / 9 / 0 / 10; 45 s and 42 s); on a fixture run that passes everything (--practices functional --pages /, 14 of 14 passed, in a throwaway copy of the script), “the same run’s verdict” failed with “no test failed (failed=0), so the failing path wasn’t run” and “exited 0, not 1”, exit 1; with --user taken out (the image run as root), “the workspace’s files are the caller’s” failed: “49 file(s) in the workspace aren’t owned by uid 1000: out (uid 0), out/evidence (uid 0), out/junit.xml (uid 0), …”. |
✅ |
| CI-26 | Manual: local Toolshop clean, then with-bugs, one start each: mr-onion run locally and from the image (docker run --init --ipc=host --network host), each with its own new --data-dir |
The same exit code, the same summary, the same JUnit totals, and every test’s title and status identical (Phase 47 Acceptance Criteria 4). Measured 2026-11-12 (Windows 11, Docker Desktop 29.4.0): clean, both exit 1, 470 planned, 406 passed, 46 failed, 18 not run, JUnit 470 / 46 / 0 / 18, the failing JUnit cases the report’s, list SHA-1 17ec8e9 in both (local 4 min 53 s, image 4 min 45 s); with-bugs, both exit 1, 481 planned, 377 passed, 84 failed, 20 not run, JUnit 481 / 84 / 0 / 20, 5ca6b7a in both (6 min 2 s, 5 min 57 s). CI-22’s counts on both. Afterwards no container, nothing listening on 4200, 8091 or 4410. |
✅ |
| CI-27 | The image as any user (automated: the ownership check in scripts/image/verify-image.mjs, on Linux, and ownershipFailures in its tests; manual, in WSL) |
docker run --user "$(id -u):$(id -g)" runs: the run folders, the runs folder used without --data-dir, and HOME are writable by any user. Every file the run writes into the mounted workspace (--out, --data-dir) is owned by that user, not root. Phase 47 (TFP-248). Measured 2026-11-13 in WSL (uid 1000), on the comparison fixture: TFP-247’s image, run as root, left out and out/report.json owned by uid 0; with --user 1000:1000 it exited 2 with “EACCES: permission denied, mkdir ‘/opt/mr-onion/.playwright-runs’”. The rebuilt image, with --user: verify-image held every check. The action’s runs (CI-29’s rehearsal) left 0 of 851 files not owned by uid 1000. |
✅ |
| CI-28 | The action’s job summary (automated: action/summary.test.mjs) |
The heading is the exit code’s verdict. Exit 0 with failing tests (report-only) says they were reported, not that nothing failed. The counts are the report’s. The failing tests are listed, or with a baseline the new ones and the fixed ones, 20 at most and the rest counted. With no report it says the run couldn’t run, or the configuration was rejected. Phase 47 (TFP-248). |
✅ |
| CI-29 | Gate A: the action in Mr Onion’s own CI (.github/workflows/action-gate.yml; manual, its rehearsal in WSL with the action’s own run.sh and summary.sh) |
The image built in the job, never pulled, against the local Toolshop started in the job. Clean, report-only: exit 0. Clean again with baseline: latest: exit 0, the step green. With-bugs: exit 1, the step failed. A URL nothing answers on: exit 2, the step failed. Each of the first three writes report.html, junit.xml and report.json, attaches them as an artifact and writes a job summary. Every file in the workspace is the job user’s (Phase 47 Acceptance Criteria 5, TFP-248). Rehearsed 2026-11-13 in WSL: clean report-only exit 0, 470 planned, 406 passed, 46 failed, 18 not run (3 m 52 s); clean against it exit 0, “0 new, 46 known, 0 fixed” (4 m 15 s); with-bugs exit 1, 481 planned, 377 passed, 84 failed, 20 not run (5 m 47 s); the broken URL exit 2; 0 of 851 files not uid 1000. CI-26’s counts on every run. On GitHub, 2026-11-13 (Action Gate, PR #31, 9b5e573, ubuntu-24.04): every step green and every check held. The image built in 40 s; the clean Toolshop started in 2 m 58 s, with-bugs in 28 s. Clean report-only: exit 0, 470 / 406 / 46 / 18 (3 m 44 s). Clean against it: exit 0, “0 new failure(s), 46 known, 0 fixed” (3 m 58 s). With-bugs: exit 1, the step failed, 481 / 377 / 84 / 20 (5 m 18 s). The broken URL: exit 2, the step failed. The three runs wrote report.html, junit.xml and report.json, and the artifacts clean-baseline, clean and with-bugs were attached. Every file was the job user’s (runner), and each run wrote a summary. The job took 17 m 12 s. |
✅ |