mr-onion

Phase 47 Decision Record — TestForge in a Target’s CI Pipeline

Date: 2026-10-27 Scope: TestForge runs as one step in another project’s CI pipeline:

Status: Locked (Emil, 2026-10-27). Complete 2026-11-13, released as 0.4.0: TFP-263, TFP-272, TFP-245, TFP-246, TFP-247 and TFP-248 done (after Phase 48, the rename to Mr Onion: Decision 11, amended 2026-11-08). Gate A passed (owner ruling, 2026-11-13). The phase completes at TFP-248, with v0.4.0 (owner decision, 2026-11-13; TFP-249 stays deferred as D31). The image’s first publish is v0.4.0’s (owner decision, 2026-11-12): tagged on 436b954 after the owner’s confirmation (2026-11-13), once Action Gate, dispatched on that commit, was green. ghcr.io/emil3663/mr-onion:0.4.0 was published (sha256:8a9b2be313aa2ae802aa55b381c699965794f049da4585e3578a2a5469119a38) and is public: it pulls with no login. The release uses the changelog’s [0.4.0] section. Tracking issue #24, opened 2026-11-08.

Context

Decisions

  1. What runs in CI, first: Standard, read-only gate. (Emil, 2026-10-27)
    • Comprehensive runs follow when D5’s all-areas run exists; one Comprehensive area may be asked for by flag before that.
    • The interactive gate is opt-in by flag. The active gate never runs in CI before Phase 43 and TFP-249.
  2. The first supported target is one the job starts itself on localhost. (Emil, 2026-10-27)
    • The usual pattern: build the app, start it in the job, test http://localhost:<port>. It is a local host, so it runs uncapped and timed checks are judged, and it needs no ownership confirmation.
    • A deployed URL (a preview environment) is allowed, and is capped like any remote host, with timed checks Not run and the report saying so, until Phase 43 gives it a confirmation.

    Amended 2026-11-02 (Emil): the CLI can start the app itself. --server "<command>" --port <n>, repeatable for a backend and a frontend, starts each command, waits until its port answers (with a timeout that ends the run with exit 2), runs TestForge against localhost, then stops every server it started, whatever the result. Modelled on scripts/with_server.py in Anthropic’s webapp-testing skill (github.com/anthropics/skills, Apache-2.0): the pattern, or the code with its notice added to THIRD_PARTY_NOTICES.md if any is reused. A team can still start the app in its own job step and pass only the URL.

  3. TestForge in CI is still single-user. (Emil, 2026-10-27) Phase 16 Decision 1 is amended, not reversed: TestForge runs on the tester’s own machine or on their own team’s CI runner, never as a shared service.
    • The CLI starts no HTTP server and opens no port of its own beyond Phase 39’s slot server on loopback.
    • It executes only code it generated in that job, or the site suite restored from that team’s own state.
    • Credentials come only from the pipeline’s secrets as environment variables (Phase 20), redacted from every output.
  4. One pipeline, two front ends. (Emil, 2026-10-27) The crawl, generate and run orchestration moves out of the API routes into src/lib/pipeline/. The routes and the CLI both call it; neither has its own copy.
    • Rejected: the CLI importing route handlers in-process, as the benchmarks do — fast to build, but it ties the CLI to Next.js request objects and to route internals.
    • The move is behaviour-preserving (behavior-preservation-verification skill): template output byte-identical on every fixed input, and the UI’s runs unchanged.
  5. What fails the build. (Emil, 2026-10-27)
    • Exit 0: no new failures. 1: test failures. 2: couldn’t run (target unreachable, crawl found nothing, browser failed) — never a pass. 3: invalid configuration.
    • With a baseline, only failures the baseline didn’t have fail the build; known failures, fixed tests and findings are reported.
    • Without a baseline (a first run), any failing test fails the build; --report-only reports and exits 0 unless it couldn’t run.
    • Findings (SEO, usability, performance improvements) never fail the build.
    • Rejected: a first run that only reports by default — a broken site would pass its first CI run.
  6. Where state lives between runs. (Emil, 2026-10-27) The data folder is pointed at a workspace folder (--data-dir) that the pipeline restores before the run and saves after it (a CI cache or artifact). Committing a site suite into the target’s repo is documented as an alternative for teams that want to review it.
    • Rejected as the default: committing state to the target’s repo, since every run’s changes would become commits someone has to make.
  7. Outputs, in --out: report.json (today’s report, plus the TestForge version and commit), the HTML export, JUnit XML, the executed suite.spec.ts, and evidence for failing tests only. Redaction applies to every file. (Emil, 2026-10-27) Amended 2026-11-08 (Emil, TFP-246): also page-data.json (TFP-245), run.json (the saved run with its verdict, for a later job’s --baseline) and baseline.json (the comparison); /api/runs/diff gains failures (known, new, fixed), an addition.

  8. Distribution: a Docker image first, an npm package later. (Emil, 2026-10-27)
    • The image pins Node, Playwright and Chromium, so “the same site gives the same suite” holds between machines. Published to GitHub’s container registry, tagged with the TestForge version (TFP-257).
    • An npm package is deferred (D32): it would leave Chromium’s version to the runner.
  9. No model in CI. (Emil, 2026-10-27) The gap writer, the local model and the OpenAI path are off in CI in this phase, whatever the site’s settings say. Candidates and promoted library entries still run: they are committed code, not model calls.

  10. GitHub Actions first. (Emil, 2026-10-27) A composite action wraps the image. Other CI systems run the image with the documented command; the guide doesn’t claim support it hasn’t tested.

  11. Order: after Phase 39, before Phase 43. (Emil, 2026-10-27)
    • Decisions 1 and 2 need nothing from Phase 43, and this answers the outside request soonest. TFP-249 waits for Phase 43.
    • Rejected: after Phase 43 (a deployed URL would get performance judged from the start, but the request waits longer); after the Standard sets (CI much later).
    • Amended 2026-11-08 (Phase 48): TFP-247 and TFP-248 publish and document mr-onion (image ghcr.io/emil3663/mr-onion, command mr-onion run); they start after Phase 48 merges. See phase-48-rename-to-mr-onion-decision-record.md, Decision 13.

Acceptance Criteria

  1. The pipeline move (TFP-263)
    • Template output is byte-identical against the recorded baseline on every fixed input (npm run check:template-unchanged).
    • The self-test and regression suites pass in full.
    • src/app/api/crawl/route.ts and generate/route.ts hold request parsing and responses only; no crawl loop or generator choice remains in them.
  2. The CLI (TFP-245)
    • Against local Toolshop clean: exit 1; JUnit counts equal report.json’s; with --report-only, exit 0. Amended 2026-11-08 (Emil): it read “exit 0”, which Decision 5 rules out (no baseline: any failing test fails the build) on a site whose Standard suite fails 46 tests in every run (Phase 39 Gate C). “Clean exits 0” moves to TFP-246, with clean as its own baseline.
    • Against local Toolshop with-bugs: exit 1; the failing JUnit cases are the report’s failing tests.
    • An unreachable URL: exit 2. A malformed config: exit 3, naming the field.
    • For the same site and inspection, the CLI’s suite is byte-identical to the UI’s.
    • No port is opened other than Phase 39’s loopback slot server (checked while a run is in progress).
  3. Baselines (TFP-246)
    • with-bugs twice, the first as baseline: the second exits 0 and lists every failure as known.
    • One bug fixed in the local copy: reported fixed, exit 0. One new defect planted: exit 1, naming only that one.
    • A changed page type is regenerated, not failed (RUN-30’s behaviour).
    • Rulings (Emil, 2026-11-08, TFP-246): a test is matched across runs by its page type, not its exact path (/product/<id> gets a new id on each start of a reseeded target), with both runs’ titles redacted the same way first; --baseline latest never takes a run that failed on a new failure against its baseline (each run records its verdict), while a first run stays a baseline; the changed-type criterion is met by generation-time regeneration (RUN-29’s path), the stale recheck having nothing to do.
  4. The image (TFP-247)
    • Criterion 2’s clean and with-bugs checks give the same results from the image as from a local run.
    • The release workflow runs the built image and asserts on its output, so a green release could have gone red (verified-release-pipeline skill).
  5. The action (TFP-248) — Gate A
    • In TestForge’s own CI, the action runs against local Toolshop clean and with-bugs started in the job: clean passes, with-bugs fails, both reports are attached, and a job summary is written.
    • A wrong target URL turns the job red with exit 2.

Regression Policy

Each of these is entered in deferred-work-register.md (D31 to D35) in the same change as this lock.

Dependencies and order

Backlog

TFP-245 to TFP-249 are as written in workflow-and-ci-backlog-2026-10-26.md Part 1, with these changes once this record locks:

TFP-263 - Move crawl and generate orchestration out of the API routes

  1. TFP-257 steps 1–5 (independent).
  2. TFP-263.
  3. TFP-245.
  4. TFP-246 and TFP-247.
  5. TFP-248 — Gate A.
  6. TFP-249 after Phase 43.

Suggested Milestone Gates

Gate A — TestForge runs in a pipeline

Gate A verification (planning session, 2026-11-13; it built none of TFP-248): every checked criterion met. The owner’s ruling (2026-11-13): Gate A passes.

Check How Result
Scope git diff --stat main e406694 The action (action/), the Gate A workflow, the guide, the Toolshop start script for CI, the image change, verify-image.mjs and its tests, the README, the test plan and records; and the handover swap with its allowlist entry, as the handover asked
Criterion 5: clean passes, with-bugs fails, a broken URL is exit 2 Read action-gate.yml Clean runs twice, the first as the baseline, and must exit 0 (with no continue-on-error, so a new failure turns the job red). With-bugs must exit 1 with its step failed. A URL nothing answers on must exit 2 with its step failed. Each Toolshop run must write report.html, junit.xml and report.json; clean must write baseline.json; both artifacts must exist. A step with shell: bash checks them all and fails on any one
Criterion 5: on GitHub PR #31’s checks; the builder’s report Action Gate green on 9b5e573 (clean 0 against its baseline, 0 new / 46 known; with-bugs 1, 84 failed; broken 2); a second run on e406694 was in progress when checked
The exit code reaches the step Read action.yml and run.sh The step ends with the run’s own exit code, after the summary and the upload; docker’s own failures (125 to 127) and a run killed by a signal count as exit 2, never as a verdict
TFP-247 note 1: root-owned files Read run.sh, the Dockerfile, the gate’s ownership check The action runs the image as the job’s user, with the out and data folders made first. The image runs as any user (its run folders sticky and world-writable, like /tmp). The gate fails if any file the runs wrote isn’t the job user’s. The guide gives --user and a chown fallback for running the image without the action
TFP-247 note 2: the failing path Read verify-image.mjs The fixture’s verdict run must fail at least one test and exit 1 (shown red by the builder on an all-passing fixture)
Secrets Read run.sh Only variables named in env are passed, by name (-e NAME), so no value is on a command line; a name that isn’t a variable name is exit 3
The guide’s claims Read docs/CI_INTEGRATION.md It says what’s tested (GitHub Actions, a server started in the job) and what isn’t (the preview example, other CI systems), and what doesn’t run in CI and why

Not checked here: the Gate A run’s logs on GitHub beyond its checks page, and the unit suite (839 of 839 reported). The verifier has no browser or Docker in its VM.

For the owner’s ruling:

  1. Gate A passes (recommended).
  2. Gate A’s trigger. It runs only when a pull request changes the action, the image or the Toolshop scripts, not the CLI (about 17 minutes a run). Recommended: keep that, and add a weekly schedule. Also dispatch it by hand on the release commit before each version tag, so a CLI change that breaks the action is caught before a release publishes it.
  3. The guide’s “Before v0.4.0” notice comes out in the v0.4.0 commit (the handover’s item 2).

Ruled (owner decision, 2026-11-13):

  1. Gate A passes.
  2. Action Gate keeps its pull-request trigger and gains a weekly schedule. It is dispatched by hand on the release commit, and seen green, before asking to push a version tag (the handover’s item 2).
  3. The guide’s “Before v0.4.0” notice is removed in the v0.4.0 commit.

Gate B — A second run can be trusted