Date: 2026-10-27 Scope: TestForge runs as one step in another project’s CI pipeline:
Status: Locked (Emil, 2026-10-27). Complete 2026-11-13, released as
0.4.0: TFP-263, TFP-272, TFP-245, TFP-246, TFP-247 and TFP-248 done (after Phase 48, the rename to
Mr Onion: Decision 11, amended 2026-11-08). Gate A passed (owner ruling,
2026-11-13). The phase completes at TFP-248, with v0.4.0 (owner decision,
2026-11-13; TFP-249 stays deferred as D31). The image’s first publish
is v0.4.0’s (owner decision, 2026-11-12): tagged on 436b954 after the
owner’s confirmation (2026-11-13), once Action Gate, dispatched on that
commit, was green. ghcr.io/emil3663/mr-onion:0.4.0 was published
(sha256:8a9b2be313aa2ae802aa55b381c699965794f049da4585e3578a2a5469119a38) and is public: it pulls with no login. The
release uses
the changelog’s [0.4.0] section. Tracking issue
#24, opened 2026-11-08.
workflow-and-ci-backlog-2026-10-26.md; this record is TFP-244 and
supersedes that backlog’s Part 1./api/run executes the code it is given, accepted because the tool runs
only on the tester’s own machine. A CI runner is neither that machine nor
the deferred SaaS (Decision 2 there): it is the target team’s own
machine, running TestForge for that team alone.6d1e854, 2026-10-27) bound the server to loopback and
guards every /api/* request. A CI run needs no server at all./api/crawl, /api/generate, POST /api/run, which returns a 202 job
the UI polls). Nothing sets a process exit code.src/lib:
src/app/api/crawl/route.ts (460 lines) holds the crawl loop, its job
map and browser handling; src/lib has the pieces it calls
(introspect/routes, selection/*);src/app/api/generate/route.ts (290 lines) chooses between template,
site-suite reuse, library, candidates, gap writer and models;src/app/api/run/route.ts (205 lines) is thin: startRunJob
(src/lib/run/executor.ts) does the work.benchmarks/2026-10-07-tfp175-selection/crawl-and-select.mts and
benchmarks/2026-10-21-gate-d/generate.mts import the route handlers
in-process with a NextRequest. That proves it can be done; it is not a
product interface.json
reporter (executor.ts:253); the report is JSON plus an HTML export.
No JUnit.src/lib/storage/data-dir.ts; TESTFORGE_DATA_DIR overrides).
diffRuns(runIdA, runIdB) (src/lib/run/diff.ts) compares two saved
runs test by test. A CI runner starts empty every time.localhost, loopback, *.localhost,
*.test) or a host with an ownership confirmation (Phase 43, not built)
runs uncapped. In CI, a target started inside the same job on
localhost is a local host; a preview deployment’s URL is not.@playwright/test, so a saved
suite can already run in any CI with npx playwright test — but without
coverage, the gap list, reuse, diffing or gates. That stays the stopgap,
documented, not the product answer.package.json 0.1.0, no tags): TFP-257.localhost. (Emil, 2026-10-27)
http://localhost:<port>. It is a local host, so it runs uncapped and
timed checks are judged, and it needs no ownership confirmation.Amended 2026-11-02 (Emil): the CLI can start the app itself.
--server "<command>" --port <n>, repeatable for a backend and a
frontend, starts each command, waits until its port answers (with a
timeout that ends the run with exit 2), runs TestForge against
localhost, then stops every server it started, whatever the result.
Modelled on scripts/with_server.py in Anthropic’s webapp-testing
skill (github.com/anthropics/skills, Apache-2.0): the pattern, or the
code with its notice added to THIRD_PARTY_NOTICES.md if any is reused.
A team can still start the app in its own job step and pass only the URL.
src/lib/pipeline/. The
routes and the CLI both call it; neither has its own copy.
behavior-preservation-verification
skill): template output byte-identical on every fixed input, and the
UI’s runs unchanged.0: no new failures. 1: test failures. 2: couldn’t run
(target unreachable, crawl found nothing, browser failed) — never a
pass. 3: invalid configuration.--report-only reports and exits 0 unless it couldn’t run.--data-dir) that the pipeline restores
before the run and saves after it (a CI cache or artifact). Committing a
site suite into the target’s repo is documented as an alternative for
teams that want to review it.
Outputs, in --out: report.json (today’s report, plus the
TestForge version and commit), the HTML export, JUnit XML, the executed
suite.spec.ts, and evidence for failing tests only. Redaction applies
to every file. (Emil, 2026-10-27)
Amended 2026-11-08 (Emil, TFP-246): also page-data.json (TFP-245),
run.json (the saved run with its verdict, for a later job’s
--baseline) and baseline.json (the comparison); /api/runs/diff
gains failures (known, new, fixed), an addition.
No model in CI. (Emil, 2026-10-27) The gap writer, the local model and the OpenAI path are off in CI in this phase, whatever the site’s settings say. Candidates and promoted library entries still run: they are committed code, not model calls.
GitHub Actions first. (Emil, 2026-10-27) A composite action wraps the image. Other CI systems run the image with the documented command; the guide doesn’t claim support it hasn’t tested.
mr-onion (image ghcr.io/emil3663/mr-onion, command
mr-onion run); they start after Phase 48 merges. See
phase-48-rename-to-mr-onion-decision-record.md, Decision 13.npm run check:template-unchanged).src/app/api/crawl/route.ts and generate/route.ts hold request
parsing and responses only; no crawl loop or generator choice remains
in them.1; JUnit counts equal
report.json’s; with --report-only, exit 0. Amended 2026-11-08
(Emil): it read “exit 0”, which Decision 5 rules out (no baseline:
any failing test fails the build) on a site whose Standard suite fails
46 tests in every run (Phase 39 Gate C). “Clean exits 0” moves to
TFP-246, with clean as its own baseline.1; the failing JUnit cases
are the report’s failing tests.2. A malformed config: exit 3, naming the
field.0 and
lists every failure as known.0. One new
defect planted: exit 1, naming only that one./product/<id> gets a new id on
each start of a reseeded target), with both runs’ titles redacted the
same way first; --baseline latest never takes a run that failed on a
new failure against its baseline (each run records its verdict), while
a first run stays a baseline; the changed-type criterion is met by
generation-time regeneration (RUN-29’s path), the stale recheck having
nothing to do.verified-release-pipeline
skill).2.TEST_PLAN.md in the same commit (a new
CI- prefix; in its own area file if TFP-253 has split the plan by
then).Each of these is entered in deferred-work-register.md (D31 to D35) in
the same change as this lock.
TFP-245 to TFP-249 are as written in workflow-and-ci-backlog-2026-10-26.md
Part 1, with these changes once this record locks:
--server/--port (Decision 2, amended
2026-11-02), with acceptance cases: the local Toolshop’s start command
given as --server, the run passes on clean and fails on with-bugs, and
no server process is left afterwards; a server that never opens its port
ends the run with exit 2, also leaving nothing running.src/lib/pipeline/ (Decision 4) instead of reimplementing
anything, and opens no server.diffRuns on runs restored from the data folder.refactor, cisrc/lib/pipeline/crawl.ts: the crawl and selection, with progress and
cancellation as callbacks, no job map;src/lib/pipeline/generate.ts: today’s generate route body, taking a
plain object, returning the suite and its generation record;5b58370, on
a Linux clone: the crawl and four generate route test files 50 of 50
passed, 0 skipped; check:template-unchanged 5b58370: 172 inputs, 172
identical; tsc exit 0; neither route mentions the browser, the crawl
loop or a generator. Read: the generate route is the old body with its
four 400s returned as { ok: false, error } and respond returning
{ ok: true, body }, nothing else; the crawl route keeps the job map and
hands the crawl progress, isCancelled and onBrowser, and DELETE
closes the browser through it. Claude Code’s self-test (5 of 5),
regression (73 of 73) and unit run (729 of 729) weren’t re-run here.
Accepted differences (Emil, 2026-11-08): the tests stay with the
routes (they are the unchanged proof the record names); progress stays an
object updated in place, not a callback; a finished crawl reports
“completed” after its browser has closed, not just before. Found, not
this ticket: a Chromium launch failure leaves the crawl job “running”
forever, before and after; TFP-272 in the workflow backlog.Gate A verification (planning session, 2026-11-13; it built none of TFP-248): every checked criterion met. The owner’s ruling (2026-11-13): Gate A passes.
| Check | How | Result |
|---|---|---|
| Scope | git diff --stat main e406694 |
The action (action/), the Gate A workflow, the guide, the Toolshop start script for CI, the image change, verify-image.mjs and its tests, the README, the test plan and records; and the handover swap with its allowlist entry, as the handover asked |
| Criterion 5: clean passes, with-bugs fails, a broken URL is exit 2 | Read action-gate.yml |
Clean runs twice, the first as the baseline, and must exit 0 (with no continue-on-error, so a new failure turns the job red). With-bugs must exit 1 with its step failed. A URL nothing answers on must exit 2 with its step failed. Each Toolshop run must write report.html, junit.xml and report.json; clean must write baseline.json; both artifacts must exist. A step with shell: bash checks them all and fails on any one |
| Criterion 5: on GitHub | PR #31’s checks; the builder’s report | Action Gate green on 9b5e573 (clean 0 against its baseline, 0 new / 46 known; with-bugs 1, 84 failed; broken 2); a second run on e406694 was in progress when checked |
| The exit code reaches the step | Read action.yml and run.sh |
The step ends with the run’s own exit code, after the summary and the upload; docker’s own failures (125 to 127) and a run killed by a signal count as exit 2, never as a verdict |
| TFP-247 note 1: root-owned files | Read run.sh, the Dockerfile, the gate’s ownership check |
The action runs the image as the job’s user, with the out and data folders made first. The image runs as any user (its run folders sticky and world-writable, like /tmp). The gate fails if any file the runs wrote isn’t the job user’s. The guide gives --user and a chown fallback for running the image without the action |
| TFP-247 note 2: the failing path | Read verify-image.mjs |
The fixture’s verdict run must fail at least one test and exit 1 (shown red by the builder on an all-passing fixture) |
| Secrets | Read run.sh |
Only variables named in env are passed, by name (-e NAME), so no value is on a command line; a name that isn’t a variable name is exit 3 |
| The guide’s claims | Read docs/CI_INTEGRATION.md |
It says what’s tested (GitHub Actions, a server started in the job) and what isn’t (the preview example, other CI systems), and what doesn’t run in CI and why |
Not checked here: the Gate A run’s logs on GitHub beyond its checks page, and the unit suite (839 of 839 reported). The verifier has no browser or Docker in its VM.
For the owner’s ruling:
v0.4.0
commit (the handover’s item 2).Ruled (owner decision, 2026-11-13):
v0.4.0 commit.