mr-onion

Phase 45 Decision Record — The Docker Sandbox for Model-Written Tests

Date: 2026-10-20 Scope: Every test a model wrote runs inside a Docker container. That means:

The container can reach nothing on the machine and nothing on the network except the run’s target hosts. Only the results leave it. The template’s tests and promoted library entries, which are reviewed code, keep running as today.

Status: Locked (Emil, 2026-10-20). Not started. Decisions, acceptance criteria and tickets approved.

Context

Decisions

  1. What runs in the sandbox, and what doesn’t.
    • Inside: every test a model wrote:
      • gap tests and their revisions;
      • candidates, bound or not;
      • scenario tests.

It doesn’t matter which run mode or which site.

  1. The container.
    • Image: Microsoft’s Playwright image, pinned to the exact version package.json resolves (@playwright/test), and to its digest.
      • A Playwright upgrade moves the pin in the same commit.
      • The tests run in Chromium only (Phase 16).
    • One container per run, not per test. It holds every model-written test of that run and is removed when they finish.
    • Mounts:
      • the run’s model-written specs and their config, read-only;
      • one empty output folder, writable.

Nothing else: not the repo, not the data folder, not the home folder.

  1. The network: only the run’s target hosts (Emil, 2026-10-20).
    • The way out: the container’s only route is a small forward proxy started with it. The proxy allows:
      • the run’s target hosts;
      • for a remote site, the hosts its pages load, as recorded by the main run’s crawl.

Anything else is refused and named in the results.

  1. What leaves, and how it is checked.
    • Only the output folder leaves:
      • Playwright’s JSON report;
      • for failures, traces, screenshots and video.
    • Capped in size. A run over the cap keeps its report and drops the largest attachments, saying so.
    • TestForge reads it as untrusted data, like the page text in TFP-202.
      • It is parsed, never executed.
      • Every captured file goes through Phase 20’s credential scan, as today.
  2. No Docker, no model-written tests (Emil, 2026-10-20).
    • If Docker isn’t running, or the image or proxy won’t start, every model-written test in the run is Not run, with the reason “sandbox unavailable: start Docker Desktop”.
    • The rest of the run goes ahead.
    • They are never run outside the sandbox. There is no setting to allow it.
  3. The cost: at most 60 seconds per run (Emil, 2026-10-20).
    • What’s measured: start, proxy, the tests’ own time, and removal, on local Toolshop. The image pull is done once and measured separately.
    • Up to 60 s more than the same tests run on the host today: kept.
    • Over 60 s: the design is revisited before it is kept, for example by keeping one container warm between runs. Recorded either way.
  4. Results are the same.
    • On the fixture, every model-written test gets the same result inside the sandbox as it got on the host, apart from tests that reach a host outside the allowed set. Those are now refused, and each is named.

Acceptance Criteria

  1. What runs where
    • The model’s tests: on the fixture, with the gap writer (a stand-in writer) and candidates on, the run record shows every model-written test ran in the sandbox.
    • The rest: the template’s tests and promoted entries ran on the host.
    • The writer: runs on the host, with TFP-202’s flags.
  2. Containment
    • Files: a stand-in test that slips past the code check (a test-only bypass) can’t read a file in the repo, the data folder or the home folder. It can’t write outside the output folder.
    • Environment: it can’t read an environment variable TestForge holds.
    • Privileges: it runs as a non-root user, with a read-only root.
  3. Network
    • Blocked: a request to a host outside the allowed set is refused, and the results name the host.
    • Local Toolshop: a sandboxed test reaches local with-bugs, including the app’s own calls to localhost:8091, and passes where it passed on the host.
  4. What leaves
    • The cap: an oversized output is cut to the report, with the drop named.
    • The scan: every captured file goes through the credential scan.
    • Untrusted data: a report holding a script-shaped string is shown as text.
  5. No Docker
    • Not run: with Docker stopped, every model-written test is Not run with Decision 5’s reason.
    • The rest: the other tests run and pass as before.
    • No bypass: no option runs them on the host.
  6. Cost and results
    • Overhead: Decision 6’s measurement on local Toolshop, with the 60 s rule applied.
    • Same results: Decision 7’s comparison on the fixture.
    • Template: check:template-unchanged reports every fixed input identical against the recorded baseline (TEMPLATE_BASELINE). The count is whatever the check’s input set holds at the time, 166 when this was written, not a fixed number. Model-written tests aren’t template output.

Regression Policy

Dependencies and order

Backlog

TFP-228 — The sandbox container and its mounts

TFP-229 — The network: target hosts only, and local targets through the host

TFP-230 — Model-written tests run in the sandbox; results come back

TFP-231 — Measure the sandbox (Gate B)

  1. TFP-228.
  2. TFP-229.
  3. TFP-230.
  4. TFP-231.

Suggested Milestone Gates

Gate A — Contained

Gate B — In use and measured