mr-onion

Phase 20 Decision Record — Authenticated Testing (Locked)

Date: 2026-09-23 Scope: Config-driven automation-account login once per run, cached Playwright storageState reused by every generated test in that run, credentials referenced by name and never inlined, captured, or displayed. Status: Complete. Gate B met 2026-09-24 (227002b); TFP-097, the non-gating login UI, landed 2026-09-24 (14dd256). Reconciled with git 2026-09-29.

Context

docs/decisions/product-roadmap.md’s Phase 20 paragraph, and its own “Explicitly out of scope” list (no autonomous login-flow discovery — this is config-driven, full stop), already lock most of this phase’s shape. This record turns that into buildable decisions. Three things were genuinely undecided and marked [PLACEHOLDER]: the credential-store mechanism beyond plain environment variables, whether the cached storageState persists to disk across separate runs or is scoped to a single run, and the exact content of the setup instructions doc the roadmap says TestForge ships. Resolved (Emil, 2026-09-24) — see Decisions 3, 5, and 6 below.

This phase is a hard dependency on Phase 19, not just a suggested order — product-roadmap.md’s own text: “Deliberately sequenced after Phases 17–19 are proven on public surface, where the blast radius of a mistake is zero.” Phase 20 tickets do not start until Phase 19’s Gate B is met, even though Phase 18 and 19 themselves run independently of each other.

Standing policy for this phase: same as Phases 18–19 — defects fixed immediately in the ticket that found them, docs updated in the same commit as the code. See product-roadmap.md’s Sequencing note.

Decisions

  1. Login is config-driven, never auto-discovered.
    • The site owner supplies a login URL and field selectors (username, password, submit) once, in run config — the same shape already locked in the roadmap’s “Explicitly out of scope” list.
    • No heuristic or AI-assisted discovery of an arbitrary site’s login flow — that remains explicitly out of scope for this roadmap.
  2. TestForge drives the configured login flow once per run and caches the resulting storageState; every generated test in that run starts already authenticated.
    • One login per run, not one login per test — the cost and risk of re-authenticating is paid once.
  3. Credentials are referenced by name, never inlined.
    • A generated test references a credential by name (e.g. an env var name), never contains the literal username/password string.
    • The immediate mechanism is an environment variable, following the existing .env.development/.env.example convention already in this repo — no new config surface for the simple case.
    • Resolved (Emil, 2026-09-24): env-var-only for this phase. No specific secret-store tool is supported now — consistent with the rest of this local-only tool, and the simplest path that still meets Decision 3’s “never inlined” requirement. Deferred explicitly, not silently: TFP-093 states this in TEST_PLAN.md’s Known Limitations rather than leaving the gap unmentioned.
  4. Credentials and storageState are never captured in traces, screenshots, or the HTML report.
    • Playwright’s own trace/screenshot config is used to exclude the login step’s captured state, and the report renderer explicitly scrubs any storageState content before it can reach HTML output — this is a redaction requirement, not an assumption that it won’t come up.
  5. The cached storageState is scoped to a single run and never persisted to disk across runs.
    • Resolved (Emil, 2026-09-24): single-run only. It’s captured at run start, held for the duration of that run (in memory, or on-disk only as a transient artifact of that run’s own working directory — never written anywhere that outlives the run), and discarded once the run finishes. No cross-run cache, no invalidation/expiry policy needed, because there’s nothing left to invalidate after the run ends. A session token never survives on disk between runs — the safer of the two risk profiles this decision named, and consistent with Decision 2’s “one login per run” framing already locked above.
  6. The setup-instructions doc is generic, not platform-specific.
    • Resolved (Emil, 2026-09-24): a new docs/AUTOMATED_ACCOUNT_SETUP.md covering the four named exemption categories (lockout, rate-limiting, MFA interruption, new-sign-in alerts) generically, as the roadmap’s own “same pattern used for Selenium automation accounts” text already implies. Not platform- or identity-provider-specific steps — TestForge can’t maintain those for every site owner’s IdP, and the roadmap never asked for that.
  7. Login config entry is API-only for Phase 20; a UI field is deferred, not required for this phase’s gate.
    • Raised by Claude Code while building TFP-092: TFP-095’s setup doc assumed a UI location for login fields to point a user at, and no such UI exists — /api/run’s login block is API-only today.
    • Resolved (Emil, 2026-09-24): API-only for this phase, same precedent as Phase 19’s named flows (API first in TFP-089, UI added after in TFP-096, not gating). See TFP-097 below — added to the backlog, not required for Gate A or Gate B.

Acceptance Criteria

  1. Config-driven login
    • Given a real target with a working login form and correct config (URL + selectors), TestForge authenticates once and every generated test in that run starts on an authenticated page — verified by an assertion only reachable when logged in (not just “no login screen shown”).
  2. Credential handling
    • A generated test file contains no literal credential string — grepping the output for the configured password value returns nothing.
    • A run’s trace, screenshots, and HTML report contain no captured storageState content or literal credential value.
  3. Sequencing
    • No Phase 20 ticket starts implementation before Phase 19’s Gate B is marked met in that phase’s own decision record.

Regression Policy

Backlog

TFP-092 — Config-driven login + storageState caching for a run

RUN-16 evidence

Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.

Test: Config-driven login: a run can log in once and every test in it starts signed in (TFP-092, automated: tests/regression/testforge.auth.spec.ts — “with a login configured, every test in the run starts signed in - and the same tests fail without it”; unit: src/lib/run/auth.test.ts)

Expected result: /api/run accepts a login block — loginUrl, usernameSelector, passwordSelector, submitSelector, usernameEnv, passwordEnv (the NAMES of environment variables), and at least one of postLoginPath / postLoginSelector — drives that login once in Chromium before the run starts, and every test in the run reuses the resulting storageState (use.storageState in playwright.config.ts, from an environment variable the executor always sets or removes, never inherits). Against a real fixture with a real cookie login: a suite of two tests that never log in themselves (/secure shows #logout and Secure area; /account shows #logout) fails both tests with no login, and passes both with it. Verified live against https://the-internet.herokuapp.com (tomsmith, from environment variables): the test asserting h2 “Secure Area” and the Logout link — only reachable when authenticated, since /secure otherwise redirects to /login — failed with Received string: "Login Page" with no login, and passed with it, along with a second test that also never logs in. Defect found live and fixed here: the first live run logged in “successfully” yet every test still landed on the login page, because that site binds a session to the User-Agent and Accept-Language it was issued to, and the login ran in a default browser context (system locale en-GB) while the tests ran as Desktop Chrome / en-US. The login and the tests now share one options object (src/lib/run/browser-context.ts, imported by both playwright.config.ts and the login), and the fixture is session-bound the same way so the regression test fails without the fix (confirmed red before green) and asserts the login request carried Desktop Chrome’s User-Agent and en-US. The state file lives in a per-run directory outside the evidence directory and is removed when the run’s process ends (Decision 5: single-run only) — asserted gone after the run — and any left by a crashed run are swept at the next run start.

RUN-17 evidence

Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.

Test: A failed login is a named run-level error, never a silent unauthenticated run (TFP-092, automated: tests/regression/testforge.auth.spec.ts — “a wrong credential is a named run-level error, and no run starts”, “a bad selector is a named run-level error that names the selector”, “a credential environment variable that is not set is a named error”, “an invalid login config is rejected before anything runs; inline credentials are never accepted”)

Expected result: The run does not start (no runId, no per-run state left behind). A wrong password returns 422 login-failed: “Login failed: after submitting the login form the page ended on /login, and the page /secure … never appeared. Check the credentials (VAR / VAR) … The run was not started, so no test ran unauthenticated.” A bad selector: “Login failed: the username field (selector "#no-such-field") was not found on . Check the login selectors." A credential variable that is not set: 422 `credential-missing`, naming the variable. A malformed config is 400: any field other than the eight named ones — so any inline credential, whatever it is called — is refused, a credential reference that is not a valid environment variable name is refused, a `loginUrl` on another origin is refused (the credentials are typed into that page), and a config with neither `postLoginPath` nor `postLoginSelector` is refused (without a signed-in signal a failed login could not be told from a successful one). Verified live against `https://the-internet.herokuapp.com`: a wrong password and a bad selector each returned the named 422 above.

TFP-093 — Named credential references, never inlined

RUN-19 evidence

Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.

Test: A login references credentials by the NAME of an environment variable and never carries a value; the literal credentials appear in no generated code, persisted run config, or API response (TFP-093, automated: tests/regression/testforge.auth.spec.ts — “the literal credentials appear in no generated code, persisted run config, or API response”, “a credential pasted where a variable name belongs is refused before it is looked up, and never echoed back”; unit: src/lib/run/auth.test.ts)

Expected result: Against the real fixture through the built app, with the configured password (Fixture-Pass-7391-zq) and username (fixture-user) as the search values: a grep of the introspect response, the suite /api/generate produced (over 200 characters of real generated code), the run’s start response, its final status, the persisted run.json, and the persisted suite.spec.ts finds neither value. The persisted run.json login block is exactly the references — the URL, selectors, and the two variable names — and nothing else. The grep is not vacuous: with run.json deliberately made to carry the password, the test fails with run.json contains the password (mutation restored, not committed). The reference mechanism itself (usernameEnv/passwordEnv, values read from the server’s environment at run time, .env.example documenting the convention) landed with TFP-092. Hardened here: a name must be upper-case letters, digits and underscores (the convention every variable in this repo’s .env files follows), so a credential pasted where a name belongs — typically mixed-case or containing punctuation — is refused with a 400 before it is ever looked up, instead of being treated as a variable name and echoed back in a “variable X is not set” message; verified for hunter2, Hunter2Hunter2, and the fixture’s real password.

TFP-094 — Redact credentials and storageState from traces, screenshots, and the report

RUN-18 evidence

Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.

Test: An authenticated run hands back none of its credentials or its session — not in the API result, the HTML report, the persisted evidence, a trace, or a download (TFP-094, automated: tests/regression/testforge.auth.spec.ts — “a run that deliberately tries to leak its credentials and session hands back none of them”; unit: src/lib/run/redact.test.ts)

Expected result: Probed first, not assumed: a real authenticated run that fails a test left its whole storageState — the session cookie and its value — inside trace.zip (recorded in the trace’s context options), even though the login itself ran in a bare, untraced context and the password and username were absent from the trace of a run that did not type them itself. So capture is not excluded by default. Every run that logged in now scrubs everything it produced before it can reach the result, the persisted evidence, or a download: the run’s secret values (the two credentials plus every cookie and web-storage value in the session, held in memory for that run only and never persisted) are redacted — raw, JSON-escaped, and percent-encoded — from text artifacts, from inside trace zips (each entry rewritten, entries and binary resources preserved), from the parsed report, and from the console output; the assembled result is then scrubbed a second time as a whole with any storageState key dropped outright, as an independent layer that does not depend on knowing the values. It fails closed: an artifact that cannot be scrubbed and might hold a secret (an unreadable zip, an unrecognized file type containing one) is deleted, not kept, and the run’s output says how many artifacts were redacted or removed. The regression test actively tries to leak: the suite prints the session cookie to stdout, types the username and password into fields (the trace records action values), and fails on an assertion whose message contains the session. It then searches the API result, the HTML report built from it, every persisted file (including inside every zip), and every artifact downloaded through the API. Confirmed red without the fix: with scrubbing neutralised it reported 31 leaks — the API result, the HTML report, error-context.md (password, username, session), five entries inside trace.zip including the network log, run.json, and the downloads. With it: none, and the trace is still a valid, useful one (still contains its context options and the failing assertion, marked [redacted] where it was scrubbed).

TFP-095 — Automation-account setup instructions

RUN-20 evidence

Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.

Test: The automation-account setup doc names all four exemption categories and matches the real login config, so it cannot drift (TFP-095, automated: npm run test:unit — src/lib/run/setup-doc.test.ts)

Expected result: docs/AUTOMATED_ACCOUNT_SETUP.md has a section for each of lockout, rate limiting, MFA interruption, and new-sign-in alerts, written generically rather than per identity provider. Its field table lists exactly the eight fields a login block accepts (LOGIN_CONFIG_FIELDS, the single list normalizeLoginConfig validates against) and no other; its JSON example uses every one and is accepted by normalizeLoginConfig; the environment-variable names it tells you to set are the ones its example references and satisfy the upper-case rule it states; every error message it quotes still exists in auth.ts (a reworded error fails the test); its claim of four workers in fast mode and two in deep matches the executor; and no example contains an inline credential. README.md and .env.example both point at it (there is no run-config UI field to point from — Decision 7, TFP-097). Confirmed the guard bites: renaming postLoginPath in the doc’s table makes it fail (reverted).

TFP-097 — Login config entry in the run-config UI

RUN-21 evidence

Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.

Test: The login can be entered in the UI, and behaves exactly as the equivalent /api/run request (TFP-097, automated: tests/regression/testforge.login-ui.spec.ts — “a login entered in the UI authenticates a real run exactly as the equivalent /api/run request body does”, “with nothing typed the run carries no login, and is signed out”, “a credential typed where a variable name belongs blocks the run and is never sent”, “an incomplete login is not run signed out…”, “the section has no field that takes a password, and says so”; unit: src/app/_lib/derive.test.ts, src/lib/run/setup-doc.test.ts)

Expected result: An “Authenticated Run” section (collapsed by default, above the generated suite) has one text input per login field, labelled as the setup doc’s field table labels them (the label list is shared code, LOGIN_FIELD_LABELS, so UI, doc and server cannot disagree), and no password field: the two credential fields take an environment-variable name, and the section says never to type a password. Against the real app and the real cookie-login fixture: typing the login and clicking Run sent /api/run a login block equal, field for field, to the API body (loginUrl, the three selectors, usernameEnv/passwordEnv, postLoginPath, postLoginSelector); the same suite sent straight to the API with that block gave identical per-test results (both Passed — a signed-out run fails both, RUN-16), and the two persisted runs carry an identical login block with the password absent. With nothing typed the request has no login at all. A credential typed into a variable-name field (Fixture-Pass-7391-zq) blocks Run — the button is disabled, a message names the field and does not echo the value — and no request is sent, so it never leaves the browser; fixing the field re-enables Run. Filling in only some of the login is not run signed out: the request is sent and the server’s named error (Login config: "usernameEnv" is required.) appears. Confirmed the equivalence test bites: with the UI’s login deliberately dropped from the request, it failed (expected the block, received undefined); reverted.

  1. TFP-092 (the core capability)
  2. TFP-093 and TFP-094 in parallel once TFP-092 lands (different concerns, no shared files)
  3. TFP-095 after TFP-092/093 land (documents the settled shape, not a moving target)
  4. TFP-097, any time after TFP-092 — doesn’t block or depend on TFP-093/094/095, so it can slot in wherever’s convenient

Suggested Milestone Gates

Gate A — Authenticated runs work and leak nothing

Met, 2026-09-24 (dad7049). TFP-092–094 landed against real targets (the-internet.herokuapp.com and a local fixture), each with a TEST_PLAN.md case proving real (not assumed) behavior — RUN-16/RUN-17 (login + failure handling), RUN-18 (nothing leaks, including the negative case: 31 leaks found with scrubbing disabled, 0 with it), and RUN-19 (grep finds no credential, not vacuous — a deliberately-mutated run.json fails the check).

Gate B — Phase 20 complete

Met, 2026-09-24 (227002b). TFP-095 landed (RUN-20): a generic setup doc covering all four exemption categories, kept from drifting by a unit test that fails if the doc’s field table, JSON example, quoted error messages, or parallelism claim stop matching the real code. Decisions 3 and 5 are reflected in TEST_PLAN.md’s Known Limitations (env-var-only credentials, single-run storageState); Decision 5 was built, not deferred.