Date: 2026-09-23
Scope: Config-driven automation-account login once per run, cached
Playwright storageState reused by every generated test in that run,
credentials referenced by name and never inlined, captured, or displayed.
Status: Complete. Gate B met 2026-09-24 (227002b); TFP-097, the non-gating login UI, landed 2026-09-24 (14dd256). Reconciled with git 2026-09-29.
docs/decisions/product-roadmap.md’s Phase 20 paragraph, and its own
“Explicitly out of scope” list (no autonomous login-flow discovery — this
is config-driven, full stop), already lock most of this phase’s shape.
This record turns that into buildable decisions. Three things were
genuinely undecided and marked [PLACEHOLDER]: the credential-store
mechanism beyond plain environment variables, whether the cached
storageState persists to disk across separate runs or is scoped to a
single run, and the exact content of the setup instructions doc the
roadmap says TestForge ships. Resolved (Emil, 2026-09-24) — see
Decisions 3, 5, and 6 below.
This phase is a hard dependency on Phase 19, not just a suggested
order — product-roadmap.md’s own text: “Deliberately sequenced after
Phases 17–19 are proven on public surface, where the blast radius of a
mistake is zero.” Phase 20 tickets do not start until Phase 19’s Gate B
is met, even though Phase 18 and 19 themselves run independently of each
other.
Standing policy for this phase: same as Phases 18–19 — defects fixed
immediately in the ticket that found them, docs updated in the same
commit as the code. See product-roadmap.md’s Sequencing note.
storageState; every generated test in that run starts
already authenticated.
.env.development/.env.example convention already in this
repo — no new config surface for the simple case.TEST_PLAN.md’s Known Limitations
rather than leaving the gap unmentioned.storageState are never captured in traces,
screenshots, or the HTML report.
storageState content before it can reach HTML output — this is a
redaction requirement, not an assumption that it won’t come up.storageState is scoped to a single run and never
persisted to disk across runs.
docs/AUTOMATED_ACCOUNT_SETUP.md
covering the four named exemption categories (lockout, rate-limiting,
MFA interruption, new-sign-in alerts) generically, as the roadmap’s
own “same pattern used for Selenium automation accounts” text already
implies. Not platform- or identity-provider-specific steps — TestForge
can’t maintain those for every site owner’s IdP, and the roadmap never
asked for that./api/run’s login block is API-only today.storageState content or literal credential value.TEST_PLAN.md gets cases for: successful authenticated run against a
real target (every generated test starts logged in); a failed login
(wrong config) producing a clear, named error rather than every test
silently running unauthenticated; and a grep-based regression case
asserting no credential value appears in generated code, traces, or
the report. Landed in the same commits as the code, per this phase’s
standing policy.feature, authstorageState.storageState for every generated test’s browser context
in the run.TEST_PLAN.md case(s) proving authenticated state actually reaches
every generated test against a real target with a login form.ca4e10c): /api/run accepts a login block (URL, three
selectors, usernameEnv/passwordEnv credential-variable names, a
post-login path or selector); the executor logs in once in Chromium
before Playwright starts and every test reuses the resulting
storageState via playwright.config.ts. The state file lives in a
per-run directory outside the evidence directory, removed when the
run’s process ends (Decision 5: single-run only), with a sweep for
any left by a crashed run. A bad selector, wrong credential,
unreachable page, or unset credential variable is a named 422 with no
run started; a malformed config (inline credentials under any field
name, a non-origin-matching loginUrl, neither post-login check set)
is a 400. Defect found and fixed live: a target that binds a
session to the User-Agent and Accept-Language it was issued to
reported a “successful” login that every test still failed, because
the login ran in a default browser context while tests ran as Desktop
Chrome/en-US — login and tests now share one options object
(src/lib/run/browser-context.ts). Judgment call, not escalated:
usernameEnv/passwordEnv accept any syntactically valid environment
variable name, with no allow-list restricting which names — consistent
with Phase 16’s already-locked unsandboxed, local-only execution
model (/api/run already runs caller-supplied code with the user’s
own privileges), not a new risk this ticket introduces.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: Config-driven login: a run can log in once and every test in it starts signed in (TFP-092, automated: tests/regression/testforge.auth.spec.ts — “with a login configured, every test in the run starts signed in - and the same tests fail without it”; unit: src/lib/run/auth.test.ts)
Expected result: /api/run accepts a login block — loginUrl, usernameSelector, passwordSelector, submitSelector, usernameEnv, passwordEnv (the NAMES of environment variables), and at least one of postLoginPath / postLoginSelector — drives that login once in Chromium before the run starts, and every test in the run reuses the resulting storageState (use.storageState in playwright.config.ts, from an environment variable the executor always sets or removes, never inherits). Against a real fixture with a real cookie login: a suite of two tests that never log in themselves (/secure shows #logout and Secure area; /account shows #logout) fails both tests with no login, and passes both with it. Verified live against https://the-internet.herokuapp.com (tomsmith, from environment variables): the test asserting h2 “Secure Area” and the Logout link — only reachable when authenticated, since /secure otherwise redirects to /login — failed with Received string: "Login Page" with no login, and passed with it, along with a second test that also never logs in. Defect found live and fixed here: the first live run logged in “successfully” yet every test still landed on the login page, because that site binds a session to the User-Agent and Accept-Language it was issued to, and the login ran in a default browser context (system locale en-GB) while the tests ran as Desktop Chrome / en-US. The login and the tests now share one options object (src/lib/run/browser-context.ts, imported by both playwright.config.ts and the login), and the fixture is session-bound the same way so the regression test fails without the fix (confirmed red before green) and asserts the login request carried Desktop Chrome’s User-Agent and en-US. The state file lives in a per-run directory outside the evidence directory and is removed when the run’s process ends (Decision 5: single-run only) — asserted gone after the run — and any left by a crashed run are swept at the next run start.
Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: A failed login is a named run-level error, never a silent unauthenticated run (TFP-092, automated: tests/regression/testforge.auth.spec.ts — “a wrong credential is a named run-level error, and no run starts”, “a bad selector is a named run-level error that names the selector”, “a credential environment variable that is not set is a named error”, “an invalid login config is rejected before anything runs; inline credentials are never accepted”)
Expected result: The run does not start (no runId, no per-run state left behind). A wrong password returns 422 login-failed: “Login failed: after submitting the login form the page ended on /login, and the page /secure … never appeared. Check the credentials (VAR / VAR) … The run was not started, so no test ran unauthenticated.” A bad selector: “Login failed: the username field (selector "#no-such-field") was not found on
feature, security, auth.env.development/.env.example convention.TEST_PLAN.md’s Known Limitations, not left silent.TEST_PLAN.md regression case grepping generated output for the
literal credential value and asserting it’s absent.622c047): grep of the real generated suite, persisted
run.json/suite.spec.ts, and API responses finds neither credential
value; confirmed not vacuous (a deliberately-mutated run.json fails
the check). Hardened: usernameEnv/passwordEnv must be upper-case
letters, digits and underscores, so a credential pasted where a name
belongs is refused with a 400 before lookup instead of being echoed
back in a “variable X is not set” error — narrows the mistake, does
not rule it out (an all-upper-case password would still pass), stated
in TEST_PLAN.md’s Known Limitations.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: A login references credentials by the NAME of an environment variable and never carries a value; the literal credentials appear in no generated code, persisted run config, or API response (TFP-093, automated: tests/regression/testforge.auth.spec.ts — “the literal credentials appear in no generated code, persisted run config, or API response”, “a credential pasted where a variable name belongs is refused before it is looked up, and never echoed back”; unit: src/lib/run/auth.test.ts)
Expected result: Against the real fixture through the built app, with the configured password (Fixture-Pass-7391-zq) and username (fixture-user) as the search values: a grep of the introspect response, the suite /api/generate produced (over 200 characters of real generated code), the run’s start response, its final status, the persisted run.json, and the persisted suite.spec.ts finds neither value. The persisted run.json login block is exactly the references — the URL, selectors, and the two variable names — and nothing else. The grep is not vacuous: with run.json deliberately made to carry the password, the test fails with run.json contains the password (mutation restored, not committed). The reference mechanism itself (usernameEnv/passwordEnv, values read from the server’s environment at run time, .env.example documenting the convention) landed with TFP-092. Hardened here: a name must be upper-case letters, digits and underscores (the convention every variable in this repo’s .env files follows), so a credential pasted where a name belongs — typically mixed-case or containing punctuation — is refused with a 400 before it is ever looked up, instead of being treated as a variable name and echoed back in a “variable X is not set” message; verified for hunter2, Hunter2Hunter2, and the fixture’s real password.
bug, security, reliabilitystorageState content from the HTML report
renderer’s input before it can reach output, as a defense-in-depth
second layer independent of the capture-exclusion above.storageState value reach the report, and asserts it can’t —
not just asserting the happy path is clean.TEST_PLAN.md case(s) per Acceptance Criteria §2’s report/trace
checks.dad7049): probed first, not assumed — a real
authenticated run’s trace.zip held the whole storageState
(session cookie and value) even though the login itself ran outside
any traced context. Two independent layers now scrub every artifact
before it can reach the result, persisted evidence, or a download:
value-based redaction (raw/JSON-escaped/percent-encoded, rewriting
inside zip entries via playwright-core’s own bundled zip tools) and
a structural pass dropping any storageState key outright. Fails
closed: an artifact that can’t be scrubbed is deleted, not kept. The
regression test deliberately tries to leak (prints the session,
types credentials, fails on an assertion echoing the session) and
found 31 leaks before the fix, 0 after, with the trace still valid
and useful. Screenshots/videos are binary and not scrubbed (Known
Limitations); values under 4 characters are not redacted (would
shred unrelated text).Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: An authenticated run hands back none of its credentials or its session — not in the API result, the HTML report, the persisted evidence, a trace, or a download (TFP-094, automated: tests/regression/testforge.auth.spec.ts — “a run that deliberately tries to leak its credentials and session hands back none of them”; unit: src/lib/run/redact.test.ts)
Expected result: Probed first, not assumed: a real authenticated run that fails a test left its whole storageState — the session cookie and its value — inside trace.zip (recorded in the trace’s context options), even though the login itself ran in a bare, untraced context and the password and username were absent from the trace of a run that did not type them itself. So capture is not excluded by default. Every run that logged in now scrubs everything it produced before it can reach the result, the persisted evidence, or a download: the run’s secret values (the two credentials plus every cookie and web-storage value in the session, held in memory for that run only and never persisted) are redacted — raw, JSON-escaped, and percent-encoded — from text artifacts, from inside trace zips (each entry rewritten, entries and binary resources preserved), from the parsed report, and from the console output; the assembled result is then scrubbed a second time as a whole with any storageState key dropped outright, as an independent layer that does not depend on knowing the values. It fails closed: an artifact that cannot be scrubbed and might hold a secret (an unreadable zip, an unrecognized file type containing one) is deleted, not kept, and the run’s output says how many artifacts were redacted or removed. The regression test actively tries to leak: the suite prints the session cookie to stdout, types the username and password into fields (the trace records action values), and fails on an assertion whose message contains the session. It then searches the API result, the HTML report built from it, every persisted file (including inside every zip), and every artifact downloaded through the API. Confirmed red without the fix: with scrubbing neutralised it reported 31 leaks — the API result, the HTML report, error-context.md (password, username, session), five entries inside trace.zip including the network log, run.json, and the downloads. With it: none, and the trace is still a valid, useful one (still contains its context options and the failing assertion, marked [redacted] where it was scrubbed).
documentationdocs/AUTOMATED_ACCOUNT_SETUP.md covering the four named exemption
categories (lockout, rate-limiting, MFA interruption, new-sign-in
alerts) generically, not platform- or identity-provider-specific
steps. Written against TFP-092’s actual config field names (login
URL, selectors) and TFP-093’s actual credential-reference shape
(env var name), not invented ahead of them.docs/AUTOMATED_ACCOUNT_SETUP.md: what an automation account is,
why it needs the four exemptions, and how to point TestForge’s run
config at one (referencing TFP-092/093’s real field names).README.md and .env.example (config entry
is API-only for this phase per Decision 7 below — there is no
run-config UI field to attach a pointer to yet).Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: The automation-account setup doc names all four exemption categories and matches the real login config, so it cannot drift (TFP-095, automated: npm run test:unit — src/lib/run/setup-doc.test.ts)
Expected result: docs/AUTOMATED_ACCOUNT_SETUP.md has a section for each of lockout, rate limiting, MFA interruption, and new-sign-in alerts, written generically rather than per identity provider. Its field table lists exactly the eight fields a login block accepts (LOGIN_CONFIG_FIELDS, the single list normalizeLoginConfig validates against) and no other; its JSON example uses every one and is accepted by normalizeLoginConfig; the environment-variable names it tells you to set are the ones its example references and satisfy the upper-case rule it states; every error message it quotes still exists in auth.ts (a reworded error fails the test); its claim of four workers in fast mode and two in deep matches the executor; and no example contains an inline credential. README.md and .env.example both point at it (there is no run-config UI field to point from — Decision 7, TFP-097). Confirmed the guard bites: renaming postLoginPath in the doc’s table makes it fail (reverted).
feature, uilogin block is API-only —
/api/run accepts it, but nothing in the app lets a user enter it, so
authenticated runs are only reachable by hand-building a request body.
Raised by Claude Code while building TFP-092, resolved by Decision 7
above (2026-09-24): API-only for this phase, UI deferred to this
ticket./api/run’s real login field names once this lands.TEST_PLAN.md case proving a login entered through the UI produces
the same authenticated run a direct API call with the same config
would.docs/AUTOMATED_ACCOUNT_SETUP.md pointer once a UI
location exists to point at./api/run request body would.login field, labelled as docs/AUTOMATED_ACCOUNT_SETUP.md
labels them (shared LOGIN_FIELD_LABELS; a unit test fails if doc,
section, and LOGIN_CONFIG_FIELDS drift). The two credential fields
take a variable name and there is no password field: a value typed
where a name belongs (the same upper-case rule the server applies)
blocks Run in the app with a message that does not echo it, and is
never sent. An incomplete block is sent as typed so the server’s named
error reaches the user rather than a quiet signed-out run. The block
applies to reruns and continued runs too. The setup doc’s field table
gained an “In the app” column and its “no UI yet” note became
instructions; README updated the same way.Moved word for word from docs/test-plan/RUN.md by TFP-253; the case’s row there links here.
Test: The login can be entered in the UI, and behaves exactly as the equivalent /api/run request (TFP-097, automated: tests/regression/testforge.login-ui.spec.ts — “a login entered in the UI authenticates a real run exactly as the equivalent /api/run request body does”, “with nothing typed the run carries no login, and is signed out”, “a credential typed where a variable name belongs blocks the run and is never sent”, “an incomplete login is not run signed out…”, “the section has no field that takes a password, and says so”; unit: src/app/_lib/derive.test.ts, src/lib/run/setup-doc.test.ts)
Expected result: An “Authenticated Run” section (collapsed by default, above the generated suite) has one text input per login field, labelled as the setup doc’s field table labels them (the label list is shared code, LOGIN_FIELD_LABELS, so UI, doc and server cannot disagree), and no password field: the two credential fields take an environment-variable name, and the section says never to type a password. Against the real app and the real cookie-login fixture: typing the login and clicking Run sent /api/run a login block equal, field for field, to the API body (loginUrl, the three selectors, usernameEnv/passwordEnv, postLoginPath, postLoginSelector); the same suite sent straight to the API with that block gave identical per-test results (both Passed — a signed-out run fails both, RUN-16), and the two persisted runs carry an identical login block with the password absent. With nothing typed the request has no login at all. A credential typed into a variable-name field (Fixture-Pass-7391-zq) blocks Run — the button is disabled, a message names the field and does not echo the value — and no request is sent, so it never leaves the browser; fixing the field re-enables Run. Filling in only some of the login is not run signed out: the request is sent and the server’s named error (Login config: "usernameEnv" is required.) appears. Confirmed the equivalence test bites: with the UI’s login deliberately dropped from the request, it failed (expected the block, received undefined); reverted.
Met, 2026-09-24 (dad7049). TFP-092–094 landed against real targets
(the-internet.herokuapp.com and a local fixture), each with a
TEST_PLAN.md case proving real (not assumed) behavior — RUN-16/RUN-17
(login + failure handling), RUN-18 (nothing leaks, including the
negative case: 31 leaks found with scrubbing disabled, 0 with it), and
RUN-19 (grep finds no credential, not vacuous — a deliberately-mutated
run.json fails the check).
Met, 2026-09-24 (227002b). TFP-095 landed (RUN-20): a generic
setup doc covering all four exemption categories, kept from drifting by
a unit test that fails if the doc’s field table, JSON example, quoted
error messages, or parallelism claim stop matching the real code.
Decisions 3 and 5 are reflected in TEST_PLAN.md’s Known Limitations
(env-var-only credentials, single-run storageState); Decision 5 was
built, not deferred.