mr-onion

Phase 17 pre-lock backlog: generator hardening from TFP-042

Version: 1.0 Date: 2026-09-17 Status: Shipped — all 11 tickets (TFP-053 through TFP-063) landed. TFP-053-061 are individually verified live against test-assist per TEST_PLAN.md’s own test case table (INT-04/05/06, GEN-11 through GEN-17, all ✅); TFP-062/063 were resolved as doc-only tickets (eb4ad3f). This doc’s header was never updated when the work landed. Corrected 2026-09-23 (Cowork verification pass).

Context

docs/decisions/product-roadmap.md’s own Sequencing note says Phase 16 is followed by “a live walkthrough of what [TestForge] actually produced … before Phase 17’s decision record is drafted,” and that the walkthrough “may change what ‘honest coverage’ needs to mean in practice before that phase is locked.” TFP-040/041/042 (examples/) already ran that walkthrough — TestForge’s real generator against test-assist, compared against a hand-authored skill-only suite and against test-assist’s own mature 55-test suite. This backlog is what that walkthrough actually found, turned into scoped tickets, per Emil’s request to extract concrete enhancements from examples/comparison-test-assist.md rather than leave it as a read-only finding.

Every ticket below is traced to a real file and line in the generator, not inferred from the comparison doc’s summary alone — re-read directly for this backlog:

What this backlog can and can’t close

TFP-053 through TFP-059 are mechanical: capture more per-element truth during introspection, then have the existing best-practice templates consume it instead of emitting the same fixed string regardless of target. TFP-060 and TFP-061 extend the same pattern to two specific behavioral signals (paired actions, drag targets) that are still discoverable statically.

TFP-042’s own clearest finding doesn’t fully close from this backlog, and the backlog doesn’t pretend it does: test-assist’s suite catches keyboard shortcuts, zoom accuracy, snapshot-gallery CRUD ordering, and verified effects of actions (does Undo actually remove the shape, not just “does clicking Undo return a response”) because it was built iteratively by someone who used the running app, not from a single static snapshot of its DOM. Static introspection has no signal for a keyboard shortcut unless the app declares aria-keyshortcuts (most don’t, including test-assist), no way to verify a canvas mutation without executing and inspecting the interaction sequence, and no concept of “the second export in the same millisecond stays distinct” without knowing the feature exists to test. That class of gap is what Phase 19 (path/journey mapping) is roadmapped to narrow — sequences of actions and their observable effects, not single-element enumeration — and TFP-062 below flags a related scope gap in Phase 18 worth resolving before that phase locks. Closing the human-tester-instinct gap entirely isn’t this backlog’s claim.

Backlog

TFP-053 — Capture per-element visibility during introspection

INT-04 evidence

Moved word for word from docs/test-plan/INT.md by TFP-253; the case’s row there links here.

Test: Actionable elements report isVisible (TFP-053, automated: “introspect reports isVisible on every discovered actionable element”)

Expected result: Every discovered actionable element carries a boolean isVisible. Verified live against test-assist, separately from this suite: #btnRecord, #btnClearSession, #uploadImage, #btnStopRecord, #textInput, #btnTextOk, #btnTextCancel all report isVisible: false — this repo’s own landing page has no naturally-hidden element to assert the false case against directly.

TFP-054 — Stop asserting unconditional visibility for conditionally-hidden elements

GEN-11 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: An element hidden at introspection time does not get an unconditional visibility claim (TFP-054, automated: “does not assert unconditional visibility for an element hidden at introspection time”)

Expected result: A synthetic element with isVisible: false generates toBeAttached() plus a “Hidden at introspection time” comment, not toBeVisible(). Verified live against test-assist: regenerating TFP-040’s exact suite now produces 0 of the 7 originally-confirmed false positives (41 of 42 passing; the 42nd, TFFU-036, is a separate, pre-existing finding unrelated to visibility).

TFP-055 — Capture ARIA state and toggle-group membership during introspection

INT-05 evidence

Moved word for word from docs/test-plan/INT.md by TFP-253; the case’s row there links here.

Test: Actionable elements report ARIA state and toggle-group membership (TFP-055, automated: “introspect reports ARIA state and toggle-group membership on actionable elements”)

Expected result: Every discovered actionable element carries ariaPressed/ariaSelected/ariaChecked/ariaExpanded/groupId as string \| null. Verified live against test-assist, separately from this suite: the seven .tool-btn elements are detected as one group (role button + shared containerLabel + 3+ members), each reporting a real ariaChecked value now that TA-252 has shipped; strokeColor/strokeSize/fillOpacity (role textbox) and Undo/Redo/Clear (one-shot action verbs) are correctly excluded from grouping.

TFP-056 — Generate accessibility coverage from real per-element data

GEN-13 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Accessibility suite asserts a real accessible name and role per discovered element (TFP-056, automated: “generates per-element accessible-name and role checks, skipping hidden elements”)

Expected result: A visible synthetic element generates a toHaveAccessibleName(/.+/) plus an ariaSnapshot()-derived role check; a hidden one (isVisible: false) gets the same toBeAttached() treatment as GEN-11 instead, since accessible-name/role computation is undefined for an element excluded from the accessibility tree. Verified live against test-assist: 24 of 31 discovered elements got a real per-element check, all passing (TFAC-002…TFAC-031), including one that caught a genuine finding first try — #uploadImage had no accessible name under a hand-rolled text/aria-label/placeholder/title heuristic; switching to Playwright’s own toHaveAccessibleName (which follows the real accname algorithm, including the wrapping <label> test-assist actually uses) resolved the false positive.

TFP-057 — Generate ARIA-state-toggle assertions for discovered groups

GEN-12 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: A detected toggle group generates an ARIA-state assertion (TFP-057, automated: “generates a toggle-group ARIA assertion for a detected selection group”)

Expected result: A synthetic 3-member groupId-sharing group generates a presence test (every member exposes an ARIA state attribute) and, since one member’s ariaChecked is "true", a toggle-behavior test. Verified live against test-assist: the generated accessibility suite’s TFAC-002/TFAC-003 both pass now that TA-252 has shipped and every .tool-btn reports a real aria-checked. The pre-TA-252 “produces a failing test naming the missing ARIA state” half of this ticket’s acceptance criteria was confirmed by tracing the generator’s own logic (no member would have activeAriaStateField(), so the toggle test wouldn’t generate at all and the presence test’s toBeTruthy() would fail), not re-verified live against a reverted target.

TFP-058 — Generate responsive coverage from real elements, add a touch smoke check

GEN-14 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Responsive suite checks real discovered elements per viewport and adds a touch smoke test (TFP-058, automated: “checks real discovered elements across viewports and adds a touch smoke test”)

Expected result: The viewport loop asserts up to 5 real isVisible: true elements stay visible at each of mobile/tablet/desktop, in addition to the existing body/heading check; a separate test dispatches touchstart/touchend at mobile viewport on one of those elements as an explicit smoke check (static introspection can’t see touch-specific handlers, so this claims only “doesn’t error on tap,” not app-specific touch behavior). Verified live against test-assist: TFRM-001 (5 real elements checked per viewport) and TFRM-002 (touch smoke) both pass.

TFP-059 — Generate usability coverage from real elements instead of duplicating functional’s count check

GEN-15 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: Usability suite asserts content-specific claims instead of an interactive-element count (TFP-059, automated: “replaces the usability interactive-count check with content-specific assertions”)

Expected result: Per-element checks assert any labeled/placeholder-bearing element’s text isn’t a generic phrase (submit, click here, ok, …); a combined check asserts up to 5 isVisible: true elements from the form/action categories have a bounding box above the desktop fold (900px). Elements hidden at introspection time are excluded from the sample entirely, not attach-only fallback like GEN-11/GEN-13 — a first pass without that filter flagged test-assist’s own hidden #btnRecord as a false “not rendered on first paint” failure. Verified live against test-assist: TFUS-002…TFUS-007 all pass.

TFP-060 — Auto-generate a paired negative assertion for discovered action pairs

GEN-16 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: A detected action pair (e.g. Undo/Redo) gets a negative-assertion regression guard, and only a real pair does (TFP-060, automated: “generates a negative-assertion test for a detected action pair, and only for it”)

Expected result: Two exactly-matching elements from a fixed opposite-word list (undo/redo, save/discard, open/close, …) generate one test asserting a wholly unrelated interaction (a mouse move to a neutral point) doesn’t change either element’s enabled state or text; an unpaired third element (no matching counterpart) generates nothing. Verified live against test-assist: exactly one pair test generated (Undo/Redo), passing, with no spurious pair among Select/Highlight/Text/Circle/Arrow/Rectangle/Pen or Save PNG/Export JSON.

TFP-061 — Detect drag-and-drop targets and generate a basic coverage test

GEN-17 evidence

Moved word for word from docs/test-plan/GEN.md by TFP-253; the case’s row there links here.

Test: A detected drop target gets valid- and invalid-content drag-and-drop tests (TFP-061, automated: “generates valid- and invalid-content drag-and-drop tests for a detected drop target”)

Expected result: A drop target generates a valid-content test (drops a real 1x1 PNG via a dragover+drop DragEvent sequence, asserting an observable DOM change and no uncaught page error) and an invalid-content test (drops a text file, asserting only no crash). Verified live against test-assist: both generated tests pass against the real canvas.

INT-06 evidence

Moved word for word from docs/test-plan/INT.md by TFP-253; the case’s row there links here.

Test: Pages report drag-and-drop targets (TFP-061, automated: “introspect reports dragDropTargets as a well-shaped array”)

Expected result: Every page carries a dragDropTargets array with isVisible/isDraggableSource/isDropTarget booleans and a string tagName. Detection dispatches a non-bubbling synthetic dragover event per candidate and checks defaultPrevented, since real drop targets (test-assist’s canvas included) attach handlers via addEventListener, invisible to an attribute-only scan. Verified live against test-assist, separately from this suite: introspecting test-assist detects exactly one drop target, the real <main class="canvas-wrap"> — a first pass with a bubbling probe also flagged 6 of its own descendants as false positives, since the probe’s event bubbled into the ancestor’s real listener; disabling bubbling on the probe (not on the generated test’s own drop simulation, which still bubbles like a real interaction) fixed it.

TFP-062 — Resolve Phase 18’s scope gap on local (non-network) failure injection

TFP-063 — Calibrate per-element enumeration depth against target relevance

TFP-062 and TFP-063 are resolved (see their Status lines above) — the remaining order is code-only, nine tickets:

  1. TFP-053 (visibility capture — unblocks TFP-054)
  2. TFP-054 (fixes the 7 confirmed false positives — highest-value, P0)
  3. TFP-055 (ARIA state capture — unblocks TFP-057, informs TFP-056)
  4. TFP-056 (accessibility coverage from real data)
  5. TFP-057 (ARIA-state-toggle assertions — the TA-252-class catch)
  6. TFP-058, TFP-059 (responsive/usability from real data — independent of each other)
  7. TFP-060, TFP-061 (negative assertions, drag-and-drop — independent, lower priority)

Suggested Milestone Gate

Gate — Ready to implement Phase 17

docs/decisions/phase-17-run-persistence-decision-record.md is drafted and locked. Its Decision 6 raises this gate’s bar beyond what this backlog originally proposed: Emil’s confirmed preference is to finish this entire backlog — all of TFP-053 through TFP-063 — before Phase 17 implementation starts, not just the subset below, and not in parallel. That’s a product-completeness call (“as complete a product as possible before stepping into more backlog”), not a technical dependency; the original, narrower reasoning for a subset gate is kept below because it’s still true on its own terms, it’s just no longer the whole bar.

The original subset this backlog proposed as functionally load-bearing for “honest coverage” specifically:

The rest of this backlog, now equally required by the confirmed gate:

Gate closed 2026-09-17. All nine code tickets landed (43718aa..47446c5), verified live against test-assist (introspect/generate/run, not just unit-level) and independently re-checked from Cowork: origin/main at 47446c5 confirmed, regression suite at 32 tests (was 22 before this batch), npx tsc --noEmit clean, and two of the higher-risk diffs (TFP-054’s visibility gating, TFP-061’s non-bubbling drag probe) read line-for-line against their own commit messages’ claims — both matched exactly, including the subtler technical details (dropping the enabled/editable/click assertions too for hidden elements, not just the visibility check; the bubbling-vs-non-bubbling distinction between the probe and the generated test’s own drop simulation). The one thing not independently re-run here was the full regression suite itself (a live Playwright + Next.js server run exceeds a single verification pass in this environment) — noted as a gap, not silently assumed passing beyond what the test count and tsc check above actually prove.

docs/decisions/phase-17-run-persistence-decision-record.md is unblocked. Phase 17 implementation can start.