mr-onion

Phase 25 — Dogfooding Against cah-pwa Backlog

Version: 1.0 Date: 2026-09-22 Status: Complete. TFP-074 1222cf5, TFP-075 47c33a1, TFP-076 d39ff34 (all 2026-09-23); results under examples/. Reconciled with git 2026-09-29.

Context

Phase 19 (docs/decisions/phase-19-dogfooding-backlog.md) established a repeatable pattern — run TestForge’s own app against a real external target, run a skill directly against the same target with no app involved, then compare both to a real baseline — and examples/README.md explicitly says this “repeats for every future skill against the same target, so results stay comparable over time.” That’s held true for new skills against test-assist; this phase is the first time it’s applied to a second target, cah-pwa.

cah-pwa is a better test of the pattern than a second run against test-assist would be, for two reasons:

Stated plainly so it isn’t assumed later: cah-pwa’s real gameplay (room creation, chat, AI-driven card behavior) sits behind Firebase-backed realtime state, and prompt-driven card behavior calls Gemini at runtime. TestForge’s crawl is unauthenticated same-origin BFS. Whether that reaches past the landing/auth surface into real gameplay is not assumed here either way — it’s exactly what TFP-074 discovers and states as fact, the same way Phase 19 discovered (rather than assumed) the browser/desktop build split in test-assist. If it turns out gameplay is reachable without real credentials (e.g. an anonymous session), record that; if not, record the specific wall hit and don’t treat it as a defect in this phase — TestForge’s own authenticated-testing support is a separate, already-tracked roadmap item, not something to silently pull into this ticket’s scope.

Scope (this phase)

In scope

  1. Run TestForge’s actual app (crawl → generate → run) against cah-pwa, served locally.
  2. Run write-playwright-test directly against cah-pwa, with no TestForge app involved.
  3. Compare all four things that now exist for the same app: TestForge’s generated suite, the skill’s generated suite, cah-pwa’s own scripts/generate_playwright_tests.js output, and its hand-authored stability matrix as context.
  4. Extend examples/README.md’s target list to include cah-pwa alongside test-assist, following the pattern it already documents rather than special-casing this run.

Out of scope

Backlog

TFP-074 - Run TestForge’s app against cah-pwa’s browser build

TFP-075 - Run write-playwright-test directly against cah-pwa

TFP-076 - Compare four suites on the same app

  1. TFP-074 and TFP-075 (can run in parallel)
  2. TFP-076

Suggested Milestone Gates

Gate A - cah-pwa dogfooding pass complete

Aside, not part of this phase’s scope

While confirming cah-pwa was a viable target, a real reliability gap turned up in js/gemini.js: the Gemini API call (line 16) has no timeout, no retry/backoff, and no catch anywhere in the file. Not actioned here — flagged so it isn’t lost. Worth its own ticket if a reliability-audit-style pass on cah-pwa is wanted later, the same way Phase 24 did for testforge itself.