Version: 1.0
Date: 2026-09-22
Status: Complete. TFP-074 1222cf5, TFP-075 47c33a1, TFP-076 d39ff34 (all 2026-09-23); results under examples/. Reconciled with git 2026-09-29.
Phase 19 (docs/decisions/phase-19-dogfooding-backlog.md) established a
repeatable pattern — run TestForge’s own app against a real external
target, run a skill directly against the same target with no app
involved, then compare both to a real baseline — and examples/README.md
explicitly says this “repeats for every future skill against the same
target, so results stay comparable over time.” That’s held true for new
skills against test-assist; this phase is the first time it’s applied
to a second target, cah-pwa.
cah-pwa is a better test of the pattern than a second run against
test-assist would be, for two reasons:
python3 -m http.server 8000 per its own
GETTING_STARTED.md. No desktop-build exclusion like Phase 19 had to
carve out.scripts/generate_playwright_tests.js,
producing the suite under tests/generated/ — plus a hand-authored
stability matrix (AUTOMATION_TEST_PLAN.md, rating each area
Stable/Moderate/Flaky/Blocked with named reasons). Phase 19’s
comparison was TestForge-generated vs. skill-generated vs.
hand-written, because test-assist had no generator of its own to
compare against. This phase can additionally compare TestForge’s and
the skill’s output against another automated approach on the same
app — a comparison Phase 19 structurally couldn’t make.Stated plainly so it isn’t assumed later: cah-pwa’s real gameplay
(room creation, chat, AI-driven card behavior) sits behind Firebase-backed
realtime state, and prompt-driven card behavior calls Gemini at runtime.
TestForge’s crawl is unauthenticated same-origin BFS. Whether that reaches
past the landing/auth surface into real gameplay is not assumed here
either way — it’s exactly what TFP-074 discovers and states as fact,
the same way Phase 19 discovered (rather than assumed) the
browser/desktop build split in test-assist. If it turns out gameplay is
reachable without real credentials (e.g. an anonymous session), record
that; if not, record the specific wall hit and don’t treat it as a defect
in this phase — TestForge’s own authenticated-testing support is a
separate, already-tracked roadmap item, not something to silently pull
into this ticket’s scope.
cah-pwa,
served locally.write-playwright-test directly against cah-pwa, with no
TestForge app involved.cah-pwa’s own
scripts/generate_playwright_tests.js output, and its hand-authored
stability matrix as context.examples/README.md’s target list to include cah-pwa
alongside test-assist, following the pattern it already documents
rather than special-casing this run.cah-pwa itself.test-assist’s desktop build, unless TFP-074’s own discovery finds
otherwise.dogfooding, integrationtest-assist’s browser build, and the first time this is checked
rather than assumed.cah-pwa locally: python3 -m http.server 8000, per its own
GETTING_STARTED.md.GETTING_STARTED.md’s
full sign-in → room → game flow.examples/testforge-app/cah-pwa/ — the generated suite, the run
report, and a notes.md stating what was and wasn’t reachable, and
why.dogfooding, skillsexamples/README.md already commits to, this time against a
target that has its own competing automated generator — so the
skill’s judgment can be compared to another tool’s output, not only to
a hand-written baseline.write-playwright-test (via Claude Code, no TestForge app
involved) directly against cah-pwa’s source and served instance.examples/README.md’s target list to name cah-pwa
explicitly alongside test-assist, per its own stated pattern.examples/skills/write-playwright-test/cah-pwa/examples/README.md.examples/README.md accurately reflects two targets, not one.dogfooding, analysiscah-pwa’s own generator, and its
hand-authored stability matrix. Without a direct, specific comparison,
having four teaches nothing. This includes the one comparison Phase
19 structurally couldn’t make: automated generator vs. automated
generator, not just automated vs. hand-written.cah-pwa’s own generator on the same
page or feature.examples/comparison-cah-pwa.md.examples/.While confirming cah-pwa was a viable target, a real reliability gap
turned up in js/gemini.js: the Gemini API call (line 16) has no
timeout, no retry/backoff, and no catch anywhere in the file. Not
actioned here — flagged so it isn’t lost. Worth its own ticket if a
reliability-audit-style pass on cah-pwa is wanted later, the same way
Phase 24 did for testforge itself.