mr-onion

Test cases: Report (RPT)

Section 3.6 of the test plan, which keeps the app overview, the feature status, the known limitations and the index of areas. Since TFP-253 each area’s cases are in a file of their own; a ticket names the area it touches.

ID Test Expected Result Status
RPT-01 Defect backlog and usability findings render in plain language (automated: “formats defect backlog and usability findings in plain language”) Report’s defects/usability tabs show human-readable findings, not raw Playwright matcher output. ✅
RPT-02 Defects, usability findings, and evidence link to the same failed test (automated: “links defects, usability findings, and captured artifacts to the same failed test”) A single failing test’s defect entry, usability finding, and captured evidence artifact are all traceable to that same test in the report. ✅
RPT-03 SEO checks grouped by page across multiple pages (automated: “groups SEO checks by page when multiple pages are under test”) SEO tab groups findings under the page they belong to when more than one page was tested. ✅
RPT-04 SEO findings summarized in plain language (automated: “summarizes SEO findings in plain language instead of raw matcher errors”) SEO tab shows a plain-language summary rather than a raw assertion failure message. ✅
RPT-05 Broken-link failures list the specific broken links (automated: “lists the specific broken links in observed outcomes for broken-link failures”) A broken-link test failure’s observed outcome names the specific URL(s) that failed, not a generic failure message. ✅
RPT-06 No structured report when Playwright executes zero tests (automated: “does not show a structured report when Playwright executes zero tests”) If the underlying Playwright run executes no tests, the UI does not present a structured report as if one exists. ✅
RPT-07 Fallback report page for runs without structured report data (automated: “opens a fallback report page for runs without structured report data”) A run lacking structured report data opens a fallback report page rather than erroring or showing a blank state. ✅
RPT-08 Report tabs, export, and page-scoped SEO drilldown after a run (automated: “covers the report tabs, HTML export, and page-scoped SEO drilldown after running a generated suite”) All seven report tabs are navigable, HTML export is available, and the SEO tab supports drilling into a specific page’s findings. ✅
RPT-09 Model-written tests are reported under their area A test’s area comes from its test.describe title. Evidence. ✅
RPT-10 A test that skips itself says why The run report’s remark gives the skip’s own reason, from Playwright’s skip annotation; a skip with no reason keeps the old wording. Evidence. ✅
RPT-11 Findings a test recorded without failing appear in the report A test can record a finding with test.info().annotations.push({ type: 'finding', description }) and still pass. Evidence. ✅
RPT-12 A test skipped by choice doesn’t count against the run A test that skipped itself with a stated reason (Playwright’s skip annotation) gets the status Not run and keeps its reason (skipReason). Evidence. ✅
RPT-13 Empty areas say why, and generation gates are recorded with the run Gate verdicts with the run (Decision 8). The app keeps the generate response’s generatedBy, model, fallbackReason, gapFill and gates and sends them with the run request as generation. Evidence. ✅
RPT-14 A run where nothing ran gives no release recommendation and a neutral badge A test ran only if it Passed, was Flaky or Failed. Evidence. ✅
RPT-15 A suite with nothing to run says so, and why, instead of “run failed” A suite with no tests ends as empty, not failed, with its report and each empty area’s reasons. Evidence. ✅
RPT-16 A Comprehensive report shows its area only, page by page, and ends with what it did not test An area summary that never reads as a clean bill of health, a per-page table, failures and findings by category, and what the run did not test. Evidence. ✅
RPT-17 Every tab of the exported Standard HTML report switches Every tab of the export switches again; from 9fbfab9 until this fix only the Summary tab could be read. Evidence. ✅
RPT-18 The report shows each test’s gate, and how many tests each gate kept from running Each execution record carries gate, read from the test’s gate annotation or its describe’s; an unknown value is ignored. Evidence. ✅
RPT-19 The run report shows coverage by page type and ends with the copyable re-run lists Page types found, covered and skipped, each with its representative page; skipped types give a reason. Evidence. ✅
RPT-20 The coverage map: each element, link and flow the inspection found, the tests that touch it, and what no test touches, per page type Read from the suite’s code, per element, link and flow. Evidence. ✅
RPT-21 Coverage in two tiers, a repeated link counted once In context: a test finds and checks the item on its page; this tier is the gap list and the index. Evidence. ✅
RPT-22 What a person sees says Mr Onion (manual: node scripts/visible-name-check.mjs --app <built app> --out <Standard run> --out <Comprehensive run>) The app’s page, the scripts and styles it loads, its manifest and /api/health; each run’s exported report and junit.xml; the CLI’s help and error output: no “testforge” in any case, apart from the allowlisted compatibility strings in the app’s JavaScript (the old interactive switch, the footer rule for runs below 0.3.0) and the user’s own paths. The page title and og:site_name say Mr Onion; a 0.3.0 run’s footer reads “Mr Onion 0.3.0”. Phase 48 Decision 1 and 7 (TFP-279). Measured 2026-11-09 against the built app and the local comparison fixture: “visible-name: no old name found”, exit 0; title “Mr Onion | AI-native Playwright test automation”, og:site_name “Mr Onion”; footers “Mr Onion 0.3.0 (2a94e6c)”; the app’s JavaScript carried the old name only in the three allowlisted strings. With the CLI’s old prefix put back (a mutant), exit 1, naming testforge: unknown command. ✅