mr-onion

Quality Gates and Evidence Rules

Status: living document, started 2026-10-01.

This page brings together, in one place:

It links to the records that hold the reasoning and does not restate it. When a rule changes, change it at its source and update the link here in the same commit (standing execution policy, rule 2).

1. What counts as evidence

What the product reports, so its own reports stay honest

Rule Where it is set Test plan
A test that skipped itself, with a reason, is “Not run”. It counts toward neither the pass ratio nor the recommendation (Option C). Phase 34 Decision 7 RPT-10, RPT-12
If no test ran, there is no release recommendation and the badge is neutral. Phase 34, GEN-44 follow-up RPT-14
A suite with nothing to run says “Nothing to run” and why, not “run failed”. Phase 36 Decision 3 RPT-15
Findings (hardening gaps, optional items) are not failures. Phase 34 Decision 3; Phase 36 Decision 2 RPT-11, GEN-52
Coverage is honest: a gap is named, never rounded up to 100%. Discovered-coverage report (src/lib/report/discovered-coverage.ts) RUN-10, RUN-11
A Comprehensive report ends with a fixed statement of what it did not test. Phase 35 Decision 6 RPT-16
Generation gate verdicts are saved with the run and shown in its report. Phase 34 Decision 8 RPT-13

2. Model output gates

Principle: detect and fall back, never rewrite (Phase 30; restated in Phase 37 Decision 3).

Every gate records a verdict in one shape: GateVerdict in src/lib/generate/gates.ts (Phase 34 Decision 8).

The gates run in this order, in src/lib/generate/local-generation.ts:

Gate (as recorded) What it rejects Code fallbackReason Introduced Test plan
time budget No answer within MRONION_OLLAMA_TIMEOUT_MS (default 180 s) ollamaTimeoutMs, local-generation.ts timeout Phase 30 (TFP-105) GEN-24
type check Any TypeScript error against the project’s Playwright types findTypeErrors, validate.ts invalid_output Phase 30 (TFP-103 follow-up) GEN-26
unawaited assertions A web-first assertion that is never awaited, so it never runs findUnawaitedAssertions, validate.ts unawaited_assertion Phase 33 (TFP-110) GEN-31
promise to plain matcher expect(<Promise>) followed by a matcher that doesn’t retry. .resolves and .rejects are exempt. findPromisesToPlainMatchers, validate.ts promise_to_plain_matcher Phase 37 Decision 2 (TFP-129) GEN-56
duplicate tests Two or more tests with the same body after whitespace and comments are normalised findDuplicateTests, validate.ts duplicate_tests Phase 37 Decision 3 (TFP-130) GEN-57

Some fallbacks happen before any gate runs:

The quality bar the gates don’t cover

The gates catch failures that are mechanical. Quality is judged by reading.

3. Template and run gates

4. Phase and ticket gates

5. Independent verification checklist

Use this for every “done” report before a gate is passed. The builder’s report is a claim; this checklist turns it into evidence.

  1. The commits exist. Run git log and git show --stat on the reported hashes. Check the changed files match the report and nothing outside the ticket changed. Run git status -sb to see what is pushed. When another agent is using the repo, use git --no-optional-locks, so no index.lock blocks its commits.
  2. Re-run the tests. Count passed, failed and skipped yourself, and compare with the reported counts. A skip needs its reason.
  3. Type check and lint:
    • node node_modules/typescript/bin/tsc --noEmit -p .;
    • eslint on the changed files.
  4. Measure output changes on fixed inputs. For the template generator, run npm run check:template-unchanged -- <base> <head> (TFP-137). Where the output should not change, require byte-identical output. Where it should change, show the difference and check it is the intended one.
  5. Read model output by hand against Phase 33’s bar, and record the result for each category.
  6. Check that each new test could have gone red (section 1’s three tiers): a bug fix’s reproduction failing before the fix, a behaviour change’s test failing with the change removed, or a stated exception. Where the report gives no such evidence, ask for it, or produce it by reverting the change locally.
  7. Check each acceptance criterion separately. Mark each one met, not met, or not checkable here, with the reason.
  8. Check the docs landed in the same commit: test-plan case, record status, roadmap, handover.
  9. State the limits. Say what couldn’t be verified, and where it can be. For example, the verifier’s VM has Node 22 and no Chromium, so the real-browser tests skip there and run on the local dev machine.
  10. Fix defects in the ticket. Any defect found goes back to the builder in that ticket before the gate passes.
  11. Record the verdict in the record’s status line, the roadmap and the status audit, so the next session does not have to verify the work again.

The generic method behind this list is the agent-report-verification skill. For refactors that claim unchanged behaviour, it is behavior-preservation-verification.

6. Known gaps in this system