Status: living document, started 2026-10-01.
This page brings together, in one place:
It links to the records that hold the reasoning and does not restate it. When a rule changes, change it at its source and update the link here in the same commit (standing execution policy, rule 2).
docs/WORKING_AGREEMENTS.md, “On measured evidence”:
“the repo should meet the bar it sells”.benchmarks/<date>-<name>/ holds the
outputs and the script that produced them. examples/ is committed too.
comparisons/ and runs/ are gitignored, so anything a decision rests
on is copied into benchmarks/ or summarised in the decision record.docs/decisions/data/labelled-failures.md holds the verdict for each one
(real, template error, model error, unclear), with its evidence, source
run and status. A failure on a real site counts as explained only once it
is recorded there.4b7120b) would have been caught sooner this way.git stash), and the
commit body says how and what it printed.check:template-unchanged and behavior-preservation-verification
instead) — the commit body says so and names what checks the change
instead.
A test that has never been seen to fail is not evidence for the change it
was written with. The dont-stub-what-you-prove skill covers the related
trap: a test that stubs what it is meant to prove.github.com/OWASP/ASVS (the 5.0 release), with
the file and version recorded. They are never written from memory (Phase
35 Decision 3).| Rule | Where it is set | Test plan |
|---|---|---|
| A test that skipped itself, with a reason, is “Not run”. It counts toward neither the pass ratio nor the recommendation (Option C). | Phase 34 Decision 7 | RPT-10, RPT-12 |
| If no test ran, there is no release recommendation and the badge is neutral. | Phase 34, GEN-44 follow-up | RPT-14 |
| A suite with nothing to run says “Nothing to run” and why, not “run failed”. | Phase 36 Decision 3 | RPT-15 |
| Findings (hardening gaps, optional items) are not failures. | Phase 34 Decision 3; Phase 36 Decision 2 | RPT-11, GEN-52 |
| Coverage is honest: a gap is named, never rounded up to 100%. | Discovered-coverage report (src/lib/report/discovered-coverage.ts) |
RUN-10, RUN-11 |
| A Comprehensive report ends with a fixed statement of what it did not test. | Phase 35 Decision 6 | RPT-16 |
| Generation gate verdicts are saved with the run and shown in its report. | Phase 34 Decision 8 | RPT-13 |
Principle: detect and fall back, never rewrite (Phase 30; restated in Phase 37 Decision 3).
generatedBy: "fallback" with a fallbackReason.Every gate records a verdict in one shape: GateVerdict in
src/lib/generate/gates.ts (Phase 34 Decision 8).
kept or rejected), evidence.countGate). The same shape takes a score and a
threshold if a calibrated gate is added later.The gates run in this order, in src/lib/generate/local-generation.ts:
| Gate (as recorded) | What it rejects | Code | fallbackReason |
Introduced | Test plan |
|---|---|---|---|---|---|
| time budget | No answer within MRONION_OLLAMA_TIMEOUT_MS (default 180 s) |
ollamaTimeoutMs, local-generation.ts |
timeout |
Phase 30 (TFP-105) | GEN-24 |
| type check | Any TypeScript error against the project’s Playwright types | findTypeErrors, validate.ts |
invalid_output |
Phase 30 (TFP-103 follow-up) | GEN-26 |
| unawaited assertions | A web-first assertion that is never awaited, so it never runs | findUnawaitedAssertions, validate.ts |
unawaited_assertion |
Phase 33 (TFP-110) | GEN-31 |
| promise to plain matcher | expect(<Promise>) followed by a matcher that doesn’t retry. .resolves and .rejects are exempt. |
findPromisesToPlainMatchers, validate.ts |
promise_to_plain_matcher |
Phase 37 Decision 2 (TFP-129) | GEN-56 |
| duplicate tests | Two or more tests with the same body after whitespace and comments are normalised | findDuplicateTests, validate.ts |
duplicate_tests |
Phase 37 Decision 3 (TFP-130) | GEN-57 |
Some fallbacks happen before any gate runs:
ollama_unreachable (GEN-22);ollama_model_missing (GEN-23);ollama_error and empty_completion (GEN-25).The gates catch failures that are mechanical. Quality is judged by reading.
benchmarks/2026-09-30-gap-fill-fix/ holds the gap-fill result./dashboard.These are left to the failure-triage gate deferred in Phase 34. That gate will be built on the labelled failure set.
npm run check:template-unchanged -- <base-ref> [<head-ref>] (TFP-137,
GEN-58): 172 fixed inputs, 28 of them with page data, each ref’s own
generator, compared byte for byte. Exit 0 only when all are identical.
Two of them run Comprehensive’s generator, so a change to Comprehensive
alone shows as different there. Every helper the generator can write into
a suite is reached by at least one, and a unit test fails if a new one
isn’t (TFP-258, GEN-107); comments between a helper’s declarations are
the known exception..github/dependabot.yml). PR Validation (lint,
the old-name check, build, smoke) and CodeQL pass on it; checked out locally after npm ci,
npm run test:unit and the regression suite (npm run test:regression,
which CI runs only by hand) pass, with their counts in the merge commit.
A failure is either explained there, with the output, or the pull request
is closed and the update ticketed. check:template-unchanged is not
evidence here: both refs link the working tree’s one node_modules, so
for a pull request that changes only package.json and the lockfile it
compares the same code against itself and is always identical. A major
update gets its own pull request and is judged on its own. A Next.js
minor or patch arrives in a pull request of its own, with react,
react-dom and eslint-config-next, and gets a ticket; a Next.js major
isn’t proposed at all (it needs a decision record). What each
held-back update waits on is the watch list in
docs/decisions/dependency-upkeep-2026-11-08.md.MRONION_INTERACTIVE=1 (GEN-44).docs/ says which gate
is next.docs/decisions/product-roadmap.md, from
Phase 18):
docs/test-plan/ (TEST_PLAN.md is the index), as each record’s
Regression Policy requires.Use this for every “done” report before a gate is passed. The builder’s report is a claim; this checklist turns it into evidence.
git log and git show --stat on the
reported hashes. Check the changed files match the report and nothing
outside the ticket changed. Run git status -sb to see what is pushed.
When another agent is using the repo, use git --no-optional-locks, so
no index.lock blocks its commits.node node_modules/typescript/bin/tsc --noEmit -p .;npm run check:template-unchanged -- <base> <head> (TFP-137).
Where the output should not change, require byte-identical output.
Where it should change, show the difference and check it is the
intended one.The generic method behind this list is the agent-report-verification
skill. For refactors that claim unchanged behaviour, it is
behavior-preservation-verification.
npm run check:template-unchanged (sections 3 and 5).
What follows is the gap as it stood: The 144-input generator
scripts used to prove Standard output unchanged exist only on the
verifier’s side. One of them reads
benchmarks/2026-09-29/content-profile.json. Until they are committed,
nobody else can repeat that check. This is tracked as TFP-137 in the
Phase 35 record, which names the command in sections 3 and 5 and closes
this gap.