Precision, on code we do not control

Recall — a vulnerable app must keep failing

Precision alone is satisfied by a scanner that reports nothing. So a deliberately vulnerable application is held to a floor: if it ever stops failing, the gate goes red.

By rule — what a senior developer actually checks

A per-repository table proves the corpus was run; it does not show which rule is trustworthy. This table groups every error-severity finding of the same run by rule: how many fired across the twenty repositories, how many of those landed on a repository held to a zero ceiling — our cleanest controls, so a finding there is the shape of a false positive even when a confidence signal kept it from blocking — and how many tests under tests/ are tagged as that rule’s control pair (Doctrine #3: no rule ships without one). Sort any column by clicking its header.

Engine v1.62.0 @ 8f2240d8, generated 2026-10-05. This table is regenerated by the nightly corpus run (.github/workflows/dogfood-nightly.yml) whenever the numbers change — not on every merge — and rides the same rolling PR as the repository table above.

Head to head — the same commits, other scanners

The same pinned clones, handed to Semgrep (its default auto ruleset) and, where the repository is JavaScript or TypeScript, to ESLint with eslint-plugin-security’s recommended rules. The counts are not comparable one-to-one: GateTest’s blocking is a gate verdict — error-severity findings at or above the confidence threshold, across code quality, security, infrastructure and documentation — while Semgrep’s error is the label a rule author chose, eslint-plugin-security reports fourteen security rules, and CodeQL’s count comes from its own CLI running the official *-security-extended query suite for the repository’s language (e.g. javascript-security-extended.qls), with a result counted as blocking-equivalent at SARIF level error or a security-severity of 7.0 or above. Read each column as what that tool says about that commit and how long it took to say it. Each tool is time-boxed at 10 minutes per repository. SonarQube has not been run yet; CodeQL runs where its CLI is installed and the repository’s language has a security-extended suite. Either column says so, and why, instead of a number when it was not measured. 18 of 20 corpus repositories measured on 2026-10-05 with GateTest --suite full.

Last measured 17 days ago; the weekly run is scheduled every Monday and this page updates automatically once it lands.

Generated 2026-10-05 by scripts/head-to-head.js on engine v1.62.0. Every tool saw the same bytes; a run that hit the time box is written as timed out with the seconds it used, and a tool the runner could not install is written as unavailable. The script and the manifest are in the repository, so anyone can re-run the table.

Confidence — the block threshold, measured on the same run

Every error finding carries a confidence score: 1.0 unless a signal fires (a test file, a fixture, a comment, a string literal), and only findings at or above the block threshold fail the gate. The threshold used to be a number someone liked. Now each corpus run sweeps the alternatives: how much would block on the clean repositories, and how much the vulnerable one would still catch.

Confidence is not a continuum: this run produced only 6 distinct values (1.00, 0.60, 0.40, 0.24, 0.20, 0.12). The shipped threshold of 0.7 sits between 0.6 and 1, so any value in that gap gives the same gate. The signals kept 159 of 315 error findings off the clean repositories and cost 0 of 53 on the vulnerable one.