Measured
Precision, on code we do not control
A scanner tuned against its own repository looks perfect on its own repository. The only honest test is code its authors did not write and cannot quietly adjust. Each repository below is cloned fresh at a pinned commit and scanned with --suite full — exactly what a paying Full Scan runs. The number is blocking findings; the ceiling is what CI holds the engine to, and it only ever moves down.
Precision — clean code must pass
| Repository | Commit | Blocking | Ceiling | Why it is in the corpus |
|---|---|---|---|---|
| express | 023767fe | 0 | 0 | the most-depended-on Node package there is; the canonical clean control |
| flask | d318b683 | 2 | 2 | Python — proves the non-JS rules are held to the same bar |
| fastify | 15ebc8e2 | 3 | 3 | large plugin-oriented server; exercises the auth and route grammars |
| got | 687eb7dc | 12 | 12 | TypeScript HTTP client with a benchmark/ tree — a known scope trap |
| hono | eebdf7be | 20 | 20 | every crossFileTaint finding here was false: c.req.query() read as SQL, RegExp.exec() as command exec; 2026-09-05 importCycle type-only elision: the four import cycles it blocked on read every binding inside function bodies — deferred, now a warning — ceiling ratchets down |
| zod | 7a002366 | 5 | 5 | monorepo with a docs site and benchmarks; was the worst case at 50, of which 33 were errorSwallow — 23 in benchmark harnesses, 7 the parsing library's coerce-then-check idiom (2026-09-04) |
| django | b3f4d83a | 60 | 60 | Python at scale — the ORM builds SQL by string in django/db, which is the one place that is the job; 74 of 76 "secrets" were test fixtures before the identifier-keyed rules learned about test paths |
| rails | 1ec64ce8 | 41 | 41 | Ruby at scale — instance_eval in the routes loader, backtick-quoted words in error messages that a naive shell-interp rule read as commands |
| spring-petclinic | 818c4136 | 8 | 8 | Java + Thymeleaf templates + a k8s manifest; the reference Spring app |
| gin | dcaa4296 | 1 | 1 | Go — the ignored-error idiom the go module could not see until 2026-09-04 |
| axum | 8b30afbf | 5 | 5 | Rust — first contact 28 blocking: todo!()/unimplemented!() inside #[cfg(test)] modules, dtolnay/rust-toolchain@stable read as a branch pin; what remains is third-party actions on @master and one real unimplemented!() |
| laravel | cdbd17f7 | 4 | 4 | PHP — first contact 28 blocking, 23 of them `$redis->eval($lua)` read as PHP eval(); what remains is real eval(var_export()) in the config cache and an unguarded rm -rf in a release script |
| cleanarchitecture | 3a06afb9 | 11 | 11 | C# + Angular + React — first contact 39 blocking, 26 of them SEO/a11y rules on the two SPA index.html shells; what remains is real: strict:false, a committed .env, unlabeled inputs, log-and-eat catches in a React component |
| ktor | f92fad04 | 7 | 7 | Kotlin — first contact 21 blocking, 14 of them a test.html under test-resources/ scored as a public page, plus a Gradle compile failure in our environment reported as failing tests; what remains is six real TODO() stubs and one third-party workflow on @main |
| vapor | 630763c5 | 0 | 0 | Swift — first contact 6 blocking: its own reusable workflows on @main and README section gaps; what remains is a *.key file the .gitignore does not cover |
| nest | 39fbddae | 9 | 9 | TypeScript monorepo (npm workspaces) with an integration/ e2e tree and sample apps — first contact 39 blocking: a logger class told it uses console, tsconfig flags as build verdicts, mongodb://localhost fixtures as secrets, `.catch(() => {})` on a stored promise the caller awaits; what remains is six `reduce(async …)` in the e2e app, two npm-audit advisories and one committed PKCS#8 key |
| trpc | 66d05445 | 7 | 7 | pnpm monorepo with members under examples/* and www — first contact 33 blocking: build configs and a CLI told they log to console, an OG-image generator scored as a web page, an Algolia search key as a secret, ternary env fallbacks as hardcoded URLs; what remains is four <img> without alt, a real import cycle, a silent connect in a constructor and `${{ github.event.release.tag_name }}` in a shell; 2026-09-05 importCycle type-only elision: its one import cycle ran through two `import { type X }` edges — elided, never a cycle in emitted JS — ceiling ratchets down |
| apollo-server | 53ac3c13 | 0 | 0 | npm-workspaces monorepo with a docs site — first contact 4 blocking: a CA bundle read as a private key, a README link the resolver could not follow, a destructured parameter the scope harvester missed; nothing remains |
| prisma | dd846dcc | 13 | 13 | pnpm monorepo, packages/** nesting, a SQL toolkit that builds SQL by design — first contact 90 blocking: `${STORAGE_TABLE}` constants as injection, seeding loops in test/ as N+1, teardown catches as swallows, tsconfig/ and turbo.json as invalid JSON, `base.sha` as shell injection, an un-runnable test:integration as failing tests; what remains is DDL with a computed database name, three import cycles, telemetry catches and a dev-tool page without a description; 2026-09-05 importCycle type-only elision: the three import cycles it blocked on read their bindings in parameter defaults and method bodies — deferred, now a warning — ceiling ratchets down |
Recall — a vulnerable app must keep failing
Precision alone is satisfied by a scanner that reports nothing. So a deliberately vulnerable application is held to a floor: if it ever stops failing, the gate goes red.
| Repository | Commit | Blocking | Floor |
|---|---|---|---|
| NodeGoat | c5cb68a7 | 53 | 40 |
Confidence — the block threshold, measured on the same run
Every error finding carries a confidence score: 1.0 unless a signal fires (a test file, a fixture, a comment, a string literal), and only findings at or above the block threshold fail the gate. The threshold used to be a number someone liked. Now each corpus run sweeps the alternatives: how much would block on the clean repositories, and how much the vulnerable one would still catch.
| Block at confidence ≥ | Blocking on clean repos | Still caught on NodeGoat |
|---|---|---|
| 0.50 | 295 | 53 |
| 0.60 | 295 | 53 |
| 0.65 | 208 | 53 |
| 0.70shipped | 208 | 53 |
| 0.75 | 208 | 53 |
| 0.80 | 208 | 53 |
| 0.85 | 208 | 53 |
| 0.90 | 208 | 53 |
| 0.95 | 208 | 53 |
| 1.00 | 208 | 53 |
Confidence is not a continuum: this run produced only 7 distinct values (1.00, 0.60, 0.40, 0.30, 0.24, 0.20, 0.12). The shipped threshold of 0.7 sits between 0.6 and 1, so any value in that gap gives the same gate. The signals kept 190 of 398 error findings off the clean repositories and cost 0 of 53 on the vulnerable one.
What remains on a repository is reported, not hidden. Django’s ORM builds SQL by string inside django/db, which is the one place that is the job; rails runner evaluates its input by contract. A scanner should say so, and the project is right to accept it.
Generated 2026-09-06 by scripts/real-world-precision.js on engine v1.61.0 @ 34107810. The corpus manifest and the runner are in the repository, so anyone can re-run the table. What the 121 modules check → Which rules teams silence →