Scott Thornton PRO

They split exactly the way you stated, I verified in the code.

The dead-code case is branch-never-ran. The knowledge-graph example defines three access-control functions, validate-cypher, filter-results, controlled-query, all correct, and the live query path calls none of them. Enforcement there is a sentence in the prompt telling the model to behave, plus a logger.warning that blocks nothing. Feed the attack at the real entry point and assert the guard's branch executed: it never does.

The other is wrong-variable. The Bedrock example has a working log scrubber, and it runs, but on a metadata string that is a version hash and a message length. The decrypted system prompt flows to the model-invocation sink without ever passing through the scrubber. Assert that the tainted value reached the guard: it did not. The scrubbed variable and the leaked variable are different variables.

So you're right, it's two oracles, not one blind spot. And the part I hadn't appreciated is that both are questions about the original artifact, not the isolated guard, which means they are static. Reachability is a call-graph query, and the taint gap is a source-to-sink-through-sanitizer query. Neither needs the example to run, so neither is bound by the executability limit that held positive controls to a fraction of the corpus. That will be the next build. Thank you for drawing the box around it so precisely.

You called it before I had the number, and the number makes your point for you!

Positive controls, run for real: isolate each secure guard, push the paired attack through it, require it to fire. Every fire is gated so the rate can't be precision in disguise: a benign input must not fire, and a kill mutation (swap the guard for a pass-through) must make the fire vanish, which also proves the harness routes through the guard instead of hardcoding a verdict. I self-tested the runner on four known-good/known-bad cases and ran a blind check: a generator built a harness for the entropy scanner without being told it was broken, and the runner returned no-fire. Measuring the
oracle was the move I skipped, and you were right that it is the load-bearing one.

And it earned its keep immediately. One bulk pass I ran (a different model, generating harnesses at scale) reused six canned attack strings across every example; it "found" 106 no-fires. The mutation score gave it away: a 39% harness false-negative rate and a 15% fire rate where careful harnesses hit 100%. Those 106 were the oracle misfiring, not defects. I threw the batch out and regenerated it properly. So: the same disease you flagged, caught by the same medicine, on my own tooling.

Measured recall, a census of the executable subset rather than a sample: AI/ML 442 of 442 (100%, 95% CI 99.1-100%), web 77 of 77 (100%, CI 95.3-100%). Every executable secure guard in both configs, 519 in all, fired on its paired attack. Zero no-fires, zero new defects.

Now for the slice, which is the real story, imho. That rate is the number for guard-shaped examples, not the corpus. AI/ML is mostly guard-shaped, about 66% runnable, so the 100% means something. Web is not: positive-control-runnable is 79 of 1,249, roughly 6%. Not because web fixes are worse, but because most are a different object. 56% have no isolable callable at all: the fix is a Spring or Flask annotation, a security-header middleware, a TLS setting, a dependency pin, or secrets moved to env variables. 14% is a language I don't run, 12% is JavaScript I haven't wrapped. There's no input to feed.

So...I'm publishing two numbers, not one, plus a caveat: measured recall on the executable subset, and the remainder as unmeasured-by-positive-control. One tweak to the framing, though. The remainder isn't unpushed-on. It got the same adversarial review, which is how the dead-code defects got caught; it just lacks a mechanized recall estimate. So it keeps the human-reviewed status rather than inheriting a rate your own argument says is probably worse.

The limit I didn't expect: even on the executable subset, positive controls catch one class. I re-ran our three removed defects as known positives. The check caught one: the entropy scanner, broken guard logic. It missed two: correct guards that were dead code or never saw the data they were meant to scrub. Isolation cannot see reachability. One of three. The fire rate is broken-guard-logic recall, complementary to the review that caught the rest, not a replacement.

Where the fix is a guard, it holds up. Where it is config, positive controls are the wrong instrument, and I would rather say so than score it anyway. Thank you, and I mean that sincerely. This thread made the dataset more honest than the audit did.

Yep, you've named the real weakness, and you are right.

The 802 and the 28 are different epistemics. The CVEs had an external referee, so rechecking all 1,379 and driving it to zero is a final result that stays final. The 28 had no referee, and "zero disputed" is a precision number: it says what the review flagged held up, not what the review walked past. Human review cannot estimate its own false-negative rate, and "a reader noticed" is the honest account of how the entropy-always-zero scanner surfaced. A reader is not a guarantee.

Positive controls are the right fix, and I'm in the process of running them now. For each secure example that claims a specific vulnerability is eliminated, push the paired vulnerable input through the secure path and require the guard to fire. The entropy scanner dies on the first one with no argument to have. That turns "does the fix work" from a review opinion into an executable assertion, and the fraction that fires is the recall figure I do not currently have.

The honest constraint: a slice of the corpus is illustrative multi-turn snippets, not standalone programs with a defined input interface, so I can't mechanize it uniformly. But the failure mode you are pointing at lives in the executable subset, a scanner, a sanitizer, an auth gate, and those are exactly the ones I'm working to wrap in positive controls. Plan: isolate the runnable, secure paths; write a positive control per stated vulnerability; publish the fire rate as a real recall number; and call out the non-executable remainder separately rather than letting it hide inside a pass.

The oracle point is the sharpest part, and I hadn't framed it that way. Mutation-scoring the oracle instead of the artifact, torch.allclose waving through buggy variants, is the same disease as a review that flags nothing on a fake fix. Measuring the oracle is the move I skipped, and I am going to steal it. Thank you for that!

If 28 was just the count review that happened to catch, the positive-control pass will say so. I would rather publish that number than keep the precision one. Thank you sincerely for taking the time to share this.

https://huggingface.co/blog/scthornton/securecode-audit-fixes