Security tools usually talk about quality in vague language: stronger models, better prompts, smarter agents. That is not enough. If a system claims to help teams trust a release, the quality story has to be measurable, reproducible, and honest about misses.
We now anchor Guardix quality updates in benchmarked behavior. The current public reference point is EVMBench: 117 high-severity vulnerabilities across 40 Code4rena audits, scored with the official detect-mode evaluation.
| Metric | Result |
|---|---|
| Recall | 59.8% |
| Detected vulnerabilities | 70 / 117 |
| Audits with 100% recall | 21 / 40 |
| Best operating band | 30–60 findings per audit |
| Recall in that band | 98.6% |
What this means
The headline is not "AI audit solved." It has not. A 59.8% recall means Guardix found the majority of high-severity benchmark bugs, missed 47, and still needs human review around high-stakes releases.
The useful signal is more specific: when the pipeline stays focused, it performs dramatically better. Audits with 30–60 produced findings reached 98.6% recall. When finding count climbs above 100, recall drops to 32.3%. Noise is not harmless; it actively buries the real bugs.
The next quality frontier is not asking the model to say more. It is forcing the pipeline to keep the right 30–60 findings and prove why each one survived.
How this changes the product
- Finding count is now treated as a quality signal, not a vanity metric.
- Deduplication and normalization are evaluated by whether they preserve benchmark recall while reducing noise.
- Coverage work focuses on the missed multi-bug audits where the pipeline finds one issue and stops exploring the area deeply enough.
- Validation must cite what kills a finding; uncertain cases stay visible as needs_manual instead of disappearing.
Why this matters for release confidence
Release audits and code audits answer different questions, but they depend on the same discipline: evidence over vibes. A release scorecard should say when audit evidence is weak. A code audit should say what it found, what it missed in evals, and which changes are meant to close those misses.
That is the standard we want for Guardix updates: show the benchmark, show the failure modes, ship the improvement, and keep the numbers attached to the product surface users actually rely on.
What we are improving next
- Noise reduction around the 30–60 finding band where recall was strongest.
- Coverage-aware re-scanning for audits with several ground-truth vulnerabilities.
- Protocol knowledge packs for common DeFi integrations such as Uniswap, Aave, Chainlink, Pendle, Gnosis Safe, and Seaport.
- Signature binding verification for chain ID, contract address, nonce, and full call data.
The benchmark post has the full table and model comparison. This update is the product implication: quality is not a single model label. It is the measured behavior of the whole pipeline under pressure.