A static analyzer sees eval(x) and asks
"is x tainted by SQL or an HTTP parameter?" It has no
concept of "x is the model's reply." That blind spot
— model output reaching a code, shell, or deserialization sink — is the
entire agent layer, and it's invisible to the tools most teams already run.
What's actually out there
eval / execeval() on the assistant's own reply (CVE-2025-51472).create(**params)) with no max_tokens —
27 call sites. Cost hygiene, not a vuln, on its own; but near-universal,
and passed through **kwargs where most scanners can't see it.pickle.loads()
as RCE. Almost none of them are. Which brings us to the part that matters
more than the findings.The part nobody publishes
What we refused to flag
A scanner that cries wolf gets demoted to advisory and ignored. The hardest — and most valuable — engineering here is not flagging the things that look scary but aren't. Every row below is a real pattern in a real, well-engineered framework that a lazy scanner reports as critical:
| Looks like HIGH | Is actually FALSE POSITIVE | Where | |
|---|---|---|---|
pickle.loads(data) "RCE from user input" | → | a LogRecord off a local multiprocessing pipe — trusted internal transport | top-tier voice-agent framework |
pickle.loads(message_ser) "RCE" | → | the framework round-tripping its own serialized objects | leading multi-agent framework |
HEADER_TOKEN = "X-…-Token" "hardcoded secret" | → | an HTTP header name, not a value | top-tier voice-agent framework |
gas_token = "0x0000…0000" "secret" | → | the Ethereum zero address — a constant | on-chain agent framework |
KEY = "YOUR-AZURE-…-KEY" "leaked key" | → | a placeholder / example value | major data framework |
57 findings, HOLD | → | all in a cookbook of 256 tutorial files; the core is clean | minimalist agent framework |
That last row is the pattern that quietly wrecks trust: grading
a framework on its tutorials instead of its code. We exclude
cookbook/, examples/, and tests/ from the
score entirely — and surface them separately, labelled unscored.
We assert HIGH · confirmed only when we can
see the dangerous input's source — a value assigned from an LLM
call in scope, or an unambiguous name like request.body. When the
danger is only inferred from a variable's name, we say so:
MEDIUM · inferred — "confirm the source." A
judge rules on the evidence in front of it, not on what a name suggests.
How we verified · what we don't claim
Every high-severity finding was traced to the exact line and read in context before we trusted it — that process is what produced the table above. Several findings that looked confirmed collapsed under a two-minute read; the local-IPC pickle and the header-name "secret" both came from a marquee framework, and both were wrong.
We're not publishing a precision/recall number yet — that needs a labeled benchmark with true negatives, and we're building it: every verified verdict now feeds a calibration corpus. What this is: a methodology and a set of hand-checked findings across the real ecosystem, plus a public account of the false positives we eliminated. Honesty about the second is the point.
One finding closed the loop the way it should — we reported an unbounded tool-call loop to the LightAgent maintainers, and they shipped a fix. A credible finding, acted on. That's the bar.
Scan your own agent
Two honest scores and one verdict — PROMOTE / HOLD / BLOCK. Every finding cites its evidence and its confidence.
# free, no config
pip install release-gate
release-gate audit .
Methodology, the analyzer, and the full false-positive test suite are open source. Findings were hand-verified; only a publicly-disclosed vulnerability (the SuperAGI CVE) and a maintainer-confirmed fix (LightAgent) are named — every other pattern is reported in aggregate, by design.