Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents

Prompt injection is the defining security problem of LLM agents. An agent with tools reads text it doesn't control — web pages, emails, tickets, calendar entries — and when an attacker plants instructions in that text, the damage isn't a bad sentence. It's an action: a leaked record, a deleted row, a transfer of money.

The field's first answer was to teach the model to say no: input classifiers, guardrail models, adversarial fine-tuning. That answer failed a specific test. These defenses reported near-zero attack success on static benchmarks — and then researchers let the attacker move second. Nasr et al. took twelve published defenses and broke most of them at over 90% success with adaptive, defense-aware attacks. The static benchmark hadn't measured resistance to attackers; it had measured resistance to a known list of attacks.

The field's second answer moved enforcement outside the model entirely. Systems like CaMeL, FIDES, Progent, RTBAS, and FORGE wrap the agent in a deterministic layer — a reference monitor, in classical security terms — that decides which actions are permitted regardless of what the model was talked into. Several report near-elimination of attacks on the standard AgentDojo benchmark.

Here's the problem our new paper starts from: these second-generation defenses are being validated with exactly the methodology that just failed for the first generation. Fixed attack sets, chosen before the defense existed. The one test that broke the in-band defenses — an attacker who knows the defense and optimizes against it — has barely been applied to the out-of-band ones.

What we did

Our paper does three things.

First, it organizes the out-of-band defenses through the classical security primitives they instantiate — Biba integrity, reference monitors, least privilege, information-flow control — and compares them across eight dimensions. This lens isn't new (others have argued agents should adopt Saltzer–Schroeder directly), but it makes the comparison sharp. One finding surprised us: retrofit is not the open problem. Progent and FORGE already deploy onto unmodified agents. The open problem is whether the gate holds when the attacker adapts.

Second, it specifies the adaptive evaluation protocol the field is missing: a defense-aware attacker, the attack families to run (string optimization, provenance spoofing, endorsement-targeting, side-channel probes), and the metrics that matter.

Third — new in this version — we ran that protocol. We took Progent, the open-source privilege-control defense, and reproduced and extended its authors' own adaptive-attack analysis (their Appendix E) in a setting they didn't test: a weak, self-hosted, open-weight agent (Qwen2.5-7B on a single H200), on AgentDojo, three full repeated runs, everything scripted.

Our adaptive attack targets the one model-based component of an otherwise deterministic defense: Progent's policy-update LLM, which widens the tool allowlist based on the content of tool results. Our injection disguises the malicious action as a benign, pre-authorized, necessary step — crafted to talk the policy model into authorizing it.

What we found

The deterministic gate held.

• Undefended, the standard AgentDojo attack succeeded 25.8% of the time on average.
• With Progent enabled, that fell roughly sixfold, to 4.2%.
• Our defense-aware adaptive attack did not beat the defense: 2.6% — no higher than the standard attack, and within the band (~0.5–4.2%) Progent's authors report for their own adaptive constructions.
• The ordering held in every one of the three runs, with small variance.

This is the opposite of what adaptive attacks did to in-band defenses, where attack success went from near-zero to above 90%. And there's a plausible reason for the difference in kind: to beat a detector, you fool a classifier; to beat a deterministic gate, you have to achieve the goal through already-authorized actions or subvert a non-model check. That's a harder problem.

Defense isn't free, though. Task utility fell (mean ~45% → ~27% under Progent), and defended runs needed roughly 15× more LLM calls per task. Out-of-band protection is sound, but it has a bill.

What this doesn't show

We want to be as careful as the paper is, because overclaiming is precisely the failure mode the paper criticizes.

This is one defense, one weak model, and one hand-crafted black-box attack family. A stronger optimized attack — white-box GCG, or an attack confined entirely to already-authorized actions — remains untested and is the obvious next experiment. There's also a possible confound: the "disguise as benign" framing aimed at the policy model may have softened the imperative aimed at the agent, so our low number may reflect a weak attack as much as a strong defense. And Qwen2.5-7B is a modest agent; a stronger model has a fatter attack surface.

So the result is consistent with, but does not establish, the hypothesis that deterministic out-of-band enforcement is a harder target for an adaptive attacker than in-band detection. It's a preliminary data point, reproducible and independently corroborating Progent's own analysis — not a proof.

Why it matters anyway

Security history keeps teaching the same lesson. Buffer overflows, SQL injection, XSS — each was a control/data confusion, each resisted detection, and each was fixed by structure enforced outside the channel the attacker controlled. The agent-security field has relearned this and built defenses that are deterministic, sound in design, and already deployable as retrofits. That's real progress.

But confidence in those defenses currently rests on an evaluation method with a demonstrated blind spot. Our paper supplies a first piece of the evidence that method can't: an independent, adaptive, defense-aware test — and, this time, the defense passed it. Either outcome would have been worth knowing. Neither was known before.

At LaunchSafe, this shapes how we build: deterministic, out-of-band action mediation as the foundation of our agent-security tooling, with the honest caveat that its robustness against a seriously optimized adaptive attacker is still unproven. The next question we're handing ourselves: can an adaptive attack that lives entirely within the agent's authorized action space still get through?

That's the next study.

Read the paper: arXiv:2606.26479

Praneeth Narisetty, Shiva Nagendra Babu Kore, Uday Kumar Reddy Kattamanchi, Jayaram Kumarapu — LaunchSafe Research