Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents

Prompt injection represents "the defining security problem of LLM agents." When agents process uncontrolled text sources like web pages or emails, attackers can embed instructions triggering harmful actions rather than producing benign output.

Historical defense approaches

Initial defenses attempted to make models reject malicious inputs through classifiers and fine-tuning. However, research by Nasr et al. revealed these approaches failed adaptive evaluation—when attackers knew the defense and optimized against it, success rates exceeded 90%, despite near-zero attack success on static benchmarks.

The second generation shifted enforcement outside the model entirely, using deterministic reference monitors. Systems including CaMeL, FIDES, Progent, RTBAS, and FORGE report substantial improvements on AgentDojo benchmarks.

The problem

"These second-generation defenses are being validated with exactly the methodology that just failed for the first generation"—fixed attack sets determined before the defense was created. Adaptive attacks have rarely tested these out-of-band solutions.

What we did

The research accomplishes three objectives:

• Organizes out-of-band defenses through classical security primitives (Biba integrity, reference monitors, least privilege, information-flow control)
• Specifies adaptive evaluation protocols including defense-aware attackers and relevant attack families
• Conducted adaptive testing of Progent using Qwen2.5-7B across three repeated runs

What we found

The deterministic gate held:

• Undefended baseline: 25.8% average attack success
• Progent-enabled: 4.2% success rate
• Adaptive attack: 2.6% success rate—comparable to standard attacks and within the band Progent's authors documented

The ordering remained consistent across all three runs. This contrasts sharply with in-band defenses, where adaptive attacks raised success rates from near-zero to above 90%.

What this doesn't show

Researchers acknowledge important limitations. This represents "one defense, one weak model, and one hand-crafted black-box attack family." Stronger white-box attacks or techniques confined to authorized actions remain untested. Qwen2.5-7B constitutes a modest agent; stronger models might present larger attack surfaces.

The result is "consistent with, but does not establish" the hypothesis that deterministic enforcement resists adaptive attacks better than in-band detection—a preliminary data point rather than conclusive proof.

Why it matters anyway

Security history demonstrates that control/data confusion vulnerabilities (buffer overflows, SQL injection, XSS) resist detection but yield to structural enforcement outside the attacker's controlled channel. Agent security has adopted deterministic, out-of-band action mediation as a foundational approach, representing genuine progress.

Current confidence in these defenses relies on evaluation methodology with a documented blind spot. This research provides the first independent, adaptive, defense-aware assessment—and the defense succeeded. Either outcome would have warranted investigation.

LaunchSafe's approach builds on deterministic, out-of-band mediation as a foundation, acknowledging that robustness against seriously optimized adaptive attacks remains unproven. The next investigation examines whether adaptive attacks confined entirely to already-authorized actions could circumvent the defense.

Read the paper: arXiv:2606.26479