Our autonomous web-exploitation engine, Mako, achieved a perfect score on the XBOW benchmark: 104 out of 104, black-box, with every solve verified against the genuine application response - never a model self-report. To our knowledge, this is the first published full-suite coverage of the XBOW-104 suite under a fabrication-proof verification regime.
The full write-up - architecture, formalism, and methodology is in our paper: Mako: A Self-Evolving Agentic Operating System (SE-AOS) for Autonomous Web Exploitation (arXiv, cs.CR). This post is the short version.
100% Success Rate - every category, every difficulty tier
Mako cleared all 104 real-world security challenges without seeing a single line of source code at run time. That places it first on the suite not just among black-box systems, but across every category. Shannon Lite, a white-box system with full source access, scored 96.2%. BoxPwnr scored 97.1%. Mako surpasses both without source access, and clears the full suite where the previous best black-box system left one challenge unsolved.
Sources: XBOW validation, MAPTA, BoxPwnr, Shannon, Strix, KinoSec, and XBOW study.
About the XBOW-104 benchmark
The XBOW validation-benchmarks are a public suite of 104 containerised, CTF-style web applications, each seeded with a single injected flag reachable only by exploiting a specific, realistic vulnerability — IDOR, SSTI, SQLi, XXE, deserialization, request smuggling, padding oracles, TOCTOU races, and more, across 26 vulnerability classes and three difficulty tiers. It measures whether a system can perform its own reconnaissance, identify the vulnerability class, chain multiple weaknesses into a working exploit, and actually cause the app to emit the flag.
Mako runs it as pure black-box: no source code, no internal documentation, no prior knowledge — only the exposed attack surface, exactly what a real attacker sees.
Results by difficulty
Results by vulnerability class
Assigning each benchmark a single primary class, the 104 solves break down as follows — every class cleared in full.
A representative case: XBEN-092
One of the case studies in the paper is XBEN-092 (SKU Service, L2), solved in 2 turns. An internal microservice can be coerced into deserialising attacker-controlled data through a language-level stream wrapper, reaching a gadget that ends in code execution, while a sibling endpoint provides the arbitrary file write needed to stage the payload. Mako recovered the gadget from disclosed source, constructed the serialized object and its container from scratch with no external tooling, staged it through the upload, and triggered the sink to read the flag. It selected the matching capability on turn one. The endpoints, wrapper, gadget, and container-building code are withheld.
The counter-intuitive part: the hardest tier is the fastest
You would expect turns-to-solve to climb with difficulty. It inverts. The hardest tier (L3) has the lowest median turn count — 2 turns because each L3 challenge was ultimately solved by a purpose-built, well-described capability the agent selected immediately.
This is the central finding. Difficulty doesn't live in the reasoner; it lives in whether the right capability exists in the library and whether the model can find it. Once a matching capability is present and its description maps cleanly onto the target's language, the per-turn probability of selecting it approaches one, and expected turns-to-solve approaches one with it. Capability, not reasoning, is what's scarce.
The engineering lesson that follows: for hard challenges, the bottleneck was tool selection, not tool capability. One benchmark, XBEN-023, went from an 18-turn failure to a 2-turn solve purely by improving a tool's description and self-discovery — no new exploit logic at all.
It runs at commodity cost
The entire 104-benchmark campaign cost $478.99 at official list pricing — roughly $4.61 per solved benchmark — processing ≈1.03 billion tokens end to end, ~95% of it input (the agent re-sends ~180 tool schemas plus its growing memory every turn, while its own output averages ~100 tokens). Full-suite, fabrication-proof web exploitation costs under five dollars per target at retail rates. That is what separates a mature pipeline from a scanner — and what makes it a force multiplier.
Methodology
- Testing mode: fully autonomous black-box exploitation
- Source-code access: none at run time — the agent interacts only over HTTP; any source analysis that informs the tool library happens out of band, never during a black-box run
- Turn budget: lean 25-turn budget per challenge
- Models: Gemini 2.5 Flash (primary) with a Gemini 3.1 Pro Preview escalation tier that engages only after 70% of the budget with no confirmed finding
- Verification: every build plants a fresh, random
FLAG{...}that does not exist when the tool code is written; the harness scans every raw tool result for that exact string, so a solve counts only when the flag literally surfaces in a genuine application response, never when the model claims success (false-positive probability ≤ 2⁻¹²⁸) - Infrastructure fixes: about 42 of the 104 upstream fixtures needed build/networking repairs to run on our platform; those changes are git-revertible and never touched the application, the vulnerability, or the flag
No challenges were excluded, no parameters were tuned per-challenge, and no manual intervention occurred during a run.
Per-benchmark results
Difficulty level (L), agent turns on the successful run (T), and primary vulnerability class, for all 104 benchmarks. Every result is a fresh-flag-verified solve. Exploit recipes, payloads, and tooling remain withheld.
Why we publish the science and withhold the weapon
This work began as an open release. Then the results arrived, and we changed our minds. A pipeline that turns "here is a URL" into a verified, working exploit at machine speed, across the full spectrum of common web vulnerabilities, with the hardest cases collapsing to a single tool call is not a scanner. It's a force multiplier for mass exploitation. The barrier protecting most of the web is not that these bugs are unknown; it's that reliably chaining reconnaissance, discovery, exploitation, and verification still takes skilled human effort. Mako removes that barrier.
So, modelled on responsible-disclosure and dual-use-research-of-concern practice, we split the release. We publish the science full-suite coverage under a fabrication-proof regime, an empirical account of where exploitation capability lives, and the self-evolving agent architecture. We withhold the weapon no payloads, no proof-of-concept code, no per-target exploit chains, no tool source.
LaunchSafe builds autonomous security agents for continuous offensive testing and agent-driven security research. Mako is our autonomous web-exploitation engine and proprietary core technology. Read the full paper on arXiv. Qualified partners and investors may request a private technical diligence under NDA - research@launchsafe.com.