{"resourceId":"arxiv-reevaluating-prompt-injection-detectors-llm-agents-benchmarks-transfer-poorly-20261002","versions":[{"version":"external-ec21175b18fc743067a484c1436ce0fc2f36fadd383467b403d7d799894b2546","resource":{"id":"arxiv-reevaluating-prompt-injection-detectors-llm-agents-benchmarks-transfer-poorly-20261002","title":"Passing the Test You Trained On: re-evaluating 15 prompt-injection detectors on agent tool outputs finds benchmark rankings transfer poorly and false-positive rates range from none to over 90%","organization":"Academic authors (arXiv preprint 2610.03448)","sector":"AI security research / LLM agent guardrails","geography":"International","publishedAt":"arXiv preprint v1, Oct. 2, 2026","publicationDate":"2026-10-02","eventDate":null,"sourceName":"arXiv","sourceLabel":"Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents","sourceUrl":"https://arxiv.org/html/2610.03448","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["data-security","developers-agents","governance-procurement"],"finding":"The authors replay ground-truth tool calls from AgentDojo and tau-bench to get benign-by-construction tool outputs, label injected outputs by differential replay, and test 15 open detectors (including Meta Prompt Guard 2) plus two task-aware LLM judges, also on BIPIA. Rankings barely transfer: Kendall tau is 0.01 between BIPIA and AgentDojo and 0.28 between the two agent benchmarks. PIGuard catches 95.1% of BIPIA injections at 1% FPR but 2.1% on AgentDojo; one detector catching 72.2% on AgentDojo catches 15.2% on tau-bench. False-positive rates on tool outputs range from none to over 90% and are consistent across agent benchmarks. Training-input form, not attack-string overlap, explains results; every approach misses classes of injections outside its training distribution. The authors recommend evaluating on the agent’s own tool outputs at low FPR and auditing training data.","sledRelevance":"Agencies evaluating AI runtime guardrails, including Prisma AIRS prompt-injection defense, should not rely on vendor or public benchmark scores; they need tests on their own agent traffic.","evidence":"Opened the arXiv HTML and read the abstract, introduction, methods, results and limitations. Code is published for reproduction.","architectureImplications":"Detector-based defenses are easiest to add but brittle; system-level separation of untrusted data from planning remains a necessary complement.","governanceImplications":"Procurement evaluations of AI guardrails should require testing on representative tool outputs, low-FPR reporting and training-data disclosure.","securityPrivacyImplications":"Missed injections let attacker-written text in email, calendars or tables redirect agents; high false positives break tasks and push users to bypass controls.","caveats":"Preprint, not peer reviewed. Commercial API detectors, including Prisma AIRS, were not tested; relevance to any specific product is by analogy only.","streamIds":["palo-alto-networks"],"roles":{"sales":"Sales framing: independent researchers show prompt-injection detector scores on public benchmarks do not predict performance inside agents. For SLED buyers of AI guardrails, the problem is choosing a product without reliable comparative evidence. A bounded engagement could be a structured bake-off on the agency’s own agent traffic. This preprint did not test Prisma AIRS or other commercial products, so it cannot support claims that any vendor’s guardrail is better or worse.","engineering":"Engineering framing: the paper gives a usable test method: replay real tool calls for clean outputs, inject attacks by differential replay, and report detection at a 1% false-positive rate. Apply that to any runtime guardrail under consideration, including commercial services, using the agency’s own tools and data formats. Pair detectors with architectural controls that keep untrusted content away from action planning, since every approach missed some injection classes.","delivery":"Delivery framing: build a repeatable guardrail evaluation harness before scaling AI agents. The AI program office or security engineering team owns it. Acceptance should include detection and false-positive rates measured on representative tool outputs and re-run after major changes. Risks are adopting a guardrail on benchmark claims, high false positives disrupting services, and the effort required to build realistic test sets."},"roleDetails":{"sales":{"customerProblem":"Buyers cannot compare AI guardrails reliably using published benchmark scores.","stakeholders":["AI program leads","Security architects","Procurement evaluators"],"discoveryQuestions":["Will the vendor test on your agent’s tool outputs?","What false-positive rate is acceptable for your services?","Will the vendor disclose training data sources?"],"valueHypothesis":"Interpretation: agency-specific evaluation reduces the risk of buying an ineffective guardrail; no product was measured here.","potentialEngagement":"A guardrail evaluation on agency traffic using the replay method described.","unsupportedClaims":["That Prisma AIRS or any commercial product performs a certain way","Generalisation beyond tested benchmarks","Peer-reviewed status"]},"engineering":{"fit":"Applies to any detector-based prompt-injection defense for agents.","architectureIntegration":"Detectors screen tool outputs before the agent; complement with system-level separation.","prerequisites":["Representative tool-output samples","Replay capability","Attack corpus"],"constraints":["Commercial detectors untested","Preprint findings"],"security":"Measure both misses and false alarms; neither alone is sufficient.","proofOfValue":["Detection at 1% FPR on agency outputs","False-positive rate on benign outputs","Re-test after tool changes"]},"delivery":{"work":"Create and maintain a guardrail evaluation harness.","dependencies":["Agent tool inventory","Sample data access","Engineering time"],"ownership":"AI program office or security engineering.","skillsAdoption":"Staff need evaluation design skills; reuse the authors’ published code where licensing allows.","governanceCheckpoints":["Pre-deployment evaluation sign-off","Periodic re-evaluation"],"acceptanceCriteria":["Proposed: detection and FPR reported for each deployed guardrail","Proposed: evaluation repeated after significant agent changes"],"risks":["Unrealistic test data","False positives breaking services","Over-trust in benchmarks"]}},"retrievedAt":"2026-10-30T03:06:25Z","enrichedAt":"2026-10-30T03:12:42Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Running replay-based evaluations requires skills most SLED teams lack; shared test harnesses could help.","procurementImplications":"Supports evaluation clauses requiring demonstrations on agency data instead of headline benchmark numbers.","operatingModelImplications":"Guardrail performance must be monitored continuously as agent tools and data change.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/html/2610.03448","referenceExcerpt":"Detection rankings transfer poorly between benchmarks: the best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}