From the CrowdStrike edition of October 30, 2026
Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents
Zhuowen Liu (arXiv preprint) · Academic research; AI agent security · Not specified (benchmark study; transferability to SLED deployments untested)
- Publisher
- arXiv
- Original publication
- 2026-10-02
- Source retrieved
- 2026-10-31
What happened
Preprint, not peer reviewed. Evaluating fifteen prompt-injection detectors (including Meta's Prompt Guard 2) and two LLM judges on benign-by-construction tool outputs from AgentDojo and tau-bench, the author finds rankings transfer poorly: the best detector on BIPIA caught 2% of AgentDojo injections at a 1% false-positive rate, and one catching 72% on AgentDojo caught 15% on tau-bench. False-positive rates on tool outputs ranged from none to over 90%. The author recommends evaluating on the agent's own tool outputs at low false-positive rates and auditing training data.
Why it matters
Independent scrutiny relevant to vendor guardrails such as Falcon Guardian's prompt and tool-call policy checks and CrowdStrike's prompt-injection taxonomy work. It suggests SLED buyers should ask for detection rates measured on their own agent workflows, not public benchmark scores. Why now: it is the most recent independent evidence found to weigh against this week's CrowdStrike agent-security claims.
Evidence and measured results
Single-author preprint with replay methodology across three benchmarks; does not test CrowdStrike or other commercial endpoint products.
Limitations and uncertainty
Preprint, not peer reviewed. Does not evaluate CrowdStrike products. Benchmarks may not represent public-sector workflows.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-30; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Buyers cannot tell from public benchmarks how well prompt-injection guardrails will work in their agents.
- Stakeholders
- CISO
- AI governance lead
- Procurement evaluator
- Developer platform owner
- Discovery questions
- How will the vendor measure detection on our agent workflows?
- What false-positive rate is acceptable?
- What data trained the guardrail model?
- Value hypothesis
- Workflow-specific testing reduces the risk of buying an ineffective guardrail; no direct savings claim.
- Potential engagement
- Design of an evaluation protocol for AI-agent guardrails during procurement.
- Unsupported claims
- Performance of any CrowdStrike product
- Peer-reviewed status
- Generalization to all agent types
Pre-sales engineering
Role takeaway
- Fit
- Any organization deploying or buying prompt-injection detection for agents.
- Architecture and integration
- Evaluation harness replaying agent tool outputs through candidate detectors.
- Prerequisites
- Sample of agent tool outputs
- Labelled injections
- Defined false-positive threshold
- Constraints
- Small labelled sets
- Workflow drift
- Vendor opacity on training data
- Security
- Measure both missed injections and false blocks.
- Proof of value
- Detection at 1% FPR on own tool outputs
- Cross-check on a second workflow
- Retest after model updates
Delivery
Role takeaway
- Work
- Build and maintain a labelled evaluation set and rerun it on changes.
- Dependencies
- Agent logs
- Analyst labelling time
- Vendor test access
- Ownership
- AI security engineer or SOC content team.
- Skills and adoption
- Staff need basic evaluation-metric skills.
- Governance checkpoints
- Evaluation protocol approval
- Results review before go-live
- Periodic retest
- Acceptance criteria
- Proposed: detection rate recorded at 1% FPR
- Proposed: retest schedule documented
- Proposed: training-data disclosure obtained
- Risks
- Unrepresentative test set
- Over-trust in benchmark scores
- Vendor refusal to disclose
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Detector choice must be validated inside the specific agent and tool environment; benchmark leadership is not a proxy.
Governance
Who approves, reviews and stays accountable for outcomes?
Procurement evaluations of AI guardrails should require low-false-positive detection metrics on representative workflows and disclosure of training data.
Security and privacy
What data, permissions and controls need testing?
High false-positive rates can block legitimate work; low detection rates leave injections through; both affect agent safety.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
No workforce data.
Procurement
What should contracts, pricing and exit terms secure?
Supports requiring workflow-specific, low-FPR evaluation in AI security procurements.
Operating model
Which teams own the service once it runs?
Guardrail tuning becomes an ongoing measurement task.
Publication history
- 2026-10-30CrowdStrike · Issue 0212 resources
Stable resource ID: arxiv-prompt-injection-detectors-reevaluation-2026