From the Palo Alto Networks edition of November 10, 2026
Preprint: 'Attack Success Rate Is Not a Number' finds most agentic-security papers omit variance and that single-run ASR comparisons often misrank defenses
Chetan Pathade, Prathamesh Pawar, Shubham Patil (arXiv authors) · Academic research / AI security measurement · Global literature review
- Publisher
- arXiv
- Original publication
- arXiv:2609.25173 v1, submitted 21 Sep 2026
- Source retrieved
- 2026-11-11
What happened
Preprint, not peer reviewed. The authors argue attack success rate (ASR) is a family of metrics shaped by six rarely specified design choices. In a meta-analysis of 259 agentic-security papers posted to arXiv from February 2025 to September 2026, most reported neither variance nor repeated runs for their headline attack metric (58%, 95% CI 44-71, in a hand-coded sample of 50; 65.3% by automated coding of all 259). Only 30.9% disclosed enough about decoding to tell whether evaluation was stochastic, and 29.7% of 64 papers using an LLM judge reported any human agreement check. Analytically, on a 100-instance benchmark the minimum detectable ASR difference at conventional power is 18.2 points, and two defenses whose true ASRs differ by 5 points are misranked by a single run about 21% of the time. They propose a ten-item reporting checklist.
Why it matters
Why now: Palo Alto's Nov. 10 Model Compass post publishes headline ASRs (0%, 7.1%, under 4%) without attack budget, repetitions or variance, and plans to offer those scores as a control input. This study gives SLED evaluators a neutral checklist for asking what such numbers mean before using them in model approval or procurement.
Evidence and measured results
Opened the arXiv abstract page (v1, 21 Sep 2026; 7 pages, 2 tables, 2 figures; DOI pending registration). Figures above are from the abstract. The full PDF and checklist items were not read.
Limitations and uncertainty
Preprint, not peer reviewed; three authors; meta-analysis limited to arXiv papers, not vendor benchmarks. Analytical power figures assume a 100-instance benchmark. Full text not reviewed in this run.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-11; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Buyers receive AI security benchmark numbers from vendors with no way to judge whether differences are real.
- Stakeholders
- CISO
- AI governance committee
- Procurement officer
- Security testing lead
- Discovery questions
- How do you currently evaluate vendor AI security claims?
- Do your RFPs require benchmark methodology disclosure?
- Who reviews test results before model or tool approval?
- Value hypothesis
- Adopting a reporting checklist can make AI security purchasing decisions more defensible and comparable across vendors.
- Potential engagement
- Help draft RFP language and an evaluation rubric for AI security benchmark claims based on the paper's checklist.
- Unsupported claims
- That any specific vendor benchmark is wrong
- That the paper evaluated Palo Alto's Model Compass
- That the findings are peer reviewed
Pre-sales engineering
Role takeaway
- Fit
- Applies to any proof of value involving prompt-injection or agent-attack testing.
- Architecture and integration
- Methodological guidance for test harnesses rather than a product integration.
- Prerequisites
- A defined attack test set
- Ability to repeat runs with logged settings
- A documented judging method
- Constraints
- Larger test sets and repeats cost time and tokens
- Vendor benchmarks may not disclose required details
- Security
- Better measurement reduces the chance of approving weaker controls on noisy evidence.
- Proof of value
- Run each candidate configuration multiple times and report variance
- Size the test set to detect the difference that matters
- Validate LLM-judge verdicts against a human-labeled sample
Delivery
Role takeaway
- Work
- Embed measurement-validity requirements into AI security pilot and acceptance test plans.
- Dependencies
- Security testing capacity
- Agreed test sets
- Governance review process
- Ownership
- Security testing lead, with AI governance committee review.
- Skills and adoption
- Testers need basic statistical literacy (confidence intervals, power) for agent security results.
- Governance checkpoints
- Test plan approval before pilot
- Results review with variance reported
- Vendor disclosure check in procurement
- Acceptance criteria
- Proposed: every reported ASR includes number of runs and an interval
- Proposed: judge agreement checked on a labeled sample
- Risks
- Pilots too small to detect real differences
- Vendors declining to disclose method
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Limited direct architecture impact; it bears on how security test results feeding model or control choices are produced and compared.
Governance
Who approves, reviews and stays accountable for outcomes?
AI governance boards and procurement teams can require vendors to disclose attack budget, number of runs, variance, judge method and harness for any ASR claim.
Security and privacy
What data, permissions and controls need testing?
Overconfident single-run scores may lead to approving models or defenses that are not meaningfully safer, leaving injection and exfiltration risk underestimated.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Testing staff need statistical literacy; smaller SLED teams may need partners for adequate test sizes.
Procurement
What should contracts, pricing and exit terms secure?
RFPs can require benchmark methodology, number of runs and confidence intervals for any AI security claim.
Operating model
Which teams own the service once it runs?
Security testing becomes repeatable, documented work rather than one-off demos.
Publication history
- 2026-11-10Palo Alto Networks · Issue 117 resources
Stable resource ID: arxiv-2609-25173-attack-success-rate-measurement-validity-20260921