{"resourceId":"arxiv-2609-25173-attack-success-rate-measurement-validity-20260921","versions":[{"version":"external-3fc48dc8fe209e49c280e2f5d42424311717817d306240cb450d2e86131826fe","resource":{"id":"arxiv-2609-25173-attack-success-rate-measurement-validity-20260921","title":"Preprint: 'Attack Success Rate Is Not a Number' finds most agentic-security papers omit variance and that single-run ASR comparisons often misrank defenses","organization":"Chetan Pathade, Prathamesh Pawar, Shubham Patil (arXiv authors)","sector":"Academic research / AI security measurement","geography":"Global literature review","publishedAt":"arXiv:2609.25173 v1, submitted 21 Sep 2026","publicationDate":"2026-09-21","eventDate":null,"sourceName":"arXiv","sourceLabel":"Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation (Preprint, not peer reviewed)","sourceUrl":"https://arxiv.org/abs/2609.25173","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","governance-procurement"],"finding":"Preprint, not peer reviewed. The authors argue attack success rate (ASR) is a family of metrics shaped by six rarely specified design choices. In a meta-analysis of 259 agentic-security papers posted to arXiv from February 2025 to September 2026, most reported neither variance nor repeated runs for their headline attack metric (58%, 95% CI 44-71, in a hand-coded sample of 50; 65.3% by automated coding of all 259). Only 30.9% disclosed enough about decoding to tell whether evaluation was stochastic, and 29.7% of 64 papers using an LLM judge reported any human agreement check. Analytically, on a 100-instance benchmark the minimum detectable ASR difference at conventional power is 18.2 points, and two defenses whose true ASRs differ by 5 points are misranked by a single run about 21% of the time. They propose a ten-item reporting checklist.","sledRelevance":"Why now: Palo Alto's Nov. 10 Model Compass post publishes headline ASRs (0%, 7.1%, under 4%) without attack budget, repetitions or variance, and plans to offer those scores as a control input. This study gives SLED evaluators a neutral checklist for asking what such numbers mean before using them in model approval or procurement.","evidence":"Opened the arXiv abstract page (v1, 21 Sep 2026; 7 pages, 2 tables, 2 figures; DOI pending registration). Figures above are from the abstract. The full PDF and checklist items were not read.","architectureImplications":"Limited direct architecture impact; it bears on how security test results feeding model or control choices are produced and compared.","governanceImplications":"Interpretation: AI governance boards and procurement teams can require vendors to disclose attack budget, number of runs, variance, judge method and harness for any ASR claim.","securityPrivacyImplications":"Overconfident single-run scores may lead to approving models or defenses that are not meaningfully safer, leaving injection and exfiltration risk underestimated.","caveats":"Preprint, not peer reviewed; three authors; meta-analysis limited to arXiv papers, not vendor benchmarks. Analytical power figures assume a 100-instance benchmark. Full text not reviewed in this run.","streamIds":["palo-alto-networks"],"roles":{"sales":"Sales framing: this preprint helps sellers stay credible when vendor benchmark numbers come up. It shows that headline attack-success rates often lack variance or repeat runs and that single runs misrank defenses about a fifth of the time when true rates differ by 5 points. Use it to suggest evaluation criteria customers can apply to every vendor, including Palo Alto, rather than to dismiss any one result. It supports a disciplined buying process, not a product claim.","engineering":"Engineering framing: before trusting any ASR, ask for the six design choices the authors identify, the number of runs, confidence intervals and the judge method. On a 100-instance benchmark, differences under about 18 points are not reliably detectable. Engineers designing proofs of value should size test sets and repeat runs accordingly, and record decoding settings so results can be reproduced and compared across candidate models or controls.","delivery":"Delivery framing: build the paper's reporting discipline into pilot test plans, including repeat runs, variance and documented judges, so results stand up in governance review. The security testing lead owns the method; governance reviewers check it. The main risk is pilots that declare a winner from one noisy run, followed by surprise when production behavior differs from the demo."},"roleDetails":{"sales":{"customerProblem":"Buyers receive AI security benchmark numbers from vendors with no way to judge whether differences are real.","stakeholders":["CISO","AI governance committee","Procurement officer","Security testing lead"],"discoveryQuestions":["How do you currently evaluate vendor AI security claims?","Do your RFPs require benchmark methodology disclosure?","Who reviews test results before model or tool approval?"],"valueHypothesis":"Adopting a reporting checklist can make AI security purchasing decisions more defensible and comparable across vendors.","potentialEngagement":"Help draft RFP language and an evaluation rubric for AI security benchmark claims based on the paper's checklist.","unsupportedClaims":["That any specific vendor benchmark is wrong","That the paper evaluated Palo Alto's Model Compass","That the findings are peer reviewed"]},"engineering":{"fit":"Applies to any proof of value involving prompt-injection or agent-attack testing.","architectureIntegration":"Methodological guidance for test harnesses rather than a product integration.","prerequisites":["A defined attack test set","Ability to repeat runs with logged settings","A documented judging method"],"constraints":["Larger test sets and repeats cost time and tokens","Vendor benchmarks may not disclose required details"],"security":"Better measurement reduces the chance of approving weaker controls on noisy evidence.","proofOfValue":["Run each candidate configuration multiple times and report variance","Size the test set to detect the difference that matters","Validate LLM-judge verdicts against a human-labeled sample"]},"delivery":{"work":"Embed measurement-validity requirements into AI security pilot and acceptance test plans.","dependencies":["Security testing capacity","Agreed test sets","Governance review process"],"ownership":"Security testing lead, with AI governance committee review.","skillsAdoption":"Testers need basic statistical literacy (confidence intervals, power) for agent security results.","governanceCheckpoints":["Test plan approval before pilot","Results review with variance reported","Vendor disclosure check in procurement"],"acceptanceCriteria":["Proposed: every reported ASR includes number of runs and an interval","Proposed: judge agreement checked on a labeled sample"],"risks":["Pilots too small to detect real differences","Vendors declining to disclose method"]}},"retrievedAt":"2026-11-11T03:51:55Z","enrichedAt":"2026-11-11T04:00:05Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Testing staff need statistical literacy; smaller SLED teams may need partners for adequate test sizes.","procurementImplications":"Interpretation: RFPs can require benchmark methodology, number of runs and confidence intervals for any AI security claim.","operatingModelImplications":"Security testing becomes repeatable, documented work rather than one-off demos.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.25173","referenceExcerpt":"two defenses whose true ASRs differ by 5 points are ranked in the wrong order by a single-run evaluation roughly 21% of the time","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}