{"resourceId":"alert-bench-aida-llm-soc-triage-study","versions":[{"version":"external-69c6a601aae446e4f4b9ee27ff0f99a0d49a594f162e85c6af5b7cc94b4db032","resource":{"id":"alert-bench-aida-llm-soc-triage-study","title":"From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage","organization":"Virginia Tech, Google and North Carolina State University (Tsegai, Kantchelian, Yao, Gao)","sector":"Academic research (security operations)","geography":"USA (public datasets)","publishedAt":"2026-10-07","publicationDate":"2026-10-07","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint arXiv:2610.10608","sourceUrl":"https://arxiv.org/abs/2610.10608","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","data-security","operating-model"],"finding":"Using Alert-Bench, which replays enterprise telemetry through a live SIEM, the authors tested five LLM agent triage approaches on 1,247 alerts from a four-day multi-stage attack; every approach missed at least 40.4% of attack-related alerts (F1 0.371–0.744). Attack alerts were often dismissed when searches returned no records, and same-context self-review had negative net correction. Their Aida design (separate challenge context, judge, append-only ledger) reached F1 0.958 and a 3.1% false-negative rate while escalating 18.4% of alerts to analysts. Authors recommend human review for high-impact decisions.","sledRelevance":"Independent counterweight to vendor triage-accuracy claims: public SOCs should measure missed attacks, not just agreement rates, before trusting AI triage.","evidence":"Preprint benchmark on public AIT datasets in replay; not peer reviewed; does not test CrowdStrike or any commercial product.","architectureImplications":"Separating proposal and challenge contexts, keeping evidence ledgers and requiring stronger evidence for dismissal improved reliability in the benchmark.","governanceImplications":"Supports human review gates for alert dismissal and high-impact actions.","securityPrivacyImplications":"Incorrect dismissals leave attacks uninvestigated; authors note evidence can be incomplete or adversarially influenced.","caveats":"Preprint; single dataset family; analyst usage not studied; not product-specific. Transfer to real SLED SOC telemetry untested.","streamIds":["crowdstrike"],"roles":{"sales":"Missed attacks are the framing point: in this independent benchmark every common LLM triage approach missed at least 40.4% of attack alerts. Interpretation: buyers evaluating Charlotte AI or any AI triage should ask for false-negative measurements on their own data. A bounded engagement could design that evaluation. The study tested no commercial product, so it cannot be used to claim CrowdStrike performs better or worse; it supports careful validation and human review.","engineering":"Evidence sufficiency is the framing point: agents dismissed attacks when searches returned nothing, and self-review introduced errors. Interpretation: engineers should check whether vendor agents distinguish missing data from benign data, use separate review contexts and log evidence. Proof of value should report recall and false-negative rates on replayed local incidents. The benchmark's public dataset limits transfer, so local replay is essential before production use.","delivery":"Escalation design is the framing point: even the best design escalated 18.4% of alerts to humans. Interpretation: delivery plans should staff for escalations rather than assume full automation, and should record analyst overrides. Ownership sits with the SOC manager. Acceptance criteria are proposals, such as a measured false-negative rate on a labelled local set and a defined escalation SLA, because the study did not measure analyst workload in deployment."},"roleDetails":{"sales":{"customerProblem":"Agencies lack independent evidence on how often AI triage dismisses real attacks.","stakeholders":["CISO","SOC manager","Risk officer","Procurement evaluator"],"discoveryQuestions":["Can you measure AI triage false negatives on your alerts?","What escalation volume can your team absorb?","How are dismissals reviewed?"],"valueHypothesis":"Structured validation can show whether AI triage is safe for a given SOC.","potentialEngagement":"AI triage evaluation plan using locally replayed incidents.","unsupportedClaims":["Any conclusion about Charlotte AI accuracy","Full automation of triage","Transfer of benchmark numbers to SLED SOCs"]},"engineering":{"fit":"Methodology applies to any AI SOC triage tool under evaluation.","architectureIntegration":"SIEM replay of telemetry, agent search tools and an evidence ledger.","prerequisites":["Labelled incident data","Replay environment","Metric definitions"],"constraints":["Public dataset family","Preprint status","No product tested"],"security":"Dismissal without evidence is a failure mode; require evidence before closing alerts.","proofOfValue":["Measure recall on attack alerts","Test empty-search handling","Compare self-review with independent challenge"]},"delivery":{"work":"Evaluation harness and escalation workflow for AI triage pilots.","dependencies":["Historical alerts","Analyst time","Vendor API access"],"ownership":"SOC manager.","skillsAdoption":"Analysts learn to review AI evidence trails and challenge verdicts.","governanceCheckpoints":["False-negative threshold before production","Dismissal review sampling","Quarterly re-evaluation"],"acceptanceCriteria":["Documented local false-negative rate (proposed)","Escalation SLA defined","Monthly sample review of dismissed alerts"],"risks":["Benchmark-to-production gap","Escalation overload","Overconfidence in agreement metrics"]}},"retrievedAt":"2026-10-29T22:09:41-05:00","enrichedAt":"2026-10-29T22:18:51-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Escalation share implies continued analyst demand.","procurementImplications":"RFPs could require vendors to support customer-run false-negative testing.","operatingModelImplications":"Human review of dismissals and high-impact decisions remains necessary.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.10608","referenceExcerpt":"Across 1,247 alerts from a four-day multi-stage attack scenario, every approach missed at least 40.4% of attack-related alerts.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}