From the CrowdStrike edition of October 29, 2026
From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage
Virginia Tech, Google and North Carolina State University (Tsegai, Kantchelian, Yao, Gao) · Academic research (security operations) · USA (public datasets)
- Publisher
- arXiv
- Original publication
- 2026-10-07
- Source retrieved
- 2026-10-30
What happened
Using Alert-Bench, which replays enterprise telemetry through a live SIEM, the authors tested five LLM agent triage approaches on 1,247 alerts from a four-day multi-stage attack; every approach missed at least 40.4% of attack-related alerts (F1 0.371–0.744). Attack alerts were often dismissed when searches returned no records, and same-context self-review had negative net correction. Their Aida design (separate challenge context, judge, append-only ledger) reached F1 0.958 and a 3.1% false-negative rate while escalating 18.4% of alerts to analysts. Authors recommend human review for high-impact decisions.
Why it matters
Independent counterweight to vendor triage-accuracy claims: public SOCs should measure missed attacks, not just agreement rates, before trusting AI triage.
Evidence and measured results
Preprint benchmark on public AIT datasets in replay; not peer reviewed; does not test CrowdStrike or any commercial product.
Limitations and uncertainty
Preprint; single dataset family; analyst usage not studied; not product-specific. Transfer to real SLED SOC telemetry untested.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-29; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Agencies lack independent evidence on how often AI triage dismisses real attacks.
- Stakeholders
- CISO
- SOC manager
- Risk officer
- Procurement evaluator
- Discovery questions
- Can you measure AI triage false negatives on your alerts?
- What escalation volume can your team absorb?
- How are dismissals reviewed?
- Value hypothesis
- Structured validation can show whether AI triage is safe for a given SOC.
- Potential engagement
- AI triage evaluation plan using locally replayed incidents.
- Unsupported claims
- Any conclusion about Charlotte AI accuracy
- Full automation of triage
- Transfer of benchmark numbers to SLED SOCs
Pre-sales engineering
Role takeaway
- Fit
- Methodology applies to any AI SOC triage tool under evaluation.
- Architecture and integration
- SIEM replay of telemetry, agent search tools and an evidence ledger.
- Prerequisites
- Labelled incident data
- Replay environment
- Metric definitions
- Constraints
- Public dataset family
- Preprint status
- No product tested
- Security
- Dismissal without evidence is a failure mode; require evidence before closing alerts.
- Proof of value
- Measure recall on attack alerts
- Test empty-search handling
- Compare self-review with independent challenge
Delivery
Role takeaway
- Work
- Evaluation harness and escalation workflow for AI triage pilots.
- Dependencies
- Historical alerts
- Analyst time
- Vendor API access
- Ownership
- SOC manager.
- Skills and adoption
- Analysts learn to review AI evidence trails and challenge verdicts.
- Governance checkpoints
- False-negative threshold before production
- Dismissal review sampling
- Quarterly re-evaluation
- Acceptance criteria
- Documented local false-negative rate (proposed)
- Escalation SLA defined
- Monthly sample review of dismissed alerts
- Risks
- Benchmark-to-production gap
- Escalation overload
- Overconfidence in agreement metrics
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Separating proposal and challenge contexts, keeping evidence ledgers and requiring stronger evidence for dismissal improved reliability in the benchmark.
Governance
Who approves, reviews and stays accountable for outcomes?
Supports human review gates for alert dismissal and high-impact actions.
Security and privacy
What data, permissions and controls need testing?
Incorrect dismissals leave attacks uninvestigated; authors note evidence can be incomplete or adversarially influenced.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Escalation share implies continued analyst demand.
Procurement
What should contracts, pricing and exit terms secure?
RFPs could require vendors to support customer-run false-negative testing.
Operating model
Which teams own the service once it runs?
Human review of dismissals and high-impact decisions remains necessary.
Publication history
- 2026-10-29CrowdStrike · Issue 0110 resources
Stable resource ID: alert-bench-aida-llm-soc-triage-study