From the Palo Alto Networks edition of November 2, 2026
APTInvestBench (preprint): across eleven LLMs, SOC investigation agents gathered sufficient evidence for 44.3% of recoverable attack actions and cited support for only 25.0%
Yu Wang, Shuhao Li, Tao Yin, Ziyang Li, Xueying Zhao, Peishuai Sun, Jiang Xie (arXiv authors) · Academic security research / LLM-based security operations · Not jurisdiction-specific (authors' affiliations not confirmed from the opened pages)
- Publisher
- arXiv
- Original publication
- arXiv preprint 2609.38954v1, submitted Sept. 30, 2026
- Source retrieved
- 2026-11-03
What happened
Preprint, not peer reviewed. The benchmark has 370 cases across seven SOC-inspired telemetry conditions, built from 56 report-informed attack reconstructions with 16.4 million log records; the main evaluation concentrates on ten scenarios. Across eleven LLMs, agents acquired sufficient evidence for 44.3% of recoverable attack actions on average, while formal citations supported only 25.0%. Moving from full to endpoint-only telemetry reduced coverage by only 1.6 percentage points, yet 35.5% of previously covered actions lost sufficient citation support. In those losses, query returns lacked a complete witness 72.7% of the time; in the remaining 27.3% the evidence was retrieved but not cited. Losses persisted across four agent frameworks even when supporting records were unchanged.
Why it matters
Why now: XSIAM 4.0 (Nov. 2) pushes the SOC toward autonomous investigation and adds ingestion filtering and federated search, which change what telemetry agents see. This study shows that agent investigation quality can shift sharply when telemetry changes, even when aggregate scores look stable. SLED SOCs, which often have uneven log coverage across agencies, campuses and districts, should test agents under their own telemetry gaps. It does not test any Palo Alto product.
Evidence and measured results
Opened the arXiv abstract page (v1, submitted Sept. 30, 2026; DOI pending registration, no journal reference) and read the HTML full text, including the setup, results and evidence-loss analyses.
Limitations and uncertainty
Preprint, not peer reviewed; general-purpose models and open frameworks, not a Palo Alto product; synthetic reconstructions; main evaluation limited to ten scenarios; author affiliations not confirmed. Older than the window but newly relevant because of the XSIAM 4.0 launch.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-03; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- SLED SOCs considering autonomous investigation agents have no evidence of how agents behave under their own incomplete telemetry.
- Stakeholders
- SOC managers
- CISOs
- Incident response leads
- Security architects
- Discovery questions
- How complete is telemetry across the agencies or campuses the SOC covers?
- How would agent reports be checked before closure?
- Will ingestion filtering remove data agents rely on?
- Value hypothesis
- Interpretation: a customer-specific evaluation can show where agents help and where human scoping remains essential.
- Potential engagement
- A replay-based agent investigation evaluation under the customer's telemetry conditions.
- Unsupported claims
- Any conclusion about XSIAM or other Palo Alto agents
- Peer-reviewed status
- Real-environment accuracy
Pre-sales engineering
Role takeaway
- Fit
- Applicable as an evaluation method for any SLED SOC piloting AI investigation agents.
- Architecture and integration
- Benchmark varies log collection, retention and sampling and scores action-level evidence and citations.
- Prerequisites
- Library of past incidents
- Ability to replay telemetry
- Scoring rubric for attack actions
- Constraints
- Preprint
- General-purpose models only
- Synthetic reconstructions, ten-scenario main evaluation
- Security
- Unscoped attack actions can mislead response; require cited evidence for every conclusion.
- Proof of value
- Compare agent reports under full and filtered telemetry
- Score citation support per attack action
- Repeat after pipeline changes
Delivery
Role takeaway
- Work
- Set up a recurring agent-evaluation harness tied to log-pipeline change management.
- Dependencies
- Incident replay data
- SOC analyst reviewers
- Change-management process
- Ownership
- The SOC manager owns evaluation; security architecture owns telemetry design.
- Skills and adoption
- Analysts learn to audit agent citations.
- Governance checkpoints
- Baseline evaluation before go-live
- Re-test after telemetry changes
- Quarterly review
- Acceptance criteria
- Proposed: action-level evidence coverage reported per test run
- Proposed: 100% of agent-closed investigations reviewed in pilot
- Proposed: no filtering change deployed without agent re-test
- Risks
- Fluent but unsupported agent reports
- Telemetry loss hidden by aggregate scores
- Overgeneralizing benchmark results
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Telemetry design (what is collected, retained or filtered) directly affects agent investigation reliability, not just human analysis.
Governance
Who approves, reviews and stays accountable for outcomes?
Agent investigation reports should be accepted only when conclusions are tied to cited records, and evaluation should be repeated after log-pipeline changes.
Security and privacy
What data, permissions and controls need testing?
Incomplete agent reports can leave attack actions unscoped, a risk for incident response and breach notification decisions.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Analysts remain necessary to check evidence chains; training should cover reviewing agent citations.
Procurement
What should contracts, pricing and exit terms secure?
Buyers can ask vendors for action-level evidence and citation metrics under varying telemetry, rather than aggregate accuracy.
Operating model
Which teams own the service once it runs?
Supports keeping humans responsible for intrusion scoping while agents assist.
Publication history
- 2026-11-02Palo Alto Networks · Issue 045 resources
Stable resource ID: arxiv-aptinvestbench-autonomous-apt-investigation-telemetry-robustness-preprint-2609-38954