{"resourceId":"arxiv-aptinvestbench-autonomous-apt-investigation-telemetry-robustness-preprint-2609-38954","versions":[{"version":"external-46f58d4f8de42f1727f9db1f4eb9d0b0302fd7c891e11d5687e1168b66278400","resource":{"id":"arxiv-aptinvestbench-autonomous-apt-investigation-telemetry-robustness-preprint-2609-38954","title":"APTInvestBench (preprint): across eleven LLMs, SOC investigation agents gathered sufficient evidence for 44.3% of recoverable attack actions and cited support for only 25.0%","organization":"Yu Wang, Shuhao Li, Tao Yin, Ziyang Li, Xueying Zhao, Peishuai Sun, Jiang Xie (arXiv authors)","sector":"Academic security research / LLM-based security operations","geography":"Not jurisdiction-specific (authors' affiliations not confirmed from the opened pages)","publishedAt":"arXiv preprint 2609.38954v1, submitted Sept. 30, 2026","publicationDate":"2026-09-30","eventDate":null,"sourceName":"arXiv","sourceLabel":"APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry (arXiv:2609.38954)","sourceUrl":"https://arxiv.org/abs/2609.38954","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["developers-agents","operating-model","infrastructure"],"finding":"Preprint, not peer reviewed. The benchmark has 370 cases across seven SOC-inspired telemetry conditions, built from 56 report-informed attack reconstructions with 16.4 million log records; the main evaluation concentrates on ten scenarios. Across eleven LLMs, agents acquired sufficient evidence for 44.3% of recoverable attack actions on average, while formal citations supported only 25.0%. Moving from full to endpoint-only telemetry reduced coverage by only 1.6 percentage points, yet 35.5% of previously covered actions lost sufficient citation support. In those losses, query returns lacked a complete witness 72.7% of the time; in the remaining 27.3% the evidence was retrieved but not cited. Losses persisted across four agent frameworks even when supporting records were unchanged.","sledRelevance":"Why now: XSIAM 4.0 (Nov. 2) pushes the SOC toward autonomous investigation and adds ingestion filtering and federated search, which change what telemetry agents see. This study shows that agent investigation quality can shift sharply when telemetry changes, even when aggregate scores look stable. SLED SOCs, which often have uneven log coverage across agencies, campuses and districts, should test agents under their own telemetry gaps. It does not test any Palo Alto product.","evidence":"Opened the arXiv abstract page (v1, submitted Sept. 30, 2026; DOI pending registration, no journal reference) and read the HTML full text, including the setup, results and evidence-loss analyses.","architectureImplications":"Telemetry design (what is collected, retained or filtered) directly affects agent investigation reliability, not just human analysis.","governanceImplications":"Interpretation: agent investigation reports should be accepted only when conclusions are tied to cited records, and evaluation should be repeated after log-pipeline changes.","securityPrivacyImplications":"Incomplete agent reports can leave attack actions unscoped, a risk for incident response and breach notification decisions.","caveats":"Preprint, not peer reviewed; general-purpose models and open frameworks, not a Palo Alto product; synthetic reconstructions; main evaluation limited to ten scenarios; author affiliations not confirmed. Older than the window but newly relevant because of the XSIAM 4.0 launch.","streamIds":["palo-alto-networks"],"roles":{"sales":"Sales framing: this preprint gives a grounded way to discuss agentic SOC claims with public-sector buyers: ask how agents perform when logs are incomplete. The customer problem is evaluating autonomous investigation features without a baseline. A bounded engagement could design a customer-specific evaluation replaying past incidents under the agency's real telemetry. The study tested no Palo Alto product, so it supports caution and testing, not conclusions about XSIAM performance.","engineering":"Engineering framing: the key lesson is that agent results depend on telemetry and that aggregate scores hide action-level losses. Engineers evaluating XSIAM agents should replay incidents with full and reduced telemetry, such as after ingestion filtering, and score whether each attack action is supported by cited records. Track citation completeness separately from detection. Note the study used open frameworks and general models, so results transfer only as a test method.","delivery":"Delivery framing: build agent evaluation into change management. Every log-pipeline or filtering change should trigger a re-test of agent investigations on a fixed incident set. Proposed acceptance includes action-level evidence coverage reported for each test and analyst review of agent reports before closure. Risks include accepting fluent but incompletely cited reports and filtering away telemetry that agents need."},"roleDetails":{"sales":{"customerProblem":"SLED SOCs considering autonomous investigation agents have no evidence of how agents behave under their own incomplete telemetry.","stakeholders":["SOC managers","CISOs","Incident response leads","Security architects"],"discoveryQuestions":["How complete is telemetry across the agencies or campuses the SOC covers?","How would agent reports be checked before closure?","Will ingestion filtering remove data agents rely on?"],"valueHypothesis":"Interpretation: a customer-specific evaluation can show where agents help and where human scoping remains essential.","potentialEngagement":"A replay-based agent investigation evaluation under the customer's telemetry conditions.","unsupportedClaims":["Any conclusion about XSIAM or other Palo Alto agents","Peer-reviewed status","Real-environment accuracy"]},"engineering":{"fit":"Applicable as an evaluation method for any SLED SOC piloting AI investigation agents.","architectureIntegration":"Benchmark varies log collection, retention and sampling and scores action-level evidence and citations.","prerequisites":["Library of past incidents","Ability to replay telemetry","Scoring rubric for attack actions"],"constraints":["Preprint","General-purpose models only","Synthetic reconstructions, ten-scenario main evaluation"],"security":"Unscoped attack actions can mislead response; require cited evidence for every conclusion.","proofOfValue":["Compare agent reports under full and filtered telemetry","Score citation support per attack action","Repeat after pipeline changes"]},"delivery":{"work":"Set up a recurring agent-evaluation harness tied to log-pipeline change management.","dependencies":["Incident replay data","SOC analyst reviewers","Change-management process"],"ownership":"The SOC manager owns evaluation; security architecture owns telemetry design.","skillsAdoption":"Analysts learn to audit agent citations.","governanceCheckpoints":["Baseline evaluation before go-live","Re-test after telemetry changes","Quarterly review"],"acceptanceCriteria":["Proposed: action-level evidence coverage reported per test run","Proposed: 100% of agent-closed investigations reviewed in pilot","Proposed: no filtering change deployed without agent re-test"],"risks":["Fluent but unsupported agent reports","Telemetry loss hidden by aggregate scores","Overgeneralizing benchmark results"]}},"retrievedAt":"2026-11-03T04:07:17Z","enrichedAt":"2026-11-03T04:10:56Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Analysts remain necessary to check evidence chains; training should cover reviewing agent citations.","procurementImplications":"Interpretation: buyers can ask vendors for action-level evidence and citation metrics under varying telemetry, rather than aggregate accuracy.","operatingModelImplications":"Supports keeping humans responsible for intrusion scoping while agents assist.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2609.38954","referenceExcerpt":"Across eleven LLMs, agents acquire sufficient evidence for 44.3% of recoverable attack actions on average, while formal citations support only 25.0%.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}