{"resourceId":"splunk-state-of-observability-2026-agent-monitoring-ai-cost-gaps-20261020","versions":[{"version":"external-1923c6450cd6c9899c9e30fd9f0d17cbd677eaec9e3a04385579a49bf06cab20","resource":{"id":"splunk-state-of-observability-2026-agent-monitoring-ai-cost-gaps-20261020","title":"Splunk State of Observability 2026: 56% cannot measure AI response quality and 55% cannot control AI costs, in a 2,050-respondent survey","organization":"Cisco / Splunk (survey fielded by Oxford Economics)","sector":"IT operations and observability (cross-industry, includes government and education respondents)","geography":"10 countries including the United States","publishedAt":"Cisco press release, Oct. 20, 2026","publicationDate":"2026-10-20","eventDate":null,"sourceName":"Cisco Investor Relations (PR Newswire release)","sourceLabel":"Splunk State of Observability 2026: Avoiding the Hidden Costs of AI Agents","sourceUrl":"https://investor.cisco.com/news/news-details/2026/Splunk-State-of-Observability-2026-Avoiding-the-Hidden-Costs-of-AI-Agents/default.aspx","evidenceClass":"vendor-claim","outcomeClass":"cautionary","topics":["operating-model","governance-procurement","developers-agents"],"finding":"Cisco released its sixth Splunk State of Observability report. Oxford Economics surveyed 2,050 ITOps and engineering professionals in 10 countries and 16 industries, including government (federal, state and local) and education, from February through March 2026. Reported results: 87% say monitoring AI is a priority but 54% lack clear business outcomes tied to it; 56% cannot measure AI workload metrics such as LLM response quality; 85% call hallucination visibility important but only 58% say they are effective at it; 55% say they cannot control AI costs; 60% say unexpected billing spikes cause more strain than high baseline costs; 57% say telemetry costs exceed the infrastructure being monitored; 83% are concerned about unintended consequences of AI agents.","sledRelevance":"Agencies and campuses piloting AI agents face the same measurement and cost-predictability gaps; the finding that most organizations cannot measure agent quality is a useful caution for SLED pilots. Sector-level results for government and education are not broken out.","evidence":"Opened the press release and read it in full, including methodology. The full report was not opened.","architectureImplications":"Agent monitoring needs new metrics (response quality, hallucination, cost per agent/model) beyond traditional application monitoring, and shared data between observability and security teams.","governanceImplications":"Ties AI spend to business outcomes; public bodies need budget controls for usage-based AI pricing.","securityPrivacyImplications":"81% say AI monitoring increased collaboration between observability and security teams, reflecting agents’ broad access.","caveats":"Vendor-sponsored survey, self-reported; private and public sectors combined; government and education results not separated; full report not reviewed; leader-versus-non-leader comparisons are correlational. Transferability caveat: mostly private-sector, multinational sample.","streamIds":["cisco"],"roles":{"sales":"Sales framing: the useful signal here is caution, not a buying trigger: most surveyed organizations cannot measure agent quality or control AI costs. For agency and campus CIOs piloting agents, the problem is budget surprises and unmeasured quality. Discovery should ask how AI usage is billed and who sees it. A bounded engagement could define cost and quality metrics for one pilot. Do not cite leader ROI figures as expected outcomes; they come from a vendor survey and are correlational.","engineering":"Engineering framing: agent observability means capturing response quality, hallucination signals, cost per model and agent, and security-relevant actions, then sharing that data with security teams. Pre-sales should check whether current monitoring can attribute AI spend to a workload at all. A proof of value would instrument one agent pilot with quality and cost metrics and alert thresholds, then test whether a billing spike is detected before the invoice arrives.","delivery":"Delivery framing: delivery is an AI operations baseline: metric definitions, dashboards, cost alerts and a joint review between operations and security. The operational owner is the IT operations or platform team, with finance for budget thresholds. Proposed acceptance criteria: every production agent has an owner, a quality metric and a cost alert. Risk: telemetry volume itself becomes a cost problem, which 57% of respondents already report."},"roleDetails":{"sales":{"customerProblem":"Public bodies piloting AI agents cannot predict AI costs or measure answer quality.","stakeholders":["CIO","IT operations lead","Budget office","CISO"],"discoveryQuestions":["How is AI usage billed and who reviews it?","Can you measure response quality for current agents?","Has an unexpected AI or telemetry bill occurred?"],"valueHypothesis":"Defining quality and cost metrics early may prevent budget surprises and unsafe agent behaviour going unseen.","potentialEngagement":"Metric-definition and instrumentation workshop for one AI pilot.","unsupportedClaims":["Leader ROI or revenue figures as expected outcomes","Government-specific results","Any savings from adopting Splunk"]},"engineering":{"fit":"Applies to any SLED organization running AI agents or LLM workloads.","architectureIntegration":"Observability pipeline extended to LLM response quality, hallucination indicators and cost per agent, shared with security tooling.","prerequisites":["Inventory of AI workloads","Billing data access","Monitoring platform able to ingest AI telemetry"],"constraints":["Self-reported survey","Sectors combined","Telemetry costs can grow with coverage"],"security":"Agent telemetry may contain prompts with personal data; set retention and redaction.","proofOfValue":["Detect a simulated billing spike before invoice","Track a quality metric for one agent for a month","Measure telemetry cost added"]},"delivery":{"work":"Build an AI operations baseline for pilots.","dependencies":["Finance thresholds","Agent owners","Security team participation"],"ownership":"IT operations or platform team with finance","skillsAdoption":"Operations staff learn AI-specific metrics; finance learns usage-based billing patterns.","governanceCheckpoints":["Owner and metric assigned before production","Monthly cost review","Joint ops-security review of agent incidents"],"acceptanceCriteria":["Each production agent has owner, quality metric and cost alert","Cost alerts fire in test","Telemetry spend tracked"],"risks":["Telemetry cost growth","Metrics that do not reflect real quality","Vendor-framed benchmarks"]}},"retrievedAt":"2026-10-30T03:11:36Z","enrichedAt":"2026-10-30T03:18:21Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Not addressed.","procurementImplications":"Usage-based AI and telemetry pricing makes spikes, not baselines, the main budget risk; contracts should include caps or alerts.","operatingModelImplications":"\"Leaders\" in the survey report better cost attribution, but the leader definition and causality are vendor-framed.","sourceVerification":{"openedUrl":"https://investor.cisco.com/news/news-details/2026/Splunk-State-of-Observability-2026-Avoiding-the-Hidden-Costs-of-AI-Agents/default.aspx","referenceExcerpt":"56% reveal they're unable to measure AI workload metrics like LLM response quality, a core metric to tracking the success of an AI agent.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}