{"resourceId":"npr-schools-struggle-measure-ai-experiments-dayton-wichita-20261019","versions":[{"version":"external-fe2a86445bca0ea06a45410bd2fee16d269dcec373ee2b15bd7469340ddec0df","resource":{"id":"npr-schools-struggle-measure-ai-experiments-dayton-wichita-20261019","title":"NPR: Dayton and Wichita AI experiments can only measure perception, and CRPE warns innovation is outrunning evidence-building","organization":"NPR (All Things Considered)","sector":"K-12 AI pilot evaluation and evidence-building","geography":"Dayton, Ohio and Wichita, Kansas, United States","publishedAt":"Monday, October 19, 2026, 4:59 p.m. EDT (NPR)","publicationDate":"2026-10-19","eventDate":null,"sourceName":"NPR","sourceLabel":"National public radio report (broadcast transcript)","sourceUrl":"https://text.npr.org/nx-s1-5991881","evidenceClass":"independent-reporting","outcomeClass":"cautionary","topics":["operating-model","governance-procurement","knowledge-work"],"finding":"NPR reports on a chatbot named Jordan used last school year at the Alternative School at Jackson Center in Dayton, Ohio, which asks students for advice to draw out their feelings and grades their responses. The consultant who helped design and evaluate it says every attempt at concrete measurement only reached perception, and the district relies on asking students and staff. He cautions against scaling it as a full solution. Wichita Public Schools' AI specialist says the district is introducing AI tools slowly, including a high school AI math tutor, and that experiments will stay small until educators can measure their value. Robin Lake of the Center on Reinventing Public Education says innovation is proceeding without much attention to evidence-building and that states and the federal government should set guardrails rather than stop experimentation.","sledRelevance":"Direct evidence that districts lack outcome measures for AI pilots, including a student-facing social-emotional chatbot in a school for students with behavioral challenges, which limits scaling decisions.","evidence":"Broadcast report with interviews; no evaluation data, sample sizes or methods published.","architectureImplications":"Not addressed. Interpretation: measuring beyond perception requires logging chatbot interactions and linking them to existing outcome data under privacy controls.","governanceImplications":"Districts are keeping AI pilots small pending evidence; CRPE calls for state and federal guardrails on experimentation.","securityPrivacyImplications":"Not discussed. Interpretation: a chatbot eliciting students' feelings in an alternative school raises sensitive-data and counseling-escalation questions the report does not cover.","caveats":"Published October 19 and not previously archived; anecdotal interviews; no quantitative results; the chatbot vendor and data handling are not described.","streamIds":["k12"],"roles":{"sales":"Sales framing: districts like Dayton and Wichita are keeping AI pilots small because they cannot show value beyond perception. That supports a bounded offer to design evaluation plans before scaling, not to promise results. Do not claim the Jordan chatbot or Wichita's math tutor work; NPR reports that nobody can yet measure whether they help or harm students.","engineering":"Engineering framing: moving past perception surveys needs interaction logs, linked outcome measures and comparison groups, all under student privacy rules. Pre-sales should ask what data each pilot tool exports, whether logs can be tied to existing assessments, and how sensitive student disclosures are handled. Proof of value is an evaluation design with defined measures and data flows agreed before launch.","delivery":"Delivery framing: build an evaluation plan into each pilot, with a logic model, baseline, comparison where possible, data-sharing terms and a decision rule for scaling or stopping. Ownership sits with the district research or accountability office. Acceptance means measures are defined before launch and reported at term end. Risks are pilots that continue on perception alone and unmanaged sensitive disclosures in social-emotional chatbots."},"roleDetails":{"sales":{"customerProblem":"Districts running AI pilots cannot show whether the tools help students, so they cannot justify scaling or stopping them.","stakeholders":["District research and accountability office","Chief academic officer","AI specialist","Student services and counseling","Pilot vendors"],"discoveryQuestions":["What outcome would justify scaling this pilot?","What data does the tool export?","Is there a baseline or comparison group?","How are sensitive student disclosures escalated?"],"valueHypothesis":"Evaluation plans set before launch could give districts defensible scale or stop decisions; unmeasured here.","potentialEngagement":"An evaluation-design sprint for one or two active AI pilots: logic model, measures, data flows and decision rules.","unsupportedClaims":["No evidence the Dayton chatbot helps students.","No results for Wichita's AI math tutor.","No sample sizes or methods."]},"engineering":{"fit":"Fits districts with active student-facing AI pilots and an assessment data system.","architectureIntegration":"Export de-identified interaction logs and join them with assessment or behavior data in the district's data warehouse under agreements.","prerequisites":["Vendor log export","Data-sharing agreement","Baseline measures","Research review process"],"constraints":["Student privacy law","Small pilot samples","Sensitive social-emotional data"],"security":"Treat chatbot conversations with students as sensitive records with retention limits and escalation paths.","proofOfValue":["Logs exportable and joinable","Measures defined before launch","Escalation path tested"]},"delivery":{"work":"Embed evaluation design into district AI pilots before they scale.","dependencies":["Research office capacity","Vendor cooperation","Teacher time for data collection"],"ownership":"District research and accountability office with the pilot's program lead.","skillsAdoption":"Train pilot leads to write logic models and choose measures beyond satisfaction surveys.","governanceCheckpoints":["Pre-launch evaluation plan approval","Mid-term data check","End-of-term scale or stop decision"],"acceptanceCriteria":["Proposed: every pilot has written outcome measures before launch","Proposed: at least one non-perception measure per pilot","Proposed: decision memo at term end"],"risks":["Perception-only evidence","Small samples","Sensitive disclosures without escalation"]}},"retrievedAt":"2026-10-22T22:11:02-05:00","enrichedAt":"2026-10-22T22:14:26-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Pilot in a school serving students with behavioral challenges; staff perception is the main measure.","procurementImplications":"Interpretation: evaluation data access should be a contract requirement for pilot tools.","operatingModelImplications":"Small, cautious pilots pending measurement; calls for state and federal guardrails on experimentation.","sourceVerification":{"openedUrl":"https://text.npr.org/nx-s1-5991881","referenceExcerpt":"Every test, every experiment that I try to measure in some sort of concrete way, I can only get as far as perception.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}