From the K–12 edition of October 22, 2026
NPR: Dayton and Wichita AI experiments can only measure perception, and CRPE warns innovation is outrunning evidence-building
NPR (All Things Considered) · K-12 AI pilot evaluation and evidence-building · Dayton, Ohio and Wichita, Kansas, United States
- Publisher
- NPR
- Original publication
- Monday, October 19, 2026, 4:59 p.m. EDT (NPR)
- Source retrieved
- 2026-10-23
What happened
NPR reports on a chatbot named Jordan used last school year at the Alternative School at Jackson Center in Dayton, Ohio, which asks students for advice to draw out their feelings and grades their responses. The consultant who helped design and evaluate it says every attempt at concrete measurement only reached perception, and the district relies on asking students and staff. He cautions against scaling it as a full solution. Wichita Public Schools' AI specialist says the district is introducing AI tools slowly, including a high school AI math tutor, and that experiments will stay small until educators can measure their value. Robin Lake of the Center on Reinventing Public Education says innovation is proceeding without much attention to evidence-building and that states and the federal government should set guardrails rather than stop experimentation.
Why it matters
Direct evidence that districts lack outcome measures for AI pilots, including a student-facing social-emotional chatbot in a school for students with behavioral challenges, which limits scaling decisions.
Evidence and measured results
Broadcast report with interviews; no evaluation data, sample sizes or methods published.
Limitations and uncertainty
Published October 19 and not previously archived; anecdotal interviews; no quantitative results; the chatbot vendor and data handling are not described.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-22; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Districts running AI pilots cannot show whether the tools help students, so they cannot justify scaling or stopping them.
- Stakeholders
- District research and accountability office
- Chief academic officer
- AI specialist
- Student services and counseling
- Pilot vendors
- Discovery questions
- What outcome would justify scaling this pilot?
- What data does the tool export?
- Is there a baseline or comparison group?
- How are sensitive student disclosures escalated?
- Value hypothesis
- Evaluation plans set before launch could give districts defensible scale or stop decisions; unmeasured here.
- Potential engagement
- An evaluation-design sprint for one or two active AI pilots: logic model, measures, data flows and decision rules.
- Unsupported claims
- No evidence the Dayton chatbot helps students.
- No results for Wichita's AI math tutor.
- No sample sizes or methods.
Pre-sales engineering
Role takeaway
- Fit
- Fits districts with active student-facing AI pilots and an assessment data system.
- Architecture and integration
- Export de-identified interaction logs and join them with assessment or behavior data in the district's data warehouse under agreements.
- Prerequisites
- Vendor log export
- Data-sharing agreement
- Baseline measures
- Research review process
- Constraints
- Student privacy law
- Small pilot samples
- Sensitive social-emotional data
- Security
- Treat chatbot conversations with students as sensitive records with retention limits and escalation paths.
- Proof of value
- Logs exportable and joinable
- Measures defined before launch
- Escalation path tested
Delivery
Role takeaway
- Work
- Embed evaluation design into district AI pilots before they scale.
- Dependencies
- Research office capacity
- Vendor cooperation
- Teacher time for data collection
- Ownership
- District research and accountability office with the pilot's program lead.
- Skills and adoption
- Train pilot leads to write logic models and choose measures beyond satisfaction surveys.
- Governance checkpoints
- Pre-launch evaluation plan approval
- Mid-term data check
- End-of-term scale or stop decision
- Acceptance criteria
- Proposed: every pilot has written outcome measures before launch
- Proposed: at least one non-perception measure per pilot
- Proposed: decision memo at term end
- Risks
- Perception-only evidence
- Small samples
- Sensitive disclosures without escalation
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Not addressed. Interpretation: measuring beyond perception requires logging chatbot interactions and linking them to existing outcome data under privacy controls.
Governance
Who approves, reviews and stays accountable for outcomes?
Districts are keeping AI pilots small pending evidence; CRPE calls for state and federal guardrails on experimentation.
Security and privacy
What data, permissions and controls need testing?
Not discussed. Interpretation: a chatbot eliciting students' feelings in an alternative school raises sensitive-data and counseling-escalation questions the report does not cover.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Pilot in a school serving students with behavioral challenges; staff perception is the main measure.
Procurement
What should contracts, pricing and exit terms secure?
Evaluation data access should be a contract requirement for pilot tools.
Operating model
Which teams own the service once it runs?
Small, cautious pilots pending measurement; calls for state and federal guardrails on experimentation.
Publication history
- 2026-10-22K–12 · Issue 464 resources
Stable resource ID: npr-schools-struggle-measure-ai-experiments-dayton-wichita-20261019