{"resourceId":"edcircuit-ai-learning-measurement-20260928","versions":[{"version":"external-c02a30acc1d919e8a0ae2b3a4f9f519cef4a0c695ab35157ec70adcac9363b1c","resource":{"id":"edcircuit-ai-learning-measurement-20260928","title":"edCircuit: Measure AI by durable learning and independence—not logins, prompts, or claimed time savings","organization":"edCircuit","sector":"K-12 education practice guidance / independent commentary","geography":"United States — K-12 districts (national practice audience)","publishedAt":"September 28, 2026","publicationDate":"2026-09-28","eventDate":null,"sourceName":"edCircuit","sourceLabel":"edCircuit analysis","sourceUrl":"https://edcircuit.com/ai-and-student-learning-how-schools-should-measure-it/","evidenceClass":"independent-reporting","outcomeClass":"cautionary","topics":["knowledge-work","operating-model","governance-procurement"],"finding":"edCircuit’s Sept. 28 analysis argues districts commonly report AI success via account activations, prompt counts, and estimated teacher time saved while failing to answer whether students learn more. Citing NCES 2025 (73% of public schools report some teacher AI use vs only 38% of leaders agreeing AI will improve outcomes) and REL Central’s note that research has not established which teacher AI uses improve learning, it urges pre-defined goals, net workload (time saved minus verification), unaided/independent student demonstrations after tutoring support is removed, equity subgroup review, and advance criteria for expand/modify/stop—including willingness to end ineffective pilots.","sledRelevance":"Operational measurement checklist that converts the assisted-vs-retained-learning brief into board-ready evaluation practice for AI pilots.","evidence":"Practice synthesis citing NCES/REL/NIST; not a new primary RCT.","architectureImplications":"Separate AI-on practice telemetry from AI-off assessments; capture verification burden and teacher overrides, not only usage dashboards.","governanceImplications":"Define go/no-go evidence before procurement renewal; publish limitations to boards and families.","securityPrivacyImplications":"Evaluation must include privacy incidents and inappropriate data entry near-misses alongside learning metrics.","caveats":"Vendor-neutral commentary; NCES/REL citations secondary; does not present new causal student-outcome estimates.","streamIds":["k12"],"roles":{"sales":"Sales framing: edCircuit’s 28 Sep piece warns that login counts and ‘hours saved’ slides do not prove AI improved learning. With NCES showing high teacher AI use but low leader confidence in better outcomes, buyers should demand unaided transfer measures, equity cuts, and stop criteria before renewal. Sell evaluation design, not dashboard vanity metrics. Interpretation of practice guidance; not a district case study.","engineering":"Engineering framing: Instrument pilots so AI-supported practice is distinguishable from independent assessment windows; log correction time and teacher overrides as first-class costs, not only engagement events; and export privacy-preserving subgroup participation and error patterns. Prefer tutoring modes that preserve productive struggle over answer-generating chat that inflates assisted polish. Build stop-the-line hooks when accuracy or accessibility failures spike. Interpretation for evaluation architecture in local pilots.","delivery":"Delivery framing: Write the measurement plan before the pilot begins—which skills, which students, what unaided demonstration, what net teacher workload, and what evidence ends or narrows the contract. Brief boards on uncertainty and limitations rather than transformational claims built from login counts. Celebrate ending a weak pilot as responsible stewardship. Interpretation for SLED delivery leads; local research capacity will vary."},"roleDetails":{"sales":{"customerProblem":"Leaders are asked to scale AI based on usage and time-savings slides without evidence of durable student learning.","stakeholders":["superintendent","board","evaluation/research","curriculum","principals","teachers"],"discoveryQuestions":["What unaided demonstration proves the skill without the AI tool?","How is verification time counted against claimed savings?","Which student groups are monitored for differential benefit or harm?","What evidence would stop or narrow the pilot?"],"valueHypothesis":"If districts require durable-learning and net-workload evidence before renewal, they avoid locking in tools that inflate assisted completion.","potentialEngagement":"Pilot evaluation design sprint producing pre-registered measures, baseline plan, and board reporting template.","unsupportedClaims":["That usage dashboards equal learning gains","Guaranteed outcome improvement from any AI tutor","That NCES percentages alone prove a specific product works"]},"engineering":{"fit":"Cross-cutting for any student- or teacher-facing AI pilot needing credible local evidence.","architectureIntegration":"Build AI-off assessment hooks; store override/error logs; support privacy-preserving subgroup analytics.","prerequisites":["Defined learning targets","Baseline data","Ability to disable AI for assessment","Privacy review of logging"],"constraints":["Limited local research capacity","Short pilot windows","Vendor dashboards biased to engagement"],"security":"Protect evaluation data; avoid shipping identifiable student traces into vendor marketing analytics.","proofOfValue":["Unaided post-test or classroom transfer task exists","Teacher time diary includes correction burden","Written expand/modify/stop criteria"]},"delivery":{"work":"Define goals, run limited classroom pilot, collect learning/workload/equity/harm signals, decide renew/narrow/stop.","dependencies":["board reporting calendar","teacher volunteers","assessment windows","privacy office"],"ownership":"Evaluation/research owns measures; curriculum owns instructional goals; IT owns telemetry privacy.","skillsAdoption":"Coach principals to ask ‘what can students do without the tool?’ in every AI status update.","governanceCheckpoints":["Pre-pilot goal lock","30-day access/privacy check","Semester learning review","Renewal decision"],"acceptanceCriteria":["Goals stated without product name","Independent performance measure collected","Public acknowledgment of limitations"],"risks":["Success redefined as enjoyment after weak learning results","Sunk-cost renewal despite null findings","Dashboard-only reporting to boards"]}},"retrievedAt":"2026-09-29T22:09:26-05:00","enrichedAt":"2026-09-29T22:18:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Equity review must include English learners, students with disabilities, and access divides; accessibility failures are evaluation evidence.","procurementImplications":"Tie renewal/payment milestones to pre-agreed learning and harm measures, not activation counts.","operatingModelImplications":"Shift AI governance from adoption theater to continuous monitoring with authority to decommission.","sourceVerification":{"openedUrl":"https://edcircuit.com/ai-and-student-learning-how-schools-should-measure-it/","referenceExcerpt":"If performance improves only when the tool is present, the system may be supporting completion rather than durable learning.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}