{"resourceId":"umd-edworkingpaper-goblins-deer-valley-math-usage-20261007","versions":[{"version":"external-1141f135455bdf42a20cc23af1c6d98ac652b9bf54d33fa0d42d35249af9c563","resource":{"id":"umd-edworkingpaper-goblins-deer-valley-math-usage-20261007","title":"University of Maryland working paper: Deer Valley (Arizona) middle schoolers who used the teacher-mediated Goblins AI math platform regularly gained 0.11 SD on MAP math, but the design is observational","organization":"Center for Educational Data Science and Innovation, University of Maryland, College Park (Anika Alam, Nicolas Dominguez-Mel, Jing Liu); EdWorkingPapers, Annenberg Institute at Brown University; funded by Accelerate","sector":"K-12 district math instruction / AI tutoring platform evaluation","geography":"Deer Valley Unified School District, Arizona, United States","publishedAt":"2026-10-07","publicationDate":"2026-10-07","eventDate":null,"sourceName":"EdWorkingPapers (Annenberg Institute at Brown University)","sourceLabel":"Academic working paper (EdWorkingPaper No. 26-1611)","sourceUrl":"https://edworkingpapers.com/sites/default/files/ai26-1611.pdf","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["operating-model","accessibility-workforce","governance-procurement"],"finding":"In Deer Valley Unified (over 32,000 students, 42 campuses), Goblins, a teacher-mediated AI math platform where students type, draw or speak and get personalised feedback, was offered to all middle-grade math teachers in 2025-26; 72.6% activated licences. Of 6,065 grade 6-8 students, 50.9% averaged at least two problems a week. Using a preregistered, machine-learning weighted propensity score design, students at that threshold scored 0.109 SD higher on within-district standardised spring MAP math (0.237 SD on NWEA's Conditional Growth Index). Students solving 10-19 and 20+ problems a week showed 0.18 SD and 0.31 SD. Students with IEPs, Hispanic students and English learners showed additional growth of 0.122, 0.084 and 0.191 SD (exploratory).","sledRelevance":"Answers a gap from recent editions: a new U.S. K-12 student-achievement study of an AI tool in normal district use. It complements the archived Tennessee Khanmigo RCT, where low engagement limited gains, by showing larger associations where about half of students reached a modest usage threshold, but with weaker causal design.","evidence":"Quasi-experimental, preregistered (OSF) weighted propensity score analysis of district administrative and platform data for one school year; baseline equivalence meets the WWC threshold according to the authors. The authors stress it is observational and vulnerable to unobserved selection; usage measures all platform problems, not AI-tutor use specifically.","architectureImplications":"Interpretation: usage-linked evaluation needs a reliable join between platform accounts and the district SIS; 13% of Goblins student users could not be matched because students self-entered names. Rostered single sign-on and usage export are prerequisites for credible local evaluation.","governanceImplications":"Evaluation committees can treat this as promising, not proven: it supports defining a minimum-use threshold and monitoring dosage, but not causal claims of effect.","securityPrivacyImplications":"Linking platform usage to student records, IEP status and English-learner status requires a data-sharing agreement and careful handling of sensitive subgroup data.","caveats":"Single district; observational with possible self-selection; teachers could turn off the AI tutor (an assessment mode added mid-year), so estimates are not AI-tutor effects; 13% of users unmatched; comparison students may have used similar edtech or Goblins-made materials; subgroup results exploratory. Vendor partnership acknowledged; study funded by Accelerate.","streamIds":["k12"],"roles":{"sales":"Lead with dosage, not access. A preregistered University of Maryland study in an Arizona district associated regular Goblins use with 0.11 SD higher MAP math scores, rising to 0.31 SD for the heaviest users, and about half of students reached the minimum threshold. That supports a bounded offer to help a district design an AI math pilot around usage targets and a local evaluation. It does not prove the tool causes gains, and it does not isolate the AI tutor.","engineering":"Treat evaluation readiness as part of the fit check. The study needed platform usage joined to SIS records, and 13% of users could not be matched because students typed their own names. Any AI math platform should come with rostered SSO, per-student usage export, and settings that record when the AI tutor is on or off, since teachers could disable it. Plan a comparison group in advance rather than relying on post-hoc matching, which the authors say remains open to selection bias.","delivery":"Deliver a teacher-mediated AI math rollout with explicit usage goals, such as at least two problems a week, and monthly dosage reports to teachers and principals. Teaching and learning owns instructional use; the data office owns the evaluation design, preferably randomised or staggered. Track subgroups such as students with IEPs and English learners, where the study saw larger associations, but treat those as hypotheses. Agree acceptance criteria before launch and keep records of which features were enabled."},"roleDetails":{"sales":{"customerProblem":"Districts are buying AI math tools with little evidence that students use them enough, or that use improves achievement.","stakeholders":["chief academic officer","math curriculum director","assessment and research office","special education director","English learner services director"],"discoveryQuestions":["What share of students actually use your current math platform each week?","Can you link platform usage to MAP or state test scores?","Do teachers control when AI help is available?","Which student groups are you most trying to reach?"],"valueHypothesis":"A usage-targeted, teacher-mediated pilot with planned evaluation could show whether regular use is associated with math growth locally before scale-up.","potentialEngagement":"One-semester AI math pilot design and evaluation plan: usage thresholds, SIS linkage, comparison group and MAP growth analysis.","unsupportedClaims":["That Goblins causes a 0.11 SD gain","That the AI tutor itself drives results","That subgroup gains will replicate elsewhere"]},"engineering":{"fit":"Relevant to districts considering student-facing AI math practice where teachers assign content and can control AI assistance.","architectureIntegration":"Rostered SSO (e.g., OneRoster/Clever-style) so platform accounts match SIS IDs, with scheduled usage export joined to assessment data in the district warehouse.","prerequisites":["SIS rostering to avoid self-entered names","per-student usage and feature-state export","MAP or comparable interim assessment data"],"constraints":["single Arizona district","observational design","AI tutor can be disabled, so usage is not AI exposure"],"security":"Usage linked to IEP and EL status is sensitive; restrict access to the analysis dataset, de-identify for reporting and cover it in the data-sharing agreement.","proofOfValue":["Confirm at least 95% of users match SIS records","Compare growth for students above and below a usage threshold with a planned comparison group","Report results separately for periods with AI tutor on and off"]},"delivery":{"work":"Roll out a teacher-mediated AI math platform with usage targets, monitoring and a pre-planned evaluation.","dependencies":["teacher licence activation and class setup","rostering integration","research office capacity"],"ownership":"Math curriculum leads own instructional use; the research and assessment office owns evaluation; IT owns rostering and data feeds.","skillsAdoption":"Teachers need time to set up classes, assign content and decide when to enable AI help; principals need dosage dashboards.","governanceCheckpoints":["pre-registered evaluation plan","mid-year dosage review","end-of-year results review before renewal"],"acceptanceCriteria":["Proposed: half of enrolled students meet the weekly usage threshold","Proposed: measurable MAP growth difference against a planned comparison group","Proposed: subgroup results reported with uncertainty"],"risks":["selection bias mistaken for impact","low teacher activation","feature changes mid-year confounding results"]}},"retrievedAt":"2026-10-07T22:06:37-05:00","enrichedAt":"2026-10-07T22:06:37-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Multimodal and seven-language support may lower barriers for students with IEPs and English learners, though the subgroup findings are exploratory; teachers carry setup and assignment work.","procurementImplications":"Contracts should require rostered accounts, usage export, feature-state logs and data access for independent evaluation; usage thresholds can be renewal criteria.","operatingModelImplications":"AI math tools perform as part of teacher-directed instruction with monitored dosage, not as unsupervised add-ons.","sourceVerification":{"openedUrl":"https://edworkingpapers.com/sites/default/files/ai26-1611.pdf","referenceExcerpt":"our estimates are still vulnerable to bias due to unobservables.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}