{"resourceId":"accelerate-seta-ed-enlightenai-ap-us-history-null-effect-no-revision-20261103","versions":[{"version":"external-dd8de93a7a046d23bab00aa966d0476b83f9f69223d52d9c680f27878c67849c","resource":{"id":"accelerate-seta-ed-enlightenai-ap-us-history-null-effect-no-revision-20261103","title":"Accelerate / SETA-ED: nine-school quasi-experiment finds no detectable effect of EnlightenAI on AP U.S. History writing or exam scores, as students almost never revised","organization":"SETA-ED (published by Accelerate)","sector":"K-12","geography":"United States (New York, New Jersey, Massachusetts charter schools)","publishedAt":"2026-11-03","publicationDate":"2026-11-03","eventDate":null,"sourceName":"Accelerate","sourceLabel":"Independent evaluation summary (SETA-ED study, Call for Effective Technology)","sourceUrl":"https://accelerate.us/research/enlightenai-ai-writing-feedback-tool-evaluation/","evidenceClass":"independent-research","outcomeClass":"mixed","topics":["knowledge-work","operating-model"],"finding":"A 2025-26 quasi-experimental study across nine Uncommon Schools high schools (five using EnlightenAI, four comparison) found no detectable effect of access to the AI writing-feedback tool on spring writing scores or AP U.S. History exam scores. Nearly all students (328 of 329) logged in and submitted 4,846 assignments, but no student completed more than one of the 6 to 8 intended revision cycles and two-thirds completed none, so the study tested access rather than the revision-based model.","sledRelevance":"Relevant to K-12 districts and charter networks buying AI grading or feedback tools: it shows high login and submission rates can coexist with almost no use of the feature meant to drive learning. Limited transferability: one network, one AP subject, grades 9 and 11.","evidence":"Matched samples of 386 students for the writing test (193 per group) and 278 for the AP exam (139 per group) from 599 rostered students; school-level assignment (5 versus 4 schools); matched on prior scores and demographics. The source reports it could detect only large effects (about 0.30 SD or more). No cost analysis. No subgroup differences detected by gender, race or ethnicity, prior achievement, or lunch status.","architectureImplications":"The source describes an AI platform with teacher review: teachers grade a sample, the AI proposes scores and comments, and teachers review before release. It does not describe integrations, hosting, or data architecture. Interpretation: districts should confirm which usage events are logged to measure revision dosage.","governanceImplications":"Interpretation: agree on intended use and success measures before deployment, monitor actual use, and act when it diverges. The source says teachers reviewed AI-proposed grades before release but gives no detail on review policy.","securityPrivacyImplications":"The source does not discuss student data handling, privacy, or security for the tool. Districts would need separate vendor review of data practices before any adoption.","caveats":"Quasi-experimental with only nine schools, so the comparison has few effective data points; the source says it cannot distinguish a tool that does not improve writing from one not used as designed. Summary page of an independent evaluation published by the funder; the full report was not opened. Single charter network and course.","streamIds":["k12"],"roles":{"sales":"Lead with the adoption-versus-use gap: nearly every student logged in, yet almost none revised work with the feedback, and scores matched comparison schools. For K-12 buyers of AI writing or feedback tools, this reframes the conversation from licenses and logins to whether the intended learning behavior happens. Interpretation: engagements should help districts define intended use up front, track it, and decide before renewal. This single nine-school quasi-experiment, in one charter network and one AP course, cannot support claims of effectiveness, ineffectiveness, or savings; it shows only that access alone did not move scores here.","engineering":"Frame the tool as a workflow, not a feature: the evaluated platform collected assignments, proposed grades with teacher review, and returned feedback, but the revise-and-resubmit loop the design depended on rarely ran. Interpretation: pre-sales engineering should ask which usage events (draft submitted, feedback viewed, revision resubmitted) are logged and exportable so a district can see dosage, not just logins. Integration with the LMS, rostering, and teacher grading review matter more than model quality here. Limits: the source reports outcomes, not architecture; student-data and privacy details were not described.","delivery":"Open with dosage monitoring: the study planned 6 to 8 revision cycles per student and observed under one on average, so delivery should plan classroom time for revision rather than assume it happens. Interpretation: a delivery team could pilot with teacher routines, a check that students act on feedback, and early course-correction when intended use is missing. The authors' account supports caution about attributing the null result to either the tool or its use; only the design limits are documented. Applicability beyond AP U.S. History in a charter network is unknown, and no cost data were collected."},"roleDetails":{"sales":{"customerProblem":"Districts buy AI feedback tools expecting better writing, but adoption metrics like logins may not reflect the intended revision behavior.","stakeholders":["Curriculum and instruction leaders","English and social studies department heads","Principals","Instructional technology director","Procurement"],"discoveryQuestions":["What does intended use look like, and how would you know it is happening?","Do you track whether students revise after feedback?","What outcome would justify renewal?"],"valueHypothesis":"A short usage-and-outcome review before renewal could help a district decide whether a feedback tool is being used as designed.","potentialEngagement":"A bounded adoption-readiness review defining intended-use metrics and a small classroom pilot design for one writing tool.","unsupportedClaims":["That the tool does or does not improve writing","Guaranteed savings or teacher time reductions","Results for other subjects, grades, or school types"]},"engineering":{"fit":"Fit depends on classroom workflow integration; this evidence concerns usage behavior, not technical capability.","architectureIntegration":"Check rostering, LMS assignment flow, teacher review of AI-proposed grades, and return of feedback to students for resubmission.","prerequisites":["Student rostering and identity","Writing assignments in a supported format","Teacher calibration samples","Usage event logging and export"],"constraints":["Source gives no hosting or jurisdictional detail","Student data rules vary by state and district","Findings limited to one AP course"],"security":"Not covered by the source; require separate privacy and security review of student data flows and vendor data use.","proofOfValue":["Measure revision cycles per student against the planned dosage","Compare unaided writing samples for users versus non-users","Log feedback views and resubmissions weekly"]},"delivery":{"work":"Plan classroom routines that build revision time into lessons and check that students act on feedback.","dependencies":["Teacher time for calibration and review","Scheduled revision periods","Usage reporting from the vendor"],"ownership":"Principal and department chair, with the district instructional technology team supporting reporting.","skillsAdoption":"Teachers need coaching on structuring revision with AI feedback; students need modeling of how to respond to feedback.","governanceCheckpoints":["Define intended use before launch","Review dosage data at set intervals","Decide to adjust or stop if intended use is missing"],"acceptanceCriteria":["Proposed: a majority of students complete at least one revision cycle per assignment window","Proposed: weekly usage report reviewed by the department chair","Proposed: unaided writing sample collected pre and post"],"risks":["Students use the tool only to submit work","Small or school-level comparisons mask effects","Overreading a null result as proof of ineffectiveness"]}},"retrievedAt":"2026-11-09T22:04:06-06:00","enrichedAt":"2026-11-09T22:04:06-06:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"The source reports the treatment group was about 80 percent Black and 15 to 21 percent Hispanic or Latino, and 82 to 86 percent economically disadvantaged, with 2 to 3 percent English learners; no differences in effect were detected by subgroup, but the study was underpowered. Teachers reviewed AI-proposed grades, so workload and training needs remain unquantified.","procurementImplications":"Interpretation: contracts could tie renewal to measured intended use and require usage-event reporting. No cost analysis was conducted in the study, so price-performance cannot be assessed.","operatingModelImplications":"Interpretation: the operating model, meaning classroom time for revision and teacher routines, may determine whether feedback tools matter. The source says making revision part of class routine is where the next implementation must start.","peerReview":"unknown","sourceVerification":{"openedUrl":"https://accelerate.us/research/enlightenai-ai-writing-feedback-tool-evaluation/","referenceExcerpt":"No student completed more than one of the 6 to 8 intended revision cycles, and two-thirds completed none.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}