{"resourceId":"contractor-reyes-genai-learning-rct-202607","versions":[{"version":"external-4678e1a3706335f89e514185b2743cd9dcbb5c757fc8cc8ff86cb2f3d7ec0d58","resource":{"id":"contractor-reyes-genai-learning-rct-202607","title":"Middlebury RCT: GenAI raises test scores 0.27 SD; unaided essay gains favor augmentation users","organization":"Middlebury College (Contractor & Reyes; IZA affiliation)","sector":"Higher education economics / experimental education research","geography":"United States (undergraduate lab experiment)","publishedAt":"2026-07-09","publicationDate":"2026-07-09","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic randomized experiment preprint","sourceUrl":"https://arxiv.org/pdf/2607.08849","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","developers-agents","operating-model","accessibility-workforce"],"finding":"In a proctored two-session RCT with 211 undergraduates, off-the-shelf generative AI access raised immediate knowledge-test scores by 0.27 SD with gains persisting one week later on unaided tests. Essay quality changed little while AI was available, but unaided later essays improved in style/relevance especially for augmentation users (AI to explain concepts); automation users’ short-run quality gains faded without AI. Authors caution results are per unit time and may not imply higher total learning if students reallocate saved time away from study.","sledRelevance":"U.S. HE causal evidence that GenAI can raise short-run conceptual test performance while writing outcomes depend on augmentation vs automation—critical for student-success and academic-integrity policy.","evidence":"IRB-approved RCT; AI-allowed vs forbidden learning/essay session; unaided follow-up about one week later; human/AI essay grading plus detector and linguistic features; arXiv:2607.08849v1 dated 9 Jul 2026.","architectureImplications":"Build assignment workflows that log use mode, encourage explanation/search over autowrite, and schedule unaided assessments; evaluation stacks need proctoring or trusted logging.","governanceImplications":"Policy should define permitted GenAI uses by learning objective and require unaided demonstration of mastery; treat vendor efficacy claims as hypotheses.","securityPrivacyImplications":"Conversation logs used for research must stay under institutional control; students need notice about monitoring in evaluation settings.","caveats":"Lab time-on-task largely fixed; selective Middlebury sample; one-week horizon; off-the-shelf tools change quickly; not a retention/advising intervention.","streamIds":["student-success"],"roles":{"sales":"For academic-affairs buyers debating open generative AI in coursework, Contractor and Reyes’s July 2026 randomized experiment with 211 undergraduates finds AI access raises immediate knowledge-test scores by 0.27 SD with persistence one week later, while unaided essay gains favor augmentation users who use AI to explain concepts rather than automate drafting. Sell guided-use designs, literacy that steers students toward augmentation, and assessment plans that measure unaided retention—not claims that any ChatGPT seat raises durable learning. Discovery must separate test-score gains from higher-order writing automation risk.","engineering":"Engineering should instrument AI sessions to distinguish augmentation (explain/search) from automation (generate prose), and pair AI-allowed assignments with later unaided assessments. Architecture implications include proctorable or logged environments for evaluation studies and time-use analytics (reading/search vs drafting). Constraints: Middlebury lab setting, one-week horizon, off-the-shelf tools, and authors’ caution that time-saving outside labs may cut total learning time. Security and privacy: capture conversation logs under IRB/institutional policy without feeding vendor training.","delivery":"Delivery is an assessment-redesign pilot: allow GenAI on a learning task, then require unaided tests/essays after a delay; teach augmentation strategies; and report score and writing outcomes by use mode. Faculty own assignment design; teaching-and-learning owns literacy modules; IR/research partners analyze logs. Acceptance criteria include documented unaided retention measures and augmentation-share metrics. Risks include students using AI to save time outside labs and mistaking short-run test lifts for curriculum-wide readiness."},"roleDetails":{"sales":{"customerProblem":"Leaders need causal evidence on whether open GenAI helps learning or mainly automates writing.","stakeholders":["provost","faculty senate","teaching & learning","academic integrity","student success"],"discoveryQuestions":["Where will unaided assessments verify learning after AI-allowed practice?","How will we teach and measure augmentation versus automation use?","Which courses prioritize conceptual tests versus higher-order writing?"],"valueHypothesis":"Guided GenAI practice plus unaided retention checks can capture test-score gains while limiting automation-only writing dependencies.","potentialEngagement":"Single-course assessment-redesign sprint with logged GenAI practice and delayed unaided measures.","unsupportedClaims":["Campus-wide learning gains from unrestricted ChatGPT seats","That essay quality improves while students have AI drafting access","Long-term retention or labor-market skill guarantees"]},"engineering":{"fit":"Fit for courses willing to combine AI-allowed practice with later locked-down assessment and optional conversation logging.","architectureIntegration":"Provide approved AI tools in learning phase; disable AI in assessment phase; classify prompts as explain vs generate where feasible.","prerequisites":["Clear academic-integrity rules","Assessment platform supporting AI on/off modes","Optional logging consent pathway"],"constraints":["Lab evidence with fixed time-on-task","One-week follow-up only","Tool drift across model versions"],"security":"Institutional control of logs; minimize PII; no vendor training on student prompts from evaluation studies.","proofOfValue":["Unaided test deltas vs control","Essay rubric changes by augmentation share","Time-use shift from drafting to reading/search"]},"delivery":{"work":"Redesign one assignment cycle for AI-allowed learning then unaided demonstration; deliver augmentation literacy; analyze outcomes by use mode.","dependencies":["Faculty redesign time","Integrity and privacy review","Analytics support for logs"],"ownership":"Course faculty with teaching-and-learning co-facilitation; integrity office consulted.","skillsAdoption":"Student workshop on explanation-first prompting; faculty norming on rubrics for unaided work.","governanceCheckpoints":["Pre-launch integrity/privacy approval","Midpoint check on automation-heavy use","Post-cycle review before expanding AI-allowed work"],"acceptanceCriteria":["Unaided assessment administered and scored","Use-mode taxonomy applied to at least a sample of logs","Public-facing claim language matches evidence limits"],"risks":["Students cutting total study time outside labs","Over-reliance on AI detectors","Equity gaps if AI proficiency predicts augmentation framing"]}},"retrievedAt":"2026-09-25T03:22:56Z","enrichedAt":"2026-09-25T03:28:16Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Augmentation skill gaps may track prior AI proficiency; pair access with explicit coaching so automation-default students are not left with vanishing unaided gains.","procurementImplications":"Contracts should allow institutional evaluation logging and AI-off assessment modes; do not purchase seats solely on vendor learning claims.","operatingModelImplications":"Shift from access policy alone to assignment design that sequences AI practice and unaided mastery checks.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2607.08849","referenceExcerpt":"AI access raises immediate test scores by 0.27 standard deviations. These gains persist one week later.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}