{"resourceId":"contractor-reyes-genai-learning-rct-iza-18792","versions":[{"version":"external-f03c030abc5d6a6f011d775b95de47d80cb4e8a571539b4b0f7bb39a77144ad9","resource":{"id":"contractor-reyes-genai-learning-rct-iza-18792","title":"Lab RCT finds 0.27 SD knowledge gains from GenAI, with augmentation lasting and automation fading","organization":"Middlebury College / IZA@LISER","sector":"Higher education learning research","geography":"United States (liberal-arts undergraduate lab experiment)","publishedAt":"July 2026","publicationDate":"2026-07-01","eventDate":null,"sourceName":"IZA Discussion Papers","sourceLabel":"IZA DP No. 18792 (Contractor & Reyes)","sourceUrl":"https://docs.iza.org/dp18792.pdf","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","operating-model","accessibility-workforce"],"finding":"In a proctored randomized experiment with 211 undergraduates learning an unfamiliar topic, generative AI access raised immediate knowledge-test scores by 0.27 SD, with gains persisting about one week later on unaided tests. Essay quality changed little while AI was available but later unaided style/relevance improved more for augmentation users (AI as explainer) than automation users (AI as text generator), whose short-run quality gains faded. Authors caution results are per unit of time in a lab and do not imply campus-wide learning rises if students reallocate saved time.","sledRelevance":"Addresses the stream’s durable-learning evidence gap with causal estimates and a usable augmentation-versus-automation distinction for teaching policy. Transferability to large public universities is limited by selective sample and lab setting; still actionable for assessment design and AI literacy.","evidence":"IRB-approved RCT; AI-allowed vs AI-forbidden in Session One; Session Two unaided ~1 week later; usage verified via proctoring, ChatGPT logs, and self-reports; human and AI essay grading plus detector features. Mechanisms include time reallocation toward reading/search and higher reported enjoyment.","architectureImplications":"Off-the-shelf GenAI without specialized tutor scaffolding produced average knowledge gains; logging how students use tools (explain vs draft) is analytically valuable. Interpretation: campus deployments should capture usage mode telemetry ethically, not only seat licenses.","governanceImplications":"Authors link incentives (grades, signaling) to students’ automation vs augmentation choices. Interpretation: policy and assessment design that reward process and unaided demonstration may matter more than blanket bans or unrestricted access.","securityPrivacyImplications":"Conversation logs were analyzed for usage typology; campus analogs raise FERPA/vendor-training risks if students paste sensitive content into consumer tools. Interpretation: prefer institution-approved tools with logging controls when studying or requiring AI use.","caveats":"Discussion paper, lab setting, N=211, selective liberal-arts sample, one-week retention only, single technical topic, time-on-task largely fixed by design. External validity to semester courses, high-stakes exams, or equity-priority populations is uncertain. Automation/augmentation labels rely on LLM coding of logs.","streamIds":["student-success"],"roles":{"sales":"Sales framing: GenAI can show real short-run learning gains, but only with careful outcome definitions and use-mode honesty. This Middlebury/IZA RCT reports 0.27 SD persistent knowledge-test gains yet warns automation-style drafting gains fade once AI is removed. Do not pitch unlimited ChatGPT access as automatic human-capital growth; sell bounded pilots that measure unaided retention and discourage pure automation. Position AI literacy and assessment redesign as the engagement, not seat-count expansion alone.","engineering":"Engineering framing: fit is strongest for controlled learning tasks with approved GenAI endpoints and usage analytics that distinguish tutoring-style prompts from draft-replacement. Architecture should isolate consumer-model risk, enable optional logging with consent, and support unaided assessment checkpoints. Prerequisites include IRB-aware evaluation design and clear course policies. Constraints: lab evidence may not survive open-ended homework time-saving. Proof of value: unaided delayed tests plus usage-mode coding, not completion speed alone.","delivery":"Delivery framing: delivery should combine AI-allowed practice with unaided demonstration and explicit coaching toward augmentation uses. Dependencies include faculty development, approved tool access, and IR/measurement support. Ownership sits with academic affairs/teaching centers jointly with IT. Acceptance criteria: pre-specified knowledge retention measures and documented student guidance on productive use; risks include grade incentives that push automation and privacy exposure from consumer chat logs."},"roleDetails":{"sales":{"customerProblem":"Institutions lack causal evidence on whether off-the-shelf GenAI builds lasting understanding or merely substitutes for student work.","stakeholders":["Provost / academic affairs","Teaching and learning center","Faculty senate","Student success analytics"],"discoveryQuestions":["Which courses will measure unaided delayed learning?","How will automation vs augmentation be taught and observed?","What approved GenAI tools avoid consumer data leakage?","How do grading incentives currently reward speed over learning?"],"valueHypothesis":"Bounded pilots that pair GenAI practice with unaided assessment may improve knowledge retention for some learners; unrestricted automation-friendly workflows are unlikely to show durable essay-skill gains.","potentialEngagement":"Interpretation: design a course-level learning pilot with delayed unaided measures and AI-literacy modules—not a campus-wide license sale justified by this lab effect size.","unsupportedClaims":["Campus-wide GPA or retention improvement","Guaranteed learning if students use AI to save time","Equality of benefits across proficiency groups","Semester-long transfer from one-week lab persistence"]},"engineering":{"fit":"Good fit for sandboxed GenAI in coursework with logging and unaided assessment hooks; weak fit as unmonitored consumer ChatGPT mandates.","architectureIntegration":"Route students through approved GenAI with optional conversation analytics; separate practice environments from proctored unaided exams.","prerequisites":["Approved GenAI tenancy","Consent model for logs","Assessment platforms for delayed tests","Faculty policy templates"],"constraints":["Lab external validity","Small selective sample","One-week horizon","LLM-based use-mode labeling error"],"security":"Block pasting of non-public data into consumer models; prefer campus tenancy with retention limits on prompts.","proofOfValue":["Unaided delayed knowledge tests","Essay quality after AI removal","Usage-mode distribution (augmentation share)","Time-use measures where feasible"]},"delivery":{"work":"Pilot selected courses with explicit augmentation coaching, unaided checkpoints, and evaluation reporting.","dependencies":["Faculty volunteers","Approved tools","IRB/ethics as required","Measurement instruments"],"ownership":"Interpretation: academic leadership owns learning outcomes; IT owns tool tenancy and logging controls.","skillsAdoption":"Teach students and faculty to prefer explanation/tutoring prompts over draft replacement; train TAs on policy.","governanceCheckpoints":["Tool approval","Assessment integrity review","Interim learning readout","Privacy review of logs"],"acceptanceCriteria":["Pre-registered primary learning endpoint","Published use-mode guidance","No unsupported productivity-to-learning claims in communications"],"risks":["Students automate under grade pressure","Privacy incidents with consumer AI","Over-generalizing 0.27 SD lab effect","Equity gaps if AI-proficient students benefit more"]}},"retrievedAt":"2026-09-15T14:15:11Z","enrichedAt":"2026-09-15T14:17:30Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Gains concentrated mid-distribution; proficiency predicts framing AI as augmentation. Interpretation: without structured guidance, less AI-experienced students may automate and lose durable benefits—an equity and workforce-readiness issue for teaching staff who must coach use modes.","procurementImplications":"Prefer institution-managed GenAI with evaluation hooks over unmanaged consumer access when learning claims are in scope; contract for data protections and research export.","operatingModelImplications":"Operating model should couple AI access with incentive-aware assessment design so students are rewarded for augmentation and unaided demonstration, not only polished AI-assisted submissions.","sourceVerification":{"openedUrl":"https://docs.iza.org/dp18792.pdf","referenceExcerpt":"AI access raises immediate test scores by 0.27 standard deviations. These gains persist one week later.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}