From the Student Success edition of September 24, 2026
Middlebury RCT: GenAI raises test scores 0.27 SD; unaided essay gains favor augmentation users
Middlebury College (Contractor & Reyes; IZA affiliation) · Higher education economics / experimental education research · United States (undergraduate lab experiment)
- Publisher
- arXiv
- Original publication
- 2026-07-09
- Source retrieved
- 2026-09-25
What happened
In a proctored two-session RCT with 211 undergraduates, off-the-shelf generative AI access raised immediate knowledge-test scores by 0.27 SD with gains persisting one week later on unaided tests. Essay quality changed little while AI was available, but unaided later essays improved in style/relevance especially for augmentation users (AI to explain concepts); automation users’ short-run quality gains faded without AI. Authors caution results are per unit time and may not imply higher total learning if students reallocate saved time away from study.
Why it matters
U.S. HE causal evidence that GenAI can raise short-run conceptual test performance while writing outcomes depend on augmentation vs automation—critical for student-success and academic-integrity policy.
Evidence and measured results
IRB-approved RCT; AI-allowed vs forbidden learning/essay session; unaided follow-up about one week later; human/AI essay grading plus detector and linguistic features; arXiv:2607.08849v1 dated 9 Jul 2026.
Limitations and uncertainty
Lab time-on-task largely fixed; selective Middlebury sample; one-week horizon; off-the-shelf tools change quickly; not a retention/advising intervention.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-25; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Leaders need causal evidence on whether open GenAI helps learning or mainly automates writing.
- Stakeholders
- provost
- faculty senate
- teaching & learning
- academic integrity
- student success
- Discovery questions
- Where will unaided assessments verify learning after AI-allowed practice?
- How will we teach and measure augmentation versus automation use?
- Which courses prioritize conceptual tests versus higher-order writing?
- Value hypothesis
- Guided GenAI practice plus unaided retention checks can capture test-score gains while limiting automation-only writing dependencies.
- Potential engagement
- Single-course assessment-redesign sprint with logged GenAI practice and delayed unaided measures.
- Unsupported claims
- Campus-wide learning gains from unrestricted ChatGPT seats
- That essay quality improves while students have AI drafting access
- Long-term retention or labor-market skill guarantees
Pre-sales engineering
Role takeaway
- Fit
- Fit for courses willing to combine AI-allowed practice with later locked-down assessment and optional conversation logging.
- Architecture and integration
- Provide approved AI tools in learning phase; disable AI in assessment phase; classify prompts as explain vs generate where feasible.
- Prerequisites
- Clear academic-integrity rules
- Assessment platform supporting AI on/off modes
- Optional logging consent pathway
- Constraints
- Lab evidence with fixed time-on-task
- One-week follow-up only
- Tool drift across model versions
- Security
- Institutional control of logs; minimize PII; no vendor training on student prompts from evaluation studies.
- Proof of value
- Unaided test deltas vs control
- Essay rubric changes by augmentation share
- Time-use shift from drafting to reading/search
Delivery
Role takeaway
- Work
- Redesign one assignment cycle for AI-allowed learning then unaided demonstration; deliver augmentation literacy; analyze outcomes by use mode.
- Dependencies
- Faculty redesign time
- Integrity and privacy review
- Analytics support for logs
- Ownership
- Course faculty with teaching-and-learning co-facilitation; integrity office consulted.
- Skills and adoption
- Student workshop on explanation-first prompting; faculty norming on rubrics for unaided work.
- Governance checkpoints
- Pre-launch integrity/privacy approval
- Midpoint check on automation-heavy use
- Post-cycle review before expanding AI-allowed work
- Acceptance criteria
- Unaided assessment administered and scored
- Use-mode taxonomy applied to at least a sample of logs
- Public-facing claim language matches evidence limits
- Risks
- Students cutting total study time outside labs
- Over-reliance on AI detectors
- Equity gaps if AI proficiency predicts augmentation framing
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Build assignment workflows that log use mode, encourage explanation/search over autowrite, and schedule unaided assessments; evaluation stacks need proctoring or trusted logging.
Governance
Who approves, reviews and stays accountable for outcomes?
Policy should define permitted GenAI uses by learning objective and require unaided demonstration of mastery; treat vendor efficacy claims as hypotheses.
Security and privacy
What data, permissions and controls need testing?
Conversation logs used for research must stay under institutional control; students need notice about monitoring in evaluation settings.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Augmentation skill gaps may track prior AI proficiency; pair access with explicit coaching so automation-default students are not left with vanishing unaided gains.
Procurement
What should contracts, pricing and exit terms secure?
Contracts should allow institutional evaluation logging and AI-off assessment modes; do not purchase seats solely on vendor learning claims.
Operating model
Which teams own the service once it runs?
Shift from access policy alone to assignment design that sequences AI practice and unaided mastery checks.
Publication history
- 2026-09-24Student Success · Issue 194 resources
Stable resource ID: contractor-reyes-genai-learning-rct-202607