From the Student Success edition of October 13, 2026
Harvard physics educator reports 62% higher learning gains with Collage AI tutor versus a prior semester, in U-M keynote summary; presenter is the vendor's chief learning officer
University of Michigan Center for Academic Innovation (host); Kelly Miller, Harvard SEAS; Collage AI (vendor) · Higher education student success / AI tutoring in flipped STEM courses · Ann Arbor, Michigan (event host) and Cambridge, Massachusetts (course), United States
- Publisher
- University of Michigan Center for Academic Innovation
- Original publication
- U-M Center for Academic Innovation blog, 13 October 2026
- Source retrieved
- 2026-10-14
What happened
In a U-M Center for Academic Innovation summary of her keynote, Harvard's Kelly Miller reported that in a flipped applied physics course where Collage AI generated pre-class formative assessments and served as a tutor, students had 62% higher learning gains than a previous semester and appreciated the AI feedback. She also recapped the earlier Harvard physics experiment in which learning gains were nearly double with a custom AI tutor. Miller is chief learning officer of Collage AI and is extending it to other courses, universities and K-12 classrooms.
Why it matters
Published 13 October, this is the newest public claim of large AI-tutor learning gains in higher education. It sits against recent controlled evidence showing harms or null effects (such as the archived Maryland trial), making the evidence standard for tutor claims a live procurement question.
Evidence and measured results
Blog summary of a talk. The 62% figure compares one course with a previous semester; sample size, assessment instrument, cohort comparability and statistical tests are not given. The earlier randomized crossover study is described only briefly.
Limitations and uncertainty
Vendor-affiliated, non-randomized cross-semester claim reported second-hand by a host blog; no methods or data published in the source; one course in one elite institution.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-14; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Institutions face strong vendor-linked claims of AI-tutor learning gains alongside controlled studies showing null or negative effects, and must decide what evidence to require.
- Stakeholders
- Provost and teaching centre
- STEM department chairs
- Procurement office
- Institutional research
- Faculty early adopters
- Discovery questions
- What study design produced the claimed gain, and with how many students?
- Is there independent or pre-registered evaluation?
- How much instructor time does configuration require?
- How does the tool perform for first-generation or underprepared students?
- Value hypothesis
- Interpretation: an instructor-configured, course-grounded tutor may support flipped-course preparation, but value must be shown in a controlled local pilot.
- Potential engagement
- A bounded controlled pilot in two to four sections of a flipped STEM course with a pre-registered outcome.
- Unsupported claims
- A 62% learning gain at other institutions
- Causal effect of the tool (cross-semester comparison)
- Independence of the evidence (vendor-affiliated presenter)
Pre-sales engineering
Role takeaway
- Fit
- Fit is for course-grounded, instructor-configured tutoring and formative assessment in flipped STEM courses.
- Architecture and integration
- Instructors upload content and learning objectives; platform generates formative assessments and tutors students; LMS integration not described.
- Prerequisites
- Prepared pre-class materials and objectives
- Instructor time for configuration
- Data agreement with the vendor
- Constraints
- Evidence limited to one course at one institution
- Privacy and hosting not described
- Conflict of interest in reported results
- Security
- Not addressed in source; confirm data retention, model-training exclusions and instructor visibility of student chats.
- Proof of value
- Randomized or matched-section comparison with a common post-test
- Audit tutor answers for accuracy in the course domain
- Measure engagement with existing course activities to detect displacement
Delivery
Role takeaway
- Work
- Run a controlled pilot with pre-registered outcomes and subgroup analysis before any wider adoption.
- Dependencies
- Willing instructors with flipped courses
- Institutional research for analysis
- Vendor configuration support
- IRB or evaluation approval
- Ownership
- Teaching centre with participating departments; institutional research owns evaluation.
- Skills and adoption
- Instructors need training on configuring lessons and interpreting student-use data.
- Governance checkpoints
- Pre-registration of outcomes
- Mid-term check on usage and displacement of other activities
- Independent analysis before scale decision
- Acceptance criteria
- Proposed: pilot reports effect sizes with confidence intervals against matched sections
- Proposed: no subgroup shows significantly lower outcomes
- Proposed: instructor configuration time recorded and reported
- Risks
- Novelty or instructor effects inflate results
- Engagement with other learning activities declines
- Vendor influence over evaluation
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Instructor-configured platform where faculty feed course content and learning objectives and control assessment type and Bloom's level; tutor guidance grounded in course materials.
Governance
Who approves, reviews and stays accountable for outcomes?
Conflict of interest: the presenter is the vendor's chief learning officer. Interpretation: institutions should require independent or pre-registered evaluation before relying on such figures.
Security and privacy
What data, permissions and controls need testing?
Not addressed.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Faculty control and content preparation are central; adoption depends on instructor time to configure lessons.
Procurement
What should contracts, pricing and exit terms secure?
Expansion to other universities and K-12 is under way; interpretation: buyers should ask for study designs and raw comparisons, not headline percentages.
Operating model
Which teams own the service once it runs?
Faculty-designed AI-supported flipped classroom; teaching centre events as a channel for diffusion of vendor-linked practice.
Publication history
- 2026-10-13Student Success · Issue 376 resources
Stable resource ID: umich-cai-miller-collage-ai-tutor-claims-20261013