{"resourceId":"umich-cai-miller-collage-ai-tutor-claims-20261013","versions":[{"version":"external-79c82baebda629970e8d5b46659deaf15709f2e7d0b990330f3010ed5c808147","resource":{"id":"umich-cai-miller-collage-ai-tutor-claims-20261013","title":"Harvard physics educator reports 62% higher learning gains with Collage AI tutor versus a prior semester, in U-M keynote summary; presenter is the vendor's chief learning officer","organization":"University of Michigan Center for Academic Innovation (host); Kelly Miller, Harvard SEAS; Collage AI (vendor)","sector":"Higher education student success / AI tutoring in flipped STEM courses","geography":"Ann Arbor, Michigan (event host) and Cambridge, Massachusetts (course), United States","publishedAt":"U-M Center for Academic Innovation blog, 13 October 2026","publicationDate":"2026-10-13","eventDate":null,"sourceName":"University of Michigan Center for Academic Innovation","sourceLabel":"Host-institution blog summarising a vendor-affiliated researcher's talk","sourceUrl":"https://ai.umich.edu/blog-posts/empowering-educators-harvard-professor-on-ai-tools-as-effective-tutors/","evidenceClass":"vendor-claim","outcomeClass":"emerging","topics":["knowledge-work","governance-procurement"],"finding":"In a U-M Center for Academic Innovation summary of her keynote, Harvard's Kelly Miller reported that in a flipped applied physics course where Collage AI generated pre-class formative assessments and served as a tutor, students had 62% higher learning gains than a previous semester and appreciated the AI feedback. She also recapped the earlier Harvard physics experiment in which learning gains were nearly double with a custom AI tutor. Miller is chief learning officer of Collage AI and is extending it to other courses, universities and K-12 classrooms.","sledRelevance":"Published 13 October, this is the newest public claim of large AI-tutor learning gains in higher education. It sits against recent controlled evidence showing harms or null effects (such as the archived Maryland trial), making the evidence standard for tutor claims a live procurement question.","evidence":"Blog summary of a talk. The 62% figure compares one course with a previous semester; sample size, assessment instrument, cohort comparability and statistical tests are not given. The earlier randomized crossover study is described only briefly.","architectureImplications":"Instructor-configured platform where faculty feed course content and learning objectives and control assessment type and Bloom's level; tutor guidance grounded in course materials.","governanceImplications":"Conflict of interest: the presenter is the vendor's chief learning officer. Interpretation: institutions should require independent or pre-registered evaluation before relying on such figures.","securityPrivacyImplications":"Not addressed.","caveats":"Vendor-affiliated, non-randomized cross-semester claim reported second-hand by a host blog; no methods or data published in the source; one course in one elite institution.","streamIds":["student-success"],"roles":{"sales":"The framing point is that headline AI-tutor gains now come with an evidence-quality question. A U-M blog reports Kelly Miller's claim of 62% higher learning gains with Collage AI versus a prior semester, but Miller is the vendor's chief learning officer and no sample or method is given. Use this to open conversations about evaluation standards, not to promise outcomes; recent controlled trials elsewhere show null or negative effects.","engineering":"For engineering, the framing point is that the design described is instructor-configured and grounded in course content. Faculty feed pre-class materials and learning objectives, choose assessment types and Bloom's levels, and the tool tutors from that content. Pre-sales work should verify how grounding is enforced, what logs instructors see, and how student data is handled, since the source does not address privacy or hosting. Validate claims through a controlled pilot.","delivery":"Delivery's framing point is converting a vendor-linked claim into a fair test. The 62% figure compares semesters without stated controls. Delivery teams piloting a similar tutor would need randomized or matched sections, a common pre/post instrument, pre-registration of outcomes and attention to subgroups such as first-generation students. Proposed acceptance criteria are targets, not observed results, and the approach applies mainly to flipped STEM courses with prepared materials."},"roleDetails":{"sales":{"customerProblem":"Institutions face strong vendor-linked claims of AI-tutor learning gains alongside controlled studies showing null or negative effects, and must decide what evidence to require.","stakeholders":["Provost and teaching centre","STEM department chairs","Procurement office","Institutional research","Faculty early adopters"],"discoveryQuestions":["What study design produced the claimed gain, and with how many students?","Is there independent or pre-registered evaluation?","How much instructor time does configuration require?","How does the tool perform for first-generation or underprepared students?"],"valueHypothesis":"Interpretation: an instructor-configured, course-grounded tutor may support flipped-course preparation, but value must be shown in a controlled local pilot.","potentialEngagement":"A bounded controlled pilot in two to four sections of a flipped STEM course with a pre-registered outcome.","unsupportedClaims":["A 62% learning gain at other institutions","Causal effect of the tool (cross-semester comparison)","Independence of the evidence (vendor-affiliated presenter)"]},"engineering":{"fit":"Fit is for course-grounded, instructor-configured tutoring and formative assessment in flipped STEM courses.","architectureIntegration":"Instructors upload content and learning objectives; platform generates formative assessments and tutors students; LMS integration not described.","prerequisites":["Prepared pre-class materials and objectives","Instructor time for configuration","Data agreement with the vendor"],"constraints":["Evidence limited to one course at one institution","Privacy and hosting not described","Conflict of interest in reported results"],"security":"Not addressed in source; confirm data retention, model-training exclusions and instructor visibility of student chats.","proofOfValue":["Randomized or matched-section comparison with a common post-test","Audit tutor answers for accuracy in the course domain","Measure engagement with existing course activities to detect displacement"]},"delivery":{"work":"Run a controlled pilot with pre-registered outcomes and subgroup analysis before any wider adoption.","dependencies":["Willing instructors with flipped courses","Institutional research for analysis","Vendor configuration support","IRB or evaluation approval"],"ownership":"Teaching centre with participating departments; institutional research owns evaluation.","skillsAdoption":"Instructors need training on configuring lessons and interpreting student-use data.","governanceCheckpoints":["Pre-registration of outcomes","Mid-term check on usage and displacement of other activities","Independent analysis before scale decision"],"acceptanceCriteria":["Proposed: pilot reports effect sizes with confidence intervals against matched sections","Proposed: no subgroup shows significantly lower outcomes","Proposed: instructor configuration time recorded and reported"],"risks":["Novelty or instructor effects inflate results","Engagement with other learning activities declines","Vendor influence over evaluation"]}},"retrievedAt":"2026-10-14T03:26:17Z","enrichedAt":"2026-10-14T03:34:21Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Faculty control and content preparation are central; adoption depends on instructor time to configure lessons.","procurementImplications":"Expansion to other universities and K-12 is under way; interpretation: buyers should ask for study designs and raw comparisons, not headline percentages.","operatingModelImplications":"Faculty-designed AI-supported flipped classroom; teaching centre events as a channel for diffusion of vendor-linked practice.","sourceVerification":{"openedUrl":"https://ai.umich.edu/blog-posts/empowering-educators-harvard-professor-on-ai-tools-as-effective-tutors/","referenceExcerpt":"compared to a previous semester, students using the Collage AI tutor had 62% higher learning gains.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}