From the Student Success edition of September 15, 2026
Michigan RCT: TA-mediated AI feedback improves revision quality but not post-test learning
University of Michigan · Higher education teaching / writing-to-learn · United States (large undergraduate economics course)
- Publisher
- arXiv / ACM CHI 2026
- Original publication
- arXiv 2602.16820v2 (24 Feb 2026); CHI ’26
- Source retrieved
- 2026-09-15
What happened
In an IRB-approved RCT in a large UMich undergraduate economics course (354 students, 11 TAs; Jan–Apr 2025), AI suggestions to TAs (FeedbackWriter) produced higher-quality student revisions (Cohen’s d≈0.50) that increased with TA adoption of AI suggestions, and feedback scored as more actionable. Assignment to AI-mediated feedback did not improve post-test scores versus human-only feedback (b=-0.007, p=0.31). Authors attribute limited learning transfer partly to single-practice dosage per knowledge component and call for higher instructional dosage and student-perception studies.
Why it matters
Shows a practical human-in-the-loop teaching pattern for large courses while reinforcing that revision-quality gains are not automatic learning gains—aligned with student-success emphasis on durable learning over engagement proxies. Newly useful beside retention-null outreach evidence for outcome-definition discipline.
Evidence and measured results
Counterbalanced RCT across two writing assignments; multivariate models; automated feedback-quality metrics; TA interviews; 1,366 essays graded with the system per abstract context. Human-only baseline matched UI without AI suggestions.
Limitations and uncertainty
Single course/domain; AI evaluator for essay quality; no AI-only comparison arm; optional post-test; limited student voice data; study period 2025 with 2026 CHI/arXiv release—backfill relative to 2026-09-15 but newly paired with retention-null evidence for outcome discipline. Not a retention study.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-15; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Large courses struggle to deliver timely, rubric-aligned writing feedback without burning out TAs, yet leaders still need learning—not only nicer drafts.
- Stakeholders
- Department chairs
- Undergraduate education deans
- Writing-across-curriculum leads
- TA coordinators
- Discovery questions
- Which courses already have detailed analytic rubrics?
- Will learning be measured beyond revision scores?
- What disclosure will students receive about AI-assisted feedback?
- How will TA adopt/edit rates be monitored?
- Value hypothesis
- TA-mediated AI suggestions may improve revision quality and feedback actionability in rubric-rich large courses; learning gains require higher practice dosage and separate measurement.
- Potential engagement
- Interpretation: a single-course feedback-copilot pilot with dual endpoints (revision quality and unaided post-test)—not a university-wide grading automation deal.
- Unsupported claims
- Automatic post-test or exam gains
- AI-only feedback equivalence
- Cross-discipline generalization from one econ course
- Retention or graduation impact
Pre-sales engineering
Role takeaway
- Fit
- Strong fit for TA copilots with human approval; weak fit for fully automated student-facing graders without human review.
- Architecture and integration
- Embed suggestion UI in grading workflow; store adopt/edit/dismiss events; keep LMS as system of record for grades.
- Prerequisites
- Detailed rubrics
- Approved LLM endpoint
- TA training environment
- Logging of suggestion provenance
- Constraints
- Single-course external validity
- AI essay-evaluator measurement limits
- Optional post-test participation
- No AI-only arm
- Security
- Process student writing in approved tenancy; disable vendor training; control TA access to identifiable submissions.
- Proof of value
- Revision quality effect with confidence intervals
- Post-test or delayed unaided measure
- TA adoption/edit rates
- Feedback actionability metrics
Delivery
Role takeaway
- Work
- Implement TA workflow, rubric workshops, disclosure language, and dual-metric evaluation.
- Dependencies
- Instructor/TA buy-in
- LMS integration
- IRB if research framing
- Teaching-center support
- Ownership
- Interpretation: academic unit owns pedagogy and acceptance; IT owns platform security.
- Skills and adoption
- Train TAs to critique AI suggestions; teach students how feedback will be produced.
- Governance checkpoints
- Disclosure approval
- Rubric quality gate
- Mid-pilot learning readout
- Privacy review
- Acceptance criteria
- Documented revision-quality improvement
- Learning endpoint analyzed even if null
- TA agency preserved (edit/dismiss)
- Risks
- Null learning misread as product failure without dosage redesign
- Hidden AI authorship erodes trust
- Over-reliance on AI quality judges
- Workload spike if UI poorly fitted
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Human-in-the-loop design where TAs adopt/edit/dismiss AI comments anchored to rubrics; requires detailed rubrics and workflow UI aligned to TA cognition. Interpretation: prioritize TA agency and rubric quality over fully automated student-facing feedback bots.
Governance
Who approves, reviews and stays accountable for outcomes?
Students knew TAs used AI but not which comments were AI-suggested—transparency tradeoffs remain. Interpretation: institutions should set disclosure norms for AI-assisted grading/feedback.
Security and privacy
What data, permissions and controls need testing?
Student essays and rubrics flow through an AI suggestion pipeline; source focuses on HCI/learning, not vendor security. Interpretation: use approved tenancy and restrict training on student writing.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
TA workload and skill mix change toward reviewing AI suggestions; students with disabilities still need accommodation-aligned feedback modalities—source does not test disability-specific outcomes.
Procurement
What should contracts, pricing and exit terms secure?
Procure TA-copilot capabilities with logging, tenancy, and evaluation access; avoid contracts that define success only as faster grading.
Operating model
Which teams own the service once it runs?
Operating model keeps humans accountable for feedback while using AI to expand coverage; requires rubric investment and disclosure norms.
Publication history
- 2026-09-15Student Success · Issue 104 resources
Stable resource ID: umich-feedbackwriter-rct-2026