{"resourceId":"umich-feedbackwriter-rct-2026","versions":[{"version":"external-f03c030abc5d6a6f011d775b95de47d80cb4e8a571539b4b0f7bb39a77144ad9","resource":{"id":"umich-feedbackwriter-rct-2026","title":"Michigan RCT: TA-mediated AI feedback improves revision quality but not post-test learning","organization":"University of Michigan","sector":"Higher education teaching / writing-to-learn","geography":"United States (large undergraduate economics course)","publishedAt":"arXiv 2602.16820v2 (24 Feb 2026); CHI ’26","publicationDate":"2026-02-24","eventDate":null,"sourceName":"arXiv / ACM CHI 2026","sourceLabel":"Lu et al., FeedbackWriter RCT (arXiv:2602.16820v2)","sourceUrl":"https://arxiv.org/html/2602.16820","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","developers-agents","operating-model"],"finding":"In an IRB-approved RCT in a large UMich undergraduate economics course (354 students, 11 TAs; Jan–Apr 2025), AI suggestions to TAs (FeedbackWriter) produced higher-quality student revisions (Cohen’s d≈0.50) that increased with TA adoption of AI suggestions, and feedback scored as more actionable. Assignment to AI-mediated feedback did not improve post-test scores versus human-only feedback (b=-0.007, p=0.31). Authors attribute limited learning transfer partly to single-practice dosage per knowledge component and call for higher instructional dosage and student-perception studies.","sledRelevance":"Shows a practical human-in-the-loop teaching pattern for large courses while reinforcing that revision-quality gains are not automatic learning gains—aligned with student-success emphasis on durable learning over engagement proxies. Newly useful beside retention-null outreach evidence for outcome-definition discipline.","evidence":"Counterbalanced RCT across two writing assignments; multivariate models; automated feedback-quality metrics; TA interviews; 1,366 essays graded with the system per abstract context. Human-only baseline matched UI without AI suggestions.","architectureImplications":"Human-in-the-loop design where TAs adopt/edit/dismiss AI comments anchored to rubrics; requires detailed rubrics and workflow UI aligned to TA cognition. Interpretation: prioritize TA agency and rubric quality over fully automated student-facing feedback bots.","governanceImplications":"Students knew TAs used AI but not which comments were AI-suggested—transparency tradeoffs remain. Interpretation: institutions should set disclosure norms for AI-assisted grading/feedback.","securityPrivacyImplications":"Student essays and rubrics flow through an AI suggestion pipeline; source focuses on HCI/learning, not vendor security. Interpretation: use approved tenancy and restrict training on student writing.","caveats":"Single course/domain; AI evaluator for essay quality; no AI-only comparison arm; optional post-test; limited student voice data; study period 2025 with 2026 CHI/arXiv release—backfill relative to 2026-09-15 but newly paired with retention-null evidence for outcome discipline. Not a retention study.","streamIds":["student-success"],"roles":{"sales":"Sales framing: AI can help TAs give better feedback at scale, but revision polish is not proof of learning. Michigan’s FeedbackWriter RCT shows medium revision-quality gains with null post-test differences. Sell human-in-the-loop feedback copilots for large writing-intensive courses with learning metrics beyond draft scores. Avoid claims that actionable AI comments automatically raise exam or concept mastery without dosage redesign.","engineering":"Engineering framing: fit is a TA-facing suggestion layer with adopt/edit/dismiss, rubric anchoring, and audit of suggestion uptake—not autonomous student chat. Integrate with LMS assignment workflows and approved LLM tenancy. Prerequisites: high-quality rubrics and TA training. Constraints: single-course evidence and AI-as-judge limitations. Proof of value must include unaided learning measures, not only revision rubrics and TA satisfaction.","delivery":"Delivery framing: roll out with TA norming, rubric refinement, disclosure policy, and pre-registered learning endpoints. Dependencies include instructional-design support and LMS integration. Ownership: course instructors/TAs for pedagogy, IT for platform. Acceptance criteria: measured revision gains plus explicit learning evaluation; risks include over-trusting AI judges, undisclosed AI comments, and mistaking draft quality for mastery."},"roleDetails":{"sales":{"customerProblem":"Large courses struggle to deliver timely, rubric-aligned writing feedback without burning out TAs, yet leaders still need learning—not only nicer drafts.","stakeholders":["Department chairs","Undergraduate education deans","Writing-across-curriculum leads","TA coordinators"],"discoveryQuestions":["Which courses already have detailed analytic rubrics?","Will learning be measured beyond revision scores?","What disclosure will students receive about AI-assisted feedback?","How will TA adopt/edit rates be monitored?"],"valueHypothesis":"TA-mediated AI suggestions may improve revision quality and feedback actionability in rubric-rich large courses; learning gains require higher practice dosage and separate measurement.","potentialEngagement":"Interpretation: a single-course feedback-copilot pilot with dual endpoints (revision quality and unaided post-test)—not a university-wide grading automation deal.","unsupportedClaims":["Automatic post-test or exam gains","AI-only feedback equivalence","Cross-discipline generalization from one econ course","Retention or graduation impact"]},"engineering":{"fit":"Strong fit for TA copilots with human approval; weak fit for fully automated student-facing graders without human review.","architectureIntegration":"Embed suggestion UI in grading workflow; store adopt/edit/dismiss events; keep LMS as system of record for grades.","prerequisites":["Detailed rubrics","Approved LLM endpoint","TA training environment","Logging of suggestion provenance"],"constraints":["Single-course external validity","AI essay-evaluator measurement limits","Optional post-test participation","No AI-only arm"],"security":"Process student writing in approved tenancy; disable vendor training; control TA access to identifiable submissions.","proofOfValue":["Revision quality effect with confidence intervals","Post-test or delayed unaided measure","TA adoption/edit rates","Feedback actionability metrics"]},"delivery":{"work":"Implement TA workflow, rubric workshops, disclosure language, and dual-metric evaluation.","dependencies":["Instructor/TA buy-in","LMS integration","IRB if research framing","Teaching-center support"],"ownership":"Interpretation: academic unit owns pedagogy and acceptance; IT owns platform security.","skillsAdoption":"Train TAs to critique AI suggestions; teach students how feedback will be produced.","governanceCheckpoints":["Disclosure approval","Rubric quality gate","Mid-pilot learning readout","Privacy review"],"acceptanceCriteria":["Documented revision-quality improvement","Learning endpoint analyzed even if null","TA agency preserved (edit/dismiss)"],"risks":["Null learning misread as product failure without dosage redesign","Hidden AI authorship erodes trust","Over-reliance on AI quality judges","Workload spike if UI poorly fitted"]}},"retrievedAt":"2026-09-15T14:15:40Z","enrichedAt":"2026-09-15T14:17:30Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"TA workload and skill mix change toward reviewing AI suggestions; students with disabilities still need accommodation-aligned feedback modalities—source does not test disability-specific outcomes.","procurementImplications":"Procure TA-copilot capabilities with logging, tenancy, and evaluation access; avoid contracts that define success only as faster grading.","operatingModelImplications":"Operating model keeps humans accountable for feedback while using AI to expand coverage; requires rubric investment and disclosure norms.","sourceVerification":{"openedUrl":"https://arxiv.org/html/2602.16820","referenceExcerpt":"there is no statistical difference between the two conditions on the post-test scores.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}