{"resourceId":"chen-gptutor-proof-feedback-transfer-2026","versions":[{"version":"external-2127cf1174f89fa41888a6820a22947a0691b240840dde8a5e9f73fca2f63c80","resource":{"id":"chen-gptutor-proof-feedback-transfer-2026","title":"GPTutor study: homework gains from earlier LLM access did not transfer to exams; chat path cautionary","organization":"Carnegie Mellon University / University of Hong Kong / arXiv","sector":"Higher education STEM tutoring / mathematical proof learning","geography":"United States (Carnegie Mellon undergraduate discrete mathematics)","publishedAt":"arXiv:2602.18807v2 [cs.HC] 31 Mar 2026","publicationDate":"2026-03-31","eventDate":null,"sourceName":"arXiv preprint (cs.HC)","sourceLabel":"Chen et al. (2026) — GPTutor conversational vs embedded LLM feedback","sourceUrl":"https://arxiv.org/pdf/2602.18807","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","developers-agents","operating-model"],"finding":"Staggered-access evaluation of GPTutor (structured proof-review tool + conversational chatbot) with 148 undergraduates in discrete mathematics: earlier access associated with higher homework performance, but benefits did not transfer to exam scores. Serial mediation: lower self-efficacy predicted higher usage of both tools; only chatbot usage constituted a negative associative pathway to subsequent exam performance, while proof-review usage showed no detectable independent association. Authors conclude conversational LLM support alone may not reliably support transfer to independent assessment.","sledRelevance":"Newly archived HE tutoring evidence after 2026-09-17 agent/perception pilots: supplies cautionary transfer and interface-design findings for student-success tutoring deployments. Why now: directly addresses durable learning vs engagement/homework proxies.","evidence":"Source findings: staggered access, mediation on usage pathways, homework vs exam outcomes. Limitations: not a fully randomized multi-arm RCT of interfaces; single course; associative pathways; preprint.","architectureImplications":"Interpretation: prefer work-anchored embedded feedback over unconstrained chat; instrument AI-free exams; log interface-specific usage.","governanceImplications":"Interpretation: approve tutors with homework-answer guardrails and explicit transfer evaluation requirements.","securityPrivacyImplications":"Logs of proofs/chat may reveal academic struggle; apply education-record controls and no-train terms.","caveats":"Preprint; CMU selective context; mediation is associative. Does not measure retention/persistence.","streamIds":["student-success"],"roles":{"sales":"Sales framing: position GPTutor-class tools as mixed evidence—CMU staggered-access study (N=148) linked earlier access to higher homework but not exam transfer, and chatbot usage formed a negative associative pathway to exams while structured proof-review did not. Do not sell open chat tutors as proven durable learning. Bound deals to work-anchored feedback designs with AI-free assessment endpoints. Keep human tutors in the value story for transfer-critical assessments.","engineering":"Engineering framing: implement work-anchored review as the default path and tightly constrain conversational solvers. Prerequisites include submission-linked feedback UIs, RAG over approved course materials, and joins from usage logs to exam outcomes. Constraints: single-course staggered design limits external validity and mediation paths are associative. Proof of value: exam transfer without negative chat-mediated pathways, plus escalation metrics to human TAs.","delivery":"Delivery framing: delivery prioritizes embedded feedback workflows over launching a free-form tutor. Ownership stays with the course team supported by teaching-learning staff. Acceptance requires pre-registered AI-free exam endpoints, interface-specific usage dashboards, and documented TA escalation. Risks include mistaking homework improvement for durable learning and amplifying over-reliance among low self-efficacy students via unconstrained chat. Schedule mid-pilot review before expanding conversational features."},"roleDetails":{"sales":{"customerProblem":"Campuses want LLM tutors for gateway STEM courses but need evidence on homework vs exam transfer.","stakeholders":["Math/CS departments","Teaching & learning","Academic integrity office","Student success"],"discoveryQuestions":["Will you measure AI-free exam transfer, not only homework?","Is feedback work-anchored or open chat?","How will low self-efficacy students be steered away from over-reliance?","What guardrails prevent answer-seeking chat?"],"valueHypothesis":"Structured, work-anchored LLM feedback may avoid the negative exam pathway seen with conversational chat; homework gains alone do not prove durable learning.","potentialEngagement":"Bounded discrete-math or proof-course pilot comparing chat vs embedded review with exam transfer as primary endpoint.","unsupportedClaims":["That homework lifts equal exam learning","That conversational tutors reliably transfer","Staff replacement savings from chat tutors"]},"engineering":{"fit":"Fits course-scoped tutoring with dual interfaces: embedded proof/work review plus optional constrained chat.","architectureIntegration":"LMS-integrated proof-review UI anchored to student submissions; RAG over course materials; usage logging separated by interface; human TA escalation.","prerequisites":["Course content corpus","Roster-scoped access","Homework and exam outcome linkage","Self-efficacy / help-seeking surveys"],"constraints":["Staggered-access not full RCT randomization of interface","Single CMU discrete-math course","Associative mediation ≠ pure causal interface effect"],"security":"Student proofs and chat logs are education records; minimize retention and block consumer model training.","proofOfValue":["Compare homework vs exam deltas by interface","Track chatbot vs review usage pathways","Test interventions that redirect low self-efficacy students"]},"delivery":{"work":"Faculty authoring of proof-review rubrics, student onboarding on productive help-seeking, and integrity messaging.","dependencies":["Instructor time","LMS admin","TA capacity","Analytics join to exams"],"ownership":"Course instructors with teaching-learning support.","skillsAdoption":"Students need help-seeking norms; TAs need review-tool workflows.","governanceCheckpoints":["Integrity review of tutor scope","Privacy review of logs","Decision gate before expanding chat features"],"acceptanceCriteria":["Exam transfer metric pre-registered","Interface-specific usage dashboards live","Escalation to human TAs documented"],"risks":["Homework gains masking weak transfer","Over-reliance among low self-efficacy students","Chatbot becoming an answer engine"]}},"retrievedAt":"2026-09-19T03:05:14Z","enrichedAt":"2026-09-19T03:11:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Interpretation: low self-efficacy students used tools more and chat associated with worse exam pathways—accessibility and academic-support offices should co-design redirects to structured help, not unrestricted chat.","procurementImplications":"Interpretation: contract for interface-level analytics and exam-transfer evaluation clauses; reject chat-only tutors sold on homework engagement.","operatingModelImplications":"Interpretation: operating model keeps TAs/faculty as owners of work-anchored feedback; chat is constrained secondary channel.","sourceVerification":{"responseId":"curl-arxiv-gptutor-2602.18807-2026-09-18","openedUrl":"https://arxiv.org/pdf/2602.18807","referenceExcerpt":"this benefit did not transfer to exam scores. A","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}