{"resourceId":"arxiv-cmu-gptutor-proof-course-chatbot-use-lower-exam-scores-20261022","versions":[{"version":"external-e08982634d40d97bbf7ffb8782a244eb578ec14f639ac56bba001ba20cdb1dc9","resource":{"id":"arxiv-cmu-gptutor-proof-course-chatbot-use-lower-exam-scores-20261022","title":"Carnegie Mellon randomized staggered-access study: AI support raised homework scores but not exam scores, and heavier chatbot use tracked lower exam results","organization":"Carnegie Mellon University (proof-based undergraduate mathematics course; GPTutor system)","sector":"Higher education tutoring, assessment and learning outcomes","geography":"Pittsburgh, Pennsylvania, United States","publishedAt":"arXiv preprint 2610.31481, submitted 22 October 2026","publicationDate":"2026-10-22","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (randomized staggered-access study)","sourceUrl":"https://arxiv.org/abs/2610.31481","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","operating-model"],"finding":"In an undergraduate proof-based mathematics course at Carnegie Mellon, 153 consenting students were randomly assigned earlier or later access to a GPTutor system with an open-ended AI chatbot and a homework-feedback interface that required a proof attempt first. While only the early group had access, its homework scores were 2.56 points higher, but scores on the next exam, taken without AI, did not differ significantly. After all students had access, each additional 100 chatbot messages was associated with a 3.02-point lower subsequent exam score, controlling for prior exam score and self-efficacy; lower self-efficacy predicted more chatbot use. Use of the attempt-first feedback interface was not negatively associated with exam scores. The authors recommend against open-ended chatbots as tutors.","sledRelevance":"Directly tests whether an AI tutor improves independent performance, separating AI-assisted homework gains from exam results; relevant to any campus deploying course chatbots for student success.","evidence":"Randomized access timing (n=153; 75 early, 78 later) for the homework and next-exam comparison; observational regression and mediation for usage (n=148); logged usage counts; 19 student interviews.","architectureImplications":"Interface design matters: an attempt-first feedback flow versus an open chat window over the same model. Usage logs were needed to see the pattern.","governanceImplications":"Evaluation of AI tutors should include performance without AI, not only assisted homework scores; usage labels should not drive discipline.","securityPrivacyImplications":"Study used logged student-AI conversations under IRB approval; student-level records are not released.","caveats":"Preprint; one course; randomization covered access timing, not interface choice or amount of use; the negative usage association may partly reflect struggling students seeking more help; data collection stopped before the final exam.","streamIds":["student-success"],"roles":{"sales":"Sales framing: this is randomized evidence that AI help can lift homework scores without improving exam performance, so tutoring claims based on assisted work are weak. Use it to ask provosts and teaching-center leaders how they judge AI tutors and whether outcomes are measured without AI. A bounded engagement could design an evaluation for one gateway course comparing attempt-first feedback with open chat. Do not claim chatbots harm learning generally; this is one course and a preprint.","engineering":"Engineering framing: the useful distinction is interface design over the same model, an open chatbot versus a feedback flow that requires a submitted attempt. Check whether a campus tutoring tool can enforce attempt-first interaction, log messages per student, and export usage for joining with grades under privacy rules. Proof of value should compare later exam performance without AI, not assisted homework scores, and test for heavy-use subgroups with low confidence.","delivery":"Delivery framing: deployment work should centre on course design and evaluation rather than tool access alone. Delivery would help instructors configure attempt-first feedback, add explain-your-reasoning checkpoints, and schedule an AI-free assessment that serves as the outcome measure. Agree data-sharing with institutional research before launch so usage logs and grades can be analysed. Review results each term with faculty and watch whether lower-confidence students rely on open chat in place of other support."},"roleDetails":{"sales":{"customerProblem":"Institutions deploying AI tutors lack evidence that assisted gains carry over to independent performance.","stakeholders":["Provost or vice provost for undergraduate education","Teaching and learning center director","Mathematics or STEM gateway course leads","Institutional research"],"discoveryQuestions":["How do you currently measure whether an AI tutor improves learning?","Do your tools let students get answers without an attempt?","Which students use the tutor most, and how do they perform without it?"],"valueHypothesis":"Interpretation: attempt-first AI feedback with AI-free outcome checks could keep benefits while reducing offloading, pending local evidence.","potentialEngagement":"A one-term evaluation design in one or two gateway courses comparing interface designs on AI-free assessments.","unsupportedClaims":["That AI tutors improve exam results","That chatbots harm learning in all subjects","Any cost savings from tutoring automation"]},"engineering":{"fit":"Fits course-integrated tutoring where the institution controls prompts, interfaces and usage logging.","architectureIntegration":"LLM API behind two interfaces (chat and attempt-first feedback), with instructor reference solutions as context and session logs joined to gradebook data for analysis.","prerequisites":["Configurable interface that can require a student attempt","Per-student usage logging","IRB or data-governance approval to link logs and grades"],"constraints":["Single proof-based course; transfer to other subjects unknown","Model snapshots and prompts differ from current products","Interface choice was self-selected after the randomized window"],"security":"Student-AI conversations and grades are sensitive education records; keep analysis de-identified and access-controlled.","proofOfValue":["Compare AI-free exam scores between access groups","Track usage intensity against later independent performance","Test attempt-first feedback separately from open chat"]},"delivery":{"work":"Configure attempt-first AI feedback in pilot courses, add reasoning checkpoints and AI-free assessments, and run a term-long evaluation.","dependencies":["Instructor buy-in","Institutional research capacity","Vendor or in-house support for interface controls"],"ownership":"Teaching and learning center with course instructors; institutional research for evaluation.","skillsAdoption":"Instructor training on reasoning-focused prompts and on interpreting usage dashboards without punitive use.","governanceCheckpoints":["Data-use approval before linking logs and grades","Mid-term review of usage by student confidence","End-of-term outcome review before scaling"],"acceptanceCriteria":["Proposed: AI-free assessment results reported by access group","Proposed: share of tutor sessions beginning with a student attempt is tracked","Proposed: no widening of gaps for lower-confidence students"],"risks":["Cognitive offloading by heavy users","Over-interpreting observational usage associations","Inaccurate tutor feedback, as seen in the paper examples"]}},"retrievedAt":"2026-10-27T03:21:11Z","enrichedAt":"2026-10-27T03:25:47Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Lower-confidence students used the chatbot most, suggesting equity concerns if open chat replaces structured support; not directly measured.","procurementImplications":"Interpretation: buyers should ask whether tutoring tools require a student attempt before feedback and whether vendors report independent-performance outcomes.","operatingModelImplications":"Interpretation: course teams should pair AI support with explain-your-reasoning tasks and assessments completed without AI.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.31481","referenceExcerpt":"every additional 100 messages sent to the AI chatbot was associated with a 3.02-point lower score on the subsequent exam","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}