{"resourceId":"uw-llteacher-llm-guided-stats-assignments-134-students-20261011","versions":[{"version":"external-ae22b8c58ae4f98d680bb320ebe2351cc93c5638eccf3064fb04aa8da5a3fe10","resource":{"id":"uw-llteacher-llm-guided-stats-assignments-134-students-20261011","title":"UW intro statistics course grades whole student-LLM conversations via LLteacher; 134 students report confidence gains, but no control group","organization":"University of Washington (Furfaro, Lovelace, Zhang, Mosciatti)","sector":"Higher education instruction / LLM-guided homework and assessment design","geography":"Seattle, Washington, United States","publishedAt":"arXiv preprint 2610.15834, submitted 11 October 2026","publicationDate":"2026-10-11","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (not peer reviewed)","sourceUrl":"https://arxiv.org/abs/2610.15834","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["knowledge-work","developers-agents","operating-model"],"finding":"A University of Washington team deployed LLteacher, an open-source tool for instructor-configured LLM-guided homework, in a large introductory statistics course serving mostly pre-major undergraduates. Assignments use recall, discovery and critical-thinking modes, and instructors grade the entire student-LLM conversation rather than only the final answer. Among 134 consenting students in an IRB-approved study, all submitted at least two of six assignments and 90% completed all six. Over 80% agreed the assignments improved their comprehension and ability to interpret statistical results, with larger self-rated confidence gains among students who started less confident. Interaction styles varied widely and reported hallucinations were rarely disruptive. The authors state the study had no control or comparison group, relied on voluntary participation in one course and quarter, and cannot establish that AI-guided homework outperforms traditional homework.","sledRelevance":"Posted 11 October and not previously captured, it offers a concrete counter-design to UMD's open-access tutor: structured, graded AI conversations. Evidence is preliminary and self-reported.","evidence":"Pre/post self-ratings and end-of-term survey items; interaction logs; descriptive results without comparison group. Authors developed the tool.","architectureImplications":"Open-source tool where instructors configure LLM behaviour per assignment and capture full transcripts for grading; interpretation: requires transcript storage, rubric tooling and model access governance.","governanceImplications":"Assessment shifts to the reasoning trace; IRB-approved research consent; course-level instructor control over AI behaviour.","securityPrivacyImplications":"Full conversation logs are graded and stored; paper does not detail retention or model provider data terms in the reviewed sections.","caveats":"No control group; self-reported confidence; single course; developer-authored preprint; not peer reviewed.","streamIds":["student-success"],"roles":{"sales":"For sales, the framing point is structured, graded AI assignments as an alternative to open AI tutors. UW's LLteacher has instructors configure the LLM and grade the whole conversation; 134 students reported comprehension and confidence gains. That supports discovery with faculty on assessment redesign. Stay bounded: this is a developer-authored preprint with no control group and self-reported outcomes, so it cannot support claims that AI homework improves grades or learning.","engineering":"Engineering's framing point is instructor-configured LLM behaviour with full transcript capture. LLteacher supports recall, discovery and critical-thinking modes and stores conversations for grading. Pre-sales teams should examine hosting of the open-source tool, model API terms, transcript retention and LMS grade passback. Validation should include hallucination sampling per assignment and grading-time measurement, since the authors report hallucinations as rarely disruptive but did not quantify their frequency or grader workload in the reviewed text.","delivery":"Delivery's framing point is piloting conversation-graded assignments with a comparison design. UW achieved high completion, 90% finishing all six assignments, but lacked a control group. A delivery engagement would set up the tool, rubrics, TA training and a comparison with traditional homework sections. Acceptance criteria below are proposals. Applicability is limited to similar large gateway courses with instructors willing to redesign assignments and grade transcripts."},"roleDetails":{"sales":{"customerProblem":"Instructors need assessments that still reveal student reasoning when LLMs can produce answers directly.","stakeholders":["Gateway course instructors","Department chairs","Centre for teaching and learning","Instructional technology","Assessment office"],"discoveryQuestions":["Which courses need reasoning-visible assessment?","Is there capacity to grade transcripts?","Would a comparison section be feasible?","Who hosts open-source tools?"],"valueHypothesis":"Interpretation: graded, structured AI conversations may preserve assessment validity while using AI productively.","potentialEngagement":"A one-term pilot of conversation-graded assignments in a gateway course with a comparison section.","unsupportedClaims":["Grade or learning gains (no control)","Equity effects (not measured)","Low hallucination rates in other subjects"]},"engineering":{"fit":"Fits courses wanting instructor-controlled LLM assignments with transcript-based grading.","architectureIntegration":"Open-source LLteacher with configurable assignment modes; full student-LLM conversation logged for grading.","prerequisites":["Hosting environment","Model API agreement","Rubrics for transcripts","LMS integration"],"constraints":["Single course evidence","Developer-authored","Modes confounded with topics"],"security":"Define transcript retention, access for graders and model-provider data terms before deployment.","proofOfValue":["Hallucination sampling per assignment","Grader time per transcript","Comparison-section learning outcomes"]},"delivery":{"work":"Deploy the tool, build transcript rubrics, train TAs and run a comparison with traditional homework.","dependencies":["Course instructor","TAs","Instructional technology","Institutional research"],"ownership":"Course instructor with teaching and learning centre support.","skillsAdoption":"TAs need calibration on grading reasoning in transcripts; students need guidance on productive interaction.","governanceCheckpoints":["IRB or evaluation approval","Mid-term grading calibration","End-of-term outcome review"],"acceptanceCriteria":["Proposed: completion rate comparable to UW's 90% of all assignments","Proposed: inter-rater agreement on transcript rubric","Proposed: exam performance compared with comparison section"],"risks":["Grading load overwhelms staff","Hallucinations mislead students","Self-report mistaken for learning evidence"]}},"retrievedAt":"2026-10-17T03:18:30Z","enrichedAt":"2026-10-17T03:23:56Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Larger perceived gains among low-confidence students suggest possible gap narrowing, unverified; TA and instructor grading load for transcripts not quantified.","procurementImplications":"Open-source option reduces licence cost but shifts hosting, model API and support to the institution.","operatingModelImplications":"Instructors design assignment modes and grade conversations; teaching teams need rubric and review capacity.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.15834","referenceExcerpt":"Interaction patterns varied widely, from long single messages to rapid iterative exchanges, and hallucinations, when present, were rarely disruptive to learning.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}