From the Student Success edition of October 16, 2026
UW intro statistics course grades whole student-LLM conversations via LLteacher; 134 students report confidence gains, but no control group
University of Washington (Furfaro, Lovelace, Zhang, Mosciatti) · Higher education instruction / LLM-guided homework and assessment design · Seattle, Washington, United States
- Publisher
- arXiv
- Original publication
- arXiv preprint 2610.15834, submitted 11 October 2026
- Source retrieved
- 2026-10-17
What happened
A University of Washington team deployed LLteacher, an open-source tool for instructor-configured LLM-guided homework, in a large introductory statistics course serving mostly pre-major undergraduates. Assignments use recall, discovery and critical-thinking modes, and instructors grade the entire student-LLM conversation rather than only the final answer. Among 134 consenting students in an IRB-approved study, all submitted at least two of six assignments and 90% completed all six. Over 80% agreed the assignments improved their comprehension and ability to interpret statistical results, with larger self-rated confidence gains among students who started less confident. Interaction styles varied widely and reported hallucinations were rarely disruptive. The authors state the study had no control or comparison group, relied on voluntary participation in one course and quarter, and cannot establish that AI-guided homework outperforms traditional homework.
Why it matters
Posted 11 October and not previously captured, it offers a concrete counter-design to UMD's open-access tutor: structured, graded AI conversations. Evidence is preliminary and self-reported.
Evidence and measured results
Pre/post self-ratings and end-of-term survey items; interaction logs; descriptive results without comparison group. Authors developed the tool.
Limitations and uncertainty
No control group; self-reported confidence; single course; developer-authored preprint; not peer reviewed.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-17; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Instructors need assessments that still reveal student reasoning when LLMs can produce answers directly.
- Stakeholders
- Gateway course instructors
- Department chairs
- Centre for teaching and learning
- Instructional technology
- Assessment office
- Discovery questions
- Which courses need reasoning-visible assessment?
- Is there capacity to grade transcripts?
- Would a comparison section be feasible?
- Who hosts open-source tools?
- Value hypothesis
- Interpretation: graded, structured AI conversations may preserve assessment validity while using AI productively.
- Potential engagement
- A one-term pilot of conversation-graded assignments in a gateway course with a comparison section.
- Unsupported claims
- Grade or learning gains (no control)
- Equity effects (not measured)
- Low hallucination rates in other subjects
Pre-sales engineering
Role takeaway
- Fit
- Fits courses wanting instructor-controlled LLM assignments with transcript-based grading.
- Architecture and integration
- Open-source LLteacher with configurable assignment modes; full student-LLM conversation logged for grading.
- Prerequisites
- Hosting environment
- Model API agreement
- Rubrics for transcripts
- LMS integration
- Constraints
- Single course evidence
- Developer-authored
- Modes confounded with topics
- Security
- Define transcript retention, access for graders and model-provider data terms before deployment.
- Proof of value
- Hallucination sampling per assignment
- Grader time per transcript
- Comparison-section learning outcomes
Delivery
Role takeaway
- Work
- Deploy the tool, build transcript rubrics, train TAs and run a comparison with traditional homework.
- Dependencies
- Course instructor
- TAs
- Instructional technology
- Institutional research
- Ownership
- Course instructor with teaching and learning centre support.
- Skills and adoption
- TAs need calibration on grading reasoning in transcripts; students need guidance on productive interaction.
- Governance checkpoints
- IRB or evaluation approval
- Mid-term grading calibration
- End-of-term outcome review
- Acceptance criteria
- Proposed: completion rate comparable to UW's 90% of all assignments
- Proposed: inter-rater agreement on transcript rubric
- Proposed: exam performance compared with comparison section
- Risks
- Grading load overwhelms staff
- Hallucinations mislead students
- Self-report mistaken for learning evidence
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Open-source tool where instructors configure LLM behaviour per assignment and capture full transcripts for grading; interpretation: requires transcript storage, rubric tooling and model access governance.
Governance
Who approves, reviews and stays accountable for outcomes?
Assessment shifts to the reasoning trace; IRB-approved research consent; course-level instructor control over AI behaviour.
Security and privacy
What data, permissions and controls need testing?
Full conversation logs are graded and stored; paper does not detail retention or model provider data terms in the reviewed sections.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Larger perceived gains among low-confidence students suggest possible gap narrowing, unverified; TA and instructor grading load for transcripts not quantified.
Procurement
What should contracts, pricing and exit terms secure?
Open-source option reduces licence cost but shifts hosting, model API and support to the institution.
Operating model
Which teams own the service once it runs?
Instructors design assignment modes and grade conversations; teaching teams need rubric and review capacity.
Publication history
- 2026-10-16Student Success · Issue 396 resources
Stable resource ID: uw-llteacher-llm-guided-stats-assignments-134-students-20261011