{"resourceId":"arxiv-vanderbilt-testcompass-feedback-only-ai-tutor-unit-testing-18-students-20261026","versions":[{"version":"external-9efe1d6239561360e186f25a6f540a261e3cbd53cb12b74abf170c93ab063d35","resource":{"id":"arxiv-vanderbilt-testcompass-feedback-only-ai-tutor-unit-testing-18-students-20261026","title":"Preprint: Vanderbilt classroom study finds a feedback-only AI tutor that does not write tests kept students doing their own testing with comparable test quality to a general AI assistant (18 students)","organization":"Vanderbilt University (authors)","sector":"Higher education computing courses and AI tutoring","geography":"Nashville, Tennessee, United States","publishedAt":"arXiv preprint 2610.36185, submitted 26 October 2026","publicationDate":"2026-10-26","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (small randomized comparison)","sourceUrl":"https://arxiv.org/abs/2610.36185","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["developers-agents","knowledge-work"],"finding":"TestCompass gives targeted feedback on weaknesses in students' unit-test suites without generating executable tests. Eighteen students recruited mainly from a sophomore computer science course (one from a senior course) were randomly assigned to TestCompass or a general-purpose AI assistant for two Java unit-testing tasks; after one participant used only general AI, 8 TestCompass and 10 general-AI participants were analysed. General-AI users produced more tests, while TestCompass users spent more time developing and verifying tests themselves. Most test-suite quality measures were comparable, and TestCompass users reached similar requirement coverage with significantly fewer test methods in one task (p=0.012). Neither group reported short-term gains in perceived testing knowledge or confidence. The general-AI condition was not standardised because IRB rules let participants use different assistants.","sledRelevance":"Tests the common design choice of restricting a course AI tutor to feedback rather than answers, a lever for preserving practice while allowing AI.","evidence":"Small randomized comparison with behavioural logs, test-suite quality measures and pre/post surveys.","architectureImplications":"Feedback-only tutor constrained from generating executable tests.","governanceImplications":"Supports course policies that steer students to feedback-oriented tools for skill-building tasks.","securityPrivacyImplications":"Not discussed beyond IRB constraints.","caveats":"Very small sample, extra-credit volunteers, fixed task order, heterogeneous control tools and short-term self-reports only.","streamIds":["student-success"],"roles":{"sales":"Sales framing: this small Vanderbilt study shows a feedback-only AI tutor kept students doing their own unit testing while reaching comparable test quality to a general assistant. It supports conversations with computing departments about constrained course tutors for practice-critical skills. A bounded engagement could compare a constrained tutor with general AI in one lab sequence. Do not claim learning gains; neither group reported improved knowledge or confidence and only 18 students took part.","engineering":"Engineering framing: the key design is a tutor that analyses a student's test suite and returns feedback on gaps without producing runnable tests. Check that the constraint holds under prompt pressure, how requirement coverage is computed, and how the tutor integrates with the course IDE or autograder. Proof of value should measure students' own testing time, coverage and defect detection against a general assistant, with standardised comparison tools unlike this study.","delivery":"Delivery framing: delivery would be a lab-level pilot. Work includes choosing assignments where hands-on practice matters, configuring the constrained tutor, explaining to students why it withholds answers, and collecting both behaviour logs and later assessment results. Plan for a larger sample and a delayed, unaided test to see whether preserved practice improves testing skill, since this study found no short-term change in self-reported knowledge."},"roleDetails":{"sales":{"customerProblem":"Computing instructors worry general AI assistants let students skip the practice that builds core skills such as testing.","stakeholders":["Computer science department chair","Introductory course instructors","Teaching assistants","Academic technology team"],"discoveryQuestions":["Which skills do students currently delegate to AI in your labs?","Would a tutor that withholds answers be acceptable to students?","How would you measure retained skill?"],"valueHypothesis":"Interpretation: constrained feedback-only tutors could preserve hands-on practice without lowering output quality.","potentialEngagement":"Lab-sequence comparison of a constrained tutor and general AI with a delayed unaided skills test.","unsupportedClaims":["Improved learning or confidence","Generalisation beyond unit testing","Results from a large sample"]},"engineering":{"fit":"Programming courses with structured lab tasks and autograded test suites.","architectureIntegration":"Tutor that analyses student test suites and returns feedback, integrated with the IDE or autograder.","prerequisites":["Requirement-level task specifications","Coverage measurement tooling","Guardrails preventing code generation"],"constraints":["18-student evidence","Heterogeneous control tools","Short tasks outside class"],"security":"Student code sent to the model should stay within approved services.","proofOfValue":["Student testing time and coverage versus general AI","Guardrail robustness tests","Delayed unaided testing assessment"]},"delivery":{"work":"Pilot the constrained tutor in selected labs, communicate its purpose, and evaluate with logs and a delayed unaided test.","dependencies":["Instructor buy-in","Tool hosting","TA support"],"ownership":"Course instructors with the department.","skillsAdoption":"Student orientation on feedback-only tools; TA guidance on interpreting tutor feedback.","governanceCheckpoints":["Department approval of tool use","Research ethics review if outcomes are studied","Term-end evaluation"],"acceptanceCriteria":["Proposed: comparable coverage to general AI","Proposed: more student-authored tests verified by students","Proposed: delayed unaided test results reported"],"risks":["Student frustration with withheld answers","Students switching to general AI","Small pilots overinterpreted"]}},"retrievedAt":"2026-10-29T03:19:50Z","enrichedAt":"2026-10-29T03:24:23Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Not discussed.","procurementImplications":"Interpretation: course tool selection can favour constrained tutors over general assistants for specific skills.","operatingModelImplications":"Interpretation: instructors may pair general AI for productivity with constrained tutors for practice-critical skills.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.36185","referenceExcerpt":"Neither group reported short-term improvements in perceived testing knowledge or confidence.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}