Programming tutor comparison finds stronger feedback uptake but mixed correct application

Computing departments may know that students submit passing code without knowing whether automated feedback helped them reason. Include course faculty, learning technology staff, institutional research and students. Ask which concepts remain difficult after a correction, how much feedback review costs and what would justify changing the tutor. A bounded engagement could evaluate one assignment sequence using a current approved tool. The value hypothesis is better diagnosis of unusable or misleading feedback, subject to local testing. Do not convert submission volume into student reach, feedback uptake into mastery, or this comparison into promised retention gains. The source supports evaluation design, not a procurement winner or staffing reduction.

Fit the evaluation to a sandboxed programming environment with stable assignment identifiers and versioned feedback. Prerequisites include an educator-authored rubric, lawful access to revision data and an independent test task. Keep execution separate from model inference and deny access to production secrets or student-record writes. Sample judge disagreements for human adjudication, including large rewrites and partially adopted advice.

Proposed proof of value

compare tutor configurations within the same cohort, report correctness and answer leakage separately, and assess an unaided transfer task. Validate latency and recovery under course deadlines. The source's observational comparison cannot establish that changing prompts alone will improve the local course.

The course lead should own instructional decisions, with an evaluator owning the analysis and IT owning service reliability. Prepare a data map, review code-handling permissions and train teaching assistants to identify harmful feedback. Dependencies include accessible coding tools, marking capacity and a manual support route.

Proposed acceptance criteria

every reported endpoint has a reconciled denominator; sampled automated scores receive human review; critical feedback errors have an owner; and independent transfer results are reviewed before expansion. These are proposed gates. Track support effort and voluntary participation as well as usage. Risks include treating repeated submissions as independent learners, review fatigue and optimizing for passing tests instead of understanding.