{"resourceId":"zhao-tc-columbia-llm-grading-diagnostic-quality-jorgpt-20261007","versions":[{"version":"external-b7d39d9830d8a58e5297ffee9049b2f3e8859c4dc0550f879ade8deee973973c","resource":{"id":"zhao-tc-columbia-llm-grading-diagnostic-quality-jorgpt-20261007","title":"LLM grading matches instructor scores fairly well but misses most misconceptions and stays upbeat on failing answers","organization":"Teachers College, Columbia University and New York University (Xi Zhao, Xinyue Jiao, Zhen Xu)","sector":"Higher education student success / AI grading and formative feedback","geography":"Spain (data from Universidad Francisco de Vitoria computer science courses); U.S.-based authors; transferable with discipline limits","publishedAt":"arXiv preprint v1 submitted 7 October 2026; header cites Proceedings of the 34th International Conference on Computers in Education (2026)","publicationDate":"2026-10-07","eventDate":null,"sourceName":"arXiv (conference paper preprint)","sourceLabel":"Academic conference paper, secondary analysis of a public dataset of instructor and LLM grades","sourceUrl":"https://arxiv.org/abs/2610.09460","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement","operating-model"],"finding":"Using the public JorGPT dataset of 3,041 student answers to 50 open-ended computer science questions from 2025-26 assessments, each graded by an instructor and by three commercial LLMs (DeepSeek-chat-V3.2, Qwen-flash, Gemini-2.5-flash-lite) given the question, answer and ideal answer, the authors find total scores correlate with instructors at r = 0.785 to 0.815, but the models differ in bias (DeepSeek strict at -0.55, Qwen lenient at +1.28, Gemini near zero at +0.06 with MAE 1.22). Accuracy varies by topic: testing questions were most reliable while architecture and Spring Boot questions had the largest errors. The four rubric sub-scores are largely redundant (inter-dimension r 0.82 to 0.99, VIF up to 45). Instructors flagged misconceptions in 15.0% of comments and 31.5% for answers graded 0 to 2; the LLMs did so in 4.8% to 7.4% overall and 5% to 7% for low scorers. For low scorers, LLMs included praise in 47% to 57% of comments versus 22.6% for instructors, and LLM sentiment stayed positive while instructor sentiment turned negative.","sledRelevance":"New this week and directly relevant as universities adopt AI grading and feedback at scale. It moves the question past score agreement to whether feedback diagnoses errors, and finds the gap is largest for the weakest students, who most need to know their answer is wrong.","evidence":"Secondary analysis of a public, ethics-approved dataset: Pearson r, MAE and signed bias by model and domain with Kruskal-Wallis tests; regression and VIF for sub-dimensions; rule-based semantic role coding and VADER sentiment of comments. Zero-shot prompting only.","architectureImplications":"Structured JSON rubric outputs look analytic but behave like one holistic score; systems should not surface sub-scores as independent diagnostics without validation.","governanceImplications":"Interpretation: AI grading policies should require human review of feedback for low-scoring work, topic-level validation, and disclosure to students, consistent with campus guidance that keeps instructors accountable for grades.","securityPrivacyImplications":"The study used anonymized public data; production grading sends student work to commercial models, which needs institutional terms and records protections.","caveats":"One institution and discipline (software engineering) in Spain; zero-shot prompts and three lower-cost model versions, so newer models or tuned prompts may differ; rule-based coding and VADER may miss nuance; no measure of how students acted on feedback.","streamIds":["student-success"],"roles":{"sales":"The framing point is that score agreement is not the same as useful feedback. This study found three commercial models tracked instructor grades reasonably well but caught far fewer misconceptions and stayed positive on failing answers. Use it to open conversations about governed AI grading with human review where diagnosis matters. Keep it bounded: one software engineering dataset, zero-shot prompts and specific model versions; it does not show that any product fails or succeeds.","engineering":"For engineering, the framing point is validation beyond correlation. Test candidate grading models on your own rubric by topic, measure bias direction, and check whether sub-scores add independent information before showing them to students. Add checks for misconception detection and tone on low-scoring answers, since that is where models diverged most. Keep instructor overrides and logs, and treat prompt or model changes as a reason to re-validate.","delivery":"Delivery's framing point is placing humans where models are weakest. Start AI grading in procedural topics, route low-scoring and design-judgment answers to instructors, and sample feedback each term for misconception coverage and tone. Tell students how AI is used in grading and how to request human review. Measure whether students revise successfully after AI feedback, which this study did not examine."},"roleDetails":{"sales":{"customerProblem":"Institutions want faster grading and feedback, but AI feedback that sounds thorough may not diagnose student errors.","stakeholders":["Provost and faculty senate","Teaching and learning center","Department chairs of large courses","Academic technology","Office of general counsel or privacy"],"discoveryQuestions":["Where is AI grading or feedback already in use, and who reviews it?","Do you validate AI grades by topic or only overall?","Can students request a human review of AI-assisted grades?"],"valueHypothesis":"AI grading used as a screen in procedural topics, with human review of low scores, may save instructor time without losing diagnosis; requires local validation.","potentialEngagement":"A validation study of a candidate AI grading tool on two courses' historical work, with topic-level accuracy and feedback-quality audits.","unsupportedClaims":["That AI feedback can replace instructor feedback","That rubric sub-scores are reliable diagnostics","That results hold for newer or tuned models"]},"engineering":{"fit":"Courses with open-ended responses, an LMS and access to commercial or hosted models.","architectureIntegration":"Grading service that receives question, answer and ideal answer, returns scores and comments to an instructor review queue in the LMS, with logs for audits.","prerequisites":["Historical instructor-graded work for validation","Institutional data terms for student work","Instructor review queue"],"constraints":["Model behavior shifts across versions","Topic-dependent accuracy","Single-discipline evidence"],"security":"Send minimal student data, exclude it from vendor training, and retain grading logs as education records.","proofOfValue":["Topic-level MAE and bias against instructor grades","Misconception-detection rate on low-scoring answers","Sub-score independence check before display"]},"delivery":{"work":"Validate and phase in AI-assisted grading with human review rules and termly feedback audits.","dependencies":["Faculty participation in validation","Historical graded data","Privacy and records review"],"ownership":"Course instructors own grades; teaching and learning center owns the review protocol; academic technology owns the tool.","skillsAdoption":"Train instructors to review AI feedback for diagnosis and tone; brief students on AI's role and appeal routes.","governanceCheckpoints":["Validation before student-facing use","Disclosure in syllabus","Termly audit of feedback quality"],"acceptanceCriteria":["Topic-level error within an agreed threshold before use","All low-scoring AI-graded work reviewed by an instructor","Human-review request route published to students"],"risks":["Overtrust in structured sub-scores","Misleadingly positive feedback for struggling students","Silent model updates changing behavior"]}},"retrievedAt":"2026-10-08T03:21:13Z","enrichedAt":"2026-10-08T03:25:13Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Uniformly positive feedback may mislead struggling students about how far off they are; instructor time is still needed where diagnosis matters.","procurementImplications":"Ask vendors for misconception-detection and tone-calibration evidence by topic, not only score correlation; model choice changed bias substantially.","operatingModelImplications":"Use LLM grading as a screen in procedural topics, keep humans on low-scoring and judgment-heavy work, and audit feedback content each term.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.09460","referenceExcerpt":"The feedback tone remains uniformly positive regardless of response quality, lacking the severity modulation observed in human feedback.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}