From the Student Success edition of October 7, 2026
LLM grading matches instructor scores fairly well but misses most misconceptions and stays upbeat on failing answers
Teachers College, Columbia University and New York University (Xi Zhao, Xinyue Jiao, Zhen Xu) · Higher education student success / AI grading and formative feedback · Spain (data from Universidad Francisco de Vitoria computer science courses); U.S.-based authors; transferable with discipline limits
- Publisher
- arXiv (conference paper preprint)
- Original publication
- arXiv preprint v1 submitted 7 October 2026; header cites Proceedings of the 34th International Conference on Computers in Education (2026)
- Source retrieved
- 2026-10-08
What happened
Using the public JorGPT dataset of 3,041 student answers to 50 open-ended computer science questions from 2025-26 assessments, each graded by an instructor and by three commercial LLMs (DeepSeek-chat-V3.2, Qwen-flash, Gemini-2.5-flash-lite) given the question, answer and ideal answer, the authors find total scores correlate with instructors at r = 0.785 to 0.815, but the models differ in bias (DeepSeek strict at -0.55, Qwen lenient at +1.28, Gemini near zero at +0.06 with MAE 1.22). Accuracy varies by topic: testing questions were most reliable while architecture and Spring Boot questions had the largest errors. The four rubric sub-scores are largely redundant (inter-dimension r 0.82 to 0.99, VIF up to 45). Instructors flagged misconceptions in 15.0% of comments and 31.5% for answers graded 0 to 2; the LLMs did so in 4.8% to 7.4% overall and 5% to 7% for low scorers. For low scorers, LLMs included praise in 47% to 57% of comments versus 22.6% for instructors, and LLM sentiment stayed positive while instructor sentiment turned negative.
Why it matters
New this week and directly relevant as universities adopt AI grading and feedback at scale. It moves the question past score agreement to whether feedback diagnoses errors, and finds the gap is largest for the weakest students, who most need to know their answer is wrong.
Evidence and measured results
Secondary analysis of a public, ethics-approved dataset: Pearson r, MAE and signed bias by model and domain with Kruskal-Wallis tests; regression and VIF for sub-dimensions; rule-based semantic role coding and VADER sentiment of comments. Zero-shot prompting only.
Limitations and uncertainty
One institution and discipline (software engineering) in Spain; zero-shot prompts and three lower-cost model versions, so newer models or tuned prompts may differ; rule-based coding and VADER may miss nuance; no measure of how students acted on feedback.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-08; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Institutions want faster grading and feedback, but AI feedback that sounds thorough may not diagnose student errors.
- Stakeholders
- Provost and faculty senate
- Teaching and learning center
- Department chairs of large courses
- Academic technology
- Office of general counsel or privacy
- Discovery questions
- Where is AI grading or feedback already in use, and who reviews it?
- Do you validate AI grades by topic or only overall?
- Can students request a human review of AI-assisted grades?
- Value hypothesis
- AI grading used as a screen in procedural topics, with human review of low scores, may save instructor time without losing diagnosis; requires local validation.
- Potential engagement
- A validation study of a candidate AI grading tool on two courses' historical work, with topic-level accuracy and feedback-quality audits.
- Unsupported claims
- That AI feedback can replace instructor feedback
- That rubric sub-scores are reliable diagnostics
- That results hold for newer or tuned models
Pre-sales engineering
Role takeaway
- Fit
- Courses with open-ended responses, an LMS and access to commercial or hosted models.
- Architecture and integration
- Grading service that receives question, answer and ideal answer, returns scores and comments to an instructor review queue in the LMS, with logs for audits.
- Prerequisites
- Historical instructor-graded work for validation
- Institutional data terms for student work
- Instructor review queue
- Constraints
- Model behavior shifts across versions
- Topic-dependent accuracy
- Single-discipline evidence
- Security
- Send minimal student data, exclude it from vendor training, and retain grading logs as education records.
- Proof of value
- Topic-level MAE and bias against instructor grades
- Misconception-detection rate on low-scoring answers
- Sub-score independence check before display
Delivery
Role takeaway
- Work
- Validate and phase in AI-assisted grading with human review rules and termly feedback audits.
- Dependencies
- Faculty participation in validation
- Historical graded data
- Privacy and records review
- Ownership
- Course instructors own grades; teaching and learning center owns the review protocol; academic technology owns the tool.
- Skills and adoption
- Train instructors to review AI feedback for diagnosis and tone; brief students on AI's role and appeal routes.
- Governance checkpoints
- Validation before student-facing use
- Disclosure in syllabus
- Termly audit of feedback quality
- Acceptance criteria
- Topic-level error within an agreed threshold before use
- All low-scoring AI-graded work reviewed by an instructor
- Human-review request route published to students
- Risks
- Overtrust in structured sub-scores
- Misleadingly positive feedback for struggling students
- Silent model updates changing behavior
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Structured JSON rubric outputs look analytic but behave like one holistic score; systems should not surface sub-scores as independent diagnostics without validation.
Governance
Who approves, reviews and stays accountable for outcomes?
AI grading policies should require human review of feedback for low-scoring work, topic-level validation, and disclosure to students, consistent with campus guidance that keeps instructors accountable for grades.
Security and privacy
What data, permissions and controls need testing?
The study used anonymized public data; production grading sends student work to commercial models, which needs institutional terms and records protections.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Uniformly positive feedback may mislead struggling students about how far off they are; instructor time is still needed where diagnosis matters.
Procurement
What should contracts, pricing and exit terms secure?
Ask vendors for misconception-detection and tone-calibration evidence by topic, not only score correlation; model choice changed bias substantially.
Operating model
Which teams own the service once it runs?
Use LLM grading as a screen in procedural topics, keep humans on low-scoring and judgment-heavy work, and audit feedback content each term.
Publication history
- 2026-10-07Student Success · Issue 314 resources
Stable resource ID: zhao-tc-columbia-llm-grading-diagnostic-quality-jorgpt-20261007