{"resourceId":"thinkpol-fong-chatgpt-feedback-ranking-diverges-students-reject-ai-marking-20261031","versions":[{"version":"external-cc49cff7e69d21e8625615a60431ddf66c8c339608d49b341958b3bf8ea9ab68","resource":{"id":"thinkpol-fong-chatgpt-feedback-ranking-diverges-students-reject-ai-marking-20261031","title":"Study reported by ThinkPol: ChatGPT ranks essay feedback differently from students (rank correlation 0.34), and undergraduates in focus groups mostly oppose instructors using AI to mark their work","organization":"Fong et al. (Computers and Education Open), reported by ThinkPol","sector":"Higher education assessment feedback and grading","geography":"United States (large public university in the Southwest; undergraduate samples)","publishedAt":"ThinkPol, 31 October 2026, reporting a Computers and Education Open article published in October 2026","publicationDate":"2026-10-31","eventDate":null,"sourceName":"ThinkPol","sourceLabel":"News report on a peer-reviewed journal study","sourceUrl":"https://thinkpol.ca/2026/10/31/students-push-back-on-ai-marking-as-study-finds-chatgpt-judges-feedback-differently/","evidenceClass":"independent-reporting","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement"],"finding":"ThinkPol (31 October 2026) reports a Computers and Education Open study by Carlton J. Fong and co-authors combining two studies of US undergraduates. In the first, 392 students compared 45 instructor-written feedback comments in pairs to produce a ranking, and ChatGPT (GPT-3.5) ranked the same comments; the rank correlation was 0.34. Specificity explained most of the students' ranking, while none of the measured features explained ChatGPT's. In the second, five focus groups with 20 undergraduates generally could not tell AI feedback from an instructor's, and most disapproved of AI marking, citing connection, expertise and fairness. Most accepted AI for grammar checks and a hybrid where instructors review AI drafts, and wanted disclosure.","sledRelevance":"Feedback quality and trust in grading are core to persistence and learning; the study warns that AI-generated marking may not match what students find useful and may erode trust if undisclosed.","evidence":"Two-part mixed-methods study as summarised by a news outlet: a ranking comparison and qualitative focus groups.","architectureImplications":"Interpretation: any AI feedback tool should keep the instructor in review and record when AI drafted feedback.","governanceImplications":"Students wanted to be told when AI is involved in feedback; the authors caution against quietly using AI feedback.","securityPrivacyImplications":"Not addressed in the report; sending student work to AI marking tools raises data handling questions that need institutional review.","caveats":"Original article not opened because of a captcha; older model; single-session ranking; small single-site focus groups on a hypothetical scenario.","streamIds":["student-success"],"roles":{"sales":"Sales framing: this study speaks to deans and teaching centres weighing AI-assisted marking to ease grading load. Students in the focus groups mostly opposed AI marking, wanted disclosure and accepted a hybrid where instructors check AI drafts. A bounded engagement could design a disclosed hybrid feedback pilot with student input. Do not claim AI feedback is as good as instructor feedback; the ranking test found only partial agreement with students, and the original paper was not available for review.","engineering":"Engineering framing: technical fit is an instructor-in-the-loop feedback workflow, not automated grading. An AI drafting step inside the LMS would need clear labelling of AI-drafted comments, instructor edit and approval, and logging. Check whether student submissions leave institutional control and how vendors retain them. Proof of value should compare student ratings of AI-drafted, instructor-edited feedback against instructor-only feedback on specificity, since specificity drove student judgments in this study, using current models rather than GPT-3.5.","delivery":"Delivery framing: delivery means setting rules and a pilot for AI-assisted feedback that keeps the instructor relationship visible. Work includes a disclosure statement, limits on which assignments may use AI drafting, and a student feedback survey. Teaching and learning centres should own the guidance, with academic integrity offices aligned so rules for staff and students feel fair. Track student trust and perceived usefulness. The main risk is quiet AI marking on major projects, which students in the study called hypocritical."},"roleDetails":{"sales":{"customerProblem":"Faculty want relief from marking load, but students distrust AI marking and may not find AI-judged feedback useful.","stakeholders":["Dean of undergraduate education","Teaching and learning center","Faculty senate","Academic integrity office","Student government"],"discoveryQuestions":["Are any instructors already using AI to draft feedback?","Do we have a disclosure rule for staff AI use in grading?","Which assignments would students accept AI help on?"],"valueHypothesis":"Interpretation: a disclosed hybrid feedback model may save instructor time while keeping student trust.","potentialEngagement":"Design of a disclosed AI-drafted feedback pilot in a few large courses, with student ratings of usefulness.","unsupportedClaims":["AI feedback matches instructor quality","Students accept AI marking","Results generalise to current models","Time savings of a specific size"]},"engineering":{"fit":"Instructor-reviewed AI drafting of feedback in the LMS; not autonomous grading.","architectureIntegration":"AI drafting step inside the LMS grading view with labelled AI comments, instructor edit and approval, and audit logs.","prerequisites":["LMS integration for feedback drafting","Labelling of AI-drafted comments","Institution-approved model with data protections"],"constraints":["Single-site evidence","Older model in the ranking test","Original paper not reviewed"],"security":"Student submissions must stay under institutional data agreements with no vendor training on student work.","proofOfValue":["Student ratings of feedback specificity for AI-drafted versus instructor-only comments","Instructor edit rates on AI drafts","Disclosure shown on every AI-assisted comment"]},"delivery":{"work":"Set disclosure rules and run a hybrid AI-drafted, instructor-edited feedback pilot with student evaluation.","dependencies":["Faculty volunteers in large courses","Approved AI tool","Student survey instrument"],"ownership":"Teaching and learning center, with academic affairs and the academic integrity office.","skillsAdoption":"Instructors need practice editing AI drafts for specificity and explaining their AI use to students.","governanceCheckpoints":["Disclosure policy approval","Exclusion list for high-stakes work","End-of-term student trust review"],"acceptanceCriteria":["Proposed: 100% of AI-assisted feedback carries a disclosure label","Proposed: student usefulness ratings no lower than instructor-only feedback","Proposed: no AI drafting on major projects without opt-in"],"risks":["Undisclosed AI marking","Generic feedback that lacks specificity","Perceived double standard between staff and students"]}},"retrievedAt":"2026-11-01T03:18:58Z","enrichedAt":"2026-11-01T03:24:15Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Instructors teaching large classes face marking load; students accepted AI help mainly for low-stakes and grammar tasks.","procurementImplications":"Interpretation: AI grading features should be evaluated against student judgments of feedback usefulness, not speed alone.","operatingModelImplications":"Interpretation: a disclosed hybrid model, where AI drafts and instructors edit, matches what students in the focus groups would accept.","peerReview":"unknown","sourceVerification":{"openedUrl":"https://thinkpol.ca/2026/10/31/students-push-back-on-ai-marking-as-study-finds-chatgpt-judges-feedback-differently/","referenceExcerpt":"The rank correlation between them was 0.34, meaning they shared about 11 per cent of their variation.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}