{"resourceId":"arxiv-edhivo-stony-brook-llm-assessment-feedback-pilot-36-students-20261027","versions":[{"version":"external-9efe1d6239561360e186f25a6f540a261e3cbd53cb12b74abf170c93ab063d35","resource":{"id":"arxiv-edhivo-stony-brook-llm-assessment-feedback-pilot-36-students-20261027","title":"Preprint: Stony Brook pilot of Edhivo, a two-model LLM platform that generates and grades assessments with instructor review, reports faster feedback and positive student survey responses in one 36-student course","organization":"Stony Brook University (authors and pilot course AMS 691.01)","sector":"Higher education assessment and feedback","geography":"Stony Brook, New York, United States","publishedAt":"arXiv preprint 2610.36654, submitted 27 October 2026","publicationDate":"2026-10-27","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (single-course pilot deployment)","sourceUrl":"https://arxiv.org/abs/2610.36654","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["knowledge-work","developers-agents"],"finding":"Edhivo uses one LLM to generate multiple-choice and open-ended assessments from instructor materials and a second to grade open responses against rubrics, with instructor review before release. In spring 2026 it ran in a Stony Brook graduate course on large language models (36 students), mainly for short formative pre- and post-quizzes around student presentations, about three per 80-minute session, which did not count toward grades. The authors estimate authoring time fell from about 30 to about 5 minutes per assessment including review, and that grading plus human audit would save about 8.3 hours for a 200-student open-ended assessment; they note these are informal single-instructor estimates. Of 30 survey respondents (83.3%), all agreed the interface was easy and that instant feedback improved their learning experience. Limitations listed include no proctoring safeguards, possible bias favouring LLM-drafted answers, STEM-only scope, and a pending comparison of AI and instructor scores.","sledRelevance":"AI-generated formative assessment and instant feedback are a common student-success proposition; this pilot shows workflow gains alongside unvalidated grading accuracy.","evidence":"Single-course pilot with timing estimates, unit-cost table and post-pilot Likert survey (30 of 36).","architectureImplications":"Dual-LLM pipeline with source-grounded generation, rubric-based evaluation, human-in-the-loop audit and instructor dashboard; low per-student model cost via a hosted open model.","governanceImplications":"Instructor review is the safeguard while AI-instructor score agreement is unvalidated.","securityPrivacyImplications":"Student responses are sent to a hosted model; data handling not detailed.","caveats":"Builder-authored preprint; graduate STEM cohort; formative only; survey measures perception, not learning; reliability data pending.","streamIds":["student-success"],"roles":{"sales":"Sales framing: this Stony Brook preprint shows AI-generated quizzes and rubric-based grading can make frequent formative feedback practical, with instructor review kept in the loop. It fits conversations with teaching centers and large-course instructors who want more low-stakes checks. A bounded engagement could pilot AI-assisted formative quizzes in one course with an accuracy audit. Do not claim learning gains or grading accuracy; the pilot covered 36 graduate students and score validation is still pending.","engineering":"Engineering framing: the design separates generation and evaluation models, grounds both in instructor materials, and adds a human audit before results reach students. Check how open-ended grading handles non-standard writing, whether the model favours LLM-styled answers as the authors warn, and where student responses are processed. Proof of value should compare AI and instructor scores on a sample, measure audit time honestly, and test behaviour on humanities-style questions before extending scope.","delivery":"Delivery framing: delivery is a course-level pilot with clear guardrails. Start with formative, ungraded quizzes as Stony Brook did, train instructors to audit AI marks efficiently, and keep high-stakes assessment out of scope given the missing proctoring safeguards. Track instructor time, agreement between AI and instructor scores, and student feedback, and extend to graded or larger courses only after agreement is acceptable across student writing styles."},"roleDetails":{"sales":{"customerProblem":"Instructors cannot author and grade enough low-stakes assessments to give students frequent, timely feedback.","stakeholders":["Teaching and learning center","Large-course instructors","Department chairs","Academic technology team"],"discoveryQuestions":["How quickly do students receive feedback on open-ended work today?","Would instructors audit AI-generated marks?","Which courses could start with ungraded formative quizzes?"],"valueHypothesis":"Interpretation: AI-assisted generation and grading with instructor audit could increase formative feedback frequency at low model cost.","potentialEngagement":"One-course pilot of AI-assisted formative quizzes with an AI-versus-instructor scoring audit.","unsupportedClaims":["Learning gains","Grading accuracy equal to instructors","Fitness for high-stakes or non-STEM assessment"]},"engineering":{"fit":"LMS-adjacent tools for formative quizzes in STEM courses with clear reference answers.","architectureIntegration":"Generation and evaluation LLMs grounded in instructor materials, with an instructor review queue and analytics dashboard.","prerequisites":["Instructor-provided materials and rubrics","Approved model hosting","Review workflow for instructors"],"constraints":["No proctoring safeguards","Possible bias toward LLM-styled writing","Single graduate STEM course evidence"],"security":"Route student responses only through approved model endpoints with retention limits.","proofOfValue":["AI-instructor score agreement on a sample","Measured audit minutes per student","Bias check across writing styles"]},"delivery":{"work":"Run a formative-only pilot, train instructors on auditing, and report agreement, time and student feedback.","dependencies":["Instructor participation","Academic technology support","Model hosting approval"],"ownership":"Course instructors with the teaching and learning center.","skillsAdoption":"Instructor training on reviewing AI marks and writing rubrics for automated evaluation.","governanceCheckpoints":["Approval for processing student responses","Agreement review before graded use","Term-end pilot review"],"acceptanceCriteria":["Proposed: AI-instructor agreement reported on a sampled set","Proposed: audit time measured, not estimated","Proposed: no graded use until agreement meets an agreed threshold"],"risks":["Unnoticed grading errors","Bias against non-standard writing","Overreliance on automated feedback"]}},"retrievedAt":"2026-10-29T03:19:50Z","enrichedAt":"2026-10-29T03:24:23Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Authors warn the grader may favour LLM-styled writing, potentially disadvantaging students with non-standard writing styles.","procurementImplications":"Per-student grading cost reported as $0.00275 for 10 questions at stated model pricing; production costs would differ.","operatingModelImplications":"Interpretation: frequent low-stakes assessment becomes feasible, but instructors must budget audit time.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.36654","referenceExcerpt":"where it supports high-frequency assessment creation and evaluation for a cohort of 36 students throughout a full semester","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}