From the Student Success edition of October 28, 2026
Preprint: Stony Brook pilot of Edhivo, a two-model LLM platform that generates and grades assessments with instructor review, reports faster feedback and positive student survey responses in one 36-student course
Stony Brook University (authors and pilot course AMS 691.01) · Higher education assessment and feedback · Stony Brook, New York, United States
- Publisher
- arXiv
- Original publication
- arXiv preprint 2610.36654, submitted 27 October 2026
- Source retrieved
- 2026-10-29
What happened
Edhivo uses one LLM to generate multiple-choice and open-ended assessments from instructor materials and a second to grade open responses against rubrics, with instructor review before release. In spring 2026 it ran in a Stony Brook graduate course on large language models (36 students), mainly for short formative pre- and post-quizzes around student presentations, about three per 80-minute session, which did not count toward grades. The authors estimate authoring time fell from about 30 to about 5 minutes per assessment including review, and that grading plus human audit would save about 8.3 hours for a 200-student open-ended assessment; they note these are informal single-instructor estimates. Of 30 survey respondents (83.3%), all agreed the interface was easy and that instant feedback improved their learning experience. Limitations listed include no proctoring safeguards, possible bias favouring LLM-drafted answers, STEM-only scope, and a pending comparison of AI and instructor scores.
Why it matters
AI-generated formative assessment and instant feedback are a common student-success proposition; this pilot shows workflow gains alongside unvalidated grading accuracy.
Evidence and measured results
Single-course pilot with timing estimates, unit-cost table and post-pilot Likert survey (30 of 36).
Limitations and uncertainty
Builder-authored preprint; graduate STEM cohort; formative only; survey measures perception, not learning; reliability data pending.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-29; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Instructors cannot author and grade enough low-stakes assessments to give students frequent, timely feedback.
- Stakeholders
- Teaching and learning center
- Large-course instructors
- Department chairs
- Academic technology team
- Discovery questions
- How quickly do students receive feedback on open-ended work today?
- Would instructors audit AI-generated marks?
- Which courses could start with ungraded formative quizzes?
- Value hypothesis
- Interpretation: AI-assisted generation and grading with instructor audit could increase formative feedback frequency at low model cost.
- Potential engagement
- One-course pilot of AI-assisted formative quizzes with an AI-versus-instructor scoring audit.
- Unsupported claims
- Learning gains
- Grading accuracy equal to instructors
- Fitness for high-stakes or non-STEM assessment
Pre-sales engineering
Role takeaway
- Fit
- LMS-adjacent tools for formative quizzes in STEM courses with clear reference answers.
- Architecture and integration
- Generation and evaluation LLMs grounded in instructor materials, with an instructor review queue and analytics dashboard.
- Prerequisites
- Instructor-provided materials and rubrics
- Approved model hosting
- Review workflow for instructors
- Constraints
- No proctoring safeguards
- Possible bias toward LLM-styled writing
- Single graduate STEM course evidence
- Security
- Route student responses only through approved model endpoints with retention limits.
- Proof of value
- AI-instructor score agreement on a sample
- Measured audit minutes per student
- Bias check across writing styles
Delivery
Role takeaway
- Work
- Run a formative-only pilot, train instructors on auditing, and report agreement, time and student feedback.
- Dependencies
- Instructor participation
- Academic technology support
- Model hosting approval
- Ownership
- Course instructors with the teaching and learning center.
- Skills and adoption
- Instructor training on reviewing AI marks and writing rubrics for automated evaluation.
- Governance checkpoints
- Approval for processing student responses
- Agreement review before graded use
- Term-end pilot review
- Acceptance criteria
- Proposed: AI-instructor agreement reported on a sampled set
- Proposed: audit time measured, not estimated
- Proposed: no graded use until agreement meets an agreed threshold
- Risks
- Unnoticed grading errors
- Bias against non-standard writing
- Overreliance on automated feedback
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Dual-LLM pipeline with source-grounded generation, rubric-based evaluation, human-in-the-loop audit and instructor dashboard; low per-student model cost via a hosted open model.
Governance
Who approves, reviews and stays accountable for outcomes?
Instructor review is the safeguard while AI-instructor score agreement is unvalidated.
Security and privacy
What data, permissions and controls need testing?
Student responses are sent to a hosted model; data handling not detailed.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Authors warn the grader may favour LLM-styled writing, potentially disadvantaging students with non-standard writing styles.
Procurement
What should contracts, pricing and exit terms secure?
Per-student grading cost reported as $0.00275 for 10 questions at stated model pricing; production costs would differ.
Operating model
Which teams own the service once it runs?
Frequent low-stakes assessment becomes feasible, but instructors must budget audit time.
Publication history
- 2026-10-28Student Success · Issue 514 resources
Stable resource ID: arxiv-edhivo-stony-brook-llm-assessment-feedback-pilot-36-students-20261027