{"resourceId":"chen-melbourne-genai-practice-feedback-20261001","versions":[{"version":"external-cdba245d4f9edcf9915bbc09f2db17a08b2a1ffe3df553ced74128f9340afcf3","resource":{"id":"chen-melbourne-genai-practice-feedback-20261001","title":"Melbourne/Monash GenAI practice platform: human-upstream answer verification; users prefer immediate imperfect feedback","organization":"University of Melbourne & Monash University (Chen, Buskes, Ren, Leong)","sector":"Higher education engineering teaching / formative feedback","geography":"Australia (University of Melbourne / Monash; large electrical engineering probability subject)","publishedAt":"Submitted to arXiv 1 October 2026 (AAEE 2026 conference paper)","publicationDate":"2026-10-01","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint / conference paper","sourceUrl":"https://arxiv.org/pdf/2610.01262","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["knowledge-work","operating-model","developers-agents","accessibility-workforce"],"finding":"Pilot of a GenAI practice platform that combines a demonstrator-verified question bank, AI-generated exam-style variants, and an iterative attempt–feedback–revise loop in a core electrical-engineering probability subject. Design relocates human oversight upstream to answer verification rather than real-time grading. Of 95 registrants from 157 enrolled, 34 attempted at least one question (230 submissions; median three, max 40). End-of-semester survey (n=53): all five constructs above neutral midpoint; engagement/active learning highest (3.97/5; 79% agreement); lowest for preferring the system over the existing problem booklet (42%). Platform users were more convinced than non-users that immediate imperfect AI feedback is more useful than delayed demonstrator feedback (79% vs 59%). Progress-test scores for a subset illustrate trajectories only—not causal learning effects.","sledRelevance":"Complementary HE design pattern to access-only tutors: scale formative feedback by verifying solutions up front and keeping GenAI in practice loops. Useful for student-success/teaching teams facing large-class feedback delays, but evidence is engagement/perception—not an RCT on grades.","evidence":"Platform analytics (95/34/230), survey of five constructs plus open responses, illustrative progress-test trajectories. No randomized control for learning impact; preprint/conference paper.","architectureImplications":"Separate verified answer store from generative variant/feedback generation; enforce attempt–feedback–revise before revealing solutions; keep humans out of real-time grading path for formative practice.","governanceImplications":"Do not use this design for high-stakes grading without additional assurance. Disclose that feedback can be imperfect; prefer formative practice scopes with verified keys.","securityPrivacyImplications":"Practice attempts and AI feedback logs are education records; minimize retention and avoid feeding them into misconduct systems.","caveats":"Voluntary uptake; survey self-report; no causal learning estimate; single Australian engineering subject; preprint.","streamIds":["student-success"],"roles":{"sales":"For engineering teaching leads drowning in delayed demonstrator feedback, this Melbourne/Monash pilot offers a bounded pattern: verify answers upstream, let GenAI run formative attempt–feedback–revise loops, and expect users to value immediacy (79%) more than non-users (59%)—with only 34 of 95 registrants actually practicing. Sell a formative practice build with verified keys, not graded AI marking. Bound claims to perception/engagement evidence and refuse guaranteed exam gains without a controlled study.","engineering":"Engineering should implement a verified-question store, AI variant generation with feed-forward prompts, and an attempt–feedback–revise state machine that never auto-grades summative work. Separate formative logs from assessment systems. Constraints include voluntary uptake and no RCT learning proof in the source. Proof-of-value is active-attempt rate, revision loops completed, and later controlled learning checks—not registration counts. Keep practice transcripts purpose-limited.","delivery":"Delivery should stand up the platform mid-semester in one large STEM subject, staff key-verification sprints, and survey users versus non-users on immediacy value. Ownership sits with the subject coordinator and teaching support. Train demonstrators to maintain the verified bank rather than chase every stuck student in real time. Acceptance means verified keys for released items and formative-only scope. Risks include low active use and students treating imperfect feedback as authoritative."},"roleDetails":{"sales":{"customerProblem":"Large STEM classes cannot deliver timely formative feedback, and GenAI grading raises trust and accountability concerns.","stakeholders":["engineering faculty","teaching & learning","academic technology","student success","demonstrators/TAs"],"discoveryQuestions":["Which items will humans verify before AI feedback goes live?","Is the tool formative-only or grading assessed work?","How will we measure active practice versus registration?"],"valueHypothesis":"Upstream human verification plus GenAI formative loops can raise perceived usefulness of immediate feedback without claiming graded-assessment automation.","potentialEngagement":"One-subject mid-semester formative pilot with verified bank, analytics, and user/non-user survey.","unsupportedClaims":["Causal exam-score gains","Replacement of demonstrators","Safe auto-grading of high-stakes work"]},"engineering":{"fit":"Formative GenAI practice platforms for large undergraduate engineering subjects with verified solution keys.","architectureIntegration":"Verified answer bank → AI variants → attempt–feedback–revise loop → analytics; human path is key verification, not live grading.","prerequisites":["demonstrator-verified items","LMS/SSO registration","formative-only policy"],"constraints":["voluntary uptake","imperfect AI feedback","no RCT learning evidence in source"],"security":"Attempt/feedback logs as education records; isolate from misconduct tooling.","proofOfValue":["Active-attempt rate among registrants","Revision-loop completion","Later controlled learning evaluation"]},"delivery":{"work":"Build verified bank, configure prompts, launch mid-semester, monitor submissions, survey users/non-users, iterate feed-forward prompts.","dependencies":["subject faculty","demonstrators for keys","IT hosting"],"ownership":"Subject coordinator with teaching-support engineering.","skillsAdoption":"Demonstrators verify keys and tune prompts; students practice attempt-then-revise habits.","governanceCheckpoints":["formative-scope signoff","key-verification complete before release","end-term claims audit"],"acceptanceCriteria":["No summative auto-grade","Verified keys for released items","Active-use metrics reported"],"risks":["Low active use","Overtrust in imperfect feedback","Scope creep into grading"]}},"retrievedAt":"2026-10-02T03:21:48Z","enrichedAt":"2026-10-02T03:28:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Immediate feedback may help stuck learners in large classes, but 34/95 active use and 42% preference versus booklet show access is not adoption. Demonstrators still needed for verified keys and edge cases.","procurementImplications":"Require demonstrator-verified item banks, exportable attempt logs, and formative-only deployment clauses—not auto-grading of assessed work.","operatingModelImplications":"Staff time shifts to upstream key verification and prompt/feed-forward design; monitor active-use rates weekly rather than registration counts.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2610.01262","referenceExcerpt":"keeping the human in the loop upstream at answer verification","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}