{"resourceId":"riazi-educator-guided-llm-erd-scaffolded-feedback-20261001","versions":[{"version":"external-a1a8cc3036f7e13952d73f1a33a88c5b51110933fea6d495d96cfee6736750a7","resource":{"id":"riazi-educator-guided-llm-erd-scaffolded-feedback-20261001","title":"UIC educator-guided ERD agent: staged disclosure yields 71.1% hidden-target incorporation among observed revisions","organization":"University of Illinois Chicago (Riazi, Rooshenas)","sector":"Higher education computing teaching / formative GenAI feedback","geography":"United States (University of Illinois Chicago; online mixed-level Database Systems)","publishedAt":"Submitted to arXiv 1 October 2026 (cs.AI); SIGCSE TS 2027 submission","publicationDate":"2026-10-01","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (classroom deployment)","sourceUrl":"https://arxiv.org/pdf/2610.00870","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["knowledge-work","developers-agents","operating-model","governance-procurement"],"finding":"Educator-guided LLM agent for conceptual ERD feedback separates hidden artifact-grounded diagnosis from a four-stage learner-controlled disclosure workflow. Summer 2026 deployment: 383 feedback episodes; among error-positive episodes with an observed target-level ERD change, 71.1% fully or partially incorporated the hidden diagnostic target, including many after Stages 1–2 before explicit correction. Qualitative cases show staged disclosure can withhold inaccurate details or enable selective uptake, though incorrect targets can still be incorporated. Survey (self-selected ≤30 users): 64.3% felt they could make their own design decisions; 39.3% found feedback too indirect; 42.9% reported repetition.","sledRelevance":"Direct design answer to yesterday's DBWorkout actionable-feedback gap and Engage-to-Unlock productive friction: educator-set disclosure stages—not unrestricted LLM tutoring—can drive diagram revisions while preserving student authorship. Relevant for HE CS teaching, writing/STEM formative tools, and GenAI literacy policies.","evidence":"Classroom deployment logs + LLM incorporation classifier checked on 20-trajectory sample; optional survey. Not RCT; incorporation ≠ diagnosis correctness; preprint.","architectureImplications":"Separate diagnosis service from disclosure controller; version ERD states per episode; educator-authored question pools/templates for early stages; credit/stage telemetry.","governanceImplications":"Require educator-defined disclosure policies before campus LLM tutors may reveal corrections; log stage, context, and disclosed content for audit; keep AI formative-only when TAs grade independently.","securityPrivacyImplications":"ERD artifacts, diagnoses, and chat are education records; purpose-limit logs; de-identify optional surveys; control who can view hidden diagnoses.","caveats":"Single course; no causal learning RCT; missing later ERDs for many episodes; self-selected survey; GPT-4o diagnosis error possible; preprint.","streamIds":["student-success"],"roles":{"sales":"For CS teaching leads reacting to frictionless tutor harms and weak actionable feedback, UIC's educator-guided ERD agent shows staged disclosure can drive revisions: among observed target-level changes in 383 episodes, 71.1% incorporated the hidden diagnostic target—often after concept and application stages alone. Sell a formative scaffold with educator-set disclosure stages and revision telemetry, not unlimited answer chat. Bound claims to this single-course deployment and refuse guaranteed exam gains.","engineering":"Engineering should separate hidden diagnosis from a disclosure controller, version artifact states per episode, ground early stages in educator-authored pools/templates, and log stage, credits, and disclosed content. Keep generative correction behind later stages. Constraints include no learning RCT and possible diagnosis error. Proof-of-value is rising target-aligned revisions after early stages without premature solution dumps—not chat counts. Treat ERDs and diagnoses as education records with audit logs.","delivery":"Delivery should pilot in one database/design course, train faculty on stage intents, keep AI formative while humans grade, and review stage-exit and revision metrics weekly. Ownership sits with the course instructor and academic-technology jointly. Acceptance means documented disclosure rules, exportable episode logs, and no unreviewed solution dumps in early stages. Risks include students finding the system too indirect and mistaking incorporation for learning gains."},"roleDetails":{"sales":{"customerProblem":"Large HE design courses lack timely formative feedback without over-revealing solutions via unrestricted GenAI tutors.","stakeholders":["CS faculty","teaching & learning","academic technology","TAs","student success"],"discoveryQuestions":["Can faculty set disclosure stages for GenAI feedback?","Do we measure revision after early-stage scaffolds?","Is AI formative-only while humans grade?"],"valueHypothesis":"Educator-controlled staged disclosure can increase target-aligned revisions while preserving student design agency, without guaranteeing grade lifts.","potentialEngagement":"One-course formative pilot with stage telemetry and revision endpoints on a design assignment.","unsupportedClaims":["Guaranteed exam gains","Diagnosis always correct","Campus-wide RCT equivalence"]},"engineering":{"fit":"Artifact-centered formative GenAI feedback for undergraduate conceptual design (ERD/database) courses.","architectureIntegration":"Structured artifact parser → hidden diagnosis → educator disclosure controller → stage generators → episode memory linked to versioned artifacts.","prerequisites":["educator rubric/requirement authoring","artifact versioning","LLM with logged prompts"],"constraints":["single-course evidence","diagnosis error risk","no learning RCT"],"security":"Artifact and diagnosis logs as education records; minimize retention of hidden diagnoses; access-control TAs/instructors.","proofOfValue":["Target-aligned revision rate after early stages","Low premature full-solution disclosure","Exportable stage telemetry"]},"delivery":{"work":"Author stage pools, configure credits, run homework pilot, review episode logs, survey indirectness/repetition, iterate transitions.","dependencies":["course instructor","TAs for independent grading","LMS/editor integration"],"ownership":"Course instructor with academic-technology operations support.","skillsAdoption":"Faculty define when corrections may appear; students practice concept/application before requesting explicit fixes.","governanceCheckpoints":["formative-scope signoff","disclosure-rule review","mid-pilot indirectness check"],"acceptanceCriteria":["Disclosure rules documented","Episode logs exportable","Early stages cannot dump full solutions"],"risks":["Too-indirect UX","Incorrect diagnosis incorporated","Overclaiming learning from incorporation"]}},"retrievedAt":"2026-10-04T03:18:46Z","enrichedAt":"2026-10-04T03:21:30Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Staged, learner-controlled disclosure may help students who need thinking time before correction, but indirectness and repetition can burden learners who need clearer next steps—test with accessibility services before scale.","procurementImplications":"Require vendors to expose educator-configurable disclosure stages, exportable episode/telemetry, and formative-only modes—not only chatbot seats.","operatingModelImplications":"Operate GenAI formative tools with weekly metrics on stage-exit distribution, incorporation/revision rates, and student-reported indirectness—not chat volume alone.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2610.00870","referenceExcerpt":"71.1% of observed target-level changes fully or partially incorporated the hidden diagnostic target","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}