{"resourceId":"azimi-ai-literacy-over-tool-design-20260915","versions":[{"version":"external-0558b6cecb290840924050e5d151e3a7bc1829ab9d7870f196fe56012f155397","resource":{"id":"azimi-ai-literacy-over-tool-design-20260915","title":"Delft pilot: scaffolded AI Study Coach matches unrestricted GenAI on assignments; literacy and self-rules matter more","organization":"Delft University of Technology","sector":"Higher education / academic research","geography":"Netherlands (master's data-analytics labs; transferable with limits)","publishedAt":"2026-09-15","publicationDate":"2026-09-15","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint (arXiv:2609.16784)","sourceUrl":"https://arxiv.org/pdf/2609.16784","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","developers-agents","accessibility-workforce","operating-model","governance-procurement"],"finding":"In a seven-week randomized pilot (N=33), a notebook-embedded AI Study Coach (stepwise hints, no code generation, hint budgets, end-of-session reflection) did not improve assignment performance versus unrestricted AI tools. Students with self-formulated AI-use rules and stronger (often self-taught) model understanding scored highest in both arms; interviewees valued awareness of reliance. Why now: submitted 2026-09-15 and absent from archive after 2026-09-20.","sledRelevance":"Challenges U.S. HE bets that building a controlled institutional tutor alone will outperform consumer GenAI—pair tool design with explicit AI literacy and assessment of reasoning.","evidence":"Mixed-methods RCT-style assignment to Coach vs unrestricted; performance plus interviews/surveys on confidence and reliance. Small single master's course.","architectureImplications":"Institutional coaches remain useful for hinting and logging, but prioritize literacy curriculum and assessments that grade process/reasoning over tool-feature races with commercial systems.","governanceImplications":"Policies should require AI-literacy outcomes and process-visible assessment, not only tool allowlists or detection.","securityPrivacyImplications":"Notebook-embedded coaches keep prompts inside course infra—prefer that over unmanaged consumer tools for student homework data.","caveats":"Small N; one scaffold design; Netherlands master's labs; does not generalize to all scaffolds or U.S. community colleges.","streamIds":["student-success"],"roles":{"sales":"For campuses shopping institutional GenAI tutors, this Delft pilot shows a carefully scaffolded Study Coach did not beat unrestricted tools on assignments—students who already governed their own AI use performed best. Reframe value toward AI literacy programs and assessment redesign rather than promising learning lifts from a controlled tutor alone. Bounded engagement is a literacy-plus-assessment discovery, not a guaranteed coach ROI.","engineering":"Engineering takeaway: if deploying a campus coach, instrument reliance rules, hint-budget behavior, and reasoning artefacts—not just completion. Architecture: notebook or LMS-embedded hinting with reflection prompts and no unrestricted code dump; still expect students to use external tools unless assessments change. Prerequisites: course team capacity and literacy modules. Constraints: small external pilot; single scaffold. Security: keep coach telemetry institutional. Proof of value: local A/B on assignments plus literacy pre/post.","delivery":"Delivery work combines a light coach pilot with a mandatory AI-literacy unit and process portfolios. Dependencies: instructors, learning designers, IR. Ownership: teaching & learning with program faculty. Skills: facilitation of student rule-setting for GenAI use. Governance: assessment committee approval of process grading. Acceptance: literacy module live and pilot report comparing coach vs status quo; risks include building another unused tutor while consumer tools dominate."},"roleDetails":{"sales":{"customerProblem":"Leaders fund controlled GenAI tutors expecting automatic learning gains versus unmanaged ChatGPT.","stakeholders":["provost / teaching & learning","program directors","student success","IT"],"discoveryQuestions":["What AI literacy is taught—not just which tools are licensed?","Do assessments grade reasoning behind AI-assisted work?","How will you know if a campus coach beats consumer tools locally?"],"valueHypothesis":"Bounded value: test literacy and assessment changes first; treat institutional coaches as optional instrumentation, not guaranteed learning engines.","potentialEngagement":"6-week literacy-and-assessment design sprint with optional coach telemetry—not a multi-year tutor build commitment.","unsupportedClaims":["Campus coach will outperform ChatGPT on grades","Scaffolding always prevents over-reliance","Transfer of Delft master's results to all U.S. undergrads"]},"engineering":{"fit":"Optional hinting coach inside notebooks/LMS plus literacy content and process capture.","architectureIntegration":"Course notebooks → coach API with hint budgets/reflections → LMS assignment + portfolio artefacts; assume shadow consumer AI exists.","prerequisites":["Instructor content for hints","Literacy learning outcomes","Assessment redesign authority"],"constraints":["Small foreign pilot evidence","One scaffold tested","Commercial tools evolve faster than campus builds"],"security":"Prefer institutional hosting of prompts; contract bans secondary training on student work.","proofOfValue":["Local assignment contrast coach vs unrestricted","Literacy pre/post","Share of students with documented personal AI-use rules"]},"delivery":{"work":"Author literacy module, optional coach, process-portfolio rubric, and end-of-term evaluation.","dependencies":["Faculty buy-in","LMS/notebook platform","Analytics support"],"ownership":"Program lead with teaching & learning co-owner.","skillsAdoption":"Teach students to write, test, and revise personal GenAI-use rules.","governanceCheckpoints":["Assessment committee review","Student communication","Mid-pilot reliance check"],"acceptanceCriteria":["Literacy outcomes assessed","Pilot comparison reported","No unverified coach-superiority claims"],"risks":["Coach unused","Equity gaps if literacy is optional","Detection-only policy rebound"]}},"retrievedAt":"2026-09-22T03:12:40Z","enrichedAt":"2026-09-22T03:16:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Students who luck into AI literacy gain advantage; making literacy explicit reduces chance-based inequity between self-taught and under-supported learners.","procurementImplications":"Do not sole-source multi-year tutor platforms without local A/B rights and bundled literacy/assessment services.","operatingModelImplications":"Student-success AI strategy should budget faculty development for AI literacy alongside any tool spend.","sourceVerification":{"responseId":"curl-arxiv-2609-16784","openedUrl":"https://arxiv.org/pdf/2609.16784","referenceExcerpt":"Assignment performance did not differ between the two conditions.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}