From the Student Success edition of September 21, 2026
Delft pilot: scaffolded AI Study Coach matches unrestricted GenAI on assignments; literacy and self-rules matter more
Delft University of Technology · Higher education / academic research · Netherlands (master's data-analytics labs; transferable with limits)
- Publisher
- arXiv
- Original publication
- 2026-09-15
- Source retrieved
- 2026-09-22
What happened
In a seven-week randomized pilot (N=33), a notebook-embedded AI Study Coach (stepwise hints, no code generation, hint budgets, end-of-session reflection) did not improve assignment performance versus unrestricted AI tools. Students with self-formulated AI-use rules and stronger (often self-taught) model understanding scored highest in both arms; interviewees valued awareness of reliance. Why now: submitted 2026-09-15 and absent from archive after 2026-09-20.
Why it matters
Challenges U.S. HE bets that building a controlled institutional tutor alone will outperform consumer GenAI—pair tool design with explicit AI literacy and assessment of reasoning.
Evidence and measured results
Mixed-methods RCT-style assignment to Coach vs unrestricted; performance plus interviews/surveys on confidence and reliance. Small single master's course.
Limitations and uncertainty
Small N; one scaffold design; Netherlands master's labs; does not generalize to all scaffolds or U.S. community colleges.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-22; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Leaders fund controlled GenAI tutors expecting automatic learning gains versus unmanaged ChatGPT.
- Stakeholders
- provost / teaching & learning
- program directors
- student success
- IT
- Discovery questions
- What AI literacy is taught—not just which tools are licensed?
- Do assessments grade reasoning behind AI-assisted work?
- How will you know if a campus coach beats consumer tools locally?
- Value hypothesis
- Bounded value: test literacy and assessment changes first; treat institutional coaches as optional instrumentation, not guaranteed learning engines.
- Potential engagement
- 6-week literacy-and-assessment design sprint with optional coach telemetry—not a multi-year tutor build commitment.
- Unsupported claims
- Campus coach will outperform ChatGPT on grades
- Scaffolding always prevents over-reliance
- Transfer of Delft master's results to all U.S. undergrads
Pre-sales engineering
Role takeaway
- Fit
- Optional hinting coach inside notebooks/LMS plus literacy content and process capture.
- Architecture and integration
- Course notebooks → coach API with hint budgets/reflections → LMS assignment + portfolio artefacts; assume shadow consumer AI exists.
- Prerequisites
- Instructor content for hints
- Literacy learning outcomes
- Assessment redesign authority
- Constraints
- Small foreign pilot evidence
- One scaffold tested
- Commercial tools evolve faster than campus builds
- Security
- Prefer institutional hosting of prompts; contract bans secondary training on student work.
- Proof of value
- Local assignment contrast coach vs unrestricted
- Literacy pre/post
- Share of students with documented personal AI-use rules
Delivery
Role takeaway
- Work
- Author literacy module, optional coach, process-portfolio rubric, and end-of-term evaluation.
- Dependencies
- Faculty buy-in
- LMS/notebook platform
- Analytics support
- Ownership
- Program lead with teaching & learning co-owner.
- Skills and adoption
- Teach students to write, test, and revise personal GenAI-use rules.
- Governance checkpoints
- Assessment committee review
- Student communication
- Mid-pilot reliance check
- Acceptance criteria
- Literacy outcomes assessed
- Pilot comparison reported
- No unverified coach-superiority claims
- Risks
- Coach unused
- Equity gaps if literacy is optional
- Detection-only policy rebound
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Institutional coaches remain useful for hinting and logging, but prioritize literacy curriculum and assessments that grade process/reasoning over tool-feature races with commercial systems.
Governance
Who approves, reviews and stays accountable for outcomes?
Policies should require AI-literacy outcomes and process-visible assessment, not only tool allowlists or detection.
Security and privacy
What data, permissions and controls need testing?
Notebook-embedded coaches keep prompts inside course infra—prefer that over unmanaged consumer tools for student homework data.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Students who luck into AI literacy gain advantage; making literacy explicit reduces chance-based inequity between self-taught and under-supported learners.
Procurement
What should contracts, pricing and exit terms secure?
Do not sole-source multi-year tutor platforms without local A/B rights and bundled literacy/assessment services.
Operating model
Which teams own the service once it runs?
Student-success AI strategy should budget faculty development for AI literacy alongside any tool spend.
Publication history
- 2026-09-21Student Success · Issue 164 resources
Stable resource ID: azimi-ai-literacy-over-tool-design-20260915