From the Student Success edition of September 21, 2026
UNSW experiment: ChatGPT raises intro coding scores but lowers recall and ownership
University of New South Wales · Higher education / academic research · Australia (single R1 CS first-year cohort; transferable with limits)
- Publisher
- arXiv
- Original publication
- 2026-09-18
- Source retrieved
- 2026-09-22
What happened
In a between-subjects experiment (N=59 enrolled; 55 analyzed), ChatGPT-4.5 assistance produced higher introductory C coding scores (89% vs 69%) than conventional web search, but lower cued recall immediately and at 48 hours and lower self-attributed ownership of submitted code (45% vs 81%). Why now: posted 2026-09-18 and absent from the student-success archive after the 2026-09-20 edition.
Why it matters
Directly warns U.S. HE student-success and CS programs that GenAI coding tutors and permissive assessment can inflate assisted performance while weakening what novices retain and claim as their work—measure unaided outcomes.
Evidence and measured results
Controlled lab tasks with ChatGPT vs web-search conditions; coding rubrics, immediate and 48-hour cued recall, ownership scales, self-reported mental effort; confirmatory physiology inconclusive. Single Australian CS cohort.
Limitations and uncertainty
Small self-selected first-year CS sample; Australian lab; does not prescribe how ChatGPT was used; physiology underpowered.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-09-22; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Campuses equate GenAI-boosted assignment scores with learning and risk silent skill loss and authorship disputes.
- Stakeholders
- CS / STEM department chairs
- teaching & learning center
- student success / academic integrity
- IT / academic technology
- Discovery questions
- Do you measure unaided recall after AI-assisted coding tasks?
- How is ownership/authorship assessed when GenAI is allowed?
- Are consumer ChatGPT and campus tools both in use?
- Value hypothesis
- Bounded advisory value: surface performance–retention gaps and redesign assessment before large GenAI coding-tutor purchases.
- Potential engagement
- 4–6 week pilot design: assisted vs unaided measures in one intro course and a leadership brief—not an outcome guarantee.
- Unsupported claims
- Guaranteed learning gains from ChatGPT coding tutors
- Causal proof that GenAI always harms retention
- Transfer of UNSW CS results to all majors or community colleges
Pre-sales engineering
Role takeaway
- Fit
- Fits as measurement and assessment-instrumentation layer around existing GenAI coding tools.
- Architecture and integration
- Approved GenAI SSO + lab IDE plugins + LMS closed quizzes; avoid training on student code without DPA.
- Prerequisites
- IRB/ethics path
- Unaided recall item bank
- Ownership/authorship survey items
- Constraints
- Small single-cohort external evidence
- Uncontrolled ChatGPT strategies
- Australian CS context
- Security
- Store prompts, code, and survey responses as education records with least-privilege access.
- Proof of value
- Reproduce assisted vs unaided score gap
- Track ownership ratings by condition
- Confirm 24–48h recall instrument reliability
Delivery
Role takeaway
- Work
- Stand up pilot instruments, run one lab cycle, analyze gaps, facilitate assessment redesign workshop.
- Dependencies
- Faculty volunteers
- Approved GenAI platform decision
- Analytics capacity
- Ownership
- CS course coordinator jointly with teaching & learning.
- Skills and adoption
- Train TAs to coach explanation-first GenAI use and discourage full-solution pasting.
- Governance checkpoints
- Syllabus AI policy update
- Integrity office review
- Student communication of assessment changes
- Acceptance criteria
- Unaided recall administered
- Ownership items collected
- Faculty decision memo on assessment changes
- Risks
- Students treat recall as low-stakes
- Faculty revert to AI-permissive homework only
- Equity issues if only some students have premium ChatGPT
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Pair coding assistants with retrieval/explanation gates and unaided checkouts; log assistance mode separately from assessed artefacts.
Governance
Who approves, reviews and stays accountable for outcomes?
Update integrity and assessment policy so AI-assisted submissions require explanation, oral, or closed retrieval components.
Security and privacy
What data, permissions and controls need testing?
Student prompts and code in GenAI sessions are education records; prefer institution-approved tools with retention limits over unmanaged consumer ChatGPT.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Students who rely heavily on GenAI may look successful on assisted tasks while lacking durable skills—widening equity gaps for those without coaching on productive use.
Procurement
What should contracts, pricing and exit terms secure?
Procure coding GenAI with evaluation rights for unaided outcomes and institutional data control; do not buy on homework-completion dashboards alone.
Operating model
Which teams own the service once it runs?
Shift CS student-success ops from completion monitoring to retrieval practice and authorship checkpoints in AI-permitted courses.
Publication history
- 2026-09-21Student Success · Issue 164 resources
Stable resource ID: bergh-chatgpt-performance-retention-ownership-20260918