{"resourceId":"bergh-chatgpt-performance-retention-ownership-20260918","versions":[{"version":"external-0558b6cecb290840924050e5d151e3a7bc1829ab9d7870f196fe56012f155397","resource":{"id":"bergh-chatgpt-performance-retention-ownership-20260918","title":"UNSW experiment: ChatGPT raises intro coding scores but lowers recall and ownership","organization":"University of New South Wales","sector":"Higher education / academic research","geography":"Australia (single R1 CS first-year cohort; transferable with limits)","publishedAt":"2026-09-18","publicationDate":"2026-09-18","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint (arXiv:2609.21194)","sourceUrl":"https://arxiv.org/pdf/2609.21194","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","developers-agents","accessibility-workforce","operating-model"],"finding":"In a between-subjects experiment (N=59 enrolled; 55 analyzed), ChatGPT-4.5 assistance produced higher introductory C coding scores (89% vs 69%) than conventional web search, but lower cued recall immediately and at 48 hours and lower self-attributed ownership of submitted code (45% vs 81%). Why now: posted 2026-09-18 and absent from the student-success archive after the 2026-09-20 edition.","sledRelevance":"Directly warns U.S. HE student-success and CS programs that GenAI coding tutors and permissive assessment can inflate assisted performance while weakening what novices retain and claim as their work—measure unaided outcomes.","evidence":"Controlled lab tasks with ChatGPT vs web-search conditions; coding rubrics, immediate and 48-hour cued recall, ownership scales, self-reported mental effort; confirmatory physiology inconclusive. Single Australian CS cohort.","architectureImplications":"Pair coding assistants with retrieval/explanation gates and unaided checkouts; log assistance mode separately from assessed artefacts.","governanceImplications":"Update integrity and assessment policy so AI-assisted submissions require explanation, oral, or closed retrieval components.","securityPrivacyImplications":"Student prompts and code in GenAI sessions are education records; prefer institution-approved tools with retention limits over unmanaged consumer ChatGPT.","caveats":"Small self-selected first-year CS sample; Australian lab; does not prescribe how ChatGPT was used; physiology underpowered.","streamIds":["student-success"],"roles":{"sales":"For teaching-and-learning and CS program buyers, this UNSW experiment shows ChatGPT can raise assisted coding scores while cutting recall and ownership—so student-success claims based on completion or homework quality alone are fragile. Position discovery around unaided retention and authorship checks before pitching GenAI coding tutors as learning wins. Bounded value is redesigning assessment and coaching productive use, not guaranteeing GPA or placement gains from ChatGPT access.","engineering":"Engineering takeaway: instrument assisted versus unaided performance, ownership surveys, and short-delay recall in programming courses adopting GenAI. Architecture should prefer campus-approved models with session logging and optional scaffold modes that require student edits before submit. Prerequisites include IRB-ready instruments and LMS hooks for closed quizzes. Constraints: single-cohort lab; ChatGPT use patterns uncontrolled. Security: keep prompts/code in FERPA-aligned tenancy. Proof of value: replicate performance–retention dissociation locally before scaling licenses.","delivery":"Delivery work is a course pilot: parallel assisted lab tasks with mandatory unaided recall and ownership items, then faculty workshop on assessment redesign. Dependencies include CS faculty, IR/ethics, and approved GenAI tooling. Ownership sits with the department plus teaching-and-learning. Skills: rubric design for explanation and retrieval practice. Governance checkpoints: syllabus AI disclosure and integrity SOP. Acceptance: documented assisted vs unaided score gaps; risks include overinterpreting homework gains as mastery."},"roleDetails":{"sales":{"customerProblem":"Campuses equate GenAI-boosted assignment scores with learning and risk silent skill loss and authorship disputes.","stakeholders":["CS / STEM department chairs","teaching & learning center","student success / academic integrity","IT / academic technology"],"discoveryQuestions":["Do you measure unaided recall after AI-assisted coding tasks?","How is ownership/authorship assessed when GenAI is allowed?","Are consumer ChatGPT and campus tools both in use?"],"valueHypothesis":"Bounded advisory value: surface performance–retention gaps and redesign assessment before large GenAI coding-tutor purchases.","potentialEngagement":"4–6 week pilot design: assisted vs unaided measures in one intro course and a leadership brief—not an outcome guarantee.","unsupportedClaims":["Guaranteed learning gains from ChatGPT coding tutors","Causal proof that GenAI always harms retention","Transfer of UNSW CS results to all majors or community colleges"]},"engineering":{"fit":"Fits as measurement and assessment-instrumentation layer around existing GenAI coding tools.","architectureIntegration":"Approved GenAI SSO + lab IDE plugins + LMS closed quizzes; avoid training on student code without DPA.","prerequisites":["IRB/ethics path","Unaided recall item bank","Ownership/authorship survey items"],"constraints":["Small single-cohort external evidence","Uncontrolled ChatGPT strategies","Australian CS context"],"security":"Store prompts, code, and survey responses as education records with least-privilege access.","proofOfValue":["Reproduce assisted vs unaided score gap","Track ownership ratings by condition","Confirm 24–48h recall instrument reliability"]},"delivery":{"work":"Stand up pilot instruments, run one lab cycle, analyze gaps, facilitate assessment redesign workshop.","dependencies":["Faculty volunteers","Approved GenAI platform decision","Analytics capacity"],"ownership":"CS course coordinator jointly with teaching & learning.","skillsAdoption":"Train TAs to coach explanation-first GenAI use and discourage full-solution pasting.","governanceCheckpoints":["Syllabus AI policy update","Integrity office review","Student communication of assessment changes"],"acceptanceCriteria":["Unaided recall administered","Ownership items collected","Faculty decision memo on assessment changes"],"risks":["Students treat recall as low-stakes","Faculty revert to AI-permissive homework only","Equity issues if only some students have premium ChatGPT"]}},"retrievedAt":"2026-09-22T03:12:40Z","enrichedAt":"2026-09-22T03:16:00Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Students who rely heavily on GenAI may look successful on assisted tasks while lacking durable skills—widening equity gaps for those without coaching on productive use.","procurementImplications":"Procure coding GenAI with evaluation rights for unaided outcomes and institutional data control; do not buy on homework-completion dashboards alone.","operatingModelImplications":"Shift CS student-success ops from completion monitoring to retrieval practice and authorship checkpoints in AI-permitted courses.","sourceVerification":{"responseId":"curl-arxiv-2609-21194","openedUrl":"https://arxiv.org/pdf/2609.21194","referenceExcerpt":"ChatGPT-assisted students achieved higher coding scores (89% vs. 69%) but lower recall scores immediately (41% vs. 53%)","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}