{"resourceId":"nature-correspondence-measure-learning-with-and-without-ai-20261103","versions":[{"version":"external-328245c513c676e3fb542cf74ea610a2db0f3cb82b86a566f10dda23a7628a8b","resource":{"id":"nature-correspondence-measure-learning-with-and-without-ai-20261103","title":"Nature correspondence: universities must assess what students can do without AI, not only how they perform with it","organization":"Christoph Trattner, University of Bergen; published in Nature","sector":"Higher education assessment and learning measurement","geography":"International (Norway author; general higher education)","publishedAt":"Nature, correspondence, 3 November 2026","publicationDate":"2026-11-03","eventDate":null,"sourceName":"Nature","sourceLabel":"Journal correspondence (opinion, not peer-reviewed research)","sourceUrl":"https://www.nature.com/articles/d41586-026-03559-1","evidenceClass":"independent-research","outcomeClass":"cautionary","topics":["knowledge-work","operating-model"],"finding":"In a Nature correspondence published 3 November 2026, Christoph Trattner (University of Bergen) responds to a proposal that universities teach students to argue with AI models and assess how well they do so. He argues one measurement is missing: assessing a student while the model is present captures assisted performance, not what the student can do once it is gone. He cites a randomized trial with secondary-school mathematics students in which unrestricted chatbot access raised practice scores but lowered performance on a later exam taken without it (Bastani et al., PNAS 2025), noting that an evaluation stopping at the assisted session would have called that tutor a success.","sledRelevance":"Sets a measurement standard for evaluating campus AI tutors and AI-integrated assessment: compare assisted and unassisted performance before claiming student-success gains.","evidence":"Argument supported by one cited randomized trial in secondary education; no new data.","architectureImplications":"Interpretation: evaluation designs need assessment points where AI is unavailable, which may require controlled-condition testing environments.","governanceImplications":"Interpretation: AI tutor pilots should pre-register unassisted outcome measures before scale-up decisions.","securityPrivacyImplications":"Not addressed.","caveats":"Opinion piece; the cited trial involves secondary-school students, so transfer to higher education is a stated limitation; no higher-education data presented.","streamIds":["student-success"],"roles":{"sales":"Sales framing: buyers of AI tutors and AI-integrated courses need evidence that students learn, not only that assisted work improves. This Nature correspondence argues universities must measure performance both with and without AI, citing a trial where chatbot access raised practice scores but lowered unaided exam results. A credible offer is evaluation design for AI pilots that includes unassisted measures. Do not use this piece as evidence about any product; it is an opinion letter citing a secondary-school trial.","engineering":"Engineering framing: evaluating AI learning tools requires data that separates assisted from unassisted performance. Fit depends on linking tutor usage logs to grades from controlled-condition assessments where AI is unavailable. Prerequisites include assessment calendars, data-sharing agreements between the LMS and institutional research, and comparison groups. Constraints include ethics approval and small course samples. Proof of value should report unaided exam performance alongside usage, since assisted scores alone can mislead, as the cited trial shows.","delivery":"Delivery framing: the operating owner is institutional research with the teaching centre. Work includes defining unassisted outcome measures before a pilot starts, scheduling controlled assessments, and reporting both assisted and unaided results to decision-makers. Acceptance should rest on a pre-agreed analysis plan. Risks are pilots declared successful on usage or practice scores alone, insufficient sample sizes, and faculty resistance to adding unassisted assessments. Applicability is limited because the cited evidence is from secondary school."},"roleDetails":{"sales":{"customerProblem":"Institutions lack evidence that AI-assisted performance reflects real learning.","stakeholders":["Institutional research","Teaching and learning centre","Provost office","Faculty assessment leads"],"discoveryQuestions":["How do you evaluate AI tutor pilots today?","Do any measures capture unaided performance?","What evidence would justify scaling an AI tool?"],"valueHypothesis":"Interpretation: adding unassisted measures can prevent scaling tools that raise practice scores but not learning.","potentialEngagement":"Evaluation-design support for one AI tutor or AI-integrated course pilot.","unsupportedClaims":["AI tutors harm university learning","Any specific product is ineffective","Higher-education effect sizes"]},"engineering":{"fit":"Evaluation data pipeline linking AI use to assisted and unassisted outcomes.","architectureIntegration":"Join LMS and tutor logs with grades from controlled-condition assessments via institutional research.","prerequisites":["Data-sharing agreement","Controlled assessments without AI","Comparison group or baseline"],"constraints":["Ethics approval","Small samples","Secondary-school evidence base"],"security":"Protect linked student usage and grade data under institutional data governance.","proofOfValue":["Unaided exam performance by usage group","Gap between practice and exam scores","Pre-registered analysis plan"]},"delivery":{"work":"Build unassisted outcome measures into AI pilot evaluation plans.","dependencies":["Faculty agreement on controlled assessments","Institutional research capacity","Pilot timeline"],"ownership":"Institutional research office with the teaching centre.","skillsAdoption":"Pilot leads need guidance on separating assisted and unaided performance.","governanceCheckpoints":["Analysis plan approved before launch","Ethics review","Scale-up decision tied to unaided results"],"acceptanceCriteria":["Proposed: unaided outcome measure in every AI pilot","Proposed: results reported by usage group","Proposed: decision memo cites both measures"],"risks":["Success declared on usage alone","Underpowered comparisons","Added assessment burden"]}},"retrievedAt":"2026-11-04T04:19:03Z","enrichedAt":"2026-11-04T04:21:37Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Not addressed.","procurementImplications":"Interpretation: vendor evidence based only on usage or assisted scores should not be treated as proof of learning.","operatingModelImplications":"Interpretation: institutional research offices should be involved in AI tutor evaluation design.","peerReview":"unknown","sourceVerification":{"openedUrl":"https://www.nature.com/articles/d41586-026-03559-1","referenceExcerpt":"in a randomized trial involving secondary-school mathematics students, unrestricted chatbot access raised practice scores, but lowered performance on a subsequent exam taken without it","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}