From the Student Success edition of November 3, 2026
Nature correspondence: universities must assess what students can do without AI, not only how they perform with it
Christoph Trattner, University of Bergen; published in Nature · Higher education assessment and learning measurement · International (Norway author; general higher education)
- Publisher
- Nature
- Original publication
- Nature, correspondence, 3 November 2026
- Source retrieved
- 2026-11-04
What happened
In a Nature correspondence published 3 November 2026, Christoph Trattner (University of Bergen) responds to a proposal that universities teach students to argue with AI models and assess how well they do so. He argues one measurement is missing: assessing a student while the model is present captures assisted performance, not what the student can do once it is gone. He cites a randomized trial with secondary-school mathematics students in which unrestricted chatbot access raised practice scores but lowered performance on a later exam taken without it (Bastani et al., PNAS 2025), noting that an evaluation stopping at the assisted session would have called that tutor a success.
Why it matters
Sets a measurement standard for evaluating campus AI tutors and AI-integrated assessment: compare assisted and unassisted performance before claiming student-success gains.
Evidence and measured results
Argument supported by one cited randomized trial in secondary education; no new data.
Limitations and uncertainty
Opinion piece; the cited trial involves secondary-school students, so transfer to higher education is a stated limitation; no higher-education data presented.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-04; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Institutions lack evidence that AI-assisted performance reflects real learning.
- Stakeholders
- Institutional research
- Teaching and learning centre
- Provost office
- Faculty assessment leads
- Discovery questions
- How do you evaluate AI tutor pilots today?
- Do any measures capture unaided performance?
- What evidence would justify scaling an AI tool?
- Value hypothesis
- Interpretation: adding unassisted measures can prevent scaling tools that raise practice scores but not learning.
- Potential engagement
- Evaluation-design support for one AI tutor or AI-integrated course pilot.
- Unsupported claims
- AI tutors harm university learning
- Any specific product is ineffective
- Higher-education effect sizes
Pre-sales engineering
Role takeaway
- Fit
- Evaluation data pipeline linking AI use to assisted and unassisted outcomes.
- Architecture and integration
- Join LMS and tutor logs with grades from controlled-condition assessments via institutional research.
- Prerequisites
- Data-sharing agreement
- Controlled assessments without AI
- Comparison group or baseline
- Constraints
- Ethics approval
- Small samples
- Secondary-school evidence base
- Security
- Protect linked student usage and grade data under institutional data governance.
- Proof of value
- Unaided exam performance by usage group
- Gap between practice and exam scores
- Pre-registered analysis plan
Delivery
Role takeaway
- Work
- Build unassisted outcome measures into AI pilot evaluation plans.
- Dependencies
- Faculty agreement on controlled assessments
- Institutional research capacity
- Pilot timeline
- Ownership
- Institutional research office with the teaching centre.
- Skills and adoption
- Pilot leads need guidance on separating assisted and unaided performance.
- Governance checkpoints
- Analysis plan approved before launch
- Ethics review
- Scale-up decision tied to unaided results
- Acceptance criteria
- Proposed: unaided outcome measure in every AI pilot
- Proposed: results reported by usage group
- Proposed: decision memo cites both measures
- Risks
- Success declared on usage alone
- Underpowered comparisons
- Added assessment burden
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Evaluation designs need assessment points where AI is unavailable, which may require controlled-condition testing environments.
Governance
Who approves, reviews and stays accountable for outcomes?
AI tutor pilots should pre-register unassisted outcome measures before scale-up decisions.
Security and privacy
What data, permissions and controls need testing?
Not addressed.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Not addressed.
Procurement
What should contracts, pricing and exit terms secure?
Vendor evidence based only on usage or assisted scores should not be treated as proof of learning.
Operating model
Which teams own the service once it runs?
Institutional research offices should be involved in AI tutor evaluation design.
Publication history
- 2026-11-03Student Success · Issue 574 resources
Stable resource ID: nature-correspondence-measure-learning-with-and-without-ai-20261103