From the SLED-wide archive edition of September 2, 2026
Randomized university study finds 90-minute AI literacy modules improve some competencies but not critical analysis
Carnegie Mellon University Eberly Center · Public-interest higher education and workforce preparation · Pennsylvania, United States
- Publisher
- Impacts of asynchronous learning modules on genAI competency in college students
- Original publication
- April 2026
- Source retrieved
- Not recorded in the historical archive
What happened
A randomized study assigned 1,368 undergraduate and graduate students in 53 courses taught by 46 instructors to either no intervention or four self-paced modules totaling about 90 minutes. The modules significantly improved knowledge of how LLMs work, prompting skill, and self-efficacy beyond the control group, but did not significantly improve responsible-use knowledge or overall skill at analyzing AI output.
Why it matters
SLED organizations often treat one training completion as evidence of AI readiness. This study shows that scalable instruction can work, but that verification, responsible use, and model knowledge are separable competencies requiring different learning designs and assessments.
Evidence and measured results
Courses were randomly assigned; 610 students were in control and 758 in treatment. Pre/post measures covered knowledge, authentic prompting and output-evaluation tasks, and self-efficacy. A blinded subset of 174 students was scored by two independent raters. Benefits were reported across discipline, sex, race or ethnicity, class year, and first-generation status, with a pre-existing female self-efficacy gap closing after the intervention.
Limitations and uncertainty
The study occurred at one selective university with instructors who volunteered their courses, measured outcomes four days after access, and does not establish durable behavior change or safer real-world AI use. The output-analysis measure used a 174-student subset, and the intervention produced no detected gain in responsible-use knowledge or overall output analysis.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source as summarized in the preserved archive. Enriched 2026-09-05; this does not change the original publication date. Labels below come from the analysis itself.
Sales
Role takeaway
Problem and stakeholders: Higher-education learning teams, faculty, student-success leaders, workforce trainers, and security owners may treat course completion as AI readiness.
- Discovery
- Must users understand models, prompt effectively, verify outputs, or handle sensitive information, and how is each demonstrated?
- Value hypothesis
- Foundational modules plus targeted practice could improve selected competencies while exposing gaps.
- Potential engagement
- Assess current training and pilot a competency-based learning and evaluation sequence.
- Evidence boundary
- The randomized university study supports short-term gains in model knowledge, prompting, and self-efficacy, with no detected gain in responsible-use knowledge or overall output analysis. It does not establish durable behavior change, safer workplace use, or equivalent effects in K12 or other institutional populations.
Pre-sales engineering
Role takeaway
- Fit
- Reuse an accessible learning platform for versioned modules, authenticated completion, practice, and feedback.
- Architecture
- Separate training records from authorization to sensitive data or consequential tools; connect follow-on exercises to approved workflows.
- Prerequisites
- Competency definitions, rubrics, instructors, and synthetic or approved practice data.
- Constraints
- Outcomes were measured soon after access; output analysis used a smaller scored subset.
- Security
- Test source checking, uncertainty, data boundaries, and escalation explicitly rather than assuming general literacy covers them.
- Proposed validation
- Measure each competency before and after training, use authentic verification tasks, and reassess after a locally appropriate interval. Compare competence and behavior with baselines without substituting confidence or attendance for demonstrated responsible use.
Delivery
Role takeaway
Work and dependencies: Define competencies, map modules to skills, add verification and responsible-use practice, and arrange direct assessment.
- Ownership
- Learning leaders own curriculum; faculty or workflow experts judge performance; security and privacy define sensitive-use expectations.
- Skills and adoption
- Prepare instructors to evaluate reasoning and sources and offer accessible paths.
- Governance checkpoints
- Approve higher-risk access separately from completion and refresh content when models or rules change.
- Proposed acceptance
- Demonstrated target-skill gains, documented unresolved gaps, successful sensitive-data scenarios, and follow-up retention or behavior evidence where required.
- Risks
- Confidence can rise without output-analysis skill, short-term gains may fade, and findings at one selective university do not guarantee wider workforce or student results.
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
A learning platform can deliver reusable foundational modules at scale, but should also support authenticated completion, versioned content as models change, authentic practice, immediate feedback, accessibility, and role-specific follow-on exercises. Training telemetry should remain separate from permission to access sensitive data or consequential tools.
Governance
Who approves, reviews and stays accountable for outcomes?
Define AI literacy as multiple testable capabilities, require demonstrated verification and responsible-use skills for higher-risk access, and refresh training when models, data rules, or workflows change. Procurement should not accept seat time or completion rates as proof that users can evaluate outputs or protect data.
Security and privacy
What data, permissions and controls need testing?
Training should use synthetic or approved data and explicitly test source checking, sensitive-data boundaries, escalation, and uncertainty. Demographic analysis can expose inequitable effects, but the data needed for it should be governed with minimization, access, and retention controls.
The preserved archive analysis covered architecture, governance and security. Not assessed for this record: accessibility and workforce, procurement, operating model.
Publication history
- 2026-09-02SLED-wide archive · Issue 065 resources
Stable resource ID: cmu-genai-literacy-randomized-study