{"resourceId":"cmu-genai-literacy-randomized-study","versions":[{"version":"legacy/2026-09-02/cmu-genai-literacy-randomized-study","resource":{"id":"cmu-genai-literacy-randomized-study","title":"Randomized university study finds 90-minute AI literacy modules improve some competencies but not critical analysis","organization":"Carnegie Mellon University Eberly Center","sector":"Public-interest higher education and workforce preparation","geography":"Pennsylvania, United States","publishedAt":"April 2026","sourceName":"Impacts of asynchronous learning modules on genAI competency in college students","sourceLabel":"Carnegie Mellon University study summary and publication record","sourceUrl":"https://www.cmu.edu/teaching/gaitar/gaitaratscale/asynchronouslearningmodules.html","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","data-security","governance-procurement","accessibility-workforce","operating-model"],"finding":"A randomized study assigned 1,368 undergraduate and graduate students in 53 courses taught by 46 instructors to either no intervention or four self-paced modules totaling about 90 minutes. The modules significantly improved knowledge of how LLMs work, prompting skill, and self-efficacy beyond the control group, but did not significantly improve responsible-use knowledge or overall skill at analyzing AI output.","sledRelevance":"SLED organizations often treat one training completion as evidence of AI readiness. This study shows that scalable instruction can work, but that verification, responsible use, and model knowledge are separable competencies requiring different learning designs and assessments.","evidence":"Courses were randomly assigned; 610 students were in control and 758 in treatment. Pre/post measures covered knowledge, authentic prompting and output-evaluation tasks, and self-efficacy. A blinded subset of 174 students was scored by two independent raters. Benefits were reported across discipline, sex, race or ethnicity, class year, and first-generation status, with a pre-existing female self-efficacy gap closing after the intervention.","architectureImplications":"A learning platform can deliver reusable foundational modules at scale, but should also support authenticated completion, versioned content as models change, authentic practice, immediate feedback, accessibility, and role-specific follow-on exercises. Training telemetry should remain separate from permission to access sensitive data or consequential tools.","governanceImplications":"Define AI literacy as multiple testable capabilities, require demonstrated verification and responsible-use skills for higher-risk access, and refresh training when models, data rules, or workflows change. Procurement should not accept seat time or completion rates as proof that users can evaluate outputs or protect data.","securityPrivacyImplications":"Training should use synthetic or approved data and explicitly test source checking, sensitive-data boundaries, escalation, and uncertainty. Demographic analysis can expose inequitable effects, but the data needed for it should be governed with minimization, access, and retention controls.","caveats":"The study occurred at one selective university with instructors who volunteered their courses, measured outcomes four days after access, and does not establish durable behavior change or safer real-world AI use. The output-analysis measure used a 174-student subset, and the intervention produced no detected gain in responsible-use knowledge or overall output analysis."}},{"version":"enrichment/2026-09-05T02:42:45.193Z/cmu-genai-literacy-randomized-study","resource":{"id":"cmu-genai-literacy-randomized-study","title":"Randomized university study finds 90-minute AI literacy modules improve some competencies but not critical analysis","organization":"Carnegie Mellon University Eberly Center","sector":"Public-interest higher education and workforce preparation","geography":"Pennsylvania, United States","publishedAt":"April 2026","publicationDate":null,"eventDate":null,"sourceName":"Impacts of asynchronous learning modules on genAI competency in college students","sourceLabel":"Carnegie Mellon University study summary and publication record","sourceUrl":"https://www.cmu.edu/teaching/gaitar/gaitaratscale/asynchronouslearningmodules.html","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","data-security","governance-procurement","accessibility-workforce","operating-model"],"finding":"A randomized study assigned 1,368 undergraduate and graduate students in 53 courses taught by 46 instructors to either no intervention or four self-paced modules totaling about 90 minutes. The modules significantly improved knowledge of how LLMs work, prompting skill, and self-efficacy beyond the control group, but did not significantly improve responsible-use knowledge or overall skill at analyzing AI output.","sledRelevance":"SLED organizations often treat one training completion as evidence of AI readiness. This study shows that scalable instruction can work, but that verification, responsible use, and model knowledge are separable competencies requiring different learning designs and assessments.","evidence":"Courses were randomly assigned; 610 students were in control and 758 in treatment. Pre/post measures covered knowledge, authentic prompting and output-evaluation tasks, and self-efficacy. A blinded subset of 174 students was scored by two independent raters. Benefits were reported across discipline, sex, race or ethnicity, class year, and first-generation status, with a pre-existing female self-efficacy gap closing after the intervention.","architectureImplications":"A learning platform can deliver reusable foundational modules at scale, but should also support authenticated completion, versioned content as models change, authentic practice, immediate feedback, accessibility, and role-specific follow-on exercises. Training telemetry should remain separate from permission to access sensitive data or consequential tools.","governanceImplications":"Define AI literacy as multiple testable capabilities, require demonstrated verification and responsible-use skills for higher-risk access, and refresh training when models, data rules, or workflows change. Procurement should not accept seat time or completion rates as proof that users can evaluate outputs or protect data.","securityPrivacyImplications":"Training should use synthetic or approved data and explicitly test source checking, sensitive-data boundaries, escalation, and uncertainty. Demographic analysis can expose inequitable effects, but the data needed for it should be governed with minimization, access, and retention controls.","caveats":"The study occurred at one selective university with instructors who volunteered their courses, measured outcomes four days after access, and does not establish durable behavior change or safer real-world AI use. The output-analysis measure used a 174-student subset, and the intervention produced no detected gain in responsible-use knowledge or overall output analysis.","streamIds":["student-success","research"],"roles":{"sales":"Interpretation — Problem and stakeholders: Higher-education learning teams, faculty, student-success leaders, workforce trainers, and security owners may treat course completion as AI readiness. Discovery: Must users understand models, prompt effectively, verify outputs, or handle sensitive information, and how is each demonstrated? Value hypothesis: Foundational modules plus targeted practice could improve selected competencies while exposing gaps. Potential engagement: Assess current training and pilot a competency-based learning and evaluation sequence. Evidence boundary: The randomized university study supports short-term gains in model knowledge, prompting, and self-efficacy, with no detected gain in responsible-use knowledge or overall output analysis. It does not establish durable behavior change, safer workplace use, or equivalent effects in K12 or other institutional populations.","engineering":"Interpretation — Fit: Reuse an accessible learning platform for versioned modules, authenticated completion, practice, and feedback. Architecture: Separate training records from authorization to sensitive data or consequential tools; connect follow-on exercises to approved workflows. Prerequisites: Competency definitions, rubrics, instructors, and synthetic or approved practice data. Constraints: Outcomes were measured soon after access; output analysis used a smaller scored subset. Security: Test source checking, uncertainty, data boundaries, and escalation explicitly rather than assuming general literacy covers them. Proposed validation: Measure each competency before and after training, use authentic verification tasks, and reassess after a locally appropriate interval. Compare competence and behavior with baselines without substituting confidence or attendance for demonstrated responsible use.","delivery":"Interpretation — Work and dependencies: Define competencies, map modules to skills, add verification and responsible-use practice, and arrange direct assessment. Ownership: Learning leaders own curriculum; faculty or workflow experts judge performance; security and privacy define sensitive-use expectations. Skills and adoption: Prepare instructors to evaluate reasoning and sources and offer accessible paths. Governance checkpoints: Approve higher-risk access separately from completion and refresh content when models or rules change. Proposed acceptance: Demonstrated target-skill gains, documented unresolved gaps, successful sensitive-data scenarios, and follow-up retention or behavior evidence where required. Risks: Confidence can rise without output-analysis skill, short-term gains may fade, and findings at one selective university do not guarantee wider workforce or student results."},"retrievedAt":null,"enrichedAt":"2026-09-05T02:42:45.193Z","enrichmentBasis":"archived evidence"}}]}