{"resourceId":"zahra-fluency-without-evidence-constraint-first-20260929","versions":[{"version":"external-1cb4b0d000477b4708642c9a14f0ccc4d7320898f132a25ee20474ed4964278d","resource":{"id":"zahra-fluency-without-evidence-constraint-first-20260929","title":"Fluency without evidence: constraint-first AI partner raises AI literacy self-reports, not CT/metacognition/knowledge","organization":"University of Tennessee at Knoxville (Zahra, Wang, Harper, He)","sector":"Higher education graduate research methods / learning sciences","geography":"United States (large public research university; two graduate research methods courses)","publishedAt":"Submitted to arXiv 29 September 2026 (cs.HC)","publicationDate":"2026-09-29","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (cs.HC)","sourceUrl":"https://arxiv.org/pdf/2609.37880","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","operating-model","accessibility-workforce","data-security"],"finding":"First DBR cycle of a constraint-first AI teaching partner in two graduate research-methods courses (demographics N=20; analytic samples ~18–19). MAILS total rose pretest M=204.00 to posttest M=228.72 (t(17)=3.97, Holm p=.009, dz=.94), concentrating in understanding AI, while critical thinking disposition, metacognitive awareness, and objectively scored knowledge items did not change. Interaction records showed brief exchanges, uneven constraint-first enactment, and missing records—leading authors to argue self-report fluency may reflect confidence, not capacity, and that interaction records are a more defensible evaluation basis.","sledRelevance":"Strong scrutiny signal for HE student-success GenAI evaluations: self-reported AI literacy can rise while critical thinking, metacognition, and knowledge stay flat, and sparse interaction logs undercut claims that constraint-first designs were actually used. Demands telemetry-backed evaluation before scaling graduate or undergraduate AI teaching partners.","evidence":"DBR with pre/post MAILS, adapted CT/metacognition scales, Course B knowledge items, and de-identified conversation transcripts. Adapted scales (CT pretest α=.64); incomplete logs; preprint.","architectureImplications":"Instrument constraint-first compliance in the product (block model help until stance/justification captured) and retain complete interaction corpora for evaluation. Do not treat MAILS-only dashboards as learning proof.","governanceImplications":"Evaluation governance should require process evidence (logs) alongside outcome surveys. Ban procurement claims of critical-thinking or metacognition gains based solely on AI-literacy self-reports.","securityPrivacyImplications":"Full conversation transcripts are sensitive education records; de-identify for research, minimize operational retention, and separate evaluation analytics from conduct systems.","caveats":"Two courses; adapted CT/metacognition instruments; incomplete interaction records; graduate/professional sample; sensitivity limits on null CT findings; preprint.","streamIds":["student-success"],"roles":{"sales":"For graduate education and student-success leaders buying AI teaching partners, this study undercuts fluency theater: MAILS rose about twenty-five points while critical thinking, metacognition, and knowledge did not move, and logs showed brief, uneven constraint-first use. Sell an evaluation engagement that wires complete interaction records and independent outcome measures—not another literacy survey dashboard. Bound claims to this DBR cycle and refuse guaranteed reasoning gains from self-report fluency alone.","engineering":"Engineering should hard-enforce constraint-first gates, capture complete transcripts, and compute enactment metrics such as turn length, stance compliance, and missing-record rates separately from MAILS totals. Keep evaluation stores purpose-limited and outside integrity pipelines. Constraints include adapted instruments and graduate-course evidence needing local revalidation before undergraduate scale-up. Proof-of-value is rising enactment fidelity plus knowledge or reasoning measures that move—or honest pre-registered nulls—not literacy confidence alone. De-identify research extracts of student dialogues.","delivery":"Delivery should implement the partner in a methods course with mandatory stance-before-help practice, weekly log audits for missing records, and pre/post knowledge plus reasoning checks. Ownership sits with graduate program faculty and the teaching-and-learning assessment lead. Train instructors to treat sparse chats as failed enactment, not success. Acceptance means complete log coverage targets and no critical-thinking claims from MAILS-only lifts. Risks include students pasting around constraints and mistaking confidence for capacity."},"roleDetails":{"sales":{"customerProblem":"AI teaching-partner pilots report rising AI literacy while lacking evidence of reasoning, metacognition, or knowledge gains.","stakeholders":["graduate studies","teaching & learning","assessment","academic technology","student success"],"discoveryQuestions":["Do we retain complete interaction logs?","Which non-self-report outcomes will we measure?","How do we verify constraint-first enactment?"],"valueHypothesis":"Telemetry-backed constraint-first designs can separate confidence gains from capacity gains and prevent overclaimed learning outcomes.","potentialEngagement":"One-semester graduate methods evaluation with enforced constraint-first UX, full logs, and knowledge/CT measures.","unsupportedClaims":["Causal critical-thinking improvement","Guaranteed metacognition gains","Generalizability beyond graduate methods"]},"engineering":{"fit":"AI teaching partners in graduate research-methods courses with evaluable interaction corpora.","architectureIntegration":"Stance/justification gate → partner questions → transcript store → enactment metrics → outcome survey linkage by hashed ID.","prerequisites":["course embedding","consent/notice for logging","assessment instruments"],"constraints":["incomplete historical logs","adapted scales","small course N"],"security":"Transcripts as education records; de-identification for research extracts; no conduct auto-triggers.","proofOfValue":["Constraint enactment rate","Missing-record rate ↓","Knowledge/CT deltas or pre-registered nulls"]},"delivery":{"work":"Configure gates, train faculty, audit logs weekly, administer measures, publish evaluation memo.","dependencies":["faculty facilitators","IT logging","IRB/assessment office as applicable"],"ownership":"Graduate course faculty with teaching-and-learning evaluation support.","skillsAdoption":"Instructors interpret sparse dialogues as enactment failures; students practice justified stances before AI help.","governanceCheckpoints":["Logging notice","mid-semester enactment audit","end-term claims review"],"acceptanceCriteria":["Complete-log target met","Non-self-report outcomes reported","No CT claims from MAILS alone"],"risks":["Workarounds pasting answers","Missing transcripts","Overtrusting literacy surveys"]}},"retrievedAt":"2026-10-01T03:23:54Z","enrichedAt":"2026-10-01T03:26:30Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"If only confident students show literacy gains while knowledge stays flat, GenAI partners may widen perceived-fluency gaps. Faculty need analytic support to read logs, not only survey summaries.","procurementImplications":"Require complete interaction logging, enactment analytics, and support for linking to independent assessments before purchasing AI teaching partners marketed for higher-order learning.","operatingModelImplications":"Make log completeness and constraint enactment standing ops metrics alongside student satisfaction; escalate when dialogues stay brief.","sourceVerification":{"openedUrl":"https://arxiv.org/pdf/2609.37880","referenceExcerpt":"they may reflect growth in confidence instead of capacity","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}