{"resourceId":"adak-llm-tutors-process-synthesis-evidence-ladder-20261014","versions":[{"version":"external-57a3753b74334ce8d41db9195b1d2b8147e1423b3c29c862654d7aa7d6d26ed8","resource":{"id":"adak-llm-tutors-process-synthesis-evidence-ladder-20261014","title":"arXiv 2610.15692 (v2 14 Oct): audit of 88 LLM-tutoring studies finds only 10.2% complete an adaptive path and 11.4% measure human learning","organization":"IIT Kharagpur; Adobe Research; Singapore Institute of Technology / NUS / TCG CREST","sector":"Higher education / educational AI research synthesis (LLM tutoring)","geography":"International research literature (NLP/AI/learning-science venues)","publishedAt":"arXiv abs 2610.15692; submitted 11 October 2026, last revised 14 October 2026 (v2)","publicationDate":"2026-10-14","eventDate":null,"sourceName":"arXiv","sourceLabel":"Preprint research synthesis / structured literature audit","sourceUrl":"https://arxiv.org/abs/2610.15692","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","developers-agents","operating-model"],"finding":"In a structured audit of 88 conversational LLM-tutoring studies, Adak and colleagues find pedagogical evaluation is widespread (87.5%) but only 9 studies (10.2%) jointly include learner-state modeling, learner-conditioned action, an explicit pedagogical policy and a downstream outcome. Only 10 studies (11.4%) directly measure human learning gains, and a single study meets their strict transfer/delayed-retention criterion. They argue LLM tutors can teach under pedagogical structure and engagement, but component implementation and high response-quality scores do not establish learning, retention or transfer.","sledRelevance":"Revised 14 Oct and not previously archived; provides an independent scrutiny lens on tutoring claims that complements positive quasi-experiments and prior UMD caution—useful for procurement evidence standards.","evidence":"Structured study-level coding of a curated corpus with explicit evidence ladder; not a meta-analysis of effect sizes; cutoff 31 Aug 2026.","architectureImplications":"Interpretation: tutoring products should expose learner-state→action links and support delayed assessment designs, not only chat quality dashboards.","governanceImplications":"Sets a research-informed bar: demand human-learning evidence before scaling tutors sold on pedagogical-quality metrics alone.","securityPrivacyImplications":"Longitudinal learner-state logging implied by adaptive tutors needs privacy review (not the paper's focus).","caveats":"Preprint; coder judgment; heterogeneous human studies; literature cutoff before some Oct 2026 papers.","streamIds":["student-success"],"roles":{"sales":"For sales, the framing point is evidence standards for tutoring AI. This 14 October arXiv audit of 88 studies finds only 10.2% complete an adaptive tutoring path and 11.4% measure human learning. Use it to reset discovery from feature demos toward evaluation design. Stay bounded: it is a literature synthesis, not proof that any specific campus tutor fails or succeeds.","engineering":"Engineering's framing point is instrumenting learner-state estimates, pedagogical actions and delayed outcome hooks—not only response-quality judges that the survey shows can Goodhart. Pre-sales should map whether a tutoring product logs state-conditioned interventions, limits answer leakage and supports comparison assessments. Proof of value should include unassisted post-tests and, where feasible, delayed measures, given that only 11.4 percent of audited studies directly measure human learning.","delivery":"Delivery's framing point is evaluation architecture for conversational tutor pilots before any campuswide enablement. Define adaptive-path logging, a comparison condition and at least one human-learning measure before scale decisions. Acceptance criteria below are proposals. Applicability is higher-education conversational tutor projects; classical non-conversational ITS and content-generation tools sit outside the paper's coded scope and should not be inferred from its rates."},"roleDetails":{"sales":{"customerProblem":"Vendors sell fluent AI tutors on pedagogical-quality scores while campuses lack evidence of learning or retention.","stakeholders":["Centre for teaching and learning","Academic technology","Discipline faculty","IR / assessment"],"discoveryQuestions":["What human-learning evidence exists for this tutor?","Is there a delayed or transfer measure?","Does the system adapt on learner state or only prompt style?","What is the comparison condition?"],"valueHypothesis":"Interpretation: requiring adaptive-path and human-learning evidence can prevent buying chat quality alone; local results still needed.","potentialEngagement":"A bounded evaluation-design package for one tutor pilot with delayed assessment.","unsupportedClaims":["That 10.2% means all tutors fail","Pooled effect sizes (not computed)","Guaranteed retention gains from structured tutors"]},"engineering":{"fit":"Fits conversational tutors that can record learner-state features, chosen pedagogical acts and outcome assessments.","architectureIntegration":"State estimate → pedagogical action logging; assignment/exam outcome hooks; optional delayed tests; avoid reward hacking on answer leakage.","prerequisites":["Learner-state schema","Action taxonomy","Assessment integration","Comparison-section capability"],"constraints":["Survey coding limits","Cutoff Aug 31","Sparse E5 evidence overall"],"security":"Learner dialogue and mastery estimates are sensitive; define access and retention before enabling dashboards.","proofOfValue":["Presence of state-conditioned actions in logs","Unassisted post-test vs control","Delayed assessment completion rate"]},"delivery":{"work":"Design a tutor pilot that logs adaptive-path elements and includes a human-learning measure with a comparison condition.","dependencies":["Faculty owner","Academic technology","IR","Teaching centre"],"ownership":"Faculty course owner with teaching-centre evaluation support.","skillsAdoption":"Instructors need training to interpret state/action logs; students need consent language for research-like evaluation.","governanceCheckpoints":["Evaluation plan IRB/ethics as required","Mid-pilot leakage/quality check","End-term learning measure review"],"acceptanceCriteria":["Proposed: adaptive-path logging enabled","Proposed: at least one unassisted learning measure","Proposed: comparison condition documented"],"risks":["Optimising chat scores only","No comparison group","Privacy pushback on logs"]}},"retrievedAt":"2026-10-18T03:16:10Z","enrichedAt":"2026-10-18T03:22:58Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Sparse transfer evidence means equity claims for durable learning remain weakly supported.","procurementImplications":"Interpretation: RFPs should require human-learning and retention metrics, not only rubric scores or engagement.","operatingModelImplications":"Teaching-and-learning centres should define evaluation horizons before IT enables campus tutors.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.15692","referenceExcerpt":"Core adaptive path 9/88 10.2%","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}