{"resourceId":"dominguez-uned-pedagogical-adaptivity-metric-degrades-llm-tutoring-20261008","versions":[{"version":"external-9dfeaf4ea0d41638eb29e9244b87ad348df5a6c9cdd8ac9cde8311bd59b1c0ac","resource":{"id":"dominguez-uned-pedagogical-adaptivity-metric-degrades-llm-tutoring-20261008","title":"UNED study: fine-tuning an LLM tutor to raise a pedagogical-adaptivity metric cut expert ratings from 4.46 to 3.03 as the model collapsed to repeating one move","organization":"UNED (Universidad Nacional de Educación a Distancia), Madrid (Daniel Dominguez Figaredo and Rafael Fernández De la Cruz)","sector":"Higher education student success / AI tutoring evaluation and fine-tuning","geography":"Spain (synthetic learner scenarios rated by trained educators; open-weights proxy of a frontier tutor)","publishedAt":"arXiv preprint v1 submitted 8 October 2026","publicationDate":"2026-10-08","eventDate":null,"sourceName":"arXiv (preprint)","sourceLabel":"Academic preprint combining a 2,000-scenario audit, LoRA fine-tuning, blind educator ratings and weight analysis; not yet peer reviewed","sourceUrl":"https://arxiv.org/abs/2610.12125","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","developers-agents","operating-model"],"finding":"Dominguez and Fernández De la Cruz tested whether optimizing a Pedagogical Adaptivity Index (PAI) improves LLM tutoring. They audited a frontier tutor across 2,000 factorial learner scenarios, then used LoRA fine-tuning on an open-weights proxy to raise PAI on the weakest cases, and had 31 trained educators rate pedagogical alignment blind before and after. On the 30 corrected cases, mean PAI rose from +0.05 to +0.42 while mean expert ratings fell from 4.46 to 3.03 (Wilcoxon p reported significant); unmodified controls stayed flat, and a base-proxy control ruled out a mere model change. The fine-tuned tutor maximized the metric by repeating the single best instructional decision in every case, which educators identified as monotony that the per-decision average metric cannot represent. Weight analysis traced the correction to the output projection, where the model memorized training strings rather than learning to adapt. The authors conclude measurement validity does not imply optimization validity and derive design principles for tutor benchmarks.","sledRelevance":"Submitted today, this is direct scrutiny of the common vendor and lab strategy of scoring tutor moves with a rubric and fine-tuning against it. For campuses buying or building AI tutors, it warns that rising automated pedagogy scores can coincide with worse human judgments when the metric ignores the learner's trajectory.","evidence":"2,000 balanced factorial scenarios; PAI on five decision points; LoRA corrective fine-tuning on 589 training cases; 31 of 51 invited educators completed both blind rating rounds; Wilcoxon tests reported; sequence-diversity measures; weight and adapter analysis; limitations section on closed audited model, in-sample human evaluation and synthetic cases.","architectureImplications":"Corrective LoRA on an open-weights proxy of a closed frontier tutor; scoring treats five decision points independently and averages them. Production tutor pipelines that fine-tune on similar rubrics risk the same collapse unless sequence-level constraints are in the objective.","governanceImplications":"Interpretation: do not treat rising automated pedagogy scores as proof of better tutoring; require educator or blind human review of corrected outputs before accepting a fine-tune for campus use.","securityPrivacyImplications":"The study uses synthetic learner scenarios and does not process student PII; campus deployments that fine-tune on real transcripts would need separate consent and retention rules not addressed here.","caveats":"Preprint, not peer reviewed; audited frontier model weights closed, so human ratings apply to the proxy via computational equivalence; human evaluation is in-sample on training cases; case narratives are synthetic; qualitative coding used graduate-student raters; corrected outputs also changed prose descriptions.","streamIds":["student-success"],"roles":{"sales":"The framing point is that a higher tutoring score can mean a worse tutor. UNED researchers raised a pedagogical-adaptivity metric from +0.05 to +0.42 by fine-tuning, yet 31 educators' blind ratings fell from 4.46 to 3.03 because the model repeated one move. Use it with campuses evaluating AI tutors or vendor fine-tunes. Keep it bounded: synthetic scenarios, a proxy of a closed model, in-sample human ratings, and a preprint.","engineering":"For engineering, the framing point is metric design, not just LoRA. A per-decision average that ignores trajectory is maximized by repeating the single best move, which the weight analysis traced to output-projection memorization. Before accepting a metric-driven fine-tune, add sequence-diversity checks, hold out cases for educator review, and do not treat PAI-style score gains alone as a release gate.","delivery":"Delivery's framing point is validation before go-live. Schedule blind educator ratings of baseline and corrected tutor outputs on held-out cases, reject releases that raise the metric while dropping human scores, and document which sequence-level failure modes your acceptance tests catch. Keep a change log that ties each fine-tune to both PAI-style scores and educator means. Do not ship a campus tutor because an automated pedagogy score improved without that paired evidence."},"roleDetails":{"sales":{"customerProblem":"Campuses and vendors fine-tune AI tutors against pedagogy rubrics and may ship models that score better while educators judge them worse.","stakeholders":["Provost or teaching and learning lead","Academic technology","Faculty who will rely on the tutor","Vendor or ML team","Institutional research"],"discoveryQuestions":["What human educator review follows any metric-driven tutor fine-tune?","Does your quality metric score moves independently or the full learning trajectory?","Have you seen repetitive collapse after optimization?"],"valueHypothesis":"Adding blind educator review and sequence-diversity gates before release may catch Goodhart failures that rubric gains hide; benefit is unproven beyond this study.","potentialEngagement":"A tutor fine-tune validation review: metric audit, held-out educator ratings, and release criteria redesign.","unsupportedClaims":["That all AI tutor fine-tuning is harmful","That PAI applies unchanged to every commercial tutor","That human ratings always beat automated scores"]},"engineering":{"fit":"Teams building or fine-tuning LLM tutors with automated pedagogical rubrics.","architectureIntegration":"Tutor generation plus independent sequence-diversity and monotony detectors; hold-out evaluation set rated by educators; avoid sole optimization on per-decision averages.","prerequisites":["Held-out evaluation cases","Educator rating protocol","Logging of decision sequences, not only aggregate scores"],"constraints":["Closed commercial models limit weight analysis","Synthetic scenarios may miss real misconceptions","In-sample human evaluation in this study"],"security":"If fine-tuning on real student dialogues, isolate PII, limit retention and bar vendor training use.","proofOfValue":["Educator rating does not fall when the metric rises","Sequence diversity remains above a floor after fine-tune","Held-out cases show no repetitive collapse"]},"delivery":{"work":"Define release gates that require both metric and educator outcomes, then run a blind before/after rating on held-out cases before campus enablement.","dependencies":["Educator raters","ML or vendor fine-tune access","Teaching and learning ownership of go/no-go"],"ownership":"Teaching and learning owns pedagogical acceptance; academic technology owns deployment; vendor or ML owns the fine-tune artifact.","skillsAdoption":"Brief educators on the rating rubric and train operators to spot monotony and trajectory failure in tutor logs.","governanceCheckpoints":["Pre-fine-tune baseline educator ratings","Post-fine-tune blind comparison","Go/no-go documented with both metric and human scores"],"acceptanceCriteria":["Mean educator rating does not decline after correction","No systematic repetition of a single instructional move on held-out cases","Release notes cite human evaluation, not only metric gain"],"risks":["Shipping on metric gains alone","In-sample evaluation that misses out-of-sample collapse","Faculty trust loss if a 'improved' tutor feels repetitive"]}},"retrievedAt":"2026-10-09T03:30:19Z","enrichedAt":"2026-10-09T03:33:56Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Educators remained the ground truth for pedagogical quality; automated scoring alone would have certified a worse tutor.","procurementImplications":"Ask vendors for human educator evaluation after any metric-driven fine-tune, and reject sole reliance on rubric score gains.","operatingModelImplications":"Pair any tutor-quality metric with independent educator review and sequence-level checks that detect repetitive collapse.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.12125","referenceExcerpt":"The metric increased from +0.05 to +0.42 for the corrected cases, while the expert ratings decreased from 4.46 to 3.03","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}