{"resourceId":"aixiv-llm-assisted-education-research-reference-audit-claude-20261107","versions":[{"version":"external-19a2f2f033744f69bbd6a0d7a18fdd19b6f9ae7939bcb8aba5c07dcc147acda3","resource":{"id":"aixiv-llm-assisted-education-research-reference-audit-claude-20261107","title":"aiXiv preprint audit: in a Claude-written education research draft, 13 of 14 references existed but only 9 were correct in every checked field","organization":"aiXiv preprint (behavioural audit of LLM-assisted education research)","sector":"Education research / higher education","geography":"Not stated on inspected page","publishedAt":"2026-11-07","publicationDate":"2026-11-07","eventDate":null,"sourceName":"aiXiv","sourceLabel":"Preprint, not peer reviewed","sourceUrl":"https://aixiv.science/abs/aixiv.261107.000001","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work"],"finding":"Source finding (preprint, not peer reviewed; abstract-level inspection): a retrospective audit of twelve published AI-in-education papers drafted with substantial language-model help (records name Claude, plus Gemini for some planning material). Evidence combined the human orchestrator's error records, a Crossref check of 21 DOIs, an independent lookup of the 14 references in one Claude-written draft, and an audit of 35 items from a second assistant. In the Claude draft, 13 references resolved to real works but only 9 were correct in every field; errors were paraphrased or truncated titles, a wrong initial, stale status or no match. Errors clustered in fine details while the register stayed confident; the checker also produced a false alarm. Authors propose a nine-type error taxonomy and an independence test for verification. Why now: posted Nov 7.","sledRelevance":"Relevant to universities, schools of education and K-12 research offices where Claude-drafted research reaches teachers and policymakers: existence checks alone overstate citation accuracy, so verification must check fields and claims.","evidence":"Small retrospective case study; one 14-reference Claude draft; no access to model internals; authors state limits. Only the abstract page was accessible; full methods and author details were not inspected.","architectureImplications":"Citation verification should be an independent step using a different tool or database than the drafter.","governanceImplications":"Supports disclosure and verification requirements for AI-assisted research and policy documents.","securityPrivacyImplications":"Limited; primarily research integrity rather than data security.","caveats":"Preprint on aiXiv (not arXiv), not peer reviewed; very small sample; abstract-only inspection; authorship and model versions not confirmed on the inspected page.","streamIds":["anthropic","research"],"roles":{"sales":"Sales framing: verification services matter as much as drafting tools. Education researchers and policy offices using Claude to draft papers face fine-grained citation errors that look credible. A bounded offer is a verification workflow that checks reference fields and claims independently of the drafting model. Do not generalize the 9-of-14 figure beyond one draft, and do not claim Claude fabricates references broadly. Cross-tagged to Research Universities for research-integrity relevance.","engineering":"Engineering framing: build independent verification into Claude-assisted writing. Resolve every reference via Crossref or OpenAlex, compare titles, authors, years and status field by field, and flag mismatches for human review. Use a different checker than the drafting model, as the authors' independence test suggests, and track false alarms because their checker produced one. Store verification results with the draft for audit.","delivery":"Delivery framing: make field-level citation checks a release gate for AI-assisted research outputs. Research offices should define who runs checks, which databases count as authoritative, and how corrections are logged. Pilot on a handful of policy briefs or papers, track error types using the authors' taxonomy, and update guidance once the full paper or peer-reviewed version provides more detail."},"roleDetails":{"sales":{"customerProblem":"Researchers and policy staff publish Claude-assisted drafts whose references may exist yet contain wrong details.","stakeholders":["Research integrity officer","Dean of education school","Institutional research office","Policy communications lead"],"discoveryQuestions":["How are AI-drafted references checked before publication?","Do checks verify fields or only existence?","Who signs off on AI-assisted policy briefs?"],"valueHypothesis":"Independent field-level checks can catch errors that existence checks miss.","potentialEngagement":"Verification workflow pilot for AI-assisted research and policy documents.","unsupportedClaims":["Error rates beyond a single 14-reference draft","Comparisons between Claude and other models","Peer-reviewed status"]},"engineering":{"fit":"Fits any group drafting with Claude and citing scholarly sources.","architectureIntegration":"Pipeline that extracts references, resolves them against Crossref/OpenAlex and diffs fields before human review.","prerequisites":["Reference extraction tooling","Crossref or OpenAlex API access","Review queue"],"constraints":["Coverage gaps in bibliographic databases","Checker false positives"],"security":"Minimal; avoid sending unpublished manuscripts to unapproved services.","proofOfValue":["Run field-level checks on recent AI-assisted drafts and count discrepancies","Measure checker false-alarm rate"]},"delivery":{"work":"Add a citation verification gate to AI-assisted research publication.","dependencies":["Library or research office support","Author buy-in","Tooling access"],"ownership":"Research integrity office with the library.","skillsAdoption":"Short guidance for authors on the nine error types and verification steps.","governanceCheckpoints":["Verification evidence attached before publication","Periodic review of error trends"],"acceptanceCriteria":["Proposed: all AI-assisted outputs pass field-level checks before release","Proposed: error log maintained by type"],"risks":["Checks skipped under deadline pressure","Overtrust in automated checkers"]}},"retrievedAt":"2026-11-08T04:04:32Z","enrichedAt":"2026-11-08T04:08:54Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Researchers need time and training for verification; drafting speed gains may be offset.","procurementImplications":"Limited; may inform selection of bibliographic verification tools.","operatingModelImplications":"Introduces a verification step owned outside the drafting author.","peerReview":"preprint","sourceVerification":{"openedUrl":"https://aixiv.science/abs/aixiv.261107.000001","referenceExcerpt":"In the 14-reference draft, 13 references resolved to a real work, but only 9 were correct in every field we checked","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}