{"resourceId":"llm-turnover-undermines-ai-writing-screening-arxiv-2610-11599","versions":[{"version":"external-b223ab084b1ec2f44bb7c6effd0c934878f131e9914a049ccd9075b866947aea","resource":{"id":"llm-turnover-undermines-ai-writing-screening-arxiv-2610-11599","title":"Preprint: AI-writing detectors for scientific abstracts collapse at LLM generation boundaries, catching 3.8% of one new model's rewrites","organization":"Kazuki Nakajima (Tokyo Metropolitan University) and Takayuki Mizuno (National Institute of Informatics)","sector":"Research integrity: screening manuscripts for AI-assisted writing","geography":"Japan-based study; PNAS abstracts; models from three U.S./Chinese vendors","publishedAt":"arXiv preprint submitted 8 October 2026","publicationDate":"2026-10-08","eventDate":null,"sourceName":"arXiv","sourceLabel":"Academic preprint (not yet peer reviewed)","sourceUrl":"https://arxiv.org/abs/2610.11599","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["governance-procurement","knowledge-work","data-security"],"finding":"The authors paired 4,000 pre-ChatGPT PNAS abstracts with rewrites by 23 LLM versions from three vendors (June 2023–August 2026) and trained RoBERTa-based detectors under maintenance scenarios. At a 1% false-flag threshold, detectors trained on a vendor's past versions caught above 99% of rewrites just before the sharpest generation boundary and 3.8% just after it (another collapse to 26.6% at GPT-5). A union of all 23 version-specific detectors falsely flagged 11.9% of held-out human abstracts (about one in eight); a pooled detector flagged 0.8% but missed 33.8% of the newest-version rewrites. A deployed commercial detector flagged 1 of 5,000 human abstracts but missed about four in five rewrites of that newest version. Vocabulary shift between versions tracked where detection failed.","sledRelevance":"Universities, journals and graduate schools using AI-writing detectors for theses, proposals or manuscripts should treat vendor accuracy claims as provisional; detection can collapse with a single model release. Relevant to research integrity policy and procurement of screening tools.","evidence":"Opened the arXiv abstract page and the full experimental HTML version via HTTPS (curl; text extracted and read, including Results, the commercial-detector case study, Discussion and limitations).","architectureImplications":"Interpretation: any screening service needs a re-validation pipeline that tests each new and older LLM version at a fixed false-positive rate before results are acted on.","governanceImplications":"Detector output should not be the sole basis for misconduct findings; policies should require documented re-verification dates against current LLM versions.","securityPrivacyImplications":"Screening uploads unpublished manuscripts to third-party detectors; false flags (up to about one in eight in one scenario) create due-process risk for authors.","caveats":"Preprint, not peer reviewed; one family of trained detectors plus one commercial detector at a fixed operating point; abstracts only and rewrite-style assistance rather than other kinds of LLM use; 23 versions from three vendors; the vocabulary association is not causal. First author disclosed using LLMs for editing and code.","streamIds":["research"],"roles":{"sales":"Sales framing: institutions relying on AI-writing detectors for manuscripts, theses or proposals face evidence that accuracy can collapse with one model release. Research integrity and graduate offices are the audience. A bounded policy and vendor-evaluation review fits. This preprint cannot support claims that any detector is reliable, or that detection should replace disclosure and human judgment. Accuracy figures come from abstracts rewritten by 23 model versions, not full manuscripts.","engineering":"Engineering framing: screening is a moving-target model-maintenance problem. Fit depends on whether the institution can test detectors against current and earlier LLM versions at a fixed false-positive rate. Constraints include vendor opacity and constant model releases. A proof of value would generate rewrites of local sample abstracts with current models and measure catch and false-flag rates before any policy relies on the tool.","delivery":"Delivery framing: the work is building a governance routine around detection, not installing a tool. Tasks include defining false-flag tolerance, scheduling re-verification after LLM releases, and requiring human review before any finding. Ownership sits with the research integrity office. Proposed acceptance measures include documented re-verification dates; risks include wrongful accusations, missed AI use and outdated vendor benchmarks being treated as current."},"roleDetails":{"sales":{"customerProblem":"Research integrity and graduate offices use AI-writing detectors whose accuracy may silently degrade with new model releases.","stakeholders":["Research integrity officer","Graduate school dean","Library/scholarly communications","General counsel"],"discoveryQuestions":["Which detectors are used and against which LLM versions were they validated?","What false-flag rate is acceptable?","Can a detector result alone trigger an investigation?"],"valueHypothesis":"Policy that treats detection as provisional and requires re-verification may reduce wrongful flags and over-reliance; unquantified.","potentialEngagement":"Two-to-three-week review of AI-writing screening policy and vendor validation evidence.","unsupportedClaims":["Any detector reliably identifies AI-assisted writing","Results apply to full manuscripts or non-English text","Detection can replace disclosure policies"]},"engineering":{"fit":"Institutions or journals running AI-writing screening on submissions.","architectureIntegration":"Periodic validation harness that generates rewrites with current and earlier LLMs and measures detection at a fixed false-flag rate (interpretation of the paper's method).","prerequisites":["Sample of pre-LLM human texts","API access to current LLMs","Vendor detector access"],"constraints":["Vendor models are opaque","New LLM releases are frequent","Abstract-only evidence"],"security":"Avoid uploading unpublished work to unvetted detectors; review data-retention terms.","proofOfValue":["Measure catch rate on rewrites from the newest models","Measure false flags on local pre-LLM texts","Repeat after each major model release"]},"delivery":{"work":"Define screening policy, false-flag tolerance, re-verification cadence and mandatory human review.","dependencies":["Vendor cooperation","Integrity office capacity","Legal review"],"ownership":"Research integrity office with graduate school.","skillsAdoption":"Train investigators that detector scores are provisional evidence, not proof.","governanceCheckpoints":["Policy approval","Re-verification after LLM releases","Appeal process review"],"acceptanceCriteria":["Proposed: re-verification documented within 30 days of major LLM releases","Proposed: no case opened on a detector score alone","Proposed: measured local false-flag rate at or below policy threshold"],"risks":["Wrongful accusations","Missed AI-assisted text","False reassurance from outdated benchmarks"]}},"retrievedAt":"2026-10-12T22:01:57-05:00","enrichedAt":"2026-10-12T22:06:05-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"False flags may fall disproportionately on authors whose writing style resembles model output, including non-native English writers (interpretation; not tested in this study).","procurementImplications":"Ask vendors which LLM versions their accuracy was last verified against and how quickly they re-validate after new releases.","operatingModelImplications":"Integrity offices should pair any detection with human review and disclosure-based policies rather than automated adjudication.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.11599","referenceExcerpt":"Research-integrity policy should therefore treat the benchmark accuracy of a detector as provisional, to be re-verified with every LLM release, including earlier versions.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}