{"resourceId":"umd-idealens-ai-idea-provenance-detector-arxiv-2610-06778","versions":[{"version":"external-8d34d21af58110dde2c4531707c8c24a774989a5700dc17bdc696229b17c3452","resource":{"id":"umd-idealens-ai-idea-provenance-detector-arxiv-2610-06778","title":"IdeaLens preprint (University of Maryland, Google DeepMind, Simon Fraser): a detector that judges whether a document's ideas, not its wording, came from AI; Nature reports experts warn against using it on individual cases","organization":"University of Maryland with Google DeepMind and Simon Fraser University (Rajendhran, Iyyer et al.)","sector":"Research integrity and AI-use screening for scholarly writing","geography":"United States and Canada (authors); multilingual evaluation","publishedAt":"arXiv preprint submitted 5 October 2026; reported by Nature news on 13 October 2026","publicationDate":"2026-10-05","eventDate":"2026-10-13","sourceName":"arXiv","sourceLabel":"Preprint (not peer reviewed)","sourceUrl":"https://arxiv.org/abs/2610.06778","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["governance-procurement","knowledge-work","accessibility-workforce"],"finding":"IdeaLens converts a document into an outline of discourse roles with paraphrased content and classifies whether the ideas are human- or AI-derived. It was trained on 1 million FineWeb documents with silver labels from Pangram, a commercial prose detector. In a controlled study, its AI flag rate fell from 95% to 7% as models wrote from increasingly detailed human plans, while Pangram 4 still flagged 92%; from AI-derived plans it stayed above 96%. On 50 stories humans wrote from AI-generated plans, it flagged 68% (95% CI 54–79%) as AI versus 8% for Pangram 4. Across 19 existing benchmarks it detected 91.1% of fully AI documents at a 0.6% false-positive rate, compared with 81.8% at 0.2% for EditLens-3B; it was weaker against DIPPER paraphrasing (77.6%). Why now: Nature news (13 Oct) presented it as a way to gauge AI use in journal and conference submissions, quoting Cornell Tech's Nikhil Garg that the tool is fallible, should not judge individual cases, and may degrade on data unlike its training data. The archive's 12 Oct edition included a preprint finding AI-writing detectors for scientific abstracts collapse with new model releases.","sledRelevance":"Universities, journals and funders are writing AI-use policies that increasingly turn on whether ideas are the researcher's own. A tool that claims to measure idea provenance will reach research integrity offices quickly; this evidence helps them treat it as a screening and research instrument, not proof in misconduct cases.","evidence":"Opened the arXiv abstract page and the arXiv HTML full text via HTTPS (curl; text extracted and read; the HTML rendering ended partway, so later sections and appendices were not fully read). Also opened the Nature news article of 13 Oct 2026 (d41586-026-03247-0); the publicly visible portion supplied the expert caution and reported framing.","architectureImplications":"Interpretation: idea-provenance scoring would sit alongside, not replace, prose detectors and human review in any submission-screening workflow; it requires an outline-extraction model step.","governanceImplications":"Policies should forbid using detector scores alone as evidence of misconduct and require re-validation on the institution's own document types and languages.","securityPrivacyImplications":"Screening unpublished manuscripts or proposals with third-party tools raises confidentiality issues similar to those funders cite for AI in peer review.","caveats":"Unreviewed preprint; labels come from another detector (Pangram), so errors can propagate; training data are web documents (FineWeb), and the inspected text did not show a dedicated evaluation on scientific papers; the human-from-AI-plan result rests on 50 stories with a wide confidence interval; weaker against paraphrasing attacks; Nature's quoted expert warns against individual-case judgments; detector accuracy may drift as models change.","streamIds":["research"],"roles":{"sales":"Sales framing: research integrity offices will be offered tools that claim to tell whether ideas, not just words, came from AI. The credible offer is helping them design a policy and evaluation protocol for any detector before it touches individual cases. This preprint cannot support claims that a detector proves misconduct, works on scientific manuscripts without local testing, or stays accurate as models change.","engineering":"Engineering framing: IdeaLens extracts an outline, then classifies it, with released models and data. Integration would be a batch screening step alongside prose detectors and human review. Constraints include silver labels from another detector, web-document training data, paraphrase weakness and drift with new models. A proof of value should measure false positives on the institution's own historical, pre-AI documents.","delivery":"Delivery framing: delivery is a governed screening process, not a tool install. Work includes local validation, threshold setting, reviewer training and appeal procedures. Ownership sits with the research integrity office with editorial or graduate school partners. Proposed acceptance measures include a locally measured false-positive rate and documented human review; risks include wrongful flags and policy drift."},"roleDetails":{"sales":{"customerProblem":"Research integrity offices and editors need to enforce AI-use policies about idea ownership but lack reliable, fair evidence.","stakeholders":["Research integrity officer","Graduate school dean","University press or journal editors","General counsel"],"discoveryQuestions":["Do your AI-use policies distinguish AI wording from AI ideas?","How would a detector score be used in a case?","Have you measured false positives on your own documents?"],"valueHypothesis":"A validated, aggregate-level screening protocol may help monitor AI use without unfair individual accusations; benefit is not measured for research settings.","potentialEngagement":"Four-week detector evaluation and policy design engagement using local historical documents.","unsupportedClaims":["Detector output proves misconduct","Accuracy on scientific manuscripts without local testing","Stable accuracy across new AI models"]},"engineering":{"fit":"Aggregate screening of submissions, theses or proposals where policies concern idea provenance.","architectureIntegration":"Outline-extraction model followed by the IdeaLens classifier, run in batch beside a prose detector, with results routed to human reviewers (per the paper's released models and data).","prerequisites":["GPU capacity for a 30B mixture-of-experts model or the smaller ModernBERT variant","Local corpus of pre-AI human documents for calibration","Confidentiality-approved hosting"],"constraints":["Silver labels from a commercial detector","Web-document training data","Weaker against paraphrasing","Unreviewed preprint"],"security":"Run on institutional infrastructure so unpublished manuscripts are not sent to third parties.","proofOfValue":["Measure false positives on pre-ChatGPT local documents","Test on disciplinary writing unlike the training data","Re-test after a new major model release"]},"delivery":{"work":"Validate the detector locally, set thresholds, write review and appeal procedures, and train reviewers.","dependencies":["Policy approval","Local calibration corpus","Hosting with confidentiality controls"],"ownership":"Research integrity office with graduate school and editorial partners.","skillsAdoption":"Train reviewers to interpret probabilistic scores and known failure modes.","governanceCheckpoints":["Policy bars score-only decisions","Local validation before use","Periodic re-validation after model releases"],"acceptanceCriteria":["Proposed: locally measured false-positive rate below an agreed threshold","Proposed: every flag reviewed by a human with documented rationale","Proposed: appeal procedure published"],"risks":["Wrongful flags against individuals","Accuracy drift with new models","Confidentiality breaches from external tools"]}},"retrievedAt":"2026-10-13T21:59:33-05:00","enrichedAt":"2026-10-13T22:03:45-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Risk of disproportionate flags for authors whose writing resembles AI patterns; integrity staff need training on error rates.","procurementImplications":"Institutions evaluating detection vendors should ask for false-positive rates on their own document types and for idea-versus-prose distinctions.","operatingModelImplications":"Shifts AI-use policy enforcement toward aggregate monitoring and disclosure rather than individual verdicts.","sourceVerification":{"openedUrl":"https://arxiv.org/abs/2610.06778","referenceExcerpt":"IdeaLens flags 68% of the stories as AI, compared to 8% for Pangram 4","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}