{"resourceId":"nyu-protocolnerd-hallucinationnerd-research-assistants-20261023","versions":[{"version":"external-aa616ba51338c25e2c33bde69ac841e89c5953b9f4bd6be37a43c3a297d09360","resource":{"id":"nyu-protocolnerd-hallucinationnerd-research-assistants-20261023","title":"NYU releases ProtocolNerd and HallucinationNerd (23 Oct): ProtocolNerd placed the cited protocol in its top 10 for 46% of 100 benchmark papers, ahead of Elicit (20%), Consensus (8%) and Perplexity (5%)","organization":"New York University, Courant Institute School of Mathematics, Computing, and Data Science (Dennis Shasha) with student and external co-authors","sector":"Research tools for laboratory scientists and research integrity","geography":"New York, United States","publishedAt":"23 October 2026 (NYU News release); ProtocolNerd paper published 22 October 2026 in MDPI Information (early access)","publicationDate":"2026-10-23","eventDate":null,"sourceName":"NYU News","sourceLabel":"NYU News, \"Computer Scientists Build Innovative Tech Tools to Find Experiments, Detect Citation “Hallucinations”\", 23 October 2026","sourceUrl":"https://www.nyu.edu/about/news-publications/news/2026/october/computer-scientists-build-innovative-tech-tools-to-find-experime.html","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","developers-agents"],"finding":"NYU announced two free web-hosted LLM tools led by Professor Dennis Shasha, described in two MDPI Information articles. HallucinationNerd breaks a document into claims, retrieves the cited sources and grades each claim-citation pair as supported, partially supported, not supported, contradicted, cited article does not exist, or could not access. ProtocolNerd lets biologists and chemists describe an experiment in plain English, asks clarifying questions, searches Protocols.io and PubMed (Europe PMC and PubMed for chemistry) with keyword and semantic retrieval, and explains its rankings. The ProtocolNerd paper (published 22 Oct) reports a citation-grounded benchmark of 100 molecular-biology papers: using a short LLM-generated query from each abstract, ProtocolNerd placed the protocol actually cited in the Methods within its top 10 results 46% of the time, significantly better than two baselines and than Elicit (20%), Consensus (8%) and Perplexity (5%); in chemistry it found 31% of protocols with Europe PMC alone and 45% including PubMed. The release says HallucinationNerd was validated against arXiv papers and compared with existing systems, but no HallucinationNerd accuracy figures were inspected.","sledRelevance":"Shows university-built, discipline-specific research assistants with published benchmarks, a useful pattern for research libraries and research computing groups. The ProtocolNerd result is better than general assistants yet misses the correct protocol in most cases, and citation checking addresses a known AI-writing failure that research integrity offices care about.","evidence":"Read the full NYU release (publication date 23 Oct 2026 in page markup): tool descriptions, grading categories, validation approach, funding (NYU Wireless; ARPA-E DE-AR0002025). Read the ProtocolNerd MDPI Information abstract and dates (received 8 Sep, accepted 4 Oct, published 22 Oct 2026; early access) via a fetch tool because direct curl returned HTTP 403: 100-paper benchmark, 46% top-10, Elicit 20%, Consensus 8%, Perplexity 5%, chemistry 31% and 45%. Opened the HallucinationNerd About page for its pipeline and team. HallucinationNerd paper not inspected.","architectureImplications":"Retrieval combining TF-IDF keyword and embedding search over open repositories (Protocols.io, PubMed, Europe PMC), LLM re-ranking with explanations, and claim-level citation checking with evidence quotes.","governanceImplications":"Citation-checking outputs are themselves LLM judgments and need human confirmation before being used in integrity decisions.","securityPrivacyImplications":"Web-hosted tools; uploading unpublished manuscripts to check citations may expose confidential content, and hosting terms were not reviewed.","caveats":"University press release plus an abstract; full ProtocolNerd methods and the HallucinationNerd paper were not inspected (MDPI returned HTTP 403 to direct fetch). Benchmark is the developers' own, 100 papers, with LLM-generated queries; top-10 recall of 46% means the cited protocol was missed in most cases. Co-authors include high-school and external students. No user adoption data. Comparison tools are general assistants, not protocol-specific systems.","streamIds":["research"],"roles":{"sales":"Lead with discipline-specific research assistance. NYU's ProtocolNerd found the cited protocol in its top 10 for 46% of 100 benchmark papers, against 20% for Elicit and 5% for Perplexity, and HallucinationNerd checks whether citations support claims. Research library deans and research integrity officers want tools better than general chatbots. Interpretation: a bounded evaluation of specialised tools fits; this evidence cannot support claims that they find the right protocol most of the time or reliably detect citation errors.","engineering":"Frame the fit around retrieval over open scientific repositories. ProtocolNerd combines keyword and semantic search over Protocols.io, PubMed and Europe PMC with clarifying questions and explained rankings; HallucinationNerd retrieves cited sources and applies a structured LLM judge per claim. Pre-sales should check hosting, data retention and whether confidential manuscripts would be uploaded. Interpretation: run a local benchmark in the institution's disciplines before endorsing either tool.","delivery":"Treat delivery as a supported tool evaluation. Work covers selecting pilot labs, building a small local benchmark of known protocols and seeded citation errors, running the tools, and producing guidance on verification. The research library or research integrity office owns guidance, with research computing advising on hosting. Interpretation: acceptance should track measured recall and citation-check accuracy on local material, since published figures come from developer benchmarks."},"roleDetails":{"sales":{"customerProblem":"Researchers spend time hunting for suitable protocols and checking citations, and general AI assistants perform poorly on both.","stakeholders":["Research library dean","Research integrity officer","Lab managers","Graduate school"],"discoveryQuestions":["How do labs find and adapt experimental protocols today?","Do you check AI-assisted manuscripts for unsupported citations?","Can unpublished text be sent to external tools?"],"valueHypothesis":"Specialised assistants may outperform general chatbots for protocol discovery and citation checking, if validated locally.","potentialEngagement":"A short evaluation of specialised research assistants against general tools on local protocols and seeded citation errors.","unsupportedClaims":["Finds the right protocol most of the time","Reliably detects citation hallucinations","Validated by independent users"]},"engineering":{"fit":"Fits life-science and chemistry research groups using open protocol and literature repositories.","architectureIntegration":"Hybrid keyword and embedding retrieval over Protocols.io, PubMed and Europe PMC with LLM re-ranking, plus claim-level citation retrieval and judging.","prerequisites":["Approval for web-hosted tool use","Local benchmark material","Guidance on confidential documents"],"constraints":["Developer benchmark only","46% top-10 recall","HallucinationNerd accuracy not inspected"],"security":"Avoid uploading unpublished manuscripts until hosting and retention terms are reviewed.","proofOfValue":["Top-10 recall on local protocol set","Citation-check accuracy on seeded errors","Comparison with currently used assistants"]},"delivery":{"work":"Select pilot labs, build benchmarks, run tools, compare results and publish usage guidance.","dependencies":["Lab participation","Tool availability","Library staff time"],"ownership":"Research library or research integrity office, with research computing advice.","skillsAdoption":"Researchers learn to answer clarifying questions well and verify suggested protocols and citation verdicts.","governanceCheckpoints":["Data handling review","Benchmark results review","Guidance approval"],"acceptanceCriteria":["Proposed: recall measured on a local protocol set","Proposed: citation-check accuracy measured on seeded errors","Proposed: guidance published to pilot labs"],"risks":["Over-trust in tool verdicts","Confidential data exposure","Tool sustainability"]}},"retrievedAt":"2026-10-25T22:05:16-05:00","enrichedAt":"2026-10-25T22:05:17-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Plain-language interface and clarifying questions may help early-career researchers; domain experts can extend ProtocolNerd without programming, per the release.","procurementImplications":"Free academic tools; institutions should check hosting, data retention and sustainability before recommending them.","operatingModelImplications":"Research libraries could curate and support such discipline-specific tools alongside general assistants, with clear guidance on verification.","sourceVerification":{"openedUrl":"https://www.nyu.edu/about/news-publications/news/2026/october/computer-scientists-build-innovative-tech-tools-to-find-experime.html","referenceExcerpt":"Hallucination Nerd permits users to check whether the claims a document makes about its citations are, in fact, backed up by the cited documents.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}