NYU releases ProtocolNerd and HallucinationNerd (23 Oct): ProtocolNerd placed the cited protocol in its top 10 for 46% of 100 benchmark papers, ahead of Elicit (20%), Consensus (8%) and Perplexity (5%)

New York University, Courant Institute School of Mathematics, Computing, and Data Science (Dennis Shasha) with student and external co-authors · Research tools for laboratory scientists and research integrity · New York, United States

Research Universities

Publisher: NYU News
Original publication: 23 October 2026 (NYU News release); ProtocolNerd paper published 22 October 2026 in MDPI Information (early access)
Source retrieved: 2026-10-26

Read original source

What happened

NYU announced two free web-hosted LLM tools led by Professor Dennis Shasha, described in two MDPI Information articles. HallucinationNerd breaks a document into claims, retrieves the cited sources and grades each claim-citation pair as supported, partially supported, not supported, contradicted, cited article does not exist, or could not access. ProtocolNerd lets biologists and chemists describe an experiment in plain English, asks clarifying questions, searches Protocols.io and PubMed (Europe PMC and PubMed for chemistry) with keyword and semantic retrieval, and explains its rankings. The ProtocolNerd paper (published 22 Oct) reports a citation-grounded benchmark of 100 molecular-biology papers: using a short LLM-generated query from each abstract, ProtocolNerd placed the protocol actually cited in the Methods within its top 10 results 46% of the time, significantly better than two baselines and than Elicit (20%), Consensus (8%) and Perplexity (5%); in chemistry it found 31% of protocols with Europe PMC alone and 45% including PubMed. The release says HallucinationNerd was validated against arXiv papers and compared with existing systems, but no HallucinationNerd accuracy figures were inspected.

Why it matters

Shows university-built, discipline-specific research assistants with published benchmarks, a useful pattern for research libraries and research computing groups. The ProtocolNerd result is better than general assistants yet misses the correct protocol in most cases, and citation checking addresses a known AI-writing failure that research integrity offices care about.

Evidence and measured results

Read the full NYU release (publication date 23 Oct 2026 in page markup): tool descriptions, grading categories, validation approach, funding (NYU Wireless; ARPA-E DE-AR0002025). Read the ProtocolNerd MDPI Information abstract and dates (received 8 Sep, accepted 4 Oct, published 22 Oct 2026; early access) via a fetch tool because direct curl returned HTTP 403: 100-paper benchmark, 46% top-10, Elicit 20%, Consensus 8%, Perplexity 5%, chemistry 31% and 45%. Opened the HallucinationNerd About page for its pipeline and team. HallucinationNerd paper not inspected.

Limitations and uncertainty

University press release plus an abstract; full ProtocolNerd methods and the HallucinationNerd paper were not inspected (MDPI returned HTTP 403 to direct fetch). Benchmark is the developers' own, 100 papers, with LLM-generated queries; top-10 recall of 46% means the cited protocol was missed in most cases. Co-authors include high-school and external students. No user adoption data. Comparison tools are general assistants, not protocol-specific systems.

Put this evidence to work

Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-25; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.

Sales

Role takeaway

Customer problem: Researchers spend time hunting for suitable protocols and checking citations, and general AI assistants perform poorly on both.
Stakeholders: Research library dean
Research integrity officer
Lab managers
Graduate school
Discovery questions: How do labs find and adapt experimental protocols today?
Do you check AI-assisted manuscripts for unsupported citations?
Can unpublished text be sent to external tools?
Value hypothesis: Specialised assistants may outperform general chatbots for protocol discovery and citation checking, if validated locally.
Potential engagement: A short evaluation of specialised research assistants against general tools on local protocols and seeded citation errors.
Unsupported claims: Finds the right protocol most of the time
Reliably detects citation hallucinations
Validated by independent users

Pre-sales engineering

Role takeaway

Fit: Fits life-science and chemistry research groups using open protocol and literature repositories.
Architecture and integration: Hybrid keyword and embedding retrieval over Protocols.io, PubMed and Europe PMC with LLM re-ranking, plus claim-level citation retrieval and judging.
Prerequisites: Approval for web-hosted tool use
Local benchmark material
Guidance on confidential documents
Constraints: Developer benchmark only
46% top-10 recall
HallucinationNerd accuracy not inspected
Security: Avoid uploading unpublished manuscripts until hosting and retention terms are reviewed.
Proof of value: Top-10 recall on local protocol set
Citation-check accuracy on seeded errors
Comparison with currently used assistants

Delivery

Role takeaway

Work: Select pilot labs, build benchmarks, run tools, compare results and publish usage guidance.
Dependencies: Lab participation
Tool availability
Library staff time
Ownership: Research library or research integrity office, with research computing advice.
Skills and adoption: Researchers learn to answer clarifying questions well and verify suggested protocols and citation verdicts.
Governance checkpoints: Data handling review
Benchmark results review
Guidance approval
Acceptance criteria: Proposed: recall measured on a local protocol set
Proposed: citation-check accuracy measured on seeded errors
Proposed: guidance published to pilot labs
Risks: Over-trust in tool verdicts
Confidential data exposure
Tool sustainability

Implementation considerations

Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.

Architecture and integration

What must connect, and where does the AI sit in the workflow?

Retrieval combining TF-IDF keyword and embedding search over open repositories (Protocols.io, PubMed, Europe PMC), LLM re-ranking with explanations, and claim-level citation checking with evidence quotes.

Governance

Who approves, reviews and stays accountable for outcomes?

Citation-checking outputs are themselves LLM judgments and need human confirmation before being used in integrity decisions.

Security and privacy

What data, permissions and controls need testing?

Web-hosted tools; uploading unpublished manuscripts to check citations may expose confidential content, and hosting terms were not reviewed.

Accessibility and workforce

Who is affected, and what skills or accommodations follow?

Plain-language interface and clarifying questions may help early-career researchers; domain experts can extend ProtocolNerd without programming, per the release.

Procurement

What should contracts, pricing and exit terms secure?

Free academic tools; institutions should check hosting, data retention and sustainability before recommending them.

Operating model

Which teams own the service once it runs?

Research libraries could curate and support such discipline-specific tools alongside general assistants, with clear guidance on verification.

Publication history

2026-10-25Research Universities · Issue 474 resources

Read preserved resource versions (JSON)

Stable resource ID: nyu-protocolnerd-hallucinationnerd-research-assistants-20261023