From the Research Universities edition of October 24, 2026
Nature Communications (St. Jude): MyGPT, an open-source local RAG platform for sensitive research and clinical documents, reaches 94.85% accuracy on a private SOP library, but reviewers found its hallucination metrics weak and superiority claims underpowered
St. Jude Children's Research Hospital (Center of Excellence for Data-Driven Discovery, Department of Structural Biology, Office of Data Science, Research Information Services) · Research institution; research data governance and knowledge work · Memphis, Tennessee, United States
- Publisher
- Nature Communications
- Original publication
- 24 October 2026 (Nature Communications, accelerated article preview; received 14 Oct 2025, accepted 21 Sep 2026)
- Source retrieved
- 2026-10-25
What happened
St. Jude researchers describe MyGPT, an open-source retrieval-augmented generation (RAG) platform that runs open-source LLMs locally or inside secure institutional environments, so users can query their own libraries (literature, confidential clinical documents, multilingual policy texts) without data leaving the environment, with source citation and three confidence metrics (question relevance, answer relevance and a "hallucination index"). The peer review file shows the first version was compared only with a standalone LLM on 40 questions and was criticised as a wrapper with an unvalidated hallucination index. The revised paper benchmarks against RAGFlow, Dify, PrivateGPT and Kotaemon on PubMedQA, BioASQ and Open RAG Benchmark subsets and reports 129 of 136 correct answers (94.85%) on a private bone-marrow-transplant SOP library, and kinase literature curation in about 2 hours versus 30-40 hours manually. Reviewers noted the margin over RAGFlow is small (91.4% vs 90.2% on BioASQ) and underpowered by the authors' own power analysis (210 documents used versus 5,106 needed), confidence-metric effect sizes are small (r = 0.11-0.17; random-forest ROC-AUC about 74%), evaluator blinding and inter-rater agreement were unclear, and kinase counts were inconsistent between sections. It was deployed on St. Jude VMs approved for protected health information.
Why it matters
Directly relevant to research universities and academic medical centres that want generative AI over confidential research, clinical or grant documents without sending data to commercial services. It shows a credible open-source pattern and a realistic evidence profile: useful accuracy in one institution's use cases, but no proof of superiority over other open-source RAG tools and confidence scores that are triage signals rather than truth checks.
Evidence and measured results
Opened the article page (open access, published 24 Oct 2026; accelerated preview showing abstract, affiliations, funding, dates). The main-text PDF link redirected to the article page, so results were taken from the open-access Supplementary Information (model hosting requirements table) and the Transparent Peer Review file: four reviewer reports in round one, two in round two, and author rebuttals, which state the benchmark datasets, comparators, 94.85% (129/136) SOP accuracy, 2 vs 30-40 hours, 91.4% vs 90.2%, power analysis (210 vs 5,106), r = 0.11-0.17, ROC-AUC about 74%, and the 282 vs 297 Q&A-pair inconsistency.
Limitations and uncertainty
Single-institution evaluation by the developers. Final main text was not accessible (accelerated preview); figures come from the peer review file and supplementary materials, which may differ slightly from the version of record. Reviewers flagged underpowered comparisons with RAGFlow, small confidence-metric effect sizes, unclear evaluator blinding, a reranker omitted in the multilingual evaluation, and inconsistent kinase Q&A counts. Cost advantage over commercial services was not quantified with installation, maintenance or staff time.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-24; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Researchers want generative AI over confidential research, clinical or policy documents, but commercial services raise data-leakage and hallucination concerns.
- Stakeholders
- Research CIO
- Chief data or data-science officer
- Information security officer
- IRB or privacy office
- Principal investigators
- Discovery questions
- Which document collections cannot leave your environment?
- Do you have approved infrastructure for protected or confidential data?
- How would you judge answer accuracy for your users?
- Value hypothesis
- A locally hosted RAG service may give researchers cited answers over sensitive documents and save curation time, if validated on local content.
- Potential engagement
- A pilot deploying an open-source private RAG service on one document library with a blinded accuracy evaluation.
- Unsupported claims
- Outperforms all other open-source RAG tools
- Hallucination index reliably detects false answers
- Guaranteed cost savings over commercial services
Pre-sales engineering
Role takeaway
- Fit
- Fits institutions with secure VM or GPU capacity and curated document libraries they cannot send to external services.
- Architecture and integration
- Ollama-served open-source LLMs with vector and BM25 retrieval, cross-encoder reranking, Docker deployment and source-highlighting interface inside institutional networks.
- Prerequisites
- Approved secure hosting environment
- GPU or memory sized to chosen model
- Curated document library
- Constraints
- Single-institution evidence
- Multilingual pipeline only partly validated
- No full cost analysis
- Security
- Host inside approved environments, enforce document-level access controls and audit logging, and keep confidential libraries off external services.
- Proof of value
- Blinded accuracy test on a local question set
- Compare against one alternative open-source RAG tool
- Check confidence scores against human-judged errors
Delivery
Role takeaway
- Work
- Deploy the platform, ingest libraries, configure retrieval and models, build an evaluation set, run blinded scoring and train users.
- Dependencies
- Security approval of hosting
- Domain experts to write and grade questions
- Research IT support staff
- Ownership
- Research IT or office of data science, with information security oversight.
- Skills and adoption
- Users need training to read citations and treat confidence scores as prompts for review; staff need container and model-serving skills.
- Governance checkpoints
- Data classification and hosting approval
- Evaluation protocol with blinding and inter-rater checks
- Go-live review of accuracy results
- Acceptance criteria
- Proposed: accuracy target met on a blinded local question set
- Proposed: no confidential data leaves the approved environment
- Proposed: low-confidence answers routed for human review
- Risks
- Users over-trusting confidence metrics
- Unquantified support costs
- Errors from outdated or wrong source documents
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Self-hosted stack: open-source LLMs served via Ollama, embedding-based vector search combined with BM25 keyword search and cross-encoder reranking, Docker Compose deployment for CPU or GPU, and options from laptops and edge devices to institutional VMs. Supplementary tables list 8B-class models needing one GPU and 8 GB of RAM and 70B models needing 64 GB.
Governance
Who approves, reviews and stays accountable for outcomes?
Institutions should treat the confidence metrics as risk indicators that trigger human review, not as hallucination detectors, and require local validation on their own document sets with blinded scoring before relying on answers.
Security and privacy
What data, permissions and controls need testing?
Data stays within the institution's environment; St. Jude ran it on VMs approved for PHI with information security team support. Local hosting still requires access control, logging and document-level permissions that the paper does not evaluate.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Reviewer installation reports suggest moderate technical skill suffices, but research IT staff time for deployment, tuning and maintenance is not quantified; multilingual support is only partly validated.
Procurement
What should contracts, pricing and exit terms secure?
An open-source alternative to licensing commercial document-chat services; total cost of ownership (hardware, support staff) must be estimated locally because the paper gives no full cost analysis.
Operating model
Which teams own the service once it runs?
Fits a model where research IT or a data-science office hosts a shared private RAG service and research groups curate their own libraries and evaluation question sets.
Publication history
- 2026-10-24Research Universities · Issue 464 resources
Stable resource ID: mygpt-st-jude-privacy-first-local-rag-research-20261024