{"resourceId":"mygpt-st-jude-privacy-first-local-rag-research-20261024","versions":[{"version":"external-5d5f56ff8fefcd7869fe04249a2625bfdd083da3f0bdafb4db724b99bb148326","resource":{"id":"mygpt-st-jude-privacy-first-local-rag-research-20261024","title":"Nature Communications (St. Jude): MyGPT, an open-source local RAG platform for sensitive research and clinical documents, reaches 94.85% accuracy on a private SOP library, but reviewers found its hallucination metrics weak and superiority claims underpowered","organization":"St. Jude Children's Research Hospital (Center of Excellence for Data-Driven Discovery, Department of Structural Biology, Office of Data Science, Research Information Services)","sector":"Research institution; research data governance and knowledge work","geography":"Memphis, Tennessee, United States","publishedAt":"24 October 2026 (Nature Communications, accelerated article preview; received 14 Oct 2025, accepted 21 Sep 2026)","publicationDate":"2026-10-24","eventDate":null,"sourceName":"Nature Communications","sourceLabel":"Patel, Downing et al., \"Democratizing reliable knowledge-seeking with MyGPT: a privacy-first, open-source retrieval-augmented generation platform\", Nature Communications, 24 October 2026","sourceUrl":"https://www.nature.com/articles/s41467-026-78509-6","evidenceClass":"academic-research","outcomeClass":"mixed","topics":["knowledge-work","data-security","infrastructure","operating-model"],"finding":"St. Jude researchers describe MyGPT, an open-source retrieval-augmented generation (RAG) platform that runs open-source LLMs locally or inside secure institutional environments, so users can query their own libraries (literature, confidential clinical documents, multilingual policy texts) without data leaving the environment, with source citation and three confidence metrics (question relevance, answer relevance and a \"hallucination index\"). The peer review file shows the first version was compared only with a standalone LLM on 40 questions and was criticised as a wrapper with an unvalidated hallucination index. The revised paper benchmarks against RAGFlow, Dify, PrivateGPT and Kotaemon on PubMedQA, BioASQ and Open RAG Benchmark subsets and reports 129 of 136 correct answers (94.85%) on a private bone-marrow-transplant SOP library, and kinase literature curation in about 2 hours versus 30-40 hours manually. Reviewers noted the margin over RAGFlow is small (91.4% vs 90.2% on BioASQ) and underpowered by the authors' own power analysis (210 documents used versus 5,106 needed), confidence-metric effect sizes are small (r = 0.11-0.17; random-forest ROC-AUC about 74%), evaluator blinding and inter-rater agreement were unclear, and kinase counts were inconsistent between sections. It was deployed on St. Jude VMs approved for protected health information.","sledRelevance":"Directly relevant to research universities and academic medical centres that want generative AI over confidential research, clinical or grant documents without sending data to commercial services. It shows a credible open-source pattern and a realistic evidence profile: useful accuracy in one institution's use cases, but no proof of superiority over other open-source RAG tools and confidence scores that are triage signals rather than truth checks.","evidence":"Opened the article page (open access, published 24 Oct 2026; accelerated preview showing abstract, affiliations, funding, dates). The main-text PDF link redirected to the article page, so results were taken from the open-access Supplementary Information (model hosting requirements table) and the Transparent Peer Review file: four reviewer reports in round one, two in round two, and author rebuttals, which state the benchmark datasets, comparators, 94.85% (129/136) SOP accuracy, 2 vs 30-40 hours, 91.4% vs 90.2%, power analysis (210 vs 5,106), r = 0.11-0.17, ROC-AUC about 74%, and the 282 vs 297 Q&A-pair inconsistency.","architectureImplications":"Self-hosted stack: open-source LLMs served via Ollama, embedding-based vector search combined with BM25 keyword search and cross-encoder reranking, Docker Compose deployment for CPU or GPU, and options from laptops and edge devices to institutional VMs. Supplementary tables list 8B-class models needing one GPU and 8 GB of RAM and 70B models needing 64 GB.","governanceImplications":"Institutions should treat the confidence metrics as risk indicators that trigger human review, not as hallucination detectors, and require local validation on their own document sets with blinded scoring before relying on answers.","securityPrivacyImplications":"Data stays within the institution's environment; St. Jude ran it on VMs approved for PHI with information security team support. Local hosting still requires access control, logging and document-level permissions that the paper does not evaluate.","caveats":"Single-institution evaluation by the developers. Final main text was not accessible (accelerated preview); figures come from the peer review file and supplementary materials, which may differ slightly from the version of record. Reviewers flagged underpowered comparisons with RAGFlow, small confidence-metric effect sizes, unclear evaluator blinding, a reranker omitted in the multilingual evaluation, and inconsistent kinase Q&A counts. Cost advantage over commercial services was not quantified with installation, maintenance or staff time.","streamIds":["research"],"roles":{"sales":"Lead with private AI over sensitive research documents. St. Jude's peer-reviewed MyGPT platform runs open-source models inside institutional environments, reports 94.85% accuracy on a private clinical SOP library and cut one literature-curation task from 30-40 hours to about 2. Research CIOs and data-governance offices want this without commercial data exposure. Interpretation: a bounded pilot on one document library fits; the evidence cannot support claims that it beats other open-source tools or that its scores detect hallucinations.","engineering":"Frame the fit around a self-hosted RAG stack. MyGPT combines vector and BM25 search with cross-encoder reranking, serves open-source models through Ollama, deploys with Docker on CPU or GPU, and ran on St. Jude VMs approved for PHI. Pre-sales should confirm GPU and memory sizing, document permissions and identity integration. Interpretation: plan a local benchmark with blinded scoring, because reviewers found the confidence metrics weakly discriminating and the comparison with RAGFlow underpowered.","delivery":"Treat delivery as standing up a governed private RAG service. Work covers hosting, library ingestion, model selection, building an institution-specific question set, blinded accuracy evaluation and user training on citations and confidence scores. Research IT or the data-science office owns operations with information security. Interpretation: acceptance should rest on measured accuracy on the client's own documents and on human review of low-confidence answers, since the published evidence comes from one institution."},"roleDetails":{"sales":{"customerProblem":"Researchers want generative AI over confidential research, clinical or policy documents, but commercial services raise data-leakage and hallucination concerns.","stakeholders":["Research CIO","Chief data or data-science officer","Information security officer","IRB or privacy office","Principal investigators"],"discoveryQuestions":["Which document collections cannot leave your environment?","Do you have approved infrastructure for protected or confidential data?","How would you judge answer accuracy for your users?"],"valueHypothesis":"A locally hosted RAG service may give researchers cited answers over sensitive documents and save curation time, if validated on local content.","potentialEngagement":"A pilot deploying an open-source private RAG service on one document library with a blinded accuracy evaluation.","unsupportedClaims":["Outperforms all other open-source RAG tools","Hallucination index reliably detects false answers","Guaranteed cost savings over commercial services"]},"engineering":{"fit":"Fits institutions with secure VM or GPU capacity and curated document libraries they cannot send to external services.","architectureIntegration":"Ollama-served open-source LLMs with vector and BM25 retrieval, cross-encoder reranking, Docker deployment and source-highlighting interface inside institutional networks.","prerequisites":["Approved secure hosting environment","GPU or memory sized to chosen model","Curated document library"],"constraints":["Single-institution evidence","Multilingual pipeline only partly validated","No full cost analysis"],"security":"Host inside approved environments, enforce document-level access controls and audit logging, and keep confidential libraries off external services.","proofOfValue":["Blinded accuracy test on a local question set","Compare against one alternative open-source RAG tool","Check confidence scores against human-judged errors"]},"delivery":{"work":"Deploy the platform, ingest libraries, configure retrieval and models, build an evaluation set, run blinded scoring and train users.","dependencies":["Security approval of hosting","Domain experts to write and grade questions","Research IT support staff"],"ownership":"Research IT or office of data science, with information security oversight.","skillsAdoption":"Users need training to read citations and treat confidence scores as prompts for review; staff need container and model-serving skills.","governanceCheckpoints":["Data classification and hosting approval","Evaluation protocol with blinding and inter-rater checks","Go-live review of accuracy results"],"acceptanceCriteria":["Proposed: accuracy target met on a blinded local question set","Proposed: no confidential data leaves the approved environment","Proposed: low-confidence answers routed for human review"],"risks":["Users over-trusting confidence metrics","Unquantified support costs","Errors from outdated or wrong source documents"]}},"retrievedAt":"2026-10-24T22:02:42-05:00","enrichedAt":"2026-10-24T22:02:45-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Reviewer installation reports suggest moderate technical skill suffices, but research IT staff time for deployment, tuning and maintenance is not quantified; multilingual support is only partly validated.","procurementImplications":"An open-source alternative to licensing commercial document-chat services; total cost of ownership (hardware, support staff) must be estimated locally because the paper gives no full cost analysis.","operatingModelImplications":"Fits a model where research IT or a data-science office hosts a shared private RAG service and research groups curate their own libraries and evaluation question sets.","sourceVerification":{"openedUrl":"https://www.nature.com/articles/s41467-026-78509-6","referenceExcerpt":"while ensuring that sensitive data never leaves the secure environment.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}