{"resourceId":"heart-lung-general-llms-unreliable-prehospital-ecg-cath-lab-activation-20260915","versions":[{"version":"external-4efd7b0692660050c4b899cda06843161d7c778c203e79d9d9690992d6409abb","resource":{"id":"heart-lung-general-llms-unreliable-prehospital-ecg-cath-lab-activation-20260915","title":"General-purpose LLMs misread prehospital ECGs for cath-lab activation; monitor algorithm outperformed all three","organization":"LSU Health Sciences Center-Shreveport","sector":"Emergency medical services research","geography":"Shreveport, Louisiana, USA","publishedAt":"2026-09-15","publicationDate":"2026-09-15","eventDate":null,"sourceName":"Heart & Lung: The Journal of Cardiopulmonary and Acute Care","sourceLabel":"Heart & Lung study of LLMs on prehospital ECGs","sourceUrl":"https://www.heartandlung.org/article/S0147-9563(26)00240-2/fulltext","evidenceClass":"academic-research","outcomeClass":"cautionary","topics":["knowledge-work","governance-procurement","data-security"],"finding":"A retrospective single-center study in Heart & Lung (published online Sept 15, 2026; LSU Health Sciences Center-Shreveport) tested three general-purpose multimodal LLMs (ChatGPT-5, Claude Sonnet 4, Gemini 2.5 Pro) on 615 prehospital 12-lead ECG images from 270 Shreveport EMS encounters (2019-2024) with suspected acute MI, against cardiology STEMI-pathway activation as the reference (101 activations, 232 positive ECGs). Gemini had 95.3% sensitivity but 9.4% specificity; ChatGPT 68.1%/42.3% and Claude 67.2%/46.5%. The monitor's built-in Glasgow algorithm was more balanced at 67.7% sensitivity and 64.2% specificity, with the highest accuracy (65.5% versus 41.8-54.3%); LLM PPV ranged 38.9-43.2%. Authors conclude general-purpose LLMs are not appropriate for cath-lab activation decisions and should be limited to education and documentation, and stress the results do not generalize to dedicated AI-ECG models. Why now: tonight's Jeju result credits a purpose-built ECG AI; this study shows why buyers must distinguish dedicated ECG tools from general chatbots that paramedics can reach on a phone.","sledRelevance":"EMS agencies need policy on clinicians using consumer chatbots for ECG reads; this U.S. study gives medical directors evidence to prohibit it while evaluating purpose-built tools separately.","evidence":"Peer-reviewed retrospective diagnostic-accuracy study with open dataset on Figshare.","architectureImplications":"General multimodal LLMs read cropped ECG images without waveform data or context; dedicated ECG models use signal-specific training.","governanceImplications":"Supports explicit EMS policy banning general-purpose LLMs for diagnostic or activation decisions and requiring validated, regulated tools.","securityPrivacyImplications":"Uploading patient ECG images to consumer AI services raises privacy concerns beyond accuracy.","caveats":"Single-center, retrospective, zero-shot prompt, static cropped images without clinical context; encounter-level labels applied to individual ECGs; reference standard is activation decisions, not angiographic occlusion for non-activated patients; no repeat runs to measure variability; published five weeks before this edition; opened via WebFetch because box curl returned 403, so publisher excerpt checks may fail.","streamIds":["emergency-services"],"roles":{"sales":"For sales, this study is a guardrail, not a pitch. It lets you help EMS medical directors separate purpose-built ECG AI from general chatbots paramedics might try on a phone. Offer a policy and tool-evaluation review, and never cite general LLM performance as support for any ECG product. Equally, do not cite this study as proof that dedicated tools work; it did not test them.","engineering":"For engineering, the lesson is input fidelity and task fit. The LLMs saw cropped images without waveforms or context, and their thresholds drifted toward false positives. Any ECG AI evaluation should use local tracings, a clear reference standard, sensitivity and specificity with PPV, comparison with the monitor algorithm, and repeated runs to measure output variability. Keep test sets separate from any tuning so results are not inflated.","delivery":"For delivery, translate the finding into policy: no general-purpose LLM for diagnostic or activation decisions, with documentation and education uses allowed only under data-protection rules. Communicate it in paramedic training, add it to QA review, and require any proposed ECG AI to show local validation against cath-lab outcomes before pilot approval. Recheck the policy each year as dedicated tools and evidence change, and record any exceptions the medical director approves."},"roleDetails":{"sales":{"customerProblem":"EMS leaders lack evidence to judge AI ECG claims and to govern informal chatbot use by clinicians.","stakeholders":["EMS medical director","Cardiology STEMI coordinator","EMS quality manager","Agency privacy officer"],"discoveryQuestions":["Do any staff use chatbots for clinical questions?","How do you validate ECG decision tools today?","What is your false-activation rate?"],"valueHypothesis":"A clear AI ECG policy and evaluation method could prevent unsafe tool use and focus pilots on validated products.","potentialEngagement":"Clinical AI use policy and ECG-tool evaluation framework for an EMS agency.","unsupportedClaims":["Performance of dedicated AI-ECG products","Any benefit of LLMs for ECG triage","Results under few-shot or context-rich prompting"]},"engineering":{"fit":"Benchmark evidence against using general multimodal LLMs for prehospital ECG triage.","architectureIntegration":"Not an integration target; informs evaluation protocols for monitor-integrated or dedicated ECG AI.","prerequisites":["Local ECG archive","Reference outcomes from cath-lab logs","Monitor algorithm outputs for comparison"],"constraints":["Single center","Static images only","Zero-shot prompting"],"security":"Block upload of patient ECGs to consumer AI services; log any sanctioned AI use.","proofOfValue":["Specificity and PPV versus monitor algorithm on local ECGs","Output consistency across repeated runs","Performance on noisy field tracings"]},"delivery":{"work":"Adopt and enforce a clinical AI use policy covering ECG interpretation.","dependencies":["Medical direction","Training staff","IT and privacy office"],"ownership":"EMS medical director.","skillsAdoption":"Paramedics learn why general chatbots are unsuitable for ECG decisions and which tools are approved.","governanceCheckpoints":["Policy approval by medical director","Training completion","Periodic audit for unsanctioned use"],"acceptanceCriteria":["Written policy published to all clinicians","100% of clinicians complete training within one quarter","Any new ECG AI pilot shows local validation before approval"],"risks":["Clinician use of consumer chatbots","Overgeneralizing results to dedicated tools","Privacy breaches from image uploads"]}},"retrievedAt":"2026-10-20T22:05:00-05:00","enrichedAt":"2026-10-20T22:05:00-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Authors note clinicians without access to dedicated tools may be tempted to use LLMs; training is the main lever.","procurementImplications":"Procurements should require task-specific validation metrics and exclude general LLMs for ECG decisions.","operatingModelImplications":"Cardiology activation remains a clinician decision; general AI limited to education and documentation.","sourceVerification":{"responseId":"webfetch-heart-lung-llm-ecg-2026-10-20","openedUrl":"https://www.heartandlung.org/article/S0147-9563(26)00240-2/fulltext","referenceExcerpt":"Multimodal LLM interpretation of prehospital ECGs demonstrated clinically unreliable performance for identifying ECGs warranting emergent cardiac catheterization laboratory activation","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}