{"resourceId":"kifse-llm-chatgpt-prehospital-triage-pre-ktas-simulation-vs-119-paramedics-20261031","versions":[{"version":"external-c5991374b135cd4e787b0a7917e74cc9e0f827596b34649cfe624327d1b06c15","resource":{"id":"kifse-llm-chatgpt-prehospital-triage-pre-ktas-simulation-vs-119-paramedics-20261031","title":"Fire Science and Engineering: ChatGPT given Pre-KTAS training materials matched 119 paramedics' triage accuracy (0.81 vs 0.80) on 100 simulated scenarios, with lower specificity","organization":"Ki-Ung Kan (Cheongju Dongbu Fire Station, Chungcheongbuk-do Fire Service HQ) and Byungwoo Kim (Korea National University of Transportation); Korean Institute of Fire Science and Engineering","sector":"EMS / prehospital triage research","geography":"South Korea","publishedAt":"2026-10-31","publicationDate":"2026-10-31","eventDate":null,"sourceName":"Fire Science and Engineering (Korean Institute of Fire Science and Engineering)","sourceLabel":"Utilization of Large Language Models in Prehospital Triage: A Focus on ChatGPT's Performance According to Pre-KTAS (Fire Sci. Eng. 2026;40(5):107-117)","sourceUrl":"https://www.kifsejournal.or.kr/journal/view.php?doi=10.7731/KIFSE.f0338352","evidenceClass":"academic-research","outcomeClass":"emerging","topics":["knowledge-work","governance-procurement"],"finding":"A simulation study published online Oct. 31 compared a non-trained LLM, an LLM given Pre-KTAS (Prehospital Korean Triage and Acuity Scale) educational materials, and 10 active-duty 119 paramedics on 100 scenarios. Each LLM ran every scenario 30 times. Overall accuracy was 0.69 ± 0.10 for the non-trained LLM, 0.81 ± 0.03 for the trained LLM and 0.80 ± 0.03 for paramedics. The trained LLM and the paramedics were more accurate than the non-trained model, with no significant difference between them. In the Korean abstract, the trained LLM had higher consistency (ICC 0.940 vs 0.836) and higher agreement with paramedics (kappa 0.66 vs 0.56), along with the lowest under-triage rate (6.1%) and the highest sensitivity (0.95) and NPV (0.97). Paramedics had the highest specificity (0.96) and PPV (0.93). AUC was 0.934 for both the trained LLM and the paramedics. The authors say field validation and institutional and ethical guidance are needed before clinical use.","sledRelevance":"It's peer-reviewed, practitioner-led evidence that a general-purpose LLM given protocol materials can match paramedics on a standard triage scale in simulation. US EMS agencies looking at LLM triage aids can use it to size a validation study, since the result is about protocol grounding and not about field performance.","evidence":"Simulation-based comparative study (100 written scenarios, 30 repetitions per LLM, 10 paramedics) analysed with Welch's ANOVA, Games-Howell, ICC, kappa, chi-square and ROC. I inspected only the article landing page with its English and Korean abstracts. I didn't read the full-text PDF, so the scenario design, ChatGPT version and prompts weren't checked.","architectureImplications":"Grounding a general LLM in the local triage protocol ('trained' here means given educational materials, not fine-tuned) materially improved accuracy and consistency, which points to retrieval or protocol-context designs.","governanceImplications":"The authors explicitly call for field validation and institutional and ethical guidance before clinical use. The higher sensitivity and lower specificity profile means more over-triage, which medical directors would need to accept explicitly.","securityPrivacyImplications":"Use of a commercial chatbot with real patient data would raise health-data disclosure issues. This study used simulated scenarios.","caveats":"Simulated text scenarios, not live calls or field encounters. Small paramedic comparison group (10). Korean triage scale. Only the abstract was inspected. Interpretation: transferability to US protocols (for example ESI or MPDS) is untested.","streamIds":["emergency-services"],"roles":{"sales":"For sales, the framing point is that a fire-service paramedic and a university paramedicine professor found a protocol-grounded ChatGPT matched 119 paramedics' triage accuracy in simulation (0.81 vs 0.80). EMS medical directors may want a structured way to test such tools. A bounded engagement is a simulation-to-field validation design using local protocols. The study can't support claims of field accuracy, safety or time savings.","engineering":"For pre-sales engineering, the lesson is that protocol grounding moved accuracy from 0.69 to 0.81 and raised consistency. Prerequisites are a digitised local protocol, scenario sets and controlled model versions. Constraints include simulated text inputs, a Korean scale and an unknown model version. Proof of value should replicate the comparison on local protocols, then measure over- and under-triage on retrospective real cases before any live use.","delivery":"For delivery, the work is a staged validation: simulation with local protocols, retrospective case review, then supervised shadow use. Ownership sits with the EMS medical director. Acceptance should set under-triage ceilings, run-to-run consistency and agreement with clinicians. Risks include over-triage burden, model updates changing behaviour, health-data exposure and over-trust by crews. Training should present the tool as advisory and capture crew disagreements for review."},"roleDetails":{"sales":{"customerProblem":"EMS agencies want consistent prehospital triage but lack evidence on whether general LLM tools are safe triage aids.","stakeholders":["EMS medical director","Fire/EMS chief","Paramedic training officers","Privacy officer","Hospital partners"],"discoveryQuestions":["Which triage scale do your crews use, and is it digitised?","What under-triage rate would your medical director accept?","Would patient data ever reach a commercial LLM?"],"valueHypothesis":"Protocol-grounded LLMs may improve triage consistency as decision support, pending field validation (interpretation).","potentialEngagement":"A simulation and retrospective-case validation study of an LLM triage aid against local protocols.","unsupportedClaims":["Field accuracy","Patient outcome improvement","Transferability to US triage scales","Safety for unsupervised use"]},"engineering":{"fit":"Fits EMS agencies exploring decision support aligned to a written triage protocol.","architectureIntegration":"General LLM supplied with protocol materials (retrieval or context) as an advisory tool beside paramedic judgment.","prerequisites":["Digitised local triage protocol","Validated scenario set","Model version control","Data protection arrangements"],"constraints":["Simulated scenarios only","Korean Pre-KTAS scale","Model version and prompts not checked (abstract only)","Lower specificity than paramedics"],"security":"Keep identifiable patient data out of commercial models unless contracts and safeguards permit it.","proofOfValue":["Replicate on local protocol scenarios","Retrospective real-case under-/over-triage rates","Run-to-run consistency across model updates"]},"delivery":{"work":"Run staged validation from simulation to supervised shadow use.","dependencies":["Medical director sponsorship","Scenario and case data","Model access"],"ownership":"EMS medical director.","skillsAdoption":"Crews learn to treat LLM output as advisory; trainers use disagreements as teaching cases.","governanceCheckpoints":["Medical director sign-off on accuracy thresholds","Privacy review","Re-validation after model updates"],"acceptanceCriteria":["Proposed: under-triage at or below an agreed ceiling on retrospective cases","Proposed: agreement with clinician reference above an agreed kappa","Proposed: stable results across repeated runs"],"risks":["Over-triage burden","Model drift","Health-data exposure","Crew over-trust"]}},"retrievedAt":"2026-10-31T22:05:12-05:00","enrichedAt":"2026-10-31T22:09:49-05:00","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Could support less-experienced EMS staff with triage consistency, but the study doesn't test use under time pressure.","procurementImplications":"Shows protocol-grounded general LLMs may be enough for triage-support pilots, though any procurement would need clinical validation and data protections.","operatingModelImplications":"Positions the LLM as decision support for paramedics, not a replacement, with medical-director oversight of over- and under-triage trade-offs.","peerReview":"peer-reviewed","sourceVerification":{"responseId":"curl-kifse-2026-10-31","openedUrl":"https://www.kifsejournal.or.kr/journal/view.php?doi=10.7731/KIFSE.f0338352","referenceExcerpt":"Overall accuracy was 0.69 ± 0.10 for the non-trained LLM, 0.81 ± 0.03 for the trained LLM, and 0.80 ± 0.03 for paramedics.","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}