From the Emergency Services edition of October 31, 2026
Fire Science and Engineering: ChatGPT given Pre-KTAS training materials matched 119 paramedics' triage accuracy (0.81 vs 0.80) on 100 simulated scenarios, with lower specificity
Ki-Ung Kan (Cheongju Dongbu Fire Station, Chungcheongbuk-do Fire Service HQ) and Byungwoo Kim (Korea National University of Transportation); Korean Institute of Fire Science and Engineering · EMS / prehospital triage research · South Korea
- Publisher
- Fire Science and Engineering (Korean Institute of Fire Science and Engineering)
- Original publication
- 2026-10-31
- Source retrieved
- 2026-11-01
What happened
A simulation study published online Oct. 31 compared a non-trained LLM, an LLM given Pre-KTAS (Prehospital Korean Triage and Acuity Scale) educational materials, and 10 active-duty 119 paramedics on 100 scenarios. Each LLM ran every scenario 30 times. Overall accuracy was 0.69 ± 0.10 for the non-trained LLM, 0.81 ± 0.03 for the trained LLM and 0.80 ± 0.03 for paramedics. The trained LLM and the paramedics were more accurate than the non-trained model, with no significant difference between them. In the Korean abstract, the trained LLM had higher consistency (ICC 0.940 vs 0.836) and higher agreement with paramedics (kappa 0.66 vs 0.56), along with the lowest under-triage rate (6.1%) and the highest sensitivity (0.95) and NPV (0.97). Paramedics had the highest specificity (0.96) and PPV (0.93). AUC was 0.934 for both the trained LLM and the paramedics. The authors say field validation and institutional and ethical guidance are needed before clinical use.
Why it matters
It's peer-reviewed, practitioner-led evidence that a general-purpose LLM given protocol materials can match paramedics on a standard triage scale in simulation. US EMS agencies looking at LLM triage aids can use it to size a validation study, since the result is about protocol grounding and not about field performance.
Evidence and measured results
Simulation-based comparative study (100 written scenarios, 30 repetitions per LLM, 10 paramedics) analysed with Welch's ANOVA, Games-Howell, ICC, kappa, chi-square and ROC. I inspected only the article landing page with its English and Korean abstracts. I didn't read the full-text PDF, so the scenario design, ChatGPT version and prompts weren't checked.
Limitations and uncertainty
Simulated text scenarios, not live calls or field encounters. Small paramedic comparison group (10). Korean triage scale. Only the abstract was inspected. Interpretation: transferability to US protocols (for example ESI or MPDS) is untested.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-10-31; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- EMS agencies want consistent prehospital triage but lack evidence on whether general LLM tools are safe triage aids.
- Stakeholders
- EMS medical director
- Fire/EMS chief
- Paramedic training officers
- Privacy officer
- Hospital partners
- Discovery questions
- Which triage scale do your crews use, and is it digitised?
- What under-triage rate would your medical director accept?
- Would patient data ever reach a commercial LLM?
- Value hypothesis
- Protocol-grounded LLMs may improve triage consistency as decision support, pending field validation (interpretation).
- Potential engagement
- A simulation and retrospective-case validation study of an LLM triage aid against local protocols.
- Unsupported claims
- Field accuracy
- Patient outcome improvement
- Transferability to US triage scales
- Safety for unsupervised use
Pre-sales engineering
Role takeaway
- Fit
- Fits EMS agencies exploring decision support aligned to a written triage protocol.
- Architecture and integration
- General LLM supplied with protocol materials (retrieval or context) as an advisory tool beside paramedic judgment.
- Prerequisites
- Digitised local triage protocol
- Validated scenario set
- Model version control
- Data protection arrangements
- Constraints
- Simulated scenarios only
- Korean Pre-KTAS scale
- Model version and prompts not checked (abstract only)
- Lower specificity than paramedics
- Security
- Keep identifiable patient data out of commercial models unless contracts and safeguards permit it.
- Proof of value
- Replicate on local protocol scenarios
- Retrospective real-case under-/over-triage rates
- Run-to-run consistency across model updates
Delivery
Role takeaway
- Work
- Run staged validation from simulation to supervised shadow use.
- Dependencies
- Medical director sponsorship
- Scenario and case data
- Model access
- Ownership
- EMS medical director.
- Skills and adoption
- Crews learn to treat LLM output as advisory; trainers use disagreements as teaching cases.
- Governance checkpoints
- Medical director sign-off on accuracy thresholds
- Privacy review
- Re-validation after model updates
- Acceptance criteria
- Proposed: under-triage at or below an agreed ceiling on retrospective cases
- Proposed: agreement with clinician reference above an agreed kappa
- Proposed: stable results across repeated runs
- Risks
- Over-triage burden
- Model drift
- Health-data exposure
- Crew over-trust
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Grounding a general LLM in the local triage protocol ('trained' here means given educational materials, not fine-tuned) materially improved accuracy and consistency, which points to retrieval or protocol-context designs.
Governance
Who approves, reviews and stays accountable for outcomes?
The authors explicitly call for field validation and institutional and ethical guidance before clinical use. The higher sensitivity and lower specificity profile means more over-triage, which medical directors would need to accept explicitly.
Security and privacy
What data, permissions and controls need testing?
Use of a commercial chatbot with real patient data would raise health-data disclosure issues. This study used simulated scenarios.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Could support less-experienced EMS staff with triage consistency, but the study doesn't test use under time pressure.
Procurement
What should contracts, pricing and exit terms secure?
Shows protocol-grounded general LLMs may be enough for triage-support pilots, though any procurement would need clinical validation and data protections.
Operating model
Which teams own the service once it runs?
Positions the LLM as decision support for paramedics, not a replacement, with medical-director oversight of over- and under-triage trade-offs.
Publication history
- 2026-10-31Emergency Services · Issue 555 resources
Stable resource ID: kifse-llm-chatgpt-prehospital-triage-pre-ktas-simulation-vs-119-paramedics-20261031