{"resourceId":"paramedic-papers-korean-prektas-chatgpt-triage-simulation-20261103","versions":[{"version":"external-7efc54f054a84ba30b20dd1ce4dabff5f3d0c355f9da65520e64084100c91ee7","resource":{"id":"paramedic-papers-korean-prektas-chatgpt-triage-simulation-20261103","title":"Korean simulation: a Pre-KTAS-trained ChatGPT matched paramedics on triage accuracy (0.81 vs 0.80), but only on text scenarios with no field validation","organization":"Fire Science and Engineering journal (summarized by Paramedic Papers)","sector":"Prehospital triage / EMS decision support","geography":"Republic of Korea","publishedAt":"2026-11-03","publicationDate":"2026-11-03","eventDate":null,"sourceName":"Paramedic Papers","sourceLabel":"Evidence-summary service describing a journal study (original journal page returned HTTP 500)","sourceUrl":"https://paramedicpapers.com/papers/doi-10-7731-kifse-f0338352/","evidenceClass":"independent-reporting","outcomeClass":"mixed","topics":["knowledge-work","governance-procurement"],"finding":"Paramedic Papers summarizes a Korean simulation study of 100 prehospital triage scenarios (Pre-KTAS scale), each run 30 times. A Pre-KTAS-trained ChatGPT matched 10 active-duty 119 paramedics on accuracy (0.81 vs 0.80), with the lowest under-triage rate (6.1%) and highest sensitivity (0.95), while paramedics had the highest specificity (0.96) and PPV (0.93). An untrained LLM scored 0.69, significantly lower (p<.001). The summary lists caveats: simulated text scenarios, a Korean triage scale, repetition that may inflate apparent precision, and no field validation, and labels the work hypothesis-generating.","sledRelevance":"A fresh data point that grounding an LLM in the local triage guideline changes results, and that comparable accuracy to paramedics in text simulation says little about live calls. It is relevant to dispatch and prehospital decision support evaluation, with strong transferability limits from Korean protocols and simulated text.","evidence":"Summary page (added 3 November 2026) reports design, participants and main findings of a simulation-based comparative study in Fire Science and Engineering. The underlying journal page could not be opened (DOI redirect returned HTTP 500), so numbers are as reported by the summary and unverified against the paper.","architectureImplications":"Interpretation: guideline grounding or training material appears to matter more than model choice alone; any deployment would need version control of protocol content and a way to test updates before release.","governanceImplications":"Because results are simulated and hypothesis-generating, governance should require prospective field testing and defined under-triage tolerances before any live use.","securityPrivacyImplications":"Not addressed in the summary. Interpretation: sending prehospital case data to a hosted LLM would need privacy review.","caveats":"Secondary summary of a paper the agent could not open; the page itself notes simulated text scenarios, a Korean triage scale, repeated runs and no field validation. Paramedic comparison group is 10 people. Peer-review status of the journal paper was not confirmed by the agent.","streamIds":["emergency-services"],"roles":{"sales":"Treat this as an early signal, not a selling point. A protocol-trained LLM matched 10 Korean paramedics on triage accuracy in text simulations, and the untrained model lagged, but there was no field testing and the source is a secondary summary. Use it to start discussions about the role of local protocol grounding and under-triage tolerance. Do not claim that AI can triage as well as paramedics in live emergencies or that results transfer to U.S. protocols.","engineering":"The takeaway for pre-sales engineering is that retrieval of or training on the local triage guideline drove the gap between the trained and untrained models. Proofs of value should test on the agency's own protocol and real call or case records, measure under-triage separately from overall accuracy, and run repeated trials to expose variability. Note repetition can inflate apparent precision. The primary paper was not accessible, so verify numbers before reuse.","delivery":"If an agency explores LLM triage support, plan a simulation phase on local protocol scenarios, then a shadow-mode field phase, with under-triage as a primary safety measure and the medical director owning the protocol content fed to the model. Governance should block live prompts until prospective results exist. Acceptance criteria must be proposed locally; the Korean figures are not a benchmark for U.S. systems."},"roleDetails":{"sales":{"customerProblem":"Agencies want to know whether LLMs can safely support triage decisions, but published evidence is early.","stakeholders":["EMS medical director","Dispatch center director","Clinical informatics lead","Risk and legal counsel"],"discoveryQuestions":["Which triage protocol would the tool need to follow, and who maintains it?","What under-triage rate is clinically acceptable?","What field testing would you require before live use?"],"valueHypothesis":"Protocol-grounded LLM decision support may merit a bounded simulation and shadow-mode evaluation; this simulation does not show live benefit.","potentialEngagement":"Bounded simulation study design using the agency's own protocol and de-identified cases, ending in a go/no-go on shadow-mode testing.","unsupportedClaims":["That LLMs triage as well as paramedics in live calls","That Korean Pre-KTAS results apply to U.S. protocols","That the paper's numbers are verified, since the primary page was inaccessible"]},"engineering":{"fit":"Exploratory only; relevant as a design input for evaluation methodology.","architectureIntegration":"Protocol-grounded LLM (guideline provided as training or retrieval context) behind a decision-support interface, with version-controlled protocol content and full logging.","prerequisites":["Machine-readable local triage protocol","De-identified scenario set with reference answers","Evaluation harness with repeated runs"],"constraints":["Text-only simulation","Hosted-model data privacy limits","Protocol differences across jurisdictions"],"security":"Not addressed; keep prehospital case data out of unapproved hosted models and review vendor data terms.","proofOfValue":["Compare trained and untrained configurations on local scenarios","Report under-triage separately from accuracy","Shadow-mode test on real cases before live prompts"]},"delivery":{"work":"Run a simulation phase on local scenarios, then shadow mode, with protocol-content ownership and under-triage monitoring.","dependencies":["Medical director approval of protocol content","Scenario dataset with expert reference labels","Privacy review of data flows"],"ownership":"EMS medical director for protocol and clinical safety; informatics lead for the evaluation harness.","skillsAdoption":"Training for evaluators on repeated-run variability and under-triage review.","governanceCheckpoints":["Approval of scenario set","Go/no-go for shadow mode","Safety review of under-triage cases"],"acceptanceCriteria":["Proposed: under-triage rate at or below a threshold set by the medical director","Proposed: results reproducible across repeated runs","Proposed: no live prompts until shadow-mode results are reviewed"],"risks":["Simulation results not generalizing","Under-triage harm","Protocol drift between model context and current guideline","Unverified source numbers"]}},"retrievedAt":"2026-11-11T04:02:08Z","enrichedAt":"2026-11-11T04:02:08Z","enrichmentBasis":"retrieved source","accessibilityWorkforceImplications":"Compares an LLM against 10 active-duty paramedics; the summary does not address workforce effects. Interpretation: positioned as support, not replacement, until field data exist.","procurementImplications":"Interpretation: require vendors to demonstrate performance on the buyer's own protocol and to disclose how protocol content is supplied to the model.","operatingModelImplications":"Interpretation: a protocol-owner role and controlled update process would be needed to keep the model's guidance aligned with current guidelines.","peerReview":"unknown","sourceVerification":{"openedUrl":"https://paramedicpapers.com/papers/doi-10-7731-kifse-f0338352/","referenceExcerpt":"found a Pre-KTAS-trained ChatGPT matched 10 active-duty paramedics on accuracy (0.81 vs 0.80) with the lowest under-triage rate (6.1%)","promptVersion":"sled-research-v4.0","model":null,"basis":"agent-reported inspection"}}}]}