OBJECTIVE: To assess the influence of an implemented artificial intelligence model predicting pediatric sepsis (defined by IPSO-Improving Pediatric Sepsis Outcomes collaborative) in the emergency department (ED) on human performance measures. MATERIALS AND METHODS: Two ED sites within a large pediatric health system in the Southeastern United States between January 1, 2021 and April 1, 2024. We interviewed ED providers and nurses within 72 hours of caring for a patient identified as potentially having sepsis by the predictive model. Thematic analysis of qualitative data was combined with electronic health record queries to assess measures of human performance, including situation awareness, explainability, human-computer agreement, workload, trust, automation bias, and relationship between staff and patients. RESULTS: We interviewed 40 clinicians. Participants found that the sepsis alert improved situation awareness, leading to changes in patient care management, resource allocation, and/or monitoring. Participants reported an average trust in the model-based alert of 3.8/5. Only 28% (555/1977) of sepsis huddles were done without alert firing, suggesting some automation bias. Treatment with antibiotics for IPSO sepsis cases was similar pre- and post-intervention without a huddle (9.3% vs 10.5%), though treatment doubled with huddle intervention (22.7%). NASA Task Load Index increased from 43 to 57 post-intervention. There was no report of adverse relationships with patients post-intervention. DISCUSSION: Human performance appeared to be generally positive with improved situation awareness and satisfaction with the alert-driven huddle. However, there was some evidence of automation bias and a slight increase in workload with the intervention. CONCLUSION: This study demonstrates the feasibility of evaluating multiple dimensions of human performance using a mixed methods approach for an AI model implemented in clinical practice. Future studies should aim to reduce the measurement burden of human performance metrics associated with AI implementation in acute care settings and assess the correlation between human performance measures and clinical outcomes.

OBJECTIVE: To conduct an independent external validation of an implemented vendor-developed emergency department (ED) pediatric sepsis predictive model. MATERIALS AND METHODS: We performed a retrospective cross-sectional study within 2 ED sites of a large pediatric health system between January 1, 2021 and April 1, 2024. A nurse-facing interruptive alert appeared when the model score exceeded the threshold, triggering clinicians to call a sepsis huddle. We compared model predictive performance with vendor-reported performance using definitions that accounted for model threshold and alert timing in clinical practice. Care processes and patient outcome measures included time to first antibiotics, time to first fluid bolus, 30-day mortality, ED to ICU admission rate, and ICU free days. RESULTS: The pre-intervention cohort consisted of 268 102 ED visits with 741 (0.28%) sepsis cases. The post-intervention cohort consisted of 331 061 ED visits with 1114 (0.34%) sepsis cases. Model predictive performance dropped from vendor-reported performance. Mean time to first antibiotic decreased from 112 to 102 minutes (P = .05, 95% confidence interval of difference, -19.1 to 0.1) and time to first bolus decreased by 16.7 minutes (P = .03, 95% confidence interval difference, -31.8 to -1.5) after the intervention. Decreases in 30-day mortality (6% [45/741] to 4% [52/1114]); ED to ICU admissions (87% [646/741] to 84% [941/1114]), and ICU free days (6 to 5) after the intervention did not meet statistical significance. DISCUSSION: Implementing the model led to significant reductions in time to fluid bolus and borderline decreases in time to antibiotics, with non-significant changes in mortality and ICU metrics. When implementing an externally developed model, local workflows, documentation patterns, and patient populations make it challenging to generalize published or reported model performance metrics to real world performance. CONCLUSION: When tailoring a vendor-developed pediatric ED sepsis model for real-world usage, predictive performance differed substantially. Post-implementation we found improvements in care process measures, suggesting such models may benefit sepsis care when adapted for specific clinical workflows.

OBJECTIVE: To assess the prevalence of recommended design elements in implemented electronic health record (EHR) interruptive alerts across pediatric care settings. MATERIALS AND METHODS: We conducted a 3-phase mixed-methods cross-sectional study. Phase 1 involved developing a codebook for alert content classification. Phase 2 identified the most frequently interruptive alerts at participating sites. Phase 3 applied the codebook to classify alerts. Inter-rater reliability (IRR) for the codebook and descriptive statistics for alert design contents were reported. RESULTS: We classified alert content on design elements such as the rationale for the alert's appearance, the hazard of ignoring it, directive versus informational content, administrative purpose, and whether it aligned with one of the Institute of Medicine's (IOM) domains of healthcare quality. Most design elements achieved an IRR above 0.7, with the exceptions for identifying directive content outside of an alert (IRR 0.58) and whether an alert was for administrative purposes only (IRR 0.36). IRR was poor for all IOM domains except equity. Institutions varied widely in the number of unique alerts and their designs. 78% of alerts stated their purpose, over half were directive, and 13% were informational. Only 2%-20% of alerts explained the consequences of inaction. DISCUSSION: This study raises important questions about the optimal balance of alert functions and desirable features of alert representation. CONCLUSION: Our study provides the first multi-center analysis of EHR alert design elements in pediatric care settings, revealing substantial variation in content and design. These findings underline the need for future research to experimentally explore EHR alert design best practices to improve efficiency and effectiveness.

BACKGROUND: Excessive electronic health record (EHR) alerts reduce the salience of actionable alerts. Little is known about the frequency of interruptive alerts across health systems and how the choice of metric affects which users appear to have the highest alert burden. OBJECTIVE: (1) Analyze alert burden by alert type, care setting, provider type, and individual provider across 6 pediatric health systems. (2) Compare alert burden using different metrics. MATERIALS AND METHODS: We analyzed interruptive alert firings logged in EHR databases at 6 pediatric health systems from 2016-2019 using 4 metrics: (1) alerts per patient encounter, (2) alerts per inpatient-day, (3) alerts per 100 orders, and (4) alerts per unique clinician days (calendar days with at least 1 EHR log in the system). We assessed intra- and interinstitutional variation and how alert burden rankings differed based on the chosen metric. RESULTS: Alert burden varied widely across institutions, ranging from 0.06 to 0.76 firings per encounter, 0.22 to 1.06 firings per inpatient-day, 0.98 to 17.42 per 100 orders, and 0.08 to 3.34 firings per clinician day logged in the EHR. Custom alerts accounted for the greatest burden at all 6 sites. The rank order of institutions by alert burden was similar regardless of which alert burden metric was chosen. Within institutions, the alert burden metric choice substantially affected which provider types and care settings appeared to experience the highest alert burden. CONCLUSION: Estimates of the clinical areas with highest alert burden varied substantially by institution and based on the metric used.

OBJECTIVES: We developed and piloted a process for sharing guideline-based clinical decision support (CDS) across institutions, using health screening of newly arrived refugees as a case example. MATERIALS AND METHODS: We developed CDS to support care of newly arrived refugees through a systematic process including a needs assessment, a 2-phase cognitive task analysis, structured preimplementation testing, local implementation, and staged dissemination. We sought consensus from prospective users on CDS scope, applicable content, basic supported workflows, and final structure. We documented processes and developed sharable artifacts from each phase of development. We publically shared CDS artifacts through online dissemination platforms. We collected feedback and implementation data from implementation sites. RESULTS: Responses from 19 organizations demonstrated a need for improved CDS for newly arrived refugee patients. A guided multicenter workflow analysis identified 2 main workflows used by organizations that would need to be supported by shared CDS. We developed CDS through an iterative design process, which was successfully disseminated to other sites using online dissemination repositories. Implementation sites had a small-to-modest analyst time commitment but reported a good match between CDS and workflow. CONCLUSION: Sharing of CDS requires overcoming technical and workflow barriers. We used a guided multicenter workflow analysis and online dissemination repositories to create flexible CDS that has been adapted at 3 sites. Organizations looking to develop sharable CDS should consider evaluating the workflows of multiple institutions and collecting feedback on scope, design, and content in order to make a more generalizable product.

Objective: Electronic health record (EHR) simulation with realistic test patients has improved recognition of safety concerns in test environments. We assessed if simulation affects EHR use patterns in real clinical settings. Materials and Methods: We created a 1-hour educational intervention of a simulated admission for pediatric interns. Data visualization and information retrieval tools were introduced to facilitate recognition of the patient's clinical status. Using EHR audit logs, we assessed the frequency with which these tools were accessed by residents prior to simulation exposure (intervention group, pre-simulation), after simulation exposure (intervention group, post-simulation), and among residents who never participated in simulation (control group). Results: From July 2015 to February 2017, 57 pediatric residents participated in a simulation and 82 did not. Residents were more likely to use the data visualization tool after simulation (73% in post-simulation weeks vs 47% of combined pre-simulation and control weeks, P <. 0001) as well as the information retrieval tool (85% vs 36%, P < .0001). After adjusting for residents' experiences measured in previously completed inpatient weeks of service, simulation remained a significant predictor of using the data visualization (OR 2.8, CI: 2.1-3.9) and information retrieval tools (OR 3.0, CI: 2.0-4.5). Tool use did not decrease in interrupted time-series analysis over a median of 19 (IQR: 8-32) weeks of post-simulation follow-up. Discussion: Simulation was associated with persistent changes to EHR use patterns among pediatric residents. Conclusion: EHR simulation is an effective educational method that can change participants' use patterns in real clinical settings.