This focus issue serves to highlight the challenging but highly valuable work of interdisciplinary teams collaborating across traditional scientific silos.1–25 There were several points of origin that motivated our choice to highlight interdisciplinary work. One point of origin that initially motivated us as guest associate editors of this focus issue was our shared personal experiences on high-impact clinical informatics projects.26–30 A second point of origin comes from talking to others who are engaged in similarly scoped efforts like the ICU Cockpit.31,32 For example, Dr Keller who led the ICU Cockpit work has spoken of similar roadblocks, shared experiences, need for time spent communicating and listening, and of how few people understand how difficult and time-consuming these efforts are—often 10-15 years from beginning to deployment. The projects we have been part of, and projects of similar scope whose leaders we have commiserated with, took years of collaborative effort from large interdisciplinary teams drawing members across the research and deployment pipelines. The collaborative clinical informatics projects that motivated this focus issue included highly engaged experts spanning a range of diverse teams of practicing clinicians, computational scientists, human–computer interaction and implementation science researchers, informaticians, and operational engineers who run the day-to-day electronic health record (EHR) systems. Interestingly, we observed that experts from diverse teams who presented components of these collaborative projects outside of their direct field of application were met with misunderstandings which manifested in dismissal of ideas, underestimation of the difficulty of another field’s problems, underestimation of the deep innovation required to translate and assemble the science, and general undervaluation of translating research to operations within the clinical informatics space. It is well known that open communication across highly distinct scientific domains is required to solve complex real-world problems and is the reason why, for instance, Oppenheimer fought so hard for open dialog between all scientists working on the Manhattan project.33 A third point of origin was our belief that clinical informatics communities could increase their engagement in interdisciplinary work and that missed opportunities for impact abound when collaborative interdisciplinary expertise is lacking. The goal of increasing dialog between distinct fields to drive increased interdisciplinary work served as a motivation for the Banff International Research Station titled: Dynamics and Data Assimilation, Physiology and Bioinformatics: Mathematics at the Interface of Theory and Clinical Applications in 2022 comprising researchers from a wide variety of interdisciplinary fields including several of the focus issue guest associate editors who were motivated to move these ideas forward through a journal focus issue. Despite some high-profile examples of interdisciplinary work, our anecdotal experience has been that examples of cross-field communication and interdisciplinary teams are hard to identify in peer-reviewed literature. We would like to see this change.

OBJECTIVE: To identify stigmatizing language in obstetric clinical notes using natural language processing (NLP). MATERIALS AND METHODS: We analyzed electronic health records from birth admissions in the Northeast United States in 2017. We annotated 1771 clinical notes to generate the initial gold standard dataset. Annotators labeled for exemplars of 5 stigmatizing and 1 positive/preferred language categories. We used a semantic similarity-based search approach to expand the initial dataset by adding additional exemplars, composing an enhanced dataset. We employed traditional classifiers (Support Vector Machine, Decision Trees, and Random Forest) and a transformer-based model, ClinicalBERT (Bidirectional Encoder Representations from Transformers) and BERT base. Models were trained and validated on initial and enhanced datasets and were tested on enhanced testing dataset. RESULTS: In the initial dataset, we annotated 963 exemplars as stigmatizing or positive/preferred. The most frequently identified category was marginalized language/identities (n = 397, 41%), and the least frequent was questioning patient credibility (n = 51, 5%). After employing a semantic similarity-based search approach, 502 additional exemplars were added, increasing the number of low-frequency categories. All NLP models also showed improved performance, with Decision Trees demonstrating the greatest improvement (21%). ClinicalBERT outperformed other models, with the highest average F1-score of 0.78. DISCUSSION: Clinical BERT seems to most effectively capture the nuanced and context-dependent stigmatizing language found in obstetric clinical notes, demonstrating its potential clinical applications for real-time monitoring and alerts to prevent usages of stigmatizing language use and reduce healthcare bias. Future research should explore stigmatizing language in diverse geographic locations and clinical settings to further contribute to high-quality and equitable perinatal care. CONCLUSION: ClinicalBERT effectively captures the nuanced stigmatizing language in obstetric clinical notes. Our semantic similarity-based search approach to rapidly extract additional exemplars enhanced the performances while reducing the need for labor-intensive annotation.

OBJECTIVES: Well-designed electronic health records (EHRs) training programs for clinical practice are known to be valuable. Training programs should be role-specific and there is a need to identify key implementation factors of EHR training programs for nurses. This scoping review (1) characterizes the EHR training programs used and (2) identifies their implementation facilitators and barriers. MATERIALS AND METHODS: We searched MEDLINE, CINAHL, PsycINFO, and Web of Science on September 3, 2023, for peer-reviewed articles that described EHR training program implementation or delivery to nurses in inpatient settings without any date restrictions. We mapped implementation factors to the Consolidated Framework for Implementation Research. Additional themes were inductively identified by reviewing these findings. RESULTS: This review included 30 articles. Healthcare systems' approaches to implementing and delivering EHR training programs were highly varied. For implementation factors, we observed themes in innovation (eg, ability to practice EHR skills after training is over, personalizing training, training pace), inner setting (eg, availability of computers, clear documentation requirements and expectations), individual (eg, computer literacy, learning preferences), and implementation process (eg, trainers and support staff hold nursing backgrounds, establishing process for dissemination of EHR updates). No themes in the outer setting were observed. DISCUSSION: We found that multilevel factors can influence the implementation and delivery of EHR training programs for inpatient nurses. Several areas for future research were identified, such as evaluating nurse preceptorship models and developing training programs for ongoing EHR training (eg, in response to new EHR workflows or features). CONCLUSIONS: This scoping review highlighted numerous factors pertaining to training interventions, healthcare systems, and implementation approaches. Meanwhile, it is unclear how external factors outside of a healthcare system influence EHR training programs. Additional studies are needed that focus on EHR retraining programs, comparing outcomes of different training models, and how to effectively disseminate updates with the EHR to nurses.

OBJECTIVES: Little is known about proactive risk assessment concerning emergency department (ED) visits and hospitalizations in patients with heart failure (HF) who receive home healthcare (HHC) services. This study developed a time series risk model for predicting ED visits and hospitalizations in patients with HF using longitudinal electronic health record data. We also explored which data sources yield the best-performing models over various time windows. MATERIALS AND METHODS: We used data collected from 9362 patients from a large HHC agency. We iteratively developed risk models using both structured (eg, standard assessment tools, vital signs, visit characteristics) and unstructured data (eg, clinical notes). Seven specific sets of variables included: (1) the Outcome and Assessment Information Set, (2) vital signs, (3) visit characteristics, (4) rule-based natural language processing-derived variables, (5) term frequency-inverse document frequency variables, (6) Bio-Clinical Bidirectional Encoder Representations from Transformers variables, and (7) topic modeling. Risk models were developed for 18 time windows (1-15, 30, 45, and 60 days) before an ED visit or hospitalization. Risk prediction performances were compared using recall, precision, accuracy, F1, and area under the receiver operating curve (AUC). RESULTS: The best-performing model was built using a combination of all 7 sets of variables and the time window of 4 days before an ED visit or hospitalization (AUC = 0.89 and F1 = 0.69). DISCUSSION AND CONCLUSION: This prediction model suggests that HHC clinicians can identify patients with HF at risk for visiting the ED or hospitalization within 4 days before the event, allowing for earlier targeted interventions.

OBJECTIVE: Understand the perceived role of electronic health records (EHR) and workflow fragmentation on clinician documentation burden in the emergency department (ED). METHODS: From February to June 2022, we conducted semistructured interviews among a national sample of US prescribing providers and registered nurses who actively practice in the adult ED setting and use Epic Systems' EHR. We recruited participants through professional listservs, social media, and email invitations sent to healthcare professionals. We analyzed interview transcripts using inductive thematic analysis and interviewed participants until we achieved thematic saturation. We finalized themes through a consensus-building process. RESULTS: We conducted interviews with 12 prescribing providers and 12 registered nurses. Six themes were identified related to EHR factors perceived to contribute to documentation burden including lack of advanced EHR capabilities, absence of EHR optimization for clinicians, poor user interface design, hindered communication, increased manual work, and added workflow blockages, and five themes associated with cognitive load. Two themes emerged in the relationship between workflow fragmentation and EHR documentation burden: underlying sources and adverse consequences. DISCUSSION: Obtaining further stakeholder input and consensus is essential to determine whether these perceived burdensome EHR factors could be extended to broader contexts and addressed through optimizing existing EHR systems alone or through a broad overhaul of the EHR's architecture and primary purpose. CONCLUSION: While most clinicians perceived that the EHR added value to patient care and care quality, our findings underscore the importance of designing EHRs that are in harmony with ED clinical workflows to alleviate the clinician documentation burden.

Clinical notes are becoming an increasingly important data source for machine learning (ML) applications in healthcare. Prior research has shown that deploying ML models can perpetuate existing biases against racial minorities, as bias can be implicitly embedded in data. In this study, we investigate the level of implicit race information available to ML models and human experts and the implications of model-detectable differences in clinical notes. Our work makes three key contributions. First, we find that models can identify patient self-reported race from clinical notes even when the notes are stripped of explicit indicators of race. Second, we determine that human experts are not able to accurately predict patient race from the same redacted clinical notes. Finally, we demonstrate the potential harm of this implicit information in a simulation study, and show that models trained on these race-redacted clinical notes can still perpetuate existing biases in clinical treatment decisions.

OBJECTIVE: To identify the risk factors home healthcare (HHC) clinicians associate with patient deterioration and understand how clinicians respond to and document these risk factors. METHODS: We interviewed multidisciplinary HHC clinicians from January to March of 2021. Risk factors were mapped to standardized terminologies (eg, Omaha System). We used directed content analysis to identify risk factors for deterioration. We used inductive thematic analysis to understand HHC clinicians' response to risk factors and documentation of risk factors. RESULTS: Fifteen HHC clinicians identified a total of 79 risk factors that were mapped to standardized terminologies. HHC clinicians most frequently responded to risk factors by communicating with the prescribing provider (86.7% of clinicians) or following up with patients and caregivers (86.7%). HHC clinicians stated that a majority of risk factors can be found in clinical notes (ie, care coordination (53.3%) or visit (46.7%)). DISCUSSION: Clinicians acknowledged that social factors play a role in deterioration risk; but these factors are infrequently studied in HHC. While a majority of risk factors were represented in the Omaha System, additional terminologies are needed to comprehensively capture risk. Since most risk factors are documented in clinical notes, methods such as natural language processing are needed to extract them. CONCLUSION: This study engaged clinicians to understand risk for deterioration during HHC. The results of our study support the development of an early warning system by providing a comprehensive list of risk factors grounded in clinician expertize and mapped to standardized terminologies.

OBJECTIVE: The study provides considerations for generating a phenotype of child abuse and neglect in Emergency Departments (ED) using secondary data from electronic health records (EHR). Implications will be provided for racial bias reduction and the development of further decision support tools to assist in identifying child abuse and neglect. MATERIALS AND METHODS: We conducted a qualitative study using in-depth interviews with 20 pediatric clinicians working in a single pediatric ED to gain insights about generating an EHR-based phenotype to identify children at risk for abuse and neglect. RESULTS: Three central themes emerged from the interviews: (1) Challenges in diagnosing child abuse and neglect, (2) Health Discipline Differences in Documentation Styles in EHR, and (3) Identification of potential racial bias through documentation. DISCUSSION: Our findings highlight important considerations for generating a phenotype for child abuse and neglect using EHR data. First, information-related challenges include lack of proper previous visit history due to limited information exchanges and scattered documentation within EHRs. Second, there are differences in documentation styles by health disciplines, and clinicians tend to document abuse in different document types within EHRs. Finally, documentation can help identify potential racial bias in suspicion of child abuse and neglect by revealing potential discrepancies in quality of care, and in the language used to document abuse and neglect. CONCLUSIONS: Our findings highlight challenges in building an EHR-based risk phenotype for child abuse and neglect. Further research is needed to validate these findings and integrate them into creation of an EHR-based risk phenotype.

Child abuse and neglect are public health issues impacting communities throughout the United States. The broad adoption of electronic health records (EHR) in health care supports the development of machine learning-based models to help identify child abuse and neglect. Employing EHR data for child abuse and neglect detection raises several critical ethical considerations. This article applied a phenomenological approach to discuss and provide recommendations for key ethical issues related to machine learning-based risk models development and evaluation: (1) biases in the data; (2) clinical documentation system design issues; (3) lack of centralized evidence base for child abuse and neglect; (4) lack of "gold standard "in assessment and diagnosis of child abuse and neglect; (5) challenges in evaluation of risk prediction performance; (6) challenges in testing predictive models in practice; and (7) challenges in presentation of machine learning-based prediction to clinicians and patients. We provide recommended solutions to each of the 7 ethical challenges and identify several areas for further policy and research.

OBJECTIVE: To propose an algorithm that utilizes only timestamps of longitudinal electronic health record data to classify clinical deterioration events. MATERIALS AND METHODS: This retrospective study explores the efficacy of machine learning algorithms in classifying clinical deterioration events among patients in intensive care units using sequences of timestamps of vital sign measurements, flowsheets comments, order entries, and nursing notes. We design a data pipeline to partition events into discrete, regular time bins that we refer to as timesteps. Logistic regressions, random forest classifiers, and recurrent neural networks are trained on datasets of different length of timesteps, respectively, against a composite outcome of death, cardiac arrest, and Rapid Response Team calls. Then these models are validated on a holdout dataset. RESULTS: A total of 6720 intensive care unit encounters meet the criteria and the final dataset includes 830 578 timestamps. The gated recurrent unit model utilizes timestamps of vital signs, order entries, flowsheet comments, and nursing notes to achieve the best performance on the time-to-outcome dataset, with an area under the precision-recall curve of 0.101 (0.06, 0.137), a sensitivity of 0.443, and a positive predictive value of 0. 092 at the threshold of 0.6. DISCUSSION AND CONCLUSION: This study demonstrates that our recurrent neural network models using only timestamps of longitudinal electronic health record data that reflect healthcare processes achieve well-performing discriminative power.

BACKGROUND: . OBJECTIVE: Electronic health records (EHRs) are linked with documentation burden resulting in clinician burnout. While clear classifications and validated measures of burnout exist, documentation burden remains ill-defined and inconsistently measured. We aim to conduct a scoping review focused on identifying approaches to documentation burden measurement and their characteristics. MATERIALS AND METHODS: Based on Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) Extension for Scoping Reviews (ScR) guidelines, we conducted a scoping review assessing MEDLINE, Embase, Web of Science, and CINAHL from inception to April 2020 for studies investigating documentation burden among physicians and nurses in ambulatory or inpatient settings. Two reviewers evaluated each potentially relevant study for inclusion/exclusion criteria. RESULTS: Of the 3482 articles retrieved, 35 studies met inclusion criteria. We identified 15 measurement characteristics, including 7 effort constructs: EHR usage and workload, clinical documentation/review, EHR work after hours and remotely, administrative tasks, cognitively cumbersome work, fragmentation of workflow, and patient interaction. We uncovered 4 time constructs: average time, proportion of time, timeliness of completion, activity rate, and 11 units of analysis. Only 45.0% of studies assessed the impact of EHRs on clinicians and/or patients and 40.0% mentioned clinician burnout. DISCUSSION: Standard and validated measures of documentation burden are lacking. While time and effort were the core concepts measured, there appears to be no consensus on the best approach nor degree of rigor to study documentation burden. CONCLUSION: Further research is needed to reliably operationalize the concept of documentation burden, explore best practices for measurement, and standardize its use.

OBJECTIVE: There are signals of clinicians' expert and knowledge-driven behaviors within clinical information systems (CIS) that can be exploited to support clinical prediction. Describe development of the Healthcare Process Modeling Framework to Phenotype Clinician Behaviors for Exploiting the Signal Gain of Clinical Expertise (HPM-ExpertSignals). MATERIALS AND METHODS: We employed an iterative framework development approach that combined data-driven modeling and simulation testing to define and refine a process for phenotyping clinician behaviors. Our framework was developed and evaluated based on the Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) predictive model to detect and leverage signals of clinician expertise for prediction of patient trajectories. RESULTS: Seven themes-identified during development and simulation testing of the CONCERN model-informed framework development. The HPM-ExpertSignals conceptual framework includes a 3-step modeling technique: (1) identify patterns of clinical behaviors from user interaction with CIS; (2) interpret patterns as proxies of an individual's decisions, knowledge, and expertise; and (3) use patterns in predictive models for associations with outcomes. The CONCERN model differentiated at risk patients earlier than other early warning scores, lending confidence to the HPM-ExpertSignals framework. DISCUSSION: The HPM-ExpertSignals framework moves beyond transactional data analytics to model clinical knowledge, decision making, and CIS interactions, which can support predictive modeling with a focus on the rapid and frequent patient surveillance cycle. CONCLUSIONS: We propose this framework as an approach to embed clinicians' knowledge-driven behaviors in predictions and inferences to facilitate capture of healthcare processes that are activated independently, and sometimes well before, physiological changes are apparent.

OBJECTIVE: The study sought to describe the prevalence and nature of clinical expert involvement in the development, evaluation, and implementation of clinical decision support systems (CDSSs) that utilize machine learning to analyze electronic health record data to assist nurses and physicians in prognostic and treatment decision making (ie, predictive CDSSs) in the hospital. MATERIALS AND METHODS: A systematic search of PubMed, CINAHL, and IEEE Xplore and hand-searching of relevant conference proceedings were conducted to identify eligible articles. Empirical studies of predictive CDSSs using electronic health record data for nurses or physicians in the hospital setting published in the last 5 years in peer-reviewed journals or conference proceedings were eligible for synthesis. Data from eligible studies regarding clinician involvement, stage in system design, predictive CDSS intention, and target clinician were charted and summarized. RESULTS: Eighty studies met eligibility criteria. Clinical expert involvement was most prevalent at the beginning and late stages of system design. Most articles (95%) described developing and evaluating machine learning models, 28% of which described involving clinical experts, with nearly half functioning to verify the clinical correctness or relevance of the model (47%). DISCUSSION: Involvement of clinical experts in predictive CDSS design should be explicitly reported in publications and evaluated for the potential to overcome predictive CDSS adoption challenges. CONCLUSIONS: If present, clinical expert involvement is most prevalent when predictive CDSS specifications are made or when system implementations are evaluated. However, clinical experts are less prevalent in developmental stages to verify clinical correctness, select model features, preprocess data, or serve as a gold standard.

Dear JAMIA Editors and Readers: We appreciate the critiques that Dr. Sendak and colleagues have brought forward regarding our scoping review of clinician involvement in predictive CDSS design.1 In their letter, Sendak and colleagues argue that our review too narrowly defined clinician involvement and that relationships established between clinician leaders, often coauthors on manuscripts, and other research team members is a valuable form of clinician involvement not adequately captured in our review.2 We recognize and agree that we should have more prominently highlighted the possibility that clinically affiliated coauthors’ contributions may have represented clinician involvement in one of the ways we charted or in a different relationship-oriented way that is also important for predictive CDSS success. We also should have consistently referred to our results finding that involvement is not widely reported instead of not widely practiced. We also acknowledge that reaching out to authors to gather...

been corrected from " [25][26][27][28]30