OBJECTIVE: To characterize the nature and consequence(s) of interdependent physician electronic health record (EHR) work across inpatient shifts. MATERIALS AND METHODS: Pooled cross-sectional analysis of EHR metadata associated with hospital medicine patients at an academic medical center, January-June 2022. Using patient-day observation data, we use a mixed effects regression model with daytime physician random effects to examine nightshift behavior (handoff time, total EHR time) as a function of behaviors by the preceding daytime team. We also assess whether nighttime patient deterioration is predicted by team coordination behaviors across shifts. RESULTS: We observed 19 671 patient days (N = 2708 encounters). Physicians used the handoff tool consistently, generally spending 8-12 minutes per shift editing patient information. When the day service team was more activated (highest tercile of handoff time, overall EHR time), nightshift experienced increased levels of EHR work and patient risk of overnight decline was elevated. (ie, Busy predicts busy). However, lower levels of dayshift activation were also associated with nightshift spillovers, including higher overnight EHR work and increased likelihood of patient clinical decline. Patient-days in the lowest and highest terciles of dayshift EHR time had a 1 percentage point increased relative risk of overnight decline (baseline prevalence of 4.4%) compared to the middle tercile (P = .04). DISCUSSION: We find evidence of spillovers in EHR work from dayshift to nightshift. Additionally, the lowest and highest levels of dayshift EHR activity are associated with increased risk of overnight patient decline. Results are associational and motivate further examination of additional confounding factors. CONCLUSION: Analyses reveal opportunities to address task interdependence across shifts, using technology to flexibly shape and support collaborative teaming practices in complex clinical environments.

OBJECTIVES: There is rapidly growing interest in learning health systems (LHSs) nationally and globally. While the critical role of informatics is recognized, the informatics community has been relatively slow to formalize LHS as a priority area. MATERIALS AND METHODS: We compiled results from a short survey of LHS leaders and American Medical Informatics Association (AMIA) members, discussion from an LHS reception at the AMIA annual meeting, and a follow-up survey to inform priorities at the intersection of LHS and informatics. RESULTS: We present opportunities between informatics and LHS which fell into themes of: Understanding and Context, Shared Resources, Collaboration, Education, Data, Evaluation, and Patient Centeredness. Immediate LHS informatics priorities identified include establishing informatics LHS forum(s), case reports of LHS informatics successes and failures, LHS informatics education resources, and improved understanding of LHS principles in informatics. CONCLUSION: Increased informatics and LHS alignment is critical for advancing this transformative national priority.

Abstract When building predictive models for real-world applications, many data are discarded because conventional learning algorithms cannot utilize it, although such data could be very informative. This paper focuses on representation learning using two types of additional data: privileged information (PI) and unlabeled data. PI refers to data available only during training but not at test time. Existing methods transfer the knowledge embedded in PI via supervised mechanisms, making them unable to use unlabeled data. In contrast, self-supervised learning methods can use unlabeled data but cannot learn from PI. While these techniques appear complementary, as we demonstrate, combining them is non-trivial. This paper introduces the privileged information regularized (PIReg) self-supervised learning framework, which utilizes both PI and unlabeled data to learn better representations.

When building predictive models for real-world applications, many data are discarded because conventional learning algorithms cannot utilize it, although such data could be very informative. This paper focuses on representation learning using two types of additional data: privileged information (PI) and unlabeled data. PI refers to data available only during training but not at test time. Existing methods transfer the knowledge embedded in PI via supervised mechanisms, making them unable to use unlabeled data. In contrast, self-supervised learning methods can use unlabeled data but cannot learn from PI. While these techniques appear complementary, as we demonstrate, combining them is non-trivial. This paper introduces the Privileged Information Regularized (PIReg) self-supervised learning framework, which utilizes both PI and unlabeled data to learn better representations.

A panel sponsored by the American College of Medical Informatics (ACMI) at the 2021 AMIA Symposium addressed the provocative question: "Are Electronic Health Records dumbing down clinicians?" After reviewing electronic health record (EHR) development and evolution, the panel discussed how EHR use can impair care delivery. Both suboptimal functionality during EHR use and longer-term effects outside of EHR use can reduce clinicians' efficiencies, reasoning abilities, and knowledge. Panel members explored potential solutions to problems discussed. Progress will require significant engagement from clinician-users, educators, health systems, commercial vendors, regulators, and policy makers. Future EHR systems must become more user-focused and scalable and enable providers to work smarter to deliver improved care.

Research Patient Data Repositories (RPDRs) have become essential infrastructure for traditional Clinical and Translational Science Award (CTSA) programs and increasingly for a wide range of research consortia and learning health system networks.1–5 Almost every institution with a CTSA or Clinical Translational Research (CTR) program (found in states with lower amounts of National Institutes of Health funding) hosts an RPDR for the benefit of affiliated researchers. These repositories aim to enable healthcare research based upon the patient populations they serve. Within the institution, RPDRs are valuable for a range of research activities. They are used to identify patients for clinical trial recruitment using privacy-preserving methods to search and extract specific cohorts of trial-eligible patients.6 They aid in developing and validating computable phenotypes that are increasingly important for accurately identifying patient cohorts in a reproducible fashion.7 RPDRs provide de-identified patient data for population health research and support a growing body of artificial intelligence to predict patient outcomes.8 Further, clinical studies can often be simulated using data from an RPDR.9 Beyond the institution, aggregates of de-identified datasets from multiple institutions linked with privacy-preserving hash codes provide an unprecedented opportunity to conduct population health research, perform comparative effectiveness analyses and apply artificial intelligence methods over large and diverse populations.10 The data contained within the RPDR vary across institutions, based on institutional strengths and weaknesses; the papers published in this issue reflect that variability (see Table 1). Data are commonly acquired from local electronic health records (EHRs) and other clinical information systems that capture information during clinical care. Data consist of diagnoses, problem lists, procedures, prescribed medications, laboratory exams, and many types of free-text reports. Overall, the benefits of the RPDR for accelerating translational research can be significant. For example, at Harvard, in 2006, between $94 and $136 million in annual research funding was linked to the use of data from the RPDR.11

OBJECTIVE: Federated learning (FL) allows multiple distributed data holders to collaboratively learn a shared model without data sharing. However, individual health system data are heterogeneous. "Personalized" FL variations have been developed to counter data heterogeneity, but few have been evaluated using real-world healthcare data. The purpose of this study is to investigate the performance of a single-site versus a 3-client federated model using a previously described Coronavirus Disease 19 (COVID-19) diagnostic model. Additionally, to investigate the effect of system heterogeneity, we evaluate the performance of 4 FL variations. MATERIALS AND METHODS: We leverage a FL healthcare collaborative including data from 5 international healthcare systems (US and Europe) encompassing 42 hospitals. We implemented a COVID-19 computer vision diagnosis system using the Federated Averaging (FedAvg) algorithm implemented on Clara Train SDK 4.0. To study the effect of data heterogeneity, training data was pooled from 3 systems locally and federation was simulated. We compared a centralized/pooled model, versus FedAvg, and 3 personalized FL variations (FedProx, FedBN, and FedAMP). RESULTS: We observed comparable model performance with respect to internal validation (local model: AUROC 0.94 vs FedAvg: 0.95, P = .5) and improved model generalizability with the FedAvg model (P < .05). When investigating the effects of model heterogeneity, we observed poor performance with FedAvg on internal validation as compared to personalized FL algorithms. FedAvg did have improved generalizability compared to personalized FL algorithms. On average, FedBN had the best rank performance on internal and external validation. CONCLUSION: FedAvg can significantly improve the generalization of the model compared to other personalization FL algorithms; however, at the cost of poor internal validity. Personalized FL may offer an opportunity to develop both internal and externally validated algorithms.

Statistical modeling of outcomes based on a patient's presenting symptoms (symptomatology) can help deliver high quality care and allocate essential resources, which is especially important during the COVID-19 pandemic. Patient symptoms are typically found in unstructured notes, and thus not readily available for clinical decision making. In an attempt to fill this gap, this study compared two methods for symptom extraction from Emergency Department (ED) admission notes. Both methods utilized a lexicon derived by expanding The Center for Disease Control and Prevention's (CDC) Symptoms of Coronavirus list. The first method utilized a word2vec model to expand the lexicon using a dictionary mapping to the Uni ed Medical Language System (UMLS). The second method utilized the expanded lexicon as a rule-based gazetteer and the UMLS. These methods were evaluated against a manually annotated reference (f1-score of 0.87 for UMLS-based ensemble; and 0.85 for rule-based gazetteer with UMLS). Through analyses of associations of extracted symptoms used as features against various outcomes, salient risks among the population of COVID-19 patients, including increased risk of in-hospital mortality (OR 1.85, p-value < 0.001), were identified for patients presenting with dyspnea. Disparities between English and non-English speaking patients were also identified, the most salient being a concerning finding of opposing risk signals between fatigue and in-hospital mortality (non-English: OR 1.95, p-value = 0.02; English: OR 0.63, p-value = 0.01). While use of symptomatology for modeling of outcomes is not unique, unlike previous studies this study showed that models built using symptoms with the outcome of in-hospital mortality were not significantly different from models using data collected during an in-patient encounter (AUC of 0.9 with 95% CI of [0.88, 0.91] using only vital signs; AUC of 0.87 with 95% CI of [0.85, 0.88] using only symptoms). These findings indicate that prognostic models based on symptomatology could aid in extending COVID-19 patient care through telemedicine, replacing the need for in-person options. The methods presented in this study have potential for use in development of symptomatology-based models for other diseases, including for the study of Post-Acute Sequelae of COVID-19 (PASC).

Clinicians face competing pressures of being clinically productive while using imperfect electronic health record (EHR) systems and maximizing face-to-face time with patients. EHR use is increasingly associated with clinician burnout and underscores the need for interventions to improve clinicians' experiences. With an aim of addressing this need, we share evidence-based informatics approaches, pragmatic next steps, and future research directions to improve 3 of the highest contributors to EHR burden: (1) documentation, (2) chart review, and (3) inbox tasks. These approaches leverage speech recognition technologies, natural language processing, artificial intelligence, and redesign of EHR workflow and user interfaces. We also offer a perspective on how EHR vendors, healthcare system leaders, and policymakers all play an integral role while sharing responsibility in helping make evidence-based sociotechnical solutions available and easy to use.

OBJECTIVE: Hospital-acquired infections (HAIs) are associated with significant morbidity, mortality, and prolonged hospital length of stay. Risk prediction models based on pre- and intraoperative data have been proposed to assess the risk of HAIs at the end of the surgery, but the performance of these models lag behind HAI detection models based on postoperative data. Postoperative data are more predictive than pre- or interoperative data since it is closer to the outcomes in time, but it is unavailable when the risk models are applied (end of surgery). The objective is to study whether such data, which is temporally unavailable at prediction time (TUP) (and thus cannot directly enter the model), can be used to improve the performance of the risk model. MATERIALS AND METHODS: An extensive array of 12 methods based on logistic/linear regression and deep learning were used to incorporate the TUP data using a variety of intermediate representations of the data. Due to the hierarchical structure of different HAI outcomes, a comparison of single and multi-task learning frameworks is also presented. RESULTS AND DISCUSSION: The use of TUP data was always advantageous as baseline methods, which cannot utilize TUP data, never achieved the top performance. The relative performances of the different models vary across the different outcomes. Regarding the intermediate representation, we found that its complexity was key and that incorporating label information was helpful. CONCLUSIONS: Using TUP data significantly helped predictive performance irrespective of the model complexity.

OBJECTIVE: The study sought to evaluate early lessons from a remote patient monitoring engagement and education technology solution for patients with coronavirus disease 2019 (COVID-19) symptoms. MATERIALS AND METHODS: A COVID-19-specific remote patient monitoring solution (GetWell Loop) was offered to patients with COVID-19 symptoms. The program engaged patients and provided educational materials and the opportunity to share concerns. Alerts were resolved through a virtual care workforce of providers and medical students. RESULTS: Between March 18 and April 20, 2020, 2255 of 3701 (60.93%) patients with COVID-19 symptoms enrolled, resulting in over 2303 alerts, 4613 messages, 13 hospital admissions, and 91 emergency room visits. A satisfaction survey was given to 300 patient respondents, 74% of whom would be extremely likely to recommend their doctor. DISCUSSION: This program provided a safe and satisfying experience for patients while minimizing COVID-19 exposure and in-person healthcare utilization. CONCLUSIONS: Remote patient monitoring appears to be an effective approach for managing COVID-19 symptoms at home.

OBJECTIVE: To build a knowledge base of dietary supplement (DS) information, called the integrated DIetary Supplement Knowledge base (iDISK), which integrates and standardizes DS-related information from 4 existing resources. MATERIALS AND METHODS: iDISK was built through an iterative process comprising 3 phases: 1) establishment of the content scope, 2) development of the data model, and 3) integration of existing resources. Four well-regarded DS resources were integrated into iDISK: The Natural Medicines Comprehensive Database, the "About Herbs" page on the Memorial Sloan Kettering Cancer Center website, the Dietary Supplement Label Database, and the Natural Health Products Database. We evaluated the iDISK build process by manually checking that the data elements associated with 50 randomly selected ingredients were correctly extracted and integrated from their respective sources. RESULTS: iDISK encompasses a terminology of 4208 DS ingredient concepts, which are linked via 6 relationship types to 495 drugs, 776 diseases, 985 symptoms, 605 therapeutic classes, 17 system organ classes, and 137 568 DS products. iDISK also contains 7 concept attribute types and 3 relationship attribute types. Evaluation of the data extraction and integration process showed average errors of 0.3%, 2.6%, and 0.4% for concepts, relationships and attributes, respectively. CONCLUSION: We developed iDISK, a publicly available standardized DS knowledge base that can facilitate more efficient and meaningful dissemination of DS knowledge.

Reports by the National Academy of Medicine and leading public health organizations advocate including occupational information as part of an individual's social context. Given recent National Academy of Medicine recommendations on occupation-related data in the electronic health record, there is a critical need for improved representation. The National Institute for Occupational Safety and Health has developed an Occupational Data for Health (ODH) model, currently in draft format. This study aimed to validate the ODH model by mapping occupation-related elements from resources representing recommendations, standards, public health reports and surveys, and research measures, along with preliminary evaluation of associated value sets. All 247 occupation-related items across 20 resources mapped to the ODH model. Recommended value sets had high variability across the evaluated resources. This study demonstrates the ODH model's value, the multifaceted nature of occupation information, and the critical need for occupation value sets to support clinical care, population health, and research.

MOTIVATION: Automatically quantifying semantic similarity and relatedness between clinical terms is an important aspect of text mining from electronic health records, which are increasingly recognized as valuable sources of phenotypic information for clinical genomics and bioinformatics research. A key obstacle to development of semantic relatedness measures is the limited availability of large quantities of clinical text to researchers and developers outside of major medical centers. Text from general English and biomedical literature are freely available; however, their validity as a substitute for clinical domain to represent semantics of clinical terms remains to be demonstrated. RESULTS: We constructed neural network representations of clinical terms found in a publicly available benchmark dataset manually labeled for semantic similarity and relatedness. Similarity and relatedness measures computed from text corpora in three domains (Clinical Notes, PubMed Central articles and Wikipedia) were compared using the benchmark as reference. We found that measures computed from full text of biomedical articles in PubMed Central repository (rho = 0.62 for similarity and 0.58 for relatedness) are on par with measures computed from clinical reports (rho = 0.60 for similarity and 0.57 for relatedness). We also evaluated the use of neural network based relatedness measures for query expansion in a clinical document retrieval task and a biomedical term word sense disambiguation task. We found that, with some limitations, biomedical articles may be used in lieu of clinical reports to represent the semantics of clinical terms and that distributional semantic methods are useful for clinical and biomedical natural language processing applications. AVAILABILITY AND IMPLEMENTATION: The software and reference standards used in this study to evaluate semantic similarity and relatedness measures are publicly available as detailed in the article. CONTACT: [email protected] information: Supplementary data are available at Bioinformatics online.

OBJECTIVE: To integrate data elements from multiple sources for informing comprehensive and standardized collection of family health history (FHH). MATERIALS AND METHODS: Three types of sources were analyzed to identify data elements associated with the collection of FHH. First, clinical notes from multiple resources were annotated for FHH information. Second, questions and responses for family members in patient-facing FHH tools were examined. Lastly, elements defined in FHH-related specifications were extracted for several standards development and related organizations. Data elements identified from the notes, tools, and specifications were subsequently combined and compared. RESULTS: In total, 891 notes from three resources, eight tools, and seven specifications associated with four organizations were analyzed. The resulting Integrated FHH Model consisted of 44 data elements for describing source of information, family members, observations, and general statements about family history. Of these elements, 16 were common to all three source types, 17 were common to two, and 11 were unique. Intra-source comparisons also revealed common and unique elements across the different notes, tools, and specifications. DISCUSSION: Through examination of multiple sources, a representative and complementary set of FHH data elements was identified. Further work is needed to create formal representations of the Integrated FHH Model, standardize values associated with each element, and inform context-specific implementations. CONCLUSIONS: There has been increased emphasis on the importance of FHH for supporting personalized medicine, biomedical research, and population health. Multi-source development of an integrated model could contribute to improving the standardized collection and use of FHH information in disparate systems.

The healthcare landscape is changing, driven by innovative care models and the emergence of new roles that are inter-professional in nature. Currently, the HL7/LOINC Document Ontology (DO) aids the use and exchange of clinical documents using a multi-axis structure of document attributes for Kind of Document, Setting, Role, Subject Matter Domain, and Type of Service. In this study, the adequacy of the Role axis for representing the type of author documenting care was assessed. Experts used a master list of 220 values created from seven resources and established mapping guidelines. Baseline certification, licensure, and didactic training were identified as key parameters that define roles and hence often need to be pre-coordinated. DO was inadequate in representing 82% of roles, and this gap was primarily due to lack of granularity in DO. Next steps include refinement of the proposed schema for the Role axis and dissemination within the larger standards community.

OBJECTIVE: To create a sense inventory of abbreviations and acronyms from clinical texts. METHODS: The most frequently occurring abbreviations and acronyms from 352,267 dictated clinical notes were used to create a clinical sense inventory. Senses of each abbreviation and acronym were manually annotated from 500 random instances and lexically matched with long forms within the Unified Medical Language System (UMLS V.2011AB), Another Database of Abbreviations in Medline (ADAM), and Stedman's Dictionary, Medical Abbreviations, Acronyms & Symbols, 4th edition (Stedman's). Redundant long forms were merged after they were lexically normalized using Lexical Variant Generation (LVG). RESULTS: The clinical sense inventory was found to have skewed sense distributions, practice-specific senses, and incorrect uses. Of 440 abbreviations and acronyms analyzed in this study, 949 long forms were identified in clinical notes. This set was mapped to 17,359, 5233, and 4879 long forms in UMLS, ADAM, and Stedman's, respectively. After merging long forms, only 2.3% matched across all medical resources. The UMLS, ADAM, and Stedman's covered 5.7%, 8.4%, and 11% of the merged clinical long forms, respectively. The sense inventory of clinical abbreviations and acronyms and anonymized datasets generated from this study are available for public use at http://www.bmhi.umn.edu/ihi/research/nlpie/resources/index.htm ('Sense Inventories', website). CONCLUSIONS: Clinical sense inventories of abbreviations and acronyms created using clinical notes and medical dictionary resources demonstrate challenges with term coverage and resource integration. Further work is needed to help with standardizing abbreviations and acronyms in clinical care and biomedicine to facilitate automated processes such as text-mining and information extraction.

The value of standards for the representation, integration, and exchange of data, information, and knowledge across the spectrum of biomedicine and health care has been widely recognized for years. Recent initiatives further underscore the importance of standards (e.g., for certification and meaningful use of electronic health records) and establishment of systematic approaches for achieving semantic interoperability. There is accordingly a need for detailed, experience-based discussions pertaining to the adoption and implementation of the breadth of standards in biomedicine and health care. The goal of this special issue has been to provide a forum for describing advanced research and development in translating standards into practice. Each paper in this issue provides a comprehensive description of methodologies employed and challenges encountered during the process of implementing a specific standard or set of standards in a practical setting. The twenty-two papers (including one methodological review) represent a broad array of experiences with standards and are categorized into the following sections: (1) Terminology Standards, (2) Document Standards, (3) Decision Support Standards, (4) Standards-Based Infrastructure, and (5) Standards Adoption Processes.

The Internet has become one of the main sources of consumer health information. Health consumers have access to ever-growing health information resources, especially since the rise of the Social Media. For example, over 20.000 videos have been uploaded by American hospitals on to YouTube. To find health videos is challenging because of factors like tags spamming and misleading information. Previous studies have found difficulties when searching for good health videos in YouTube, including false information (e.g., herbal cures for diabetes or cancer).

Family history information has emerged as an increasingly important tool for clinical care and research. While recent standards provide for structured entry of family history, many clinicians record family history data in text. The authors sought to characterize family history information within clinical documents to assess the adequacy of existing models and create a more comprehensive model for its representation. Models were evaluated on 100 documents containing 238 sentences and 410 statements relevant to family history. Most statements were of family member plus disease or of disease only. Statement coverage was 91%, 77%, and 95% for HL7 Clinical Genomics Family History Model, HL7 Clinical Statement Model, and the newly created Merged Family History Model, respectively. Negation (18%) and inexact family member specification (9.5%) occurred commonly. Overall, both HL7 models could represent most family history statements in clinical reports; however, refinements are needed to represent the full breadth of family history data.

The authors summarize their experience in iteratively testing the adequacy of three versions of the Health Level Seven (HL7) Logical Observation Identifiers Names and Codes (LOINC) Clinical Document Ontology (CDO) to represent document names at Columbia University Medical Center. The percentage of documents fully represented increased from 23.4% (Version 1) to 98.5% (Version 3). The proportion of unique representations increased from 7.9% (Analysis 1) to 39.4% (Analysis 4); the proportion reflects the level of specificity in the document names as well as the completeness and level of granularity of the CDO. The authors shared the findings of each analysis with the Clinical LOINC committee and participated in the decision-making regarding changes to the CDO on the basis of those analyses and those conducted by the Department of Veterans Affairs. The authors encourage other institutions to actively engage in testing healthcare standards and participating in standards development activities to increase the likelihood that the evolving standards will meet institutional needs.

OBJECTIVE: To determine whether natural language processing (NLP) can effectively detect adverse events defined in the New York Patient Occurrence Reporting and Tracking System (NYPORTS) using discharge summaries. DESIGN: An adverse event detection system for discharge summaries using the NLP system MedLEE was constructed to identify 45 NYPORTS event types. The system was first applied to a random sample of 1,000 manually reviewed charts. The system then processed all inpatient cases with electronic discharge summaries for two years. All system-identified events were reviewed, and performance was compared with traditional reporting. MEASUREMENTS: System sensitivity, specificity, and predictive value, with manual review serving as the gold standard. RESULTS: The system correctly identified 16 of 65 events in 1,000 charts. Of 57,452 total electronic discharge summaries, the system identified 1,590 events in 1,461 cases, and manual review verified 704 events in 652 cases, resulting in an overall sensitivity of 0.28 (95% confidence interval [CI]: 0.17-0.42), specificity of 0.985 (CI: 0.984-0.986), and positive predictive value of 0.45 (CI: 0.42-0.47) for detecting cases with events and an average specificity of 0.9996 (CI: 0.9996-0.9997) per event type. Traditional event reporting detected 322 events during the period (sensitivity 0.09), of which the system identified 110 as well as 594 additional events missed by traditional methods. CONCLUSION: NLP is an effective technique for detecting a broad range of adverse events in text documents and outperformed traditional and previous automated adverse event detection methods.