BACKGROUND: The use of generative large language models (LLMs) with electronic health record (EHR) data is rapidly expanding to support clinical and research tasks. This systematic review characterizes the clinical fields and use cases that have been studied and evaluated to date. METHODS: We followed the Preferred Reporting Items for Systematic Review and Meta-Analyses guidelines to conduct a systematic review of articles from PubMed and Web of Science published between January 1, 2023, and November 9, 2024. Studies were included if they used generative LLMs to analyze real-world EHR data and reported quantitative performance evaluations. Through data extraction, we identified clinical specialties and tasks for each included article, and summarized evaluation methods. RESULTS: Of the 18 735 articles retrieved, 196 met our criteria. Most studies focused on radiology (26.0%), oncology (10.7%), and emergency medicine (6.6%). Regarding clinical tasks, clinical decision support made up the largest proportion of studies (62.2%), while summarizations and patient communications made up the smallest, at 5.6% and 5.1%, respectively. In addition, GPT-4 and GPT-3.5 were the most commonly used generative LLMs, appearing in 60.2% and 57.7% of studies, respectively. Across these studies, we identified 22 unique non-NLP metrics and 35 unique NLP metrics. While NLP metrics offer greater scalability, none demonstrated a strong correlation with gold-standard human evaluations. CONCLUSION: Our findings highlight the need to evaluate generative LLMs on EHR data across a broader range of clinical specialties and tasks, as well as the urgent need for standardized, scalable, and clinically meaningful evaluation frameworks.

OBJECTIVE: Automated literature screening in biomedical research is often hindered by domain shifts and scarcity of labeled data, which limit model accuracy and generalizability. While large language models (LLMs) perform well in zero-shot settings, they often fail to capture complex, domain-specific reasoning patterns. To address this limitation, this study investigates whether an interactive, weakly supervised learning framework combining GPT (generative pre-trained transformer)'s fine-tuning adaptability with DeepSeek's reasoning capabilities can improve literature screening performance across biomedical domains. MATERIALS AND METHODS: We developed an active learning framework that leverages model disagreement between GPT-4o and DeepSeek to improve literature screening performance. This process began with a labeled corpus of 6331 articles on large language models, from which a model disagreement analysis was performed to identify cases where GPT-4o misclassified and DeepSeek produced correct predictions. Three GPT variants-GPT-4o, GPT-4o-mini, and GPT-4.1-nano, were fine-tuned under standard supervised learning settings using these disagreement-based samples. Fine-tuning prompts incorporated classification labels and, when available, rationale traces generated by DeepSeek to provide reasoning-augmented weak supervision. Model performance was evaluated on an independent benchmark set of 291 annotated articles across 10 topic queries in cancer immunotherapy and LLMs in medicine, using standard evaluation metrics, with recall as the primary measure. RESULTS: Fine-tuning GPT models using disagreement-based examples significantly improved performance. GPT-4o-mini achieved the best overall results after fine-tuning, especially with the highest F1 score (0.93, P < .001) and recall (0.95, P < .001). Across the biomedical topics, fine-tuned models consistently outperformed their zero-shot counterparts without increasing reviewer workload. DISCUSSION: These findings demonstrate the effectiveness of disagreement-driven active learning in enhancing GPT-based biomedical literature screening. Lightweight models like GPT-4o-mini benefit most from targeted, reasoning-enriched training, highlighting their suitability for scalable deployment. CONCLUSION: This study introduces an interactive active learning framework that leverages fine-tuned LLMs with reasoning capabilities to enhance literature screening. The approach offers a scalable solution to more efficient and reliable information retrieval in systematic reviews.

OBJECTIVE: Accurate, complete allergy histories are critical for decision-making and medication prescription. However, allergy information is often spread across the electronic health record (EHR); thus, allergy lists are often inaccurate or incomplete. Discrepant allergy information can lead to suboptimal or unsafe clinical care and contribute to alert fatigue. We developed an allergy reconciliation module within Mass General Brigham (MGB)'s EHR to support accurate and intuitive reconciliation of discrepancies in the allergy list, thereby enhancing patient safety. MATERIALS AND METHODS: We combined data-driven methods and knowledge from domain experts to develop 5 mechanisms to compare allergy information across the EHR and designed a user interface to display discrepancies and suggested reconciliation actions, with links to relevant data sources. Qualitative and quantitative analyses were conducted to assess the module's performance and measure user acceptance. RESULTS: We implemented and tested the proposed allergy reconciliation mechanisms and module. A comprehensive integration workflow was developed for the module, which was piloted among 111 primary care physicians at MGB. F1 scores of the reconciliation mechanisms range from 0.86 to 1.0. Qualitative analysis showed majority positive feedback from pilot users. DISCUSSION: Our allergy reconciliation module achieved high performance, and physicians who used it largely accepted its recommendations. However, 56% of the pilot group ultimately did not use the module. User engagement and education are likely needed to increase adoption. CONCLUSION: We built a module to automatically identify discrepancies within patients' allergy records and remind providers to reconcile and update the allergy list. Its high accuracy shows promise for enhancing patient safety and utility of drug allergy alerts.

OBJECTIVES: Leveraging artificial intelligence (AI) in conjunction with electronic health records (EHRs) holds transformative potential to improve healthcare. However, addressing bias in AI, which risks worsening healthcare disparities, cannot be overlooked. This study reviews methods to handle various biases in AI models developed using EHR data. MATERIALS AND METHODS: We conducted a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines, analyzing articles from PubMed, Web of Science, and IEEE published between January 01, 2010 and December 17, 2023. The review identified key biases, outlined strategies for detecting and mitigating bias throughout the AI model development, and analyzed metrics for bias assessment. RESULTS: Of the 450 articles retrieved, 20 met our criteria, revealing 6 major bias types: algorithmic, confounding, implicit, measurement, selection, and temporal. The AI models were primarily developed for predictive tasks, yet none have been deployed in real-world healthcare settings. Five studies concentrated on the detection of implicit and algorithmic biases employing fairness metrics like statistical parity, equal opportunity, and predictive equity. Fifteen studies proposed strategies for mitigating biases, especially targeting implicit and selection biases. These strategies, evaluated through both performance and fairness metrics, predominantly involved data collection and preprocessing techniques like resampling and reweighting. DISCUSSION: This review highlights evolving strategies to mitigate bias in EHR-based AI models, emphasizing the urgent need for both standardized and detailed reporting of the methodologies and systematic real-world testing and evaluation. Such measures are essential for gauging models' practical impact and fostering ethical AI that ensures fairness and equity in healthcare.

OBJECTIVES: Despite federally mandated collection of sex and gender demographics in the electronic health record (EHR), longitudinal assessments are lacking. We assessed sex and gender demographic field utilization using EHR metadata. MATERIALS AND METHODS: Patients ≥18 years of age in the Mass General Brigham health system with a first Legal Sex entry (registration requirement) between January 8, 2018 and January 1, 2022 were included in this retrospective study. Metadata for all sex and gender fields (Legal Sex, Sex Assigned at Birth [SAAB], Gender Identity) were quantified by completion rates, user types, and longitudinal change. A nested qualitative study of providers from specialties with high and low field use identified themes related to utilization. RESULTS: 1 576 120 patients met inclusion criteria: 100% had a Legal Sex, 20% a Gender Identity, and 19% a SAAB; 321 185 patients had field changes other than initial Legal Sex entry. About 2% of patients had a subsequent Legal Sex change, and 25% of those had ≥2 changes; 20% of patients had ≥1 update to Gender Identity and 19% to SAAB. Excluding the first Legal Sex entry, administrators made most changes (67%) across all fields, followed by patients (25%), providers (7.2%), and automated Health Level-7 (HL7) interface messages (0.7%). Provider utilization varied by subspecialty; themes related to systems barriers and personal perceptions were identified. DISCUSSION: Sex and gender demographic fields are primarily used by administrators and raise concern about data accuracy; provider use is heterogenous and lacking. Provider awareness of field availability and variable workflows may impede use. CONCLUSION: EHR metadata highlights areas for improvement of sex and gender field utilization.

OBJECTIVE: Social media-based public health research is crucial for epidemic surveillance, but most studies identify relevant corpora with keyword-matching. This study develops a system to streamline the process of curating colloquial medical dictionaries. We demonstrate the pipeline by curating a Unified Medical Language System (UMLS)-colloquial symptom dictionary from COVID-19-related tweets as proof of concept. METHODS: COVID-19-related tweets from February 1, 2020, to April 30, 2022 were used. The pipeline includes three modules: a named entity recognition module to detect symptoms in tweets; an entity normalization module to aggregate detected entities; and a mapping module that iteratively maps entities to Unified Medical Language System concepts. A random 500 entity samples were drawn from the final dictionary for accuracy validation. Additionally, we conducted a symptom frequency distribution analysis to compare our dictionary to a pre-defined lexicon from previous research. RESULTS: We identified 498 480 unique symptom entity expressions from the tweets. Pre-processing reduces the number to 18 226. The final dictionary contains 38 175 unique expressions of symptoms that can be mapped to 966 UMLS concepts (accuracy = 95%). Symptom distribution analysis found that our dictionary detects more symptoms and is effective at identifying psychiatric disorders like anxiety and depression, often missed by pre-defined lexicons. CONCLUSIONS: This study advances public health research by implementing a novel, systematic pipeline for curating symptom lexicons from social media data. The final lexicon's high accuracy, validated by medical professionals, underscores the potential of this methodology to reliably interpret, and categorize vast amounts of unstructured social media data into actionable medical insights across diverse linguistic and regional landscapes.

OBJECTIVES: To develop an unbiased objective for learning automatic coding algorithms from clinical records annotated with only partial relevant International Classification of Diseases codes, as annotation noise in undercoded clinical records used as training data can mislead the learning process of deep neural networks. MATERIALS AND METHODS: We use Medical Information Mart for Intensive Care III as our dataset. We employ positive-unlabeled learning to achieve unbiased loss estimation, which is free of misleading training signal. We then utilize reweighting mechanism to compensate for the imbalance between positive and negative samples. To further close the performance gap caused by poor quality annotation, we integrate the supervision provided by the automatic annotation tool Medical Concept Annotation Toolkit which can ease the heavy burden of manual validation. RESULTS: Our benchmarking results show that positive-unlabeled learning with reweighting outperforms competitive baseline methods over a range of missing label ratios. Integrating supervision provided by annotation tool further boosted the performance. DISCUSSION: Considering the annotation noise and severe imbalance, unbiased loss estimation and reweighting mechanism are both important for learning from undercoded clinical records. Unbiased loss requires the estimation of false negative ratios and estimation through trained models is practical and competitive. CONCLUSIONS: The combination of positive-unlabeled learning with reweighting and supervision provided by the annotation tool is a promising solution to learn from undercoded clinical records.

OBJECTIVE: Understanding public discourse on emergency use of unproven therapeutics is essential to monitor safe use and combat misinformation. We developed a natural language processing-based pipeline to understand public perceptions of and stances on coronavirus disease 2019 (COVID-19)-related drugs on Twitter across time. METHODS: This retrospective study included 609 189 US-based tweets between January 29, 2020 and November 30, 2021 on 4 drugs that gained wide public attention during the COVID-19 pandemic: (1) Hydroxychloroquine and Ivermectin, drug therapies with anecdotal evidence; and (2) Molnupiravir and Remdesivir, FDA-approved treatment options for eligible patients. Time-trend analysis was used to understand the popularity and related events. Content and demographic analyses were conducted to explore potential rationales of people's stances on each drug. RESULTS: Time-trend analysis revealed that Hydroxychloroquine and Ivermectin received much more discussion than Molnupiravir and Remdesivir, particularly during COVID-19 surges. Hydroxychloroquine and Ivermectin were highly politicized, related to conspiracy theories, hearsay, celebrity effects, etc. The distribution of stance between the 2 major US political parties was significantly different (P < .001); Republicans were much more likely to support Hydroxychloroquine (+55%) and Ivermectin (+30%) than Democrats. People with healthcare backgrounds tended to oppose Hydroxychloroquine (+7%) more than the general population; in contrast, the general population was more likely to support Ivermectin (+14%). CONCLUSION: Our study found that social media users with have different perceptions and stances on off-label versus FDA-authorized drug use across different stages of COVID-19, indicating that health systems, regulatory agencies, and policymakers should design tailored strategies to monitor and reduce misinformation for promoting safe drug use. Our analysis pipeline and stance detection models are made public at https://github.com/ningkko/COVID-drug.

OBJECTIVE: There are signals of clinicians' expert and knowledge-driven behaviors within clinical information systems (CIS) that can be exploited to support clinical prediction. Describe development of the Healthcare Process Modeling Framework to Phenotype Clinician Behaviors for Exploiting the Signal Gain of Clinical Expertise (HPM-ExpertSignals). MATERIALS AND METHODS: We employed an iterative framework development approach that combined data-driven modeling and simulation testing to define and refine a process for phenotyping clinician behaviors. Our framework was developed and evaluated based on the Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) predictive model to detect and leverage signals of clinician expertise for prediction of patient trajectories. RESULTS: Seven themes-identified during development and simulation testing of the CONCERN model-informed framework development. The HPM-ExpertSignals conceptual framework includes a 3-step modeling technique: (1) identify patterns of clinical behaviors from user interaction with CIS; (2) interpret patterns as proxies of an individual's decisions, knowledge, and expertise; and (3) use patterns in predictive models for associations with outcomes. The CONCERN model differentiated at risk patients earlier than other early warning scores, lending confidence to the HPM-ExpertSignals framework. DISCUSSION: The HPM-ExpertSignals framework moves beyond transactional data analytics to model clinical knowledge, decision making, and CIS interactions, which can support predictive modeling with a focus on the rapid and frequent patient surveillance cycle. CONCLUSIONS: We propose this framework as an approach to embed clinicians' knowledge-driven behaviors in predictions and inferences to facilitate capture of healthcare processes that are activated independently, and sometimes well before, physiological changes are apparent.

We defined tolerance range as the distance of observing similar disease conditions or functional status from the upper to the lower boundaries of a specified time interval. A tolerance range was identified for linear regression and support vector machines to optimize the improvement rate (defined as IR) on accuracy in predicting mortality risk in patients with chronic obstructive pulmonary disease using clinical notes. The corpus includes pulmonary, cardiology, and radiology reports of 15,500 patients who died between 2011 and 2017. Their performance was compared against a long short-term memory recurrent neural network. The results demonstrate an overall improvement by those basic machine learning approaches after considering an optimal tolerance range: the average IR of linear regression was 90.1% and the maximum IR of support vector machines was 66.2%. There was a similitude between the time segments produced by our tolerance algorithms and those produced by the long short-term memory.

OBJECTIVE: Incomplete and static reaction picklists in the allergy module led to free-text and missing entries that inhibit the clinical decision support intended to prevent adverse drug reactions. We developed a novel, data-driven, "dynamic" reaction picklist to improve allergy documentation in the electronic health record (EHR). MATERIALS AND METHODS: We split 3 decades of allergy entries in the EHR of a large Massachusetts healthcare system into development and validation datasets. We consolidated duplicate allergens and those with the same ingredients or allergen groups. We created a reaction value set via expert review of a previously developed value set and then applied natural language processing to reconcile reactions from structured and free-text entries. Three association rule-mining measures were used to develop a comprehensive reaction picklist dynamically ranked by allergen. The dynamic picklist was assessed using recall at top k suggested reactions, comparing performance to the static picklist. RESULTS: The modified reaction value set contained 490 reaction concepts. Among 4 234 327 allergy entries collected, 7463 unique consolidated allergens and 469 unique reactions were identified. Of the 3 dynamic reaction picklists developed, the 1 with the optimal ranking achieved recalls of 0.632, 0.763, and 0.822 at the top 5, 10, and 15, respectively, significantly outperforming the static reaction picklist ranked by reaction frequency. CONCLUSION: The dynamic reaction picklist developed using EHR data and a statistical measure was superior to the static picklist and suggested proper reactions for allergy documentation. Further studies might evaluate the usability and impact on allergy documentation in the EHR.

Learning representations of clinical notes poses challenges in handling complex content that necessitates preprocessing steps to make the data more suitable for data mining. An important issue, addressed here, is that of temporal expressions, where cues indicate the time when clinical events occur. We present a three-step data reconstruction algorithm for transforming similar clinical entities (e.g., symptoms, complications) into sequential data through unsupervised annotation of temporal expressions. First, the data reconstruction algorithm detects if an expression has temporal intent. Second, it decomposes and rewrites the expression into non-temporal sub-expression and temporal constraints. Finally, it clusters similar non-temporal sub-expressions by using unsupervised sentence embedding under the modified K-medoids paradigm. We experimented with our proposed algorithm on clinical notes associated with chronic obstructive pulmonary disease (COPD). Visualizing reconstruction results of cardiology reports for a longitudinal cohort of patients with COPD demonstrated that this algorithm is feasible.

OBJECTIVE: The study sought to review recent literature regarding use of speech recognition (SR) technology for clinical documentation and to understand the impact of SR on document accuracy, provider efficiency, institutional cost, and more. MATERIALS AND METHODS: We searched 10 scientific and medical literature databases to find articles about clinician use of SR for documentation published between January 1, 1990, and October 15, 2018. We annotated included articles with their research topic(s), medical domain(s), and SR system(s) evaluated and analyzed the results. RESULTS: One hundred twenty-two articles were included. Forty-eight (39.3%) involved the radiology department exclusively and 10 (8.2%) involved emergency medicine; 10 (8.2%) mentioned multiple departments. Forty-eight (39.3%) articles studied productivity; 20 (16.4%) studied the effect of SR on documentation time, with mixed findings. Decreased turnaround time was reported in all 19 (15.6%) studies in which it was evaluated. Twenty-nine (23.8%) studies conducted error analyses, though various evaluation metrics were used. Reported percentage of documents with errors ranged from 4.8% to 71%; reported word error rates ranged from 7.4% to 38.7%. Seven (5.7%) studies assessed documentation-associated costs; 5 reported decreases and 2 reported increases. Many studies (44.3%) used products by Nuance Communications. Other vendors included IBM (9.0%) and Philips (6.6%); 7 (5.7%) used self-developed systems. CONCLUSION: Despite widespread use of SR for clinical documentation, research on this topic remains largely heterogeneous, often using different evaluation metrics with mixed findings. Further, that SR-assisted documentation has become increasingly common in clinical settings beyond radiology warrants further investigation of its use and effectiveness in these settings.

Objective: To develop a comprehensive value set for documenting and encoding adverse reactions in the allergy module of an electronic health record. Materials and Methods: We analyzed 2 471 004 adverse reactions stored in Partners Healthcare's Enterprise-wide Allergy Repository (PEAR) of 2.7 million patients. Using the Medical Text Extraction, Reasoning, and Mapping System, we processed both structured and free-text reaction entries and mapped them to Systematized Nomenclature of Medicine - Clinical Terms. We calculated the frequencies of reaction concepts, including rare, severe, and hypersensitivity reactions. We compared PEAR concepts to a Federal Health Information Modeling and Standards value set and University of Nebraska Medical Center data, and then created an integrated value set. Results: We identified 787 reaction concepts in PEAR. Frequently reported reactions included: rash (14.0%), hives (8.2%), gastrointestinal irritation (5.5%), itching (3.2%), and anaphylaxis (2.5%). We identified an additional 320 concepts from Federal Health Information Modeling and Standards and the University of Nebraska Medical Center to resolve gaps due to missing and partial matches when comparing these external resources to PEAR. This yielded 1106 concepts in our final integrated value set. The presence of rare, severe, and hypersensitivity reactions was limited in both external datasets. Hypersensitivity reactions represented roughly 20% of the reactions within our data. Discussion: We developed a value set for encoding adverse reactions using a large dataset from one health system, enriched by reactions from 2 large external resources. This integrated value set includes clinically important severe and hypersensitivity reactions. Conclusion: This work contributes a value set, harmonized with existing data, to improve the consistency and accuracy of reaction documentation in electronic health records, providing the necessary building blocks for more intelligent clinical decision support for allergies and adverse reactions.

Background: Adequate clinical information provided with radiology orders is important for an accurate interpretation of imaging studies. Nonetheless, high percentage of radiology orders lack adequate information. Assessment of the adequacy of the information associated with radiology orders could be achieved manually. However, manual assessment is costly and inefficient. Novel approaches using machine learning and text mining to assess the adequacy of radiology order information could reduce the costs and improve efficiency. We aimed to test the application of machine learning algorithms to identify radiology orders with adequate/inadequate information. Methods: We extracted 1,967 electronic chest computed tomography (CT) orders at an academic tertiary hospital during January 2014, and manually classified them into containing adequate or inadequate information based on the American College of Radiology guidelines. We used text mining (text parsing and vectorization) and machine learning (Naïve Bayes, Support Vector Machines and Decision Tree classifiers) to automate order adequacy classification, and evaluated the system performance against the manual review. Results: Surprisingly, only 30.6% of orders had adequate information when evaluated manually. Non-resident physicians provided the least number of adequate order information (26.7%). Classifiers achieved high classification accuracy. Naïve Bayes classifier performed slightly better overall (Accuracy= .9) than Support Vector Machines (Accuracy= .89) and Decision Trees (J48, Accuracy= .85). Conclusions: High percentage of orders lack adequate information in chest CT. Machine learning classifiers could be utilized to assess the adequacy of radiology order information.

Mutual information clustering is an agglomerative hierarchical clustering method that has been used to group random variables or sets thereof. Some researchers have found that the normalization method used can lead to oddly-sized clusters that do not line up with expected results. We introduce a new normalization parameter to control the size of the clusters, and apply it to food allergy data from a large allergy repository from an electronic health record, treating the distributions of food allergies in our population as random variables. Our method was able to identify previously known food cross-reaction groups (with an adjusted Rand index of 0.971, outperforming alternative clustering algorithms), in addition to proposing possible new groups. Our results demonstrate the viability of mutual information clustering as an approach for discovering possible food cross-reactions.

Research clues can be expressed as coherent chains of keywords grouped by theme. Capturing clues to research from the vast and expanding medical literature is valuable. Yet, it is difficult to automatically create clear visualizations of research clues despite the presence of many competing summarization tools. In this paper, we propose a linear classifier based on a spiral, which we call a regional classifier. The study emphasizes the development of visualization methods and the process of finding a specific research clue to track patient needs reported in medical literature. When timelines are combined with a spiral geographical map, they show a geometric shape that helps to reveal the clues from different spatial viewpoints and periodical constraints. Our evaluation showed that the regional classifier produces better visual effects than support vector machine classifiers. It covers important concepts of each theme and is able to represent the relationships among papers in a way that captures continuous developments and changes in key themes.

Objective: Health Information Exchange (HIE) enables providers to share healthcare information electronically across different organizations to promote safer, more efficient, and less costly patient-centered care. This paper describes the development and implementation of a city-wide HIE system in Shanghai, China. Methods: In 2006, as a product of the Chinese healthcare reform, the Health Information Exchange and Sharing Platform was proposed as a means to facilitate HIE within Shanghai. In collaboration with the Shanghai Hospital Development Center, a state-run nonprofit corporate, the HIE project was implemented across multiple levels within the city. The HIE system is based on the Service-oriented Architecture and complies with industry standards. Results: On September 2010, the first and largest Chinese HIE system was established. As of 2016 the system includes all of Shanghai's 38 tertiary hospitals (highest level hospitals in China), plus 6 district hospitals, and 40 community health centers, with coverage for 39 million patients. The system currently provides a rich source of patient information including encounter history, medication history, laboratory results, radiology images and reports, and clinical notes. Initial outcomes indicate a significant reduction in medication errors and duplication of tests, saving at least 48 million RMB a year, with overall improvement in the quality of care following implementation. Conclusion: The adoption of HIE in Shanghai resulted in improved access to accurate, complete, and relevant clinical information in real time, thus facilitating delivery of high-quality, cost-effective, and efficient care.

OBJECTIVE: Accurate food adverse sensitivity documentation in electronic health records (EHRs) is crucial to patient safety. This study examined, encoded, and grouped foods that caused any adverse sensitivity in a large allergy repository using natural language processing and standard terminologies. METHODS: Using the Medical Text Extraction, Reasoning, and Mapping System (MTERMS), we processed both structured and free-text entries stored in an enterprise-wide allergy repository (Partners' Enterprise-wide Allergy Repository), normalized diverse food allergen terms into concepts, and encoded these concepts using the Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) and Unique Ingredient Identifiers (UNII) terminologies. Concept coverage also was assessed for these two terminologies. We further categorized allergen concepts into groups and calculated the frequencies of these concepts by group. Finally, we conducted an external validation of MTERMS's performance when identifying food allergen terms, using a randomized sample from a different institution. RESULTS: We identified 158 552 food allergen records (2140 unique terms) in the Partners repository, corresponding to 672 food allergen concepts. High-frequency groups included shellfish (19.3%), fruits or vegetables (18.4%), dairy (9.0%), peanuts (8.5%), tree nuts (8.5%), eggs (6.0%), grains (5.1%), and additives (4.7%). Ambiguous, generic concepts such as "nuts" and "seafood" accounted for 8.8% of the records. SNOMED-CT covered more concepts than UNII in terms of exact (81.7% vs 68.0%) and partial (14.3% vs 9.7%) matches. DISCUSSION: Adverse sensitivities to food are diverse, and existing standard terminologies have gaps in their coverage of the breadth of allergy concepts. CONCLUSION: New strategies are needed to represent and standardize food adverse sensitivity concepts, to improve documentation in EHRs.

OBJECTIVE: There have been growing concerns about the impact of drug allergy alerts on patient safety and provider alert fatigue. The authors aimed to explore the common drug allergy alerts over the last 10 years and the reasons why providers tend to override these alerts. DESIGN: Retrospective observational cross-sectional study (2004-2013). MATERIALS AND METHODS: Drug allergy alert data (n = 611,192) were collected from two large academic hospitals in Boston, MA (USA). RESULTS: Overall, the authors found an increase in the rate of drug allergy alert overrides, from 83.3% in 2004 to 87.6% in 2013 (P < .001). Alarmingly, alerts for immune mediated and life threatening reactions with definite allergen and prescribed medication matches were overridden 72.8% and 74.1% of the time, respectively. However, providers were less likely to override these alerts compared to possible (cross-sensitivity) or probable (allergen group) matches (P < .001). The most common drug allergy alerts were triggered by allergies to narcotics (48%) and other analgesics (6%), antibiotics (10%), and statins (2%). Only slightly more than one-third of the reactions (34.2%) were potentially immune mediated. Finally, more than half of the overrides reasons pointed to irrelevant alerts (i.e., patient has tolerated the medication before, 50.9%) and providers were significantly more likely to override repeated alerts (89.7%) rather than first time alerts (77.4%, P < .001). DISCUSSION AND CONCLUSIONS: These findings underline the urgent need for more efforts to provide more accurate and relevant drug allergy alerts to help reduce alert override rates and improve alert fatigue.

Accurate electronic health records are important for clinical care and research as well as ensuring patient safety. It is crucial for misspelled words to be corrected in order to ensure that medical records are interpreted correctly. This paper describes the development of a spelling correction system for medical text. Our spell checker is based on Shannon's noisy channel model, and uses an extensive dictionary compiled from many sources. We also use named entity recognition, so that names are not wrongly corrected as misspellings. We apply our spell checker to three different types of free-text data: clinical notes, allergy entries, and medication orders; and evaluate its performance on both misspelling detection and correction. Our spell checker achieves detection performance of up to 94.4% and correction accuracy of up to 88.2%. We show that high-performance spelling correction is possible on a variety of clinical documents.

OBJECTIVE: Allergy documentation and exchange are vital to ensuring patient safety. This study aims to analyze and compare various existing standard terminologies for representing allergy information. METHODS: Five terminologies were identified, including the Systemized Nomenclature of Medical Clinical Terms (SNOMED CT), National Drug File-Reference Terminology (NDF-RT), Medication Dictionary for Regulatory Activities (MedDRA), Unique Ingredient Identifier (UNII), and RxNorm. A qualitative analysis was conducted to compare desirable characteristics of each terminology, including content coverage, concept orientation, formal definitions, multiple granularities, vocabulary structure, subset capability, and maintainability. A quantitative analysis was also performed to compare the content coverage of each terminology for (1) common food, drug, and environmental allergens and (2) descriptive concepts for common drug allergies, adverse reactions (AR), and no known allergies. RESULTS: Our qualitative results show that SNOMED CT fulfilled the greatest number of desirable characteristics, followed by NDF-RT, RxNorm, UNII, and MedDRA. Our quantitative results demonstrate that RxNorm had the highest concept coverage for representing drug allergens, followed by UNII, SNOMED CT, NDF-RT, and MedDRA. For food and environmental allergens, UNII demonstrated the highest concept coverage, followed by SNOMED CT. For representing descriptive allergy concepts and adverse reactions, SNOMED CT and NDF-RT showed the highest coverage. Only SNOMED CT was capable of representing unique concepts for encoding no known allergies. CONCLUSIONS: The proper terminology for encoding a patient's allergy is complex, as multiple elements need to be captured to form a fully structured clinical finding. Our results suggest that while gaps still exist, a combination of SNOMED CT and RxNorm can satisfy most criteria for encoding common allergies and provide sufficient content coverage.

OBJECTIVE: To measure the uncertainty of temporal assertions like "3 weeks ago" in clinical texts. DESIGN: Temporal assertions extracted from narrative clinical reports were compared to facts extracted from a structured clinical database for the same patients. MEASUREMENTS: The authors correlated the assertions and the facts to determine the dependence of the uncertainty of the assertions on the semantic and lexical properties of the assertions. RESULTS: The observed deviation between the stated duration and actual duration averaged about 20% of the stated deviation. Linear regression revealed that assertions about events further in the past tend to be more uncertain, smaller numeric values tend to be more uncertain (1 mo v. 30 d), and round numbers tend to be more uncertain (10 versus 11 yrs). CONCLUSIONS: The authors empirically derived semantics behind statements of duration using "ago," and verified intuitions about how numbers are used.

OBJECTIVE Electronic health records (EHRs) have the potential to advance the quality of care, but studies have shown mixed results. The authors sought to examine the extent of EHR usage and how the quality of care delivered in ambulatory care practices varied according to duration of EHR availability. METHODS The study linked two data sources: a statewide survey of physicians' adoption and use of EHR and claims data reflecting quality of care as indicated by physicians' performance on widely used quality measures. Using four years of measurement, we combined 18 quality measures into 6 clinical condition categories. While the survey of physicians was cross-sectional, respondents indicated the year in which they adopted EHR. In an analysis accounting for duration of EHR use, we examined the relationship between EHR adoption and quality of care. RESULTS The percent of physicians reporting adoption of EHR and availability of EHR core functions more than doubled between 2000 and 2005. Among EHR users in 2005, the average duration of EHR use was 4.8 years. For all 6 clinical conditions, there was no difference in performance between EHR users and non-users. In addition, for these 6 clinical conditions, there was no consistent pattern between length of time using an EHR and physicians performance on quality measures in both bivariate and multivariate analyses. CONCLUSIONS In this cross-sectional study, we found no association between duration of using an EHR and performance with respect to quality of care, although power was limited. Intensifying the use of key EHR features, such as clinical decision support, may be needed to realize quality improvement from EHRs. Future studies should examine the relationship between the extent to which physicians use key EHR functions and their performance on quality measures over time.

CONTEXT: TimeText is a temporal reasoning system designed to represent, extract, and reason about temporal information in clinical text. OBJECTIVE: To measure the accuracy of the TimeText for processing clinical discharge summaries. DESIGN: Six physicians with biomedical informatics training served as domain experts. Twenty discharge summaries were randomly selected for the evaluation. For each of the first 14 reports, 5 to 8 clinically important medical events were chosen. The temporal reasoning system generated temporal relations about the endpoints (start or finish) of pairs of medical events. Two experts (subjects) manually generated temporal relations for these medical events. The system and expert-generated results were assessed by four other experts (raters). All of the twenty discharge summaries were used to assess the system's accuracy in answering time-oriented clinical questions. For each report, five to ten clinically plausible temporal questions about events were generated. Two experts generated answers to the questions to serve as the gold standard. We wrote queries to retrieve answers from system's output. MEASUREMENTS: Correctness of generated temporal relations, recall of clinically important relations, and accuracy in answering temporal questions. RESULTS: The raters determined that 97% of subjects' 295 generated temporal relations were correct and that 96.5% of the system's 995 generated temporal relations were correct. The system captured 79% of 307 temporal relations determined to be clinically important by the subjects and raters. The system answered 84% of the temporal questions correctly. CONCLUSION: The system encoded the majority of information identified by experts, and was able to answer simple temporal questions.

OBJECTIVE: To model the temporal information contained in medical narrative reports as a simple temporal constraint satisfaction problem. DESIGN: A constraint satisfaction problem is defined by time points and constraints (inequalities between points). A time interval comprises a pair of points and a constraint. Five complete electronic discharge summaries and paragraphs from 226 other discharge summaries were studied. Medical events were represented as intervals, and assertions about events were represented as constraints. Through a consensus process, a set of encoding procedures and a list of issues related to encoding were generated. MEASUREMENTS: Instances of temporal disjunction and contradiction and distribution of temporal constraints were used. RESULTS: An average of 95 medical events (range, 46-151) and 234 temporal assertions (range, 118-388) were identified per complete discharge summary. Nondefinitional assertions were explicit (36%) or implicit (64%) and absolute (17%), qualitative (72%), or metric (11%). Implicit assertions were based on domain knowledge and assumptions, e.g., the section of the report determined the ordering of events. Issues included linking events, intermittence, periodicity, granularity, vagueness, ambiguity, uncertainty, and plans. ions such as intermittence were not represented explicitly. The temporal network was sparse: Only 0.80% (range, 0.42%-1.38%) of possible constraints were instantiated. No instances of discontinuous temporal disjunction were found in the complete summaries or the 226 paragraphs. One instance of temporal contradiction was found (intrareport rate of 0.2 with a 95% confidence interval of 0.005-1.114). CONCLUSION: A simple temporal constraint satisfaction problem appears sufficient to represent most temporal assertions in discharge summaries and may be useful for encoding electronic medical records.