BACKGROUND: The use of generative large language models (LLMs) with electronic health record (EHR) data is rapidly expanding to support clinical and research tasks. This systematic review characterizes the clinical fields and use cases that have been studied and evaluated to date. METHODS: We followed the Preferred Reporting Items for Systematic Review and Meta-Analyses guidelines to conduct a systematic review of articles from PubMed and Web of Science published between January 1, 2023, and November 9, 2024. Studies were included if they used generative LLMs to analyze real-world EHR data and reported quantitative performance evaluations. Through data extraction, we identified clinical specialties and tasks for each included article, and summarized evaluation methods. RESULTS: Of the 18 735 articles retrieved, 196 met our criteria. Most studies focused on radiology (26.0%), oncology (10.7%), and emergency medicine (6.6%). Regarding clinical tasks, clinical decision support made up the largest proportion of studies (62.2%), while summarizations and patient communications made up the smallest, at 5.6% and 5.1%, respectively. In addition, GPT-4 and GPT-3.5 were the most commonly used generative LLMs, appearing in 60.2% and 57.7% of studies, respectively. Across these studies, we identified 22 unique non-NLP metrics and 35 unique NLP metrics. While NLP metrics offer greater scalability, none demonstrated a strong correlation with gold-standard human evaluations. CONCLUSION: Our findings highlight the need to evaluate generative LLMs on EHR data across a broader range of clinical specialties and tasks, as well as the urgent need for standardized, scalable, and clinically meaningful evaluation frameworks.

OBJECTIVES: Despite federally mandated collection of sex and gender demographics in the electronic health record (EHR), longitudinal assessments are lacking. We assessed sex and gender demographic field utilization using EHR metadata. MATERIALS AND METHODS: Patients ≥18 years of age in the Mass General Brigham health system with a first Legal Sex entry (registration requirement) between January 8, 2018 and January 1, 2022 were included in this retrospective study. Metadata for all sex and gender fields (Legal Sex, Sex Assigned at Birth [SAAB], Gender Identity) were quantified by completion rates, user types, and longitudinal change. A nested qualitative study of providers from specialties with high and low field use identified themes related to utilization. RESULTS: 1 576 120 patients met inclusion criteria: 100% had a Legal Sex, 20% a Gender Identity, and 19% a SAAB; 321 185 patients had field changes other than initial Legal Sex entry. About 2% of patients had a subsequent Legal Sex change, and 25% of those had ≥2 changes; 20% of patients had ≥1 update to Gender Identity and 19% to SAAB. Excluding the first Legal Sex entry, administrators made most changes (67%) across all fields, followed by patients (25%), providers (7.2%), and automated Health Level-7 (HL7) interface messages (0.7%). Provider utilization varied by subspecialty; themes related to systems barriers and personal perceptions were identified. DISCUSSION: Sex and gender demographic fields are primarily used by administrators and raise concern about data accuracy; provider use is heterogenous and lacking. Provider awareness of field availability and variable workflows may impede use. CONCLUSION: EHR metadata highlights areas for improvement of sex and gender field utilization.

OBJECTIVES: Large language models (LLMs) are poised to change care delivery, but their impact on health equity is unclear. While marginalized populations have been historically excluded from early technology developments, LLMs present an opportunity to change our approach to developing, evaluating, and implementing new technologies. In this perspective, we describe the role of LLMs in supporting health equity. MATERIALS AND METHODS: We apply the National Institute on Minority Health and Health Disparities (NIMHD) research framework to explore the use of LLMs for health equity. RESULTS: We present opportunities for how LLMs can improve health equity across individual, family and organizational, community, and population health. We describe emerging concerns including biased data, limited technology diffusion, and privacy. Finally, we highlight recommendations focused on prompt engineering, retrieval augmentation, digital inclusion, transparency, and bias mitigation. CONCLUSION: The potential of LLMs to support health equity depends on making health equity a focus from the start.

Informatics researchers and practitioners have started exploring racism related to the implementation and use of electronic health records (EHRs). While this work has begun to expose structural racism which is a fundamental driver of racial and ethnic disparities, there is a lack of inclusion of concepts of racism in this work. This perspective provides a classification of racism at 3 levels-individual, organizational, and structural-and offers recommendations for future research, practice, and policy. Our recommendations include the need to capture and use structural measures of social determinants of health to address structural racism, intersectionality as a theoretical framework for research, structural competency training, research on the role of prejudice and stereotyping in stigmatizing documentation in EHRs, and actions to increase the diversity of private sector informatics workforce and participation of minority scholars in specialty groups. Informaticians have an ethical and moral obligation to address racism, and private and public sector organizations have a transformative role in addressing equity and racism associated with EHR implementation and use.

OBJECTIVES: To develop an unbiased objective for learning automatic coding algorithms from clinical records annotated with only partial relevant International Classification of Diseases codes, as annotation noise in undercoded clinical records used as training data can mislead the learning process of deep neural networks. MATERIALS AND METHODS: We use Medical Information Mart for Intensive Care III as our dataset. We employ positive-unlabeled learning to achieve unbiased loss estimation, which is free of misleading training signal. We then utilize reweighting mechanism to compensate for the imbalance between positive and negative samples. To further close the performance gap caused by poor quality annotation, we integrate the supervision provided by the automatic annotation tool Medical Concept Annotation Toolkit which can ease the heavy burden of manual validation. RESULTS: Our benchmarking results show that positive-unlabeled learning with reweighting outperforms competitive baseline methods over a range of missing label ratios. Integrating supervision provided by annotation tool further boosted the performance. DISCUSSION: Considering the annotation noise and severe imbalance, unbiased loss estimation and reweighting mechanism are both important for learning from undercoded clinical records. Unbiased loss requires the estimation of false negative ratios and estimation through trained models is practical and competitive. CONCLUSIONS: The combination of positive-unlabeled learning with reweighting and supervision provided by the annotation tool is a promising solution to learn from undercoded clinical records.

The 21st Century Cures Act and the rise of telemedicine led to renewed focus on patient portals. However, portal use disparities persist and are in part driven by limited digital literacy. To address digital disparities in primary care, we implemented an integrated digital health navigator program supporting portal use among patients with type II diabetes. During our pilot, we were able to enroll 121 (30.9%) patients onto the portal. Of newly enrolled or trained patients, 75 (62.0%) were Black, 13 (10.7%) were White, 23 (19.0%) were Hispanic/Latinx, 4 (3.3%) were Asian, 3 (2.5%) were of another race or ethnicity, and 3 (2.5%) had missing data. Our overall portal enrollment for clinic patients with type II diabetes increased for Hispanic/Latinx patients from 30% to 42% and Black patients from 49% to 61%. We used the Consolidated Framework for Implementation Research to understand key implementation components. Using our approach, other clinics can implement an integrated digital health navigator to support patient portal use.

OBJECTIVE: To improve problem list documentation and care quality. MATERIALS AND METHODS: We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. RESULTS: There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. DISCUSSION: The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. CONCLUSION: An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.

OBJECTIVE: To evaluate and understand pregnant patients' perspectives on the implementation of artificial intelligence (AI) in clinical care with a focus on opportunities to improve healthcare technologies and healthcare delivery. MATERIALS AND METHODS: We developed an anonymous survey and enrolled patients presenting to the labor and delivery unit at a tertiary care center September 2019-June 2020. We investigated the role and interplay of patient demographic factors, healthcare literacy, understanding of AI, comfort levels with various AI scenarios, and preferences for AI use in clinical care. RESULTS: Of the 349 parturients, 57.6% were between the ages of 25-34 years, 90.1% reported college or graduate education and 69.2% believed the benefits of AI use in clinical care outweighed the risks. Cluster analysis revealed 2 distinct groups: patients more comfortable with clinical AI use (Pro-AI) and those who preferred physician presence (AI-Cautious). Pro-AI patients had a higher degree of education, were more knowledgeable about AI use in their daily lives and saw AI use as a significant advancement in medicine. AI-Cautious patients reported a lack of human qualities and low trust in the technology as detriments to AI use. DISCUSSION: Patient trust and the preservation of the human physician-patient relationship are critical in moving forward with AI implementation in healthcare. Pregnant individuals are cautiously optimistic about AI use in their care. CONCLUSION: Our findings provide insights into the status of AI use in perinatal care and provide a platform for driving patient-centered innovations.

OBJECTIVE: Clinicians spend significant time working in the electronic health record (EHR). The US is an outlier in EHR time, suggesting that EHR-related work may be driven in part by the legal environment and threat of malpractice. To assess this, we evaluate the association between state-level malpractice climate and clinician time spent in the EHR. MATERIALS AND METHODS: We use EHR metadata from 351 ambulatory care health systems in the United States using Epic from January-August 2019 combined with state-level data on malpractice incidence and payouts. We used descriptive statistics to measure variation in clinician EHR time, including total EHR time, documentation time per day, and after-hours EHR time per day. Multi-variable regression evaluated the association between clinicians in high malpractice states and EHR use. RESULTS: We found no association between location in a state in the top-quartile of malpractice payouts and time spent in the EHR per day, time spent in the EHR outside of scheduled hours, or time spent documenting per day, except for a subgroup of the clinicians in the highest malpractice specialties, where there was a small increase in EHR time per day (B = 6.08 min, P < 0.001) and time spent documenting notes (B = 2.77 min, P < 0.001). DISCUSSION: State-level differences in malpractice incidence are unlikely to be a significant driver of EHR work for most clinicians. CONCLUSION: Policymakers seeking to address EHR documentation burden should examine burden driven by other socio-technical demands on clinician time, such as billing or quality measurement.

OBJECTIVE: Understanding public discourse on emergency use of unproven therapeutics is essential to monitor safe use and combat misinformation. We developed a natural language processing-based pipeline to understand public perceptions of and stances on coronavirus disease 2019 (COVID-19)-related drugs on Twitter across time. METHODS: This retrospective study included 609 189 US-based tweets between January 29, 2020 and November 30, 2021 on 4 drugs that gained wide public attention during the COVID-19 pandemic: (1) Hydroxychloroquine and Ivermectin, drug therapies with anecdotal evidence; and (2) Molnupiravir and Remdesivir, FDA-approved treatment options for eligible patients. Time-trend analysis was used to understand the popularity and related events. Content and demographic analyses were conducted to explore potential rationales of people's stances on each drug. RESULTS: Time-trend analysis revealed that Hydroxychloroquine and Ivermectin received much more discussion than Molnupiravir and Remdesivir, particularly during COVID-19 surges. Hydroxychloroquine and Ivermectin were highly politicized, related to conspiracy theories, hearsay, celebrity effects, etc. The distribution of stance between the 2 major US political parties was significantly different (P < .001); Republicans were much more likely to support Hydroxychloroquine (+55%) and Ivermectin (+30%) than Democrats. People with healthcare backgrounds tended to oppose Hydroxychloroquine (+7%) more than the general population; in contrast, the general population was more likely to support Ivermectin (+14%). CONCLUSION: Our study found that social media users with have different perceptions and stances on off-label versus FDA-authorized drug use across different stages of COVID-19, indicating that health systems, regulatory agencies, and policymakers should design tailored strategies to monitor and reduce misinformation for promoting safe drug use. Our analysis pipeline and stance detection models are made public at https://github.com/ningkko/COVID-drug.

OBJECTIVE: We developed a comprehensive, medication-related clinical decision support (CDS) software prototype for use in the operating room. The purpose of this study was to compare the usability of the CDS software to the current standard electronic health record (EHR) medication administration and documentation workflow. MATERIALS AND METHODS: The primary outcome was the time taken to complete all simulation tasks. Secondary outcomes were the total number of mouse clicks and the total distance traveled on the screen in pixels. Forty participants were randomized and assigned to complete 7 simulation tasks in 1 of 2 groups: (1) the CDS group (n = 20), who completed tasks using the CDS and (2) the Control group (n = 20), who completed tasks using the standard medication workflow with retrospective manual documentation in our anesthesia information management system. Blinding was not possible. We video- and audio-recorded the participants to capture quantitative data (time on task, mouse clicks, and pixels traveled on the screen) and qualitative data (think-aloud verbalization). RESULTS: The CDS group mean total task time (402.2 ± 85.9 s) was less than the Control group (509.8 ± 103.6 s), with a mean difference of 107.6 s (95% confidence interval [CI], 60.5-179.5 s, P < .001). The CDS group used fewer mouse clicks (26.4 ± 4.5 clicks) than the Control group (56.0 ± 15.0 clicks) with a mean difference of 29.6 clicks (95% CI, 23.2-37.6, P < .001). The CDS group had fewer pixels traveled on the computer monitor (59.5 ± 20.0 thousand pixels) than the Control group (109.3 ± 40.8 thousand pixels) with a mean difference of 49.8 thousand pixels (95% CI, 33.0-73.7, P < .001). CONCLUSIONS: The perioperative medication-related CDS software prototype substantially outperformed standard EHR workflow by decreasing task time and improving efficiency and quality of care in a simulation setting.

Open discussions of social justice and health inequities may be an uncommon focus within information technology science, business, and health care delivery partnerships. However, the COVID-19 pandemic-which disproportionately affected Black, indigenous, and people of color-has reinforced the need to examine and define roles that technology partners should play to lead anti-racism efforts through our work. In our perspective piece, we describe the imperative to prioritize TechQuity-equity and social justice as a technology business strategy-through collaborating in partnerships that focus on eliminating racial and social inequities.

OBJECTIVE: Little is known regarding variation among electronic health record (EHR) vendors in quality performance. This issue is compounded by selection effects in which high-quality hospitals coalesce to a subset of market leading vendors. We measured hospital performance, stratified by EHR vendor, across 4 quality metrics. MATERIALS AND METHODS: We used data on 1272 hospitals in 2018 across 4 quality measures: Leapfrog Computerized Provider Order Entry/EHR Evaluation, Centers for Medicare and Medicaid Services Hospital Compare Star Ratings, Hospital-Acquired Condition (HAC) score, and Hospital Readmission Reduction Program (HRRP) ratio. We examined score distributions and used multivariable regression to evaluate the association between vendor and score, recovering partial R2 to assess the proportion of quality variation explained by vendor. RESULTS: We found significant variation across and within EHR vendors. The largest vendor, vendor A, had the highest mean score on the Leapfrog Computerized Provider Order Entry/EHR Evaluation and HRRP ratio, vendor G had the highest Hospital Compare score, and vendor F had the highest HAC score. In adjusted models, no vendor was significantly associated with higher performance on more than 2 measures. EHR vendor explained between 1.2% (HAC) and 7.6 (HRRP) of the variation in quality performance. DISCUSSION: No EHR vendor was associated with higher quality across all measures, and the 2 largest vendors were not associated with the highest scores. Only a small fraction of quality variation was explained by EHR vendor choice. CONCLUSIONS: Top performance on quality measures can be achieved with any EHR vendor; much of quality performance is driven by the hospital and how it uses the EHR.

OBJECTIVE: The study sought to develop an in-depth understanding of how hospitals with a long history of health information technology (HIT) use have responded to the COVID-19 (coronavirus disease 2019) pandemic from an HIT perspective. MATERIALS AND METHODS: We undertook interviews with 44 healthcare professionals with a background in informatics from 6 hospitals internationally. Interviews were informed by a topic guide and were conducted via videoconferencing software. Thematic analysis was employed to develop a coding framework and identify emerging themes. RESULTS: Three themes and 6 subthemes were identified. HITs were employed to manage time and resources during a surge in patient numbers through fast-tracked governance procedures, and the creation of real-time bed capacity tracking within electronic health records. Improving the integration of different hospital systems was identified as important across sites. The use of hard-stop alerts and order sets were perceived as being effective at helping to respond to potential medication shortages and selecting available drug treatments. Utilizing information from multiple data sources to develop alerts facilitated treatment. Finally, the upscaling/optimization of telehealth and remote working capabilities was used to reduce the risk of nosocomial infection within hospitals. DISCUSSION: A number of the HIT-related changes implemented at these sites were perceived to have facilitated more effective patient treatment and management of resources. Informaticians generally felt more valued by hospital management as a result. CONCLUSIONS: Improving integration between data systems, utilizing specialized alerts, and expanding telehealth represent strategies that hospitals should consider when using HIT for delivering hospital care in the context of the COVID-19 pandemic.

OBJECTIVE: To determine user and electronic health records (EHR) integration requirements for a scalable remote symptom monitoring intervention for asthma patients and their providers. METHODS: Guided by the Non-Adoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework, we conducted a user-centered design process involving English- and Spanish-speaking patients and providers affiliated with an academic medical center. We conducted a secondary analysis of interview transcripts from our prior study, new design sessions with patients and primary care providers (PCPs), and a survey of PCPs. We determined EHR integration requirements as part of the asthma app design and development process. RESULTS: Analysis of 26 transcripts (21 patients, 5 providers) from the prior study, 21 new design sessions (15 patients, 6 providers), and survey responses from 55 PCPs (71% of 78) identified requirements. Patient-facing requirements included: 1- or 5-item symptom questionnaires each week, depending on asthma control; option to request a callback; ability to enter notes, triggers, and peak flows; and tips pushed via the app prior to a clinic visit. PCP-facing requirements included a clinician-facing dashboard accessible from the EHR and an EHR inbox message preceding the visit. PCP preferences diverged regarding graphical presentations of patient-reported outcomes (PROs). Nurse-facing requirements included callback requests sent as an EHR inbox message. Requirements were consistent for English- and Spanish-speaking patients. EHR integration required use of custom application programming interfaces (APIs). CONCLUSION: Using the NASSS framework to guide our user-centered design process, we identified patient and provider requirements for scaling an EHR-integrated remote symptom monitoring intervention in primary care. These requirements met the needs of patients and providers. Additional standards for PRO displays and EHR inbox APIs are needed to facilitate spread.

OBJECTIVE: To assess the appropriateness of medication-related clinical decision support (CDS) alerts associated with renal insufficiency and the potential/actual harm from overriding the alerts. MATERIALS AND METHODS: Override rate frequency was recorded for all inpatients who had a renal CDS alert trigger between 05/2017 and 04/2018. Two random samples of 300 for each of 2 types of medication-related CDS alerts associated with renal insufficiency-"dose change" and "avoid medication"-were evaluated by 2 independent reviewers using predetermined criteria for appropriateness of alert trigger, appropriateness of override, and patient harm. RESULTS: We identified 37 100 "dose change" and 5095 "avoid medication" alerts in the population evaluated, and 100% of each were overridden. Dose change triggers were classified as 12.5% appropriate and overrides of these alerts classified as 90.5% appropriate. Avoid medication triggers were classified as 29.6% appropriate and overrides 76.5% appropriate. We identified 5 adverse drug events, and, of these, 4 of the 5 were due to inappropriately overridden alerts. CONCLUSION: Alerts were nearly always presented inappropriately and were all overridden during the 1-year period studied. Alert fatigue resulting from receiving too many poor-quality alerts may result in failure to recognize errors that could lead to patient harm. Although medication-related CDS alerts associated with renal insufficiency had previously been found to be the most clinically beneficial alerts in a legacy system, in this system they were ineffective. These findings underscore the need for improvements in alert design, implementation, and monitoring of alert performance to make alerts more patient-specific and clinically appropriate.

OBJECTIVE: Pressure injuries are common and serious complications for hospitalized patients. The pressure injury rate is an important patient safety metric and an indicator of the quality of nursing care. Timely and accurate prediction of pressure injury risk can significantly facilitate early prevention and treatment and avoid adverse outcomes. While many pressure injury risk assessment tools exist, most were developed before there was access to large clinical datasets and advanced statistical methods, limiting their accuracy. In this paper, we describe the development of machine learning-based predictive models, using phenotypes derived from nurse-entered direct patient assessment data. METHODS: We utilized rich electronic health record data, including full assessment records entered by nurses, from 5 different hospitals affiliated with a large integrated healthcare organization to develop machine learning-based prediction models for pressure injury. Five-fold cross-validation was conducted to evaluate model performance. RESULTS: Two pressure injury phenotypes were defined for model development: nonhospital acquired pressure injury (N = 4398) and hospital acquired pressure injury (N = 1767), representing 2 distinct clinical scenarios. A total of 28 clinical features were extracted and multiple machine learning predictive models were developed for both pressure injury phenotypes. The random forest model performed best and achieved an AUC of 0.92 and 0.94 in 2 test sets, respectively. The Glasgow coma scale, a nurse-entered level of consciousness measurement, was the most important feature for both groups. CONCLUSIONS: This model accurately predicts pressure injury development and, if validated externally, may be helpful in widespread pressure injury prevention.

We defined tolerance range as the distance of observing similar disease conditions or functional status from the upper to the lower boundaries of a specified time interval. A tolerance range was identified for linear regression and support vector machines to optimize the improvement rate (defined as IR) on accuracy in predicting mortality risk in patients with chronic obstructive pulmonary disease using clinical notes. The corpus includes pulmonary, cardiology, and radiology reports of 15,500 patients who died between 2011 and 2017. Their performance was compared against a long short-term memory recurrent neural network. The results demonstrate an overall improvement by those basic machine learning approaches after considering an optimal tolerance range: the average IR of linear regression was 90.1% and the maximum IR of support vector machines was 66.2%. There was a similitude between the time segments produced by our tolerance algorithms and those produced by the long short-term memory.

OBJECTIVE: The objective of this paper is to share challenges, recommendations, and lessons learned regarding the development and implementation of a Patient Safety Learning Laboratory (PSLL) project, an innovative and complex intervention comprised of a suite of Health Information Technology (HIT) tools integrated with a newly implemented Electronic Health Record (EHR) vendor system in the acute care setting at a large academic center. MATERIALS AND METHODS: The PSLL Administrative Core engaged stakeholders and study personnel throughout all phases of the project: problem analysis, design, development, implementation, and evaluation. Implementation challenges and recommendations were derived from direct observations and the collective experience of PSLL study personnel. RESULTS: The PSLL intervention was implemented on 12 inpatient units during the 18-month study period, potentially impacting 12,628 patient admissions. Challenges to implementation included stakeholder engagement, project scope/complexity, technology/governance, and team structure. Recommendations to address each of these challenges were generated, some enacted during the trial, others as lessons learned for future iterative refinements of the intervention and its implementation. CONCLUSION: Designing, implementing, and evaluating a suite of tools integrated within a vendor EHR to improve patient safety has a variety of challenges. Keys to success include continuous stakeholder engagement, involvement of systems and human factors engineers within a multidisciplinary team, an iterative approach to user-centered design, and a willingness to think outside of current workflows and processes to change health system culture around adverse event prevention.

OBJECTIVE: The study sought to evaluate the overall performance of hospitals that used the Computerized Physician Order Entry Evaluation Tool in both 2017 and 2018, along with their performance against fatal orders and nuisance orders. MATERIALS AND METHODS: We evaluated 1599 hospitals that took the test in both 2017 and 2018 by using their overall percentage scores on the test, along with the percentage of fatal orders appropriately alerted on, and the percentage of nuisance orders incorrectly alerted on. RESULTS: Hospitals showed overall improvement; the mean score in 2017 was 58.1%, and this increased to 66.2% in 2018. Fatal order performance improved slightly from 78.8% to 83.0% (P < .001), though there was almost no change in nuisance order performance (89.0% to 89.7%; P = .43). Hospitals alerting on one or more nuisance orders had a 3-percentage-point increase in their overall score. DISCUSSION: Despite the improvement of overall scores in 2017 and 2018, there was little improvement in fatal order performance, suggesting that hospitals are not targeting the deadliest orders first. Nuisance order performance showed almost no improvement, and some hospitals may be achieving higher scores by overalerting, suggesting that the thresholds for which alerts are fired from are too low. CONCLUSIONS: Although hospitals improved overall from 2017 to 2018, there is still important room for improvement for both fatal and nuisance orders. Hospitals that incorrectly alerted on one or more nuisance orders had slightly higher overall performance, suggesting that some hospitals may be achieving higher scores at the cost of overalerting, which has the potential to cause clinician burnout and even worsen safety.

OBJECTIVE: The study sought to determine frequency and appropriateness of overrides of high-priority drug-drug interaction (DDI) alerts and whether adverse drug events (ADEs) were associated with overrides in a newly implemented electronic health record. MATERIALS AND METHODS: We conducted a retrospective study of overridden high-priority DDI alerts occurring from April 1, 2016, to March 31, 2017, from inpatient and outpatient settings at an academic health center. We studied highest-severity DDIs that were previously designated as "hard stops" and additional high-priority DDIs identified from clinical experience and literature review. All highest-severity alert overrides (n = 193) plus a stratified random sample of additional overrides (n = 371) were evaluated for override appropriateness, using predetermined criteria. Charts were reviewed to identify ADEs for overrides that resulted in medication administration. A chi-square test was used to compare ADE rate by override appropriateness. RESULTS: Of 16 011 alerts presented to providers, 15 318 (95.7%) were overridden, including 193 (87.3%) of the highest-severity DDIs and 15 125 (95.8%) of additional DDIs. Override appropriateness was 45.4% overall, 0.5% for highest-severity DDIs and 68.7% for additional DDIs. For alerts that resulted in medication administration (n = 423, 75.0%), 29 ADEs were identified (6.9%, 5.1 per 100 overrides). The rate of ADEs was higher with inappropriate vs appropriate overrides (9.4% vs 4.3%; P = .038). CONCLUSIONS: The override rate was nearly 90% for even the highest-severity DDI alerts, indicating that stronger suggestions should be made for these alerts, while other alerts should be evaluated for potential suppression.

OBJECTIVE: Nursing time represents one of the highest costs for most health services. We conducted a systematic review of the literature on the impact of health information technology on nurses' time. MATERIALS AND METHODS: We followed PRISMA guidelines and searched 6 large databases for relevant articles published between Jan 2004 and December 2019. Two authors reviewed the titles, abstracts, and full texts. We included articles that included a comparison group in the design, measured the time taken to carry out documentation or medication administration, documented the quantitative estimates of time differences between the 2, had nurses as subjects, and was conducted in either a care home, hospital, or community clinic. RESULTS: We identified a total of 1647 articles, of which 33 met our inclusion criteria. Twenty-one studies reported the impact of 12 different health information technology (HIT) implementations on nurses' documentation time. Weighted averages were calculated for studies that implemented barcode medication administration (BCMA) and 2 weighted averages for those that implemented EHRs, as these studies used different sampling units; both showed an increase in the time spent in documentation (+22% and +46%). However, the time spent carrying out medication administration following BCMA implementation fell by 33% (P < .05). HIT also caused a redistribution of nurses' time which, in some cases, was spent in more "value-adding" activities, such as delivering direct patient care as well as inter-professional communication. DISCUSSION AND CONCLUSIONS: Most of the HIT systems increased nursing documentation time, although time fell for medication administration following BCMA. Many HIT systems also resulted in nurses spending more time in direct care and "value-adding" activities.

Data change the game in terms of how we respond to pandemics. Global data on disease trajectories and the effectiveness and economic impact of different social distancing measures are essential to facilitate effective local responses to pandemics. COVID-19 data flowing across geographic borders are extremely useful to public health professionals for many purposes such as accelerating the pharmaceutical development pipeline, and for making vital decisions about intensive care unit rooms, where to build temporary hospitals, or where to boost supplies of personal protection equipment, ventilators, or diagnostic tests. Sharing data enables quicker dissemination and validation of pharmaceutical innovations, as well as improved knowledge of what prevention and mitigation measures work. Even if physical borders around the globe are closed, it is crucial that data continues to transparently flow across borders to enable a data economy to thrive, which will promote global public health through global cooperation and solidarity.

Learning representations of clinical notes poses challenges in handling complex content that necessitates preprocessing steps to make the data more suitable for data mining. An important issue, addressed here, is that of temporal expressions, where cues indicate the time when clinical events occur. We present a three-step data reconstruction algorithm for transforming similar clinical entities (e.g., symptoms, complications) into sequential data through unsupervised annotation of temporal expressions. First, the data reconstruction algorithm detects if an expression has temporal intent. Second, it decomposes and rewrites the expression into non-temporal sub-expression and temporal constraints. Finally, it clusters similar non-temporal sub-expressions by using unsupervised sentence embedding under the modified K-medoids paradigm. We experimented with our proposed algorithm on clinical notes associated with chronic obstructive pulmonary disease (COPD). Visualizing reconstruction results of cardiology reports for a longitudinal cohort of patients with COPD demonstrated that this algorithm is feasible.

We established a Patient Safety Learning Laboratory comprising 2 core and 3 individual project teams to introduce a suite of digital health tools integrated with our electronic health record to identify, assess, and mitigate threats to patient safety in real time. One of the core teams employed systems engineering (SE) and human factors (HF) methods to analyze problems, design and develop improvements to intervention components, support implementation, and evaluate the system of systems as an integrated whole. Of the 29 participants, 19 and 16 participated in surveys and focus groups, respectively, about their perception of SE and HF. We identified 7 themes regarding use of the 12 SE and HF methods over the 4-year project. Qualitative methods (interviews, focus, groups, observations, usability testing) were most frequently used, typically by individual project teams, and generated the most insight. Quantitative methods (failure mode and effects analysis, simulation modeling) typically were used by the SE and HF core team but generated variable insight. A decentralized project structure led to challenges using these SE and HF methods at the project and systems level. We offer recommendations and insights for using SE and HF to support digital health patient safety initiatives.

Predictive analytics in health care has generated increasing enthusiasm recently, as reflected in a rapidly growing body of predictive models reported in literature and in real-time embedded models using electronic health record data. However, estimating the benefit of applying any single model to a specific clinical problem remains challenging today. Developing a shared framework for estimating model value is therefore critical to facilitate the effective, safe, and sustainable use of predictive tools into the future. We highlight key concepts within the prediction-action dyad that together are expected to impact model benefit. These include factors relevant to model prediction (including the number needed to screen) as well as those relevant to the subsequent action (number needed to treat). In the simplest terms, a number needed to benefit contextualizes the numbers needed to screen and treat, offering an opportunity to estimate the value of a clinical predictive model in action.

As part of an interdisciplinary acute care patient portal task force with members from 10 academic medical centers and professional organizations, we held a national workshop with 71 attendees representing over 30 health systems, professional organizations, and technology companies. Our consensus approach identified 7 key sociotechnical and evaluation research focus areas related to the consumption and capture of information from patients, care partners (eg, family, friends), and clinicians through portals in the acute and post-acute care settings. The 7 research areas were: (1) standards, (2) privacy and security, (3) user-centered design, (4) implementation, (5) data and content, (6) clinical decision support, and (7) measurement. Patient portals are not yet in routine use in the acute and post-acute setting, and research focused on the identified domains should increase the likelihood that they will deliver benefit, especially as there are differences between needs in acute and post-acute care compared to the ambulatory setting.

Objective: To develop a comprehensive value set for documenting and encoding adverse reactions in the allergy module of an electronic health record. Materials and Methods: We analyzed 2 471 004 adverse reactions stored in Partners Healthcare's Enterprise-wide Allergy Repository (PEAR) of 2.7 million patients. Using the Medical Text Extraction, Reasoning, and Mapping System, we processed both structured and free-text reaction entries and mapped them to Systematized Nomenclature of Medicine - Clinical Terms. We calculated the frequencies of reaction concepts, including rare, severe, and hypersensitivity reactions. We compared PEAR concepts to a Federal Health Information Modeling and Standards value set and University of Nebraska Medical Center data, and then created an integrated value set. Results: We identified 787 reaction concepts in PEAR. Frequently reported reactions included: rash (14.0%), hives (8.2%), gastrointestinal irritation (5.5%), itching (3.2%), and anaphylaxis (2.5%). We identified an additional 320 concepts from Federal Health Information Modeling and Standards and the University of Nebraska Medical Center to resolve gaps due to missing and partial matches when comparing these external resources to PEAR. This yielded 1106 concepts in our final integrated value set. The presence of rare, severe, and hypersensitivity reactions was limited in both external datasets. Hypersensitivity reactions represented roughly 20% of the reactions within our data. Discussion: We developed a value set for encoding adverse reactions using a large dataset from one health system, enriched by reactions from 2 large external resources. This integrated value set includes clinically important severe and hypersensitivity reactions. Conclusion: This work contributes a value set, harmonized with existing data, to improve the consistency and accuracy of reaction documentation in electronic health records, providing the necessary building blocks for more intelligent clinical decision support for allergies and adverse reactions.

Objective: To define the types and numbers of inpatient clinical decision support alerts, measure the frequency with which they are overridden, and describe providers' reasons for overriding them and the appropriateness of those reasons. Materials and Methods: We conducted a cross-sectional study of medication-related clinical decision support alerts over a 3-year period at a 793-bed tertiary-care teaching institution. We measured the rate of alert overrides, the rate of overrides by alert type, the reasons cited for overrides, and the appropriateness of those reasons. Results: Overall, 73.3% of patient allergy, drug-drug interaction, and duplicate drug alerts were overridden, though the rate of overrides varied by alert type (P < .0001). About 60% of overrides were appropriate, and that proportion also varied by alert type (P < .0001). Few overrides of renal- (2.2%) or age-based (26.4%) medication substitutions were appropriate, while most duplicate drug (98%), patient allergy (96.5%), and formulary substitution (82.5%) alerts were appropriate. Discussion: Despite warnings of potential significant harm, certain categories of alert overrides were inappropriate >75% of the time. The vast majority of duplicate drug, patient allergy, and formulary substitution alerts were appropriate, suggesting that these categories of alerts might be good targets for refinement to reduce alert fatigue. Conclusion: Almost three-quarters of alerts were overridden, and 40% of the overrides were not appropriate. Future research should optimize alert types and frequencies to increase their clinical relevance, reducing alert fatigue so that important alerts are not inappropriately overridden.

Objective: To evaluate the impact of a referral manager tool on primary care practices. Materials and Methods: We evaluated a referral manager module in a locally developed electronic health record (EHR) that was enhanced to improve the referral management process in primary care practices. Baseline (n = 61) and follow-up (n = 35) provider and staff surveys focused on the ease of performing various steps in the referral process, confidence in completing those steps, and user satisfaction. Additional metrics were calculated that focused on completed specialist visits, acknowledged notes, and patient communication. Results: Of 1341 referrals that were initiated during the course of the study, 76.8% were completed. All the steps of the referral process were easier to accomplish following implementation of the enhanced referral manager module in the EHR. Specifically, tracking the status of an in-network referral became much easier (+1.43 [3.91-2.48] on a 5-point scale, P < .0001). Although we found improvement in the ease of performing out-of-network referrals, there was a greater impact on in-network referrals. Discussion: Implementation of an electronic tool developed using user-centered design principles along with adequate staff to monitor and intervene when necessary made it easier for primary care practices to track referrals and to identify if a breakdown in the process occurred. This is especially important for high-priority referrals. Out-of-network referrals continue to present challenges, which may eventually be helped by improving interoperability among EHRs and scheduling systems. Conclusion: An enhanced referral manager system can improve referral workflows, leading to enhanced efficiency and patient safety and reduced malpractice risk.

Objective: To estimate the national cost of ADEs resulting from inappropriate medication-related alert overrides in the U.S. inpatient setting. Materials and Methods: We used three different regression models (Basic, Model 1, Model 2) with model inputs taken from the medical literature. A random sample of 40 990 adult inpatients at the Brigham and Women's Hospital (BWH) in Boston with a total of 1 639 294 medication orders was taken. We extrapolated BWH medication orders using 2014 National Inpatient Sample (NIS) data. Results: Using three regression models, we estimated that 29.7 million adult inpatient discharges in 2014 resulted in between 1.02 billion and 1.07 billion medication orders, which in turn generated between 75.1 million and 78.8 million medication alerts, respectively. Taking the basic model (78.8 million), we estimated that 5.5 million medication-related alerts might have been inappropriately overridden, resulting in approximately 196 600 ADEs nationally. This was projected to cost between $871 million and $1.8 billion for treating preventable ADEs. We also estimated that clinicians and pharmacists would have jointly spent 175 000 hours responding to 78.8 million alerts with an opportunity cost of $16.9 million. Discussion and Conclusion: These data suggest that further optimization of hospitals computerized provider order entry systems and their associated clinical decision support is needed and would result in substantial savings. We have erred on the side of caution in developing this range, taking two conservative cost estimates for a preventable ADE that did not include malpractice or litigation costs, or costs of injuries to patients.

Objective: To identify and understand the factors that contribute to medication errors associated with the use of computerized provider order entry (CPOE) in pediatrics and provide recommendations on how CPOE systems could be improved. Materials and Methods: We conducted a systematic literature review across 3 large databases: the Cumulative Index to Nursing and Allied Health Literature, Embase, and Medline. Three independent reviewers screened the titles, and 2 authors then independently reviewed all abstracts and full texts, with 1 author acting as a constant across all publications. Data were extracted onto a customized data extraction sheet, and a narrative synthesis of all eligible studies was undertaken. Results: A total of 47 articles were included in this review. We identified 5 factors that contributed to errors with the use of a CPOE system: (1) lack of drug dosing alerts, which failed to detect calculation errors; (2) generation of inappropriate dosing alerts, such as warnings based on incorrect drug indications; (3) inappropriate drug duplication alerts, as a result of the system failing to consider factors such as the route of administration; (4) dropdown menu selection errors; and (5) system design issues, such as a lack of suitable dosing options for a particular drug. Discussion and Conclusions: This review highlights 5 key factors that contributed to the occurrence of CPOE-related medication errors in pediatrics. Dosing support is the most important. More advanced clinical decision support that can suggest doses based on the drug indication is needed.

Objective: To study the association of workarounds with medication administration errors using barcode-assisted medication administration (BCMA), and to determine the frequency and types of workarounds and medication administration errors. Materials and Methods: A prospective observational study in Dutch hospitals using BCMA to administer medication. Direct observation was used to collect data. Primary outcome measure was the proportion of medication administrations with one or more medication administration errors. Secondary outcome was the frequency and types of workarounds and medication administration errors. Univariate and multivariate multilevel logistic regression analysis were used to assess the association between workarounds and medication administration errors. Descriptive statistics were used for the secondary outcomes. Results: We included 5793 medication administrations for 1230 inpatients. Workarounds were associated with medication administration errors (adjusted odds ratio 3.06 [95% CI: 2.49-3.78]). Most commonly, procedural workarounds were observed, such as not scanning at all (36%), not scanning patients because they did not wear a wristband (28%), incorrect medication scanning, multiple medication scanning, and ignoring alert signals (11%). Common types of medication administration errors were omissions (78%), administration of non-ordered drugs (8.0%), and wrong doses given (6.0%). Discussion: Workarounds are associated with medication administration errors in hospitals using BCMA. These data suggest that BCMA needs more post-implementation evaluation if it is to achieve the intended benefits for medication safety. Conclusion: In hospitals using barcode-assisted medication administration, workarounds occurred in 66% of medication administrations and were associated with large numbers of medication administration errors.

Objective: To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods: We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results: We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion: Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion: CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

Objective: To examine medication errors potentially related to computerized prescriber order entry (CPOE) and refine a previously published taxonomy to classify them. Materials and Methods: We reviewed all patient safety medication reports that occurred in the medication ordering phase from 6 sites participating in a United States Food and Drug Administration-sponsored project examining CPOE safety. Two pharmacists independently reviewed each report to confirm whether the error occurred in the ordering/prescribing phase and was related to CPOE. For those related to CPOE, we assessed whether CPOE facilitated (actively contributed to) the error or failed to prevent the error (did not directly cause it, but optimal systems could have potentially prevented it). A previously developed taxonomy was iteratively refined to classify the reports. Results: Of 2522 medication error reports, 1308 (51.9%) were related to CPOE. Of these, CPOE facilitated the error in 171 (13.1%) and potentially could have prevented the error in 1137 (86.9%). The most frequent categories of "what happened to the patient" were delays in medication reaching the patient, potentially receiving duplicate drugs, or receiving a higher dose than indicated. The most frequent categories for "what happened in CPOE" included orders not routed to or received at the intended location, wrong dose ordered, and duplicate orders. Variations were seen in the format, categorization, and quality of reports, resulting in error causation being assignable in only 403 instances (31%). Discussion and Conclusion: Errors related to CPOE commonly involved transmission errors, erroneous dosing, and duplicate orders. More standardized safety reporting using a common taxonomy could help health care systems and vendors learn and implement prevention strategies.

OBJECTIVE: To understand the different types and causes of prescribing errors associated with computerized provider order entry (CPOE) systems, and recommend improvements in these systems. MATERIALS AND METHODS: We conducted a systematic review of the literature published between January 2004 and June 2015 using three large databases: the Cumulative Index to Nursing and Allied Health Literature, Embase, and Medline. Studies that reported qualitative data about the types and causes of these errors were included. A narrative synthesis of all eligible studies was undertaken. RESULTS: A total of 1185 publications were identified, of which 34 were included in the review. We identified 8 key themes associated with CPOE-related prescribing errors: computer screen display, drop-down menus and auto-population, wording, default settings, nonintuitive or inflexible ordering, repeat prescriptions and automated processes, users' work processes, and clinical decision support systems. Displaying an incomplete list of a patient's medications on the computer screen often contributed to prescribing errors. Lack of system flexibility resulted in users employing error-prone workarounds, such as the addition of contradictory free-text comments. Users' misinterpretations of how text was presented in CPOE systems were also linked with the occurrence of prescribing errors. DISCUSSION AND CONCLUSIONS: Human factors design is important to reduce error rates. Drop-down menus should be designed with safeguards to decrease the likelihood of selection errors. Development of more sophisticated clinical decision support, which can perform checks on free-text, may also prevent errors. Further research is needed to ensure that systems minimize error likelihood and meet users' workflow expectations.

Implementation and adoption of complex health information technology (HIT) is gaining momentum internationally. This is underpinned by the drive to improve the safety, quality, and efficiency of care. Although most of the benefits associated with HIT will only be realized through optimization of these systems, relatively few health care organizations currently have the expertise or experience needed to undertake this. It is extremely important to have systems working before embarking on HIT optimization, which, much like implementation, is an ongoing, difficult, and often expensive process. We discuss some key organization-level activities that are important in optimizing large-scale HIT systems. These include considerations relating to leadership, strategy, vision, and continuous cycles of improvement. Although these alone are not sufficient to fully optimize complex HIT, they provide a starting point for conceptualizing this important area.

Communication in acute care settings is fragmented and occurs asynchronously via a variety of electronic modalities. Providers are often not on the same page with regard to the plan of care. We designed and developed a secure, patient-centered "microblog" messaging platform that identifies care team members by synchronizing with the electronic health record, and directs providers to a single forum where they can communicate about the plan of care. The system was used for 35% of patients admitted to a medical intensive care unit over a 6-month period. Major themes in messages included care coordination (49%), clinical summarization (29%), and care team collaboration (27%). Message transparency and persistence were seen as useful features by 83% and 62% of respondents, respectively. Availability of alternative messaging tools and variable use by non-unit providers were seen as main barriers to adoption by 83% and 62% of respondents, respectively. This approach has much potential to improve communication across settings once barriers are addressed.

Background: The Centers for Medicare and Medicaid Services (CMS) canceled Meaningful Use (MU), replacing it with Advancing Care Information, which preserves many MU elements. Therefore, transitioning from MU stage 1 to MU stage 2 has important implications for the new policy, yet the quality of care provided by physicians transitioning from MU1 to MU2 is unknown. Methods: Retrospective longitudinal evaluation of the quality of care delivered by outpatient physicians at an academic medical center in the transition between MU1 and MU2. Results: Between MU1 and MU2, 4 measures improved: hypertension control (35% vs 40%), influenza immunization (63% vs 68%), tobacco use assessment/counseling (86% vs 96%), and diabetes control (93% vs 96%; P all <.01). One worsened: senior weight screening/follow-up (54% vs 49%; P < .01). Two were unchanged: chlamydia screening and adult weight screening/follow-up. Conclusion: In this single-site study, when clinicians progressed from MU1 to MU2, 4 quality measures improved, 2 were unchanged, and 1 worsened. Analysis of national data should guide policy decisions about the content of MU's successor.

BACKGROUND AND OBJECTIVE: Clinical decision support tools for risk prediction are readily available, but typically require workflow interruptions and manual data entry so are rarely used. Due to new data interoperability standards for electronic health records (EHRs), other options are available. As a clinical case study, we sought to build a scalable, web-based system that would automate calculation of kidney failure risk and display clinical decision support to users in primary care practices. MATERIALS AND METHODS: We developed a single-page application, web server, database, and application programming interface to calculate and display kidney failure risk. Data were extracted from the EHR using the Consolidated Clinical Document Architecture interoperability standard for Continuity of Care Documents (CCDs). EHR users were presented with a noninterruptive alert on the patient's summary screen and a hyperlink to details and recommendations provided through a web application. Clinic schedules and CCDs were retrieved using existing application programming interfaces to the EHR, and we provided a clinical decision support hyperlink to the EHR as a service. RESULTS: We debugged a series of terminology and technical issues. The application was validated with data from 255 patients and subsequently deployed to 10 primary care clinics where, over the course of 1 year, 569 533 CCD documents were processed. CONCLUSIONS: We validated the use of interoperable documents and open-source components to develop a low-cost tool for automated clinical decision support. Since Consolidated Clinical Document Architecture-based data extraction extends to any certified EHR, this demonstrates a successful modular approach to clinical decision support.

Objective: The study objective was to evaluate the accuracy, validity, and clinical usefulness of medication error alerts generated by an alerting system using outlier detection screening. Materials and Methods: Five years of clinical data were extracted from an electronic health record system for 747 985 patients who had at least one visit during 2012-2013 at practices affiliated with 2 academic medical centers. Data were screened using the system to detect outliers suggestive of potential medication errors. A sample of 300 charts was selected for review from the 15 693 alerts generated. A coding system was developed and codes assigned based on chart review to reflect the accuracy, validity, and clinical value of the alerts. Results: Three-quarters of the chart-reviewed alerts generated by the screening system were found to be valid in which potential medication errors were identified. Of these valid alerts, the majority (75.0%) were found to be clinically useful in flagging potential medication errors or issues. Discussion: A clinical decision support (CDS) system that used a probabilistic, machine-learning approach based on statistically derived outliers to detect medication errors generated potentially useful alerts with a modest rate of false positives. The performance of such a surveillance and alerting system is critically dependent on the quality and completeness of the underlying data. Conclusion: The screening system was able to generate alerts that might otherwise be missed with existing CDS systems and did so with a reasonably high degree of alert usefulness when subjected to review of patients' clinical contexts and details.

BACKGROUND: Practice guidelines recommend anticoagulation therapy for patients with atrial fibrillation (AF) who have other risk factors putting them at an elevated risk of stroke. These patients remain undertreated, but, with increasing use of electronic healthcare records (EHRs), it may be possible to identify candidates for treatment. OBJECTIVE: To test algorithms for identifying AF patients who also have known risk factors for stroke and major bleeding using EHR data. MATERIALS AND METHODS: We evaluated the performance of algorithms using EHR data from the Partners Healthcare System at identifying AF patients and 16 additional conditions that are risk factors in the CHA 2 DS 2 -VASc and HAS-BLED risk scores for stroke and major bleeding. Algorithms were based on information contained in problem lists, billing codes, laboratory data, prescription data, vital status, and clinical notes. The performance of candidate algorithms in 1000 bootstrap resamples was compared to a gold standard of manual chart review by experienced resident physicians. RESULTS: : Physicians reviewed 480 patient charts. For 11 conditions, the median positive predictive value (PPV) of the EHR-derived algorithms was greater than 0.90. Although the PPV for some risk factors was poor, the median PPV for identifying patients with a CHA 2 DS 2 -VASc score ≥2 or a HAS-BLED score ≥3 was 1.00 and 0.92, respectively. DISCUSSION: We developed and tested a set of algorithms to identify AF patients and known risk factors for stroke and major bleeding using EHR data. Algorithms such as these can be built into EHR systems to facilitate informed decision making and help shift population health management efforts towards patients with the greatest need.

Objective: Health Information Exchange (HIE) enables providers to share healthcare information electronically across different organizations to promote safer, more efficient, and less costly patient-centered care. This paper describes the development and implementation of a city-wide HIE system in Shanghai, China. Methods: In 2006, as a product of the Chinese healthcare reform, the Health Information Exchange and Sharing Platform was proposed as a means to facilitate HIE within Shanghai. In collaboration with the Shanghai Hospital Development Center, a state-run nonprofit corporate, the HIE project was implemented across multiple levels within the city. The HIE system is based on the Service-oriented Architecture and complies with industry standards. Results: On September 2010, the first and largest Chinese HIE system was established. As of 2016 the system includes all of Shanghai's 38 tertiary hospitals (highest level hospitals in China), plus 6 district hospitals, and 40 community health centers, with coverage for 39 million patients. The system currently provides a rich source of patient information including encounter history, medication history, laboratory results, radiology images and reports, and clinical notes. Initial outcomes indicate a significant reduction in medication errors and duplication of tests, saving at least 48 million RMB a year, with overall improvement in the quality of care following implementation. Conclusion: The adoption of HIE in Shanghai resulted in improved access to accurate, complete, and relevant clinical information in real time, thus facilitating delivery of high-quality, cost-effective, and efficient care.

OBJECTIVE: Variation in the use of tests and treatments has been demonstrated to be substantial between providers and geographic regions. This study assessed variation between outpatient providers in overriding electronic prescribing warnings. METHODS: Responses to warnings were prospectively logged. Random effects models were used to calculate provider-to-provider variation in the rates for the decisions to override warnings in 6 different clinical domains: medication allergies, drug-drug interactions, duplicate drugs, renal recommendations, age-based recommendations, and formulary substitutions. RESULTS: A total of 157 482 responses were logged. Differences between 1717 providers accounted for 11% of the overall variability in override rates, so that while the average override rate was 45.2%, individual provider rates had a wide range with a 95% confidence interval (CI) (13.7%-76.7% ). The highest variations between providers were observed in the categories age-based (25.4% of total variability; average override rate 70.2% [95% CI, 29.1%-100% ]) and renal recommendations (24.2%; average 70% [95% CI, 29.5%-100% ]), and provider responses within these 2 categories were most often clinically inappropriate according to prior work. Among providers who received at least 10 age-based recommendations, 64 of 238 (27%) overrode ≥ 90% of the warnings and 13 of 238 (5%) overrode all of them. Of those who received at least 10 renal recommendations, 36 of 92 (39%) overrode ≥ 90% of the alerts and 9 of 92 (10%) overrode all of them. CONCLUSIONS: The decision to override prescribing warnings shows variation between providers, and the magnitude of variation differs among the clinical domains of the warnings; more variation was observed in areas with more inappropriate overrides.

We implemented a web-based, patient-centered toolkit that engages patients/caregivers in the hospital plan of care by facilitating education and patient-provider communication. Of the 585 eligible patients approached on medical intensive care and oncology units, 239 were enrolled (119 patients, 120 caregivers). The most common reason for not approaching the patient was our inability to identify a health care proxy when a patient was incapacitated. Significantly more caregivers were enrolled in medical intensive care units compared with oncology units (75% vs 32%; P < .01). Of the 239 patient/caregivers, 158 (66%) and 97 (41%) inputted a daily and overall goal, respectively. Use of educational content was highest for medications and test results and infrequent for problems. The most common clinical theme identified in 291 messages sent by 158 patients/caregivers was health concerns, needs, preferences, or questions (19%, 55 of 291). The average system usability scores and satisfaction ratings of a sample of surveyed enrollees were favorable. From analysis of feedback, we identified barriers to adoption and outlined strategies to promote use.

BACKGROUND: Experts suggest that formulary alerts at the time of medication order entry are the most effective form of clinical decision support to automate formulary management. OBJECTIVE: Our objectives were to quantify the frequency of inappropriate nonformulary medication (NFM) alert overrides in the inpatient setting and provide insight on how the design of formulary alerts could be improved. METHODS: Alert overrides of the top 11 (n = 206) most-utilized and highest-costing NFMs, from January 1 to December 31, 2012, were randomly selected for appropriateness evaluation. Using an empirically developed appropriateness algorithm, appropriateness of NFM alert overrides was assessed by 2 pharmacists via chart review. Appropriateness agreement of overrides was assessed with a Cohen's kappa. We also assessed which types of NFMs were most likely to be inappropriately overridden, the override reasons that were disproportionately provided in the inappropriate overrides, and the specific reasons the overrides were considered inappropriate. RESULTS: Approximately 17.2% (n = 35.4/206) of NFM alerts were inappropriately overridden. Non-oral NFM alerts were more likely to be inappropriately overridden compared to orals. Alerts overridden with "blank" reasons were more likely to be inappropriate. The failure to first try a formulary alternative was the most common reason for alerts being overridden inappropriately. CONCLUSION: Approximately 1 in 5 NFM alert overrides are overridden inappropriately. Future research should evaluate the impact of mandating a valid override reason and adding a list of formulary alternatives to each NFM alert; we speculate these NFM alert features may decrease the frequency of inappropriate overrides.

OBJECTIVE: To understand the evolving market of commercial off-the-shelf Computerized Physician Order Entry (CPOE) and Computerized Decision Support (CDS) applications and its effects on their uptake and implementation in English hospitals. METHODS: Although CPOE and CDS vendors have been quick to enter the English market, uptake has been slow and uneven. To investigate this, the authors undertook qualitative ethnography of vendors and adopters of hospital CPOE/CDS systems in England. The authors collected data from semi-structured interviews with 11 individuals from 4 vendors, including the 2 most entrenched suppliers, and 6 adopter hospitals, and 21 h of ethnographic observation of 2 user groups, and 1 vendor event. The research and analysis was informed by insights from studies of the evolution of technology fields and the emergence of generic COTS enterprise solutions. RESULTS: Four key themes emerged: (1) adoption of systems that had been developed outside of England, (2) vendors' configuration and customization strategies, (3) localized adopter practices vs generic systems, and (4) unrealistic adopter demands. Evidence for our over-arching finding concerning the current immaturity of the market was derived from vendors' strategies, adopters' reactions to the technology, and policy makers' incomplete insights. CONCLUSIONS: The CPOE/CDS market in England is still in an emergent phase. The rapid entrance of diverse products, triggered by federal policy initiatives, has resulted in premature adoption of systems that do not yet adequately meet the needs of hospitals. Vendors and adopters lacked understanding of how to design and implement generic solutions to meet diverse user needs.

OBJECTIVE: Accurate food adverse sensitivity documentation in electronic health records (EHRs) is crucial to patient safety. This study examined, encoded, and grouped foods that caused any adverse sensitivity in a large allergy repository using natural language processing and standard terminologies. METHODS: Using the Medical Text Extraction, Reasoning, and Mapping System (MTERMS), we processed both structured and free-text entries stored in an enterprise-wide allergy repository (Partners' Enterprise-wide Allergy Repository), normalized diverse food allergen terms into concepts, and encoded these concepts using the Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) and Unique Ingredient Identifiers (UNII) terminologies. Concept coverage also was assessed for these two terminologies. We further categorized allergen concepts into groups and calculated the frequencies of these concepts by group. Finally, we conducted an external validation of MTERMS's performance when identifying food allergen terms, using a randomized sample from a different institution. RESULTS: We identified 158 552 food allergen records (2140 unique terms) in the Partners repository, corresponding to 672 food allergen concepts. High-frequency groups included shellfish (19.3%), fruits or vegetables (18.4%), dairy (9.0%), peanuts (8.5%), tree nuts (8.5%), eggs (6.0%), grains (5.1%), and additives (4.7%). Ambiguous, generic concepts such as "nuts" and "seafood" accounted for 8.8% of the records. SNOMED-CT covered more concepts than UNII in terms of exact (81.7% vs 68.0%) and partial (14.3% vs 9.7%) matches. DISCUSSION: Adverse sensitivities to food are diverse, and existing standard terminologies have gaps in their coverage of the breadth of allergy concepts. CONCLUSION: New strategies are needed to represent and standardize food adverse sensitivity concepts, to improve documentation in EHRs.

OBJECTIVE: To test the vulnerabilities of a wide range of computerized physician order entry (CPOE) systems to different types of medication errors, and develop a more comprehensive qualitative understanding of how their design could be improved. MATERIALS AND METHODS: The authors reviewed a random sample of 63,040 medication error reports from the US Pharmacopeia (USP) MEDMARX reporting system where CPOE systems were considered a "contributing factor" to errors and flagged test scenarios that could be tested in current CPOE systems. Testers entered these orders in 13 commercial and homegrown CPOE systems across 16 different sites in the United States and Canada, using both usual practice and where-needed workarounds. Overarching themes relevant to interface design and usability/workflow issues were identified. RESULTS: CPOE systems often failed to detect and prevent important medication errors. Generation of electronic alert warnings varied widely between systems, and depended on a number of factors, including how the order information was entered. Alerts were often confusing, with unrelated warnings appearing on the same screen as those more relevant to the current erroneous entry. Dangerous drug-drug interaction warnings were displayed only after the order was placed rather than at the time of ordering. Testers illustrated various workarounds that allowed them to enter these erroneous orders. DISCUSSION AND CONCLUSION: The authors found high variability in ordering approaches between different CPOE systems, with major deficiencies identified in some systems. It is important that developers reflect on these findings and build in safeguards to ensure safer prescribing for patients.

OBJECTIVE: There have been growing concerns about the impact of drug allergy alerts on patient safety and provider alert fatigue. The authors aimed to explore the common drug allergy alerts over the last 10 years and the reasons why providers tend to override these alerts. DESIGN: Retrospective observational cross-sectional study (2004-2013). MATERIALS AND METHODS: Drug allergy alert data (n = 611,192) were collected from two large academic hospitals in Boston, MA (USA). RESULTS: Overall, the authors found an increase in the rate of drug allergy alert overrides, from 83.3% in 2004 to 87.6% in 2013 (P < .001). Alarmingly, alerts for immune mediated and life threatening reactions with definite allergen and prescribed medication matches were overridden 72.8% and 74.1% of the time, respectively. However, providers were less likely to override these alerts compared to possible (cross-sensitivity) or probable (allergen group) matches (P < .001). The most common drug allergy alerts were triggered by allergies to narcotics (48%) and other analgesics (6%), antibiotics (10%), and statins (2%). Only slightly more than one-third of the reactions (34.2%) were potentially immune mediated. Finally, more than half of the overrides reasons pointed to irrelevant alerts (i.e., patient has tolerated the medication before, 50.9%) and providers were significantly more likely to override repeated alerts (89.7%) rather than first time alerts (77.4%, P < .001). DISCUSSION AND CONCLUSIONS: These findings underline the urgent need for more efforts to provide more accurate and relevant drug allergy alerts to help reduce alert override rates and improve alert fatigue.

OBJECTIVE: To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. MATERIALS AND METHODS: We identified and investigated several CDSS malfunctions at Brigham and Women's Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. RESULTS: We identified four CDSS malfunctions at Brigham and Women's Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually. DISCUSSION: CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate. CONCLUSION: CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

OBJECTIVE: To investigate experiences with leveraging health information technology (HIT) to improve patient care and population health, and reduce healthcare expenditures. MATERIALS AND METHODS: In-depth qualitative interviews with federal government employees, health policy, HIT and medico-legal experts, health providers, physicians, purchasers, payers, patient advocates, and vendors from across the United States. RESULTS: The authors undertook 47 interviews. There was a widely shared belief that Health Information Technology for Economic and Clinical Health (HITECH) had catalyzed the creation of a digital infrastructure, which was being used in innovative ways to improve quality of care and curtail costs. There were however major concerns about the poor usability of electronic health records (EHRs), their limited ability to support multi-disciplinary care, and major difficulties with health information exchange, which undermined efforts to deliver integrated patient-centered care. Proposed strategies for enhancing the benefits of HIT included federal stimulation of competition by mandating vendors to open-up their application program interfaces, incenting development of low-cost consumer informatics tools, and promoting Congressional review of the The Health Insurance Portability and Accountability Act (HIPPA) to optimize the balance between data privacy and reuse. Many underscored the need to "kick the legs from underneath the fee-for-service model" and replace it with a data-driven reimbursement system that rewards high quality care. CONCLUSIONS: The HITECH Act has stimulated unprecedented, multi-stakeholder interest in HIT. Early experiences indicate that the resulting digital infrastructure is being used to improve quality of care and curtail costs. Reform efforts are however severely limited by problems with usability, limited interoperability and the persistence of the fee-for-service paradigm-addressing these issues therefore needs to be the federal government's main policy target.

OBJECTIVE: To investigate organizational strategies to promote personal health records (PHRs) adoption with a focus on patients with chronic disease. METHODS: Using semi-structured interviews and a web-based survey, we sampled US health delivery organizations which had implemented PHRs for at least 12 months, were recognized as PHR innovators, and had scored highly in national patient satisfaction surveys. Respondents had lead positions for clinical information systems or high-risk population management. Using grounded theory approach, thematic categories were derived from interviews and coupled with data from the survey. RESULTS: Interviews were conducted with 30 informants from 16 identified organizations. Organizational strategies were directed towards raising patient awareness via multimedia communications, and provider acceptance and uptake. Strategies for providers were grouped into six main themes: organizational vision, governance and policies, work process redesign, staff training, information technology (IT) support, and monitoring and incentives. Successful organizations actively communicated their vision, engaged leaders at all levels, had clear governance, planning, and protocols, set targets, and celebrated achievement. The most effective strategy for patient uptake was through health professional encouragement. No specific outreach efforts targeted patients with chronic disease. Registration and PHR activity was routinely measured but without reference to a denominator population or high risk subpopulations. DISCUSSION AND CONCLUSION: Successful PHR implementation represents a social change and operational project catalyzed by a technical solution. The key to clinician acceptance is making their work easier. However, organizations will likely not achieve the value they want from PHRs unless they target specific populations and monitor their uptake.

BACKGROUND AND OBJECTIVE: Physician awareness of the results of tests pending at discharge (TPADs) is poor. We developed an automated system that notifies responsible physicians of TPAD results via secure, network email. We sought to evaluate the impact of this system on self-reported awareness of TPAD results by responsible physicians, a necessary intermediary step to improve management of TPAD results. METHODS: We conducted a cluster-randomized controlled trial at a major hospital affiliated with an integrated healthcare delivery network in Boston, Massachusetts. Adult patients with TPADs who were discharged from inpatient general medicine and cardiology services were assigned to the intervention or usual care arm if their inpatient attending physician and primary care physician (PCP) were both randomized to the same study arm. Patients of physicians randomized to discordant study arms were excluded. We surveyed these physicians 72 h after all TPAD results were finalized. The primary outcome was awareness of TPAD results by attending physicians. Secondary outcomes included awareness of TPAD results by PCPs, awareness of actionable TPAD results, and provider satisfaction. RESULTS: We analyzed data on 441 patients. We sent 441 surveys to attending physicians and 353 surveys to PCPs and received 275 and 152 responses from 83 different attending physicians and 112 different PCPs, respectively (attending physician survey response rate of 63%). Intervention attending physicians and PCPs were significantly more aware of TPAD results (76% vs 38%, adjusted/clustered OR 6.30 (95% CI 3.02 to 13.16), p<0.001; 57% vs 33%, adjusted/clustered OR 3.08 (95% CI 1.43 to 6.66), p=0.004, respectively). Intervention attending physicians tended to be more aware of actionable TPAD results (59% vs 29%, adjusted/clustered OR 4.25 (0.65, 27.85), p=0.13). One hundred and eighteen (85%) and 43 (63%) intervention attending physician and PCP survey respondents, respectively, were satisfied with this intervention. CONCLUSIONS: Automated email notification represents a promising strategy for managing TPAD results, potentially mitigating an unresolved patient safety concern. CLINICAL TRIAL REGISTRATION: ClinicalTrials.gov (NCT01153451).

OBJECTIVE: As healthcare systems and providers move toward meaningful use of electronic health records, longitudinal care plans (LCPs) may provide a means to improve communication and coordination as patients transition across settings. The objective of this study was to determine the current state of communication of LCPs across settings and levels of care. MATERIALS AND METHODS: We conducted surveys and interviews with professionals from emergency departments, acute care hospitals, skilled nursing facilities, and home health agency settings in six regions in the USA. We coded the transcripts according to the Agency for Healthcare Research and Quality (AHRQ) 'Broad Approaches' to care coordination to understand the degree to which current practice meets the definition of an LCP. RESULTS: Participants (n=22) from all settings reported that LCPs do not exist in their current state. We found LCPs in practice, and none of these were shared or reconciled across settings. Moreover, we found wide variation in the types and formats of care plan information that was communicated as patients transitioned. The most common formats, even when care plan information was communicated within the same healthcare system, were paper and fax. DISCUSSION: These findings have implications for data reuse, interoperability, and achieving widespread adoption of LCPs. CONCLUSIONS: The use of LCPs to support care transitions is suboptimal. Strategies are needed to transform the LCP from vision to reality.

BACKGROUND: Electronic prescribing is increasingly used, in part because of government incentives for its use. Many of its benefits come from clinical decision support (CDS), but often too many alerts are displayed, resulting in alert fatigue. OBJECTIVE: To characterize the override rates for medication-related CDS alerts in the outpatient setting, the reasons cited for overrides at the time of prescribing, and the appropriateness of overrides. METHODS: We measured CDS alert override rates and the coded reasons for overrides cited by providers at the time of prescribing. Our primary outcome was the rate of CDS alert overrides; our secondary outcomes were the rate of overrides by alert type, reasons cited for overrides at the time of prescribing, and override appropriateness for a subset of 600 alert overrides. Through detailed chart reviews of alert override cases, and selective literature review, we developed appropriateness criteria for each alert type, which were modified iteratively as necessary until consensus was reached on all criteria. RESULTS: We reviewed 157,483 CDS alerts (7.9% alert rate) on 2,004,069 medication orders during the study period. 82,889 (52.6%) of alerts were overridden. The most common alerts were duplicate drug (33.1%), patient allergy (16.8%), and drug-drug interactions (15.8%). The most likely alerts to be overridden were formulary substitutions (85.0%), age-based recommendations (79.0%), renal recommendations (78.0%), and patient allergies (77.4%). An average of 53% of overrides were classified as appropriate, and rates of appropriateness varied by alert type (p<0.0001) from 12% for renal recommendations to 92% for patient allergies. DISCUSSION: About half of CDS alerts were overridden by providers and about half of the overrides were classified as appropriate, but the likelihood of overriding an alert varied widely by alert type. Refinement of these alerts has the potential to improve the relevance of alerts and reduce alert fatigue.

Despite the potential for electronic health records to help providers coordinate care, the current marketplace has failed to provide adequate solutions. Using a simple framework, we describe a vision of information technology capabilities that could substantially improve four care coordination activities: identifying collaborators, contacting collaborators, collaborating, and monitoring. Collaborators can include any individual clinician, caregiver, or provider organization involved in care for a given patient. This vision can be used to guide the development of care coordination tools and help policymakers track and promote their adoption.

At the 2011 American College of Medical Informatics (ACMI) Winter Symposium we studied the overlap between health IT and economics and what leading healthcare delivery organizations are achieving today using IT that might offer paths for the nation to follow for using health IT in healthcare reform. We recognized that health IT by itself can improve health value, but its main contribution to health value may be that it can make possible new care delivery models to achieve much larger value. Health IT is a critically important enabler to fundamental healthcare system changes that may be a way out of our current, severe problem of rising costs and national deficit. We review the current state of healthcare costs, federal health IT stimulus programs, and experiences of several leading organizations, and offer a model for how health IT fits into our health economic future.

OBJECTIVE: Alert fatigue represents a common problem associated with the use of clinical decision support systems in electronic health records (EHR). This problem is particularly profound with drug-drug interaction (DDI) alerts for which studies have reported override rates of approximately 90%. The objective of this study is to report consensus-based recommendations of an expert panel on DDI that can be safely made non-interruptive to the provider's workflow, in EHR, in an attempt to reduce alert fatigue. METHODS: We utilized an expert panel process to rate the interactions. Panelists had expertise in medicine, pharmacy, pharmacology and clinical informatics, and represented both academic institutions and vendors of medication knowledge bases and EHR. In addition, representatives from the US Food and Drug Administration and the American Society of Health-System Pharmacy contributed to the discussions. RESULTS: Recommendations and considerations of the panel resulted in the creation of a list of 33 class-based low-priority DDI that do not warrant being interruptive alerts in EHR. In one institution, these accounted for 36% of the interactions displayed. DISCUSSION: Development and customization of the content of medication knowledge bases that drive DDI alerting represents a resource-intensive task. Creation of a standardized list of low-priority DDI may help reduce alert fatigue across EHR. CONCLUSIONS: Future efforts might include the development of a consortium to maintain this list over time. Such a list could also be used in conjunction with financial incentives tied to its adoption in EHR.

OBJECTIVE: Little is known about the frequency and types of prescribing errors in the ambulatory setting among community-based, primary care providers. Therefore, the rates and types of prescribing errors were assessed among community-based, primary care providers in two states. MATERIAL AND METHODS: A non-randomized cross-sectional study was conducted of 48 providers in New York and 30 providers in Massachusetts, all of whom used paper prescriptions, from September 2005 to November 2006. Using standardized methodology, prescriptions and medical records were reviewed to identify errors. RESULTS: 9385 prescriptions were analyzed from 5955 patients. The overall prescribing error rate, excluding illegibility errors, was 36.7 per 100 prescriptions (95% CI 30.7 to 44.0) and did not vary significantly between providers from each state (p=0.39). One or more non-illegibility errors were found in 28% of prescriptions. Rates of illegibility errors were very high (175.0 per 100 prescriptions, 95% CI 169.1 to 181.3). Inappropriate abbreviation and direction errors also occurred frequently (13.4 and 4.2 errors per 100 prescriptions, respectively). Reviewers determined that the vast majority of errors could have been eliminated through the use of e-prescribing with clinical decision support. DISCUSSION: Prescribing errors appear to occur at very high rates among community-based primary care providers, especially when compared with studies of academic-affiliated providers that have found nearly threefold lower error rates. Illegibility errors are particularly problematical. CONCLUSIONS: Further characterizing prescribing errors of community-based providers may inform strategies to improve ambulatory medication safety, especially e-prescribing. TRIAL REGISTRATION NUMBER: http://www.clinicaltrials.gov, NCT00225576.

OBJECTIVE: Computerized provider order entry (CPOE) with clinical decision support (CDS) can help hospitals improve care. Little is known about what CDS is presently in use and how it is managed, however, especially in community hospitals. This study sought to address this knowledge gap by identifying standard practices related to CDS in US community hospitals with mature CPOE systems. MATERIALS AND METHODS: Representatives of 34 community hospitals, each of which had over 5 years experience with CPOE, were interviewed to identify standard practices related to CDS. Data were analyzed with a mix of descriptive statistics and qualitative approaches to the identification of patterns, themes and trends. RESULTS: This broad sample of community hospitals had robust levels of CDS despite their small size and the independent nature of many of their physician staff members. The hospitals uniformly used medication alerts and order sets, had sophisticated governance procedures for CDS, and employed staff to customize CDS. DISCUSSION: The level of customization needed for most CDS before implementation was greater than expected. Customization requires skilled individuals who represent an emerging manpower need at this type of hospital. CONCLUSION: These results bode well for robust diffusion of CDS to similar hospitals in the process of adopting CDS and suggest that national policies to promote CDS use may be successful.

The last few years have clearly been the most exciting ever for health information technology (HIT) policy. The nation has made a huge investment in HIT through the Recovery Act of 2009 and its HITECH provisions, on the premise that electronic health records and widespread information exchange can improve the quality, safety, and efficiency of our healthcare system and transform the care delivery experience for providers, patients, and families—all while helping to improve population health and health data systems. But implementation of such an ambitious program brings many challenges. We think that the next few years will be even more important for AMIA and other HIT stakeholders as we realistically face uncertainty about returns on the national investment. Our goals in writing this column are to describe the role of the AMIA and its Public Policy Committee (PPC), to highlight some accomplishments of past years, and to discuss some of the new activities going forward. Through many educational and advocacy efforts, AMIA and its members play a significant role in helping shape HIT policy, and AMIA serves as an independent voice around issues relating to HIT and informatics more broadly. AMIA's influence is enhanced by the diversity of its members' expertise, which includes clinical, public health, and consumer informatics, research, education, health administration, and computer sciences, among many others. Members in academic, government, corporate, and community settings all benefit from AMIA's extensive educational and information-sharing activities at the Annual Symposium, in e-News, on the web site, and in other forums. AMIA members serve on a host of influential policy-relevant committees, including the Health Information Technology Policy Committee, the Health Information Standards Committee, the National Committee on Vital and Health Statistics, and various Institute of Medicine and National Quality Forum committees. In particular, the Policy and Standards committees—which are federal advisory committees—have provided direct input to the Office of the National Coordinator around policy issues based on a body of evidence developed by AMIA members and others over many years. Testimony provided to Congress and the Policy and Standards Committees by AMIA members also has helped to inform policy-makers about issues of interest to the broader HIT community. The key role of the AMIA PPC is to make recommendations to the AMIA Board of Directors regarding AMIA positions on public policy issues at a national level. This is carried out in a variety of ways but primarily focuses on tracking agency and legislative activities, developing policy statements for Board approval, holding discussions at AMIA health policy meetings, and developing a variety of white papers and position statements to help educate the AMIA membership, other stakeholders, and the media about key policy issues. For example, in carrying out their responsibilities to implement provisions of the Recovery Act and the Affordable Care Act (healthcare reform), many federal agencies have undertaken activities related to HIT. These include the Office of the National Coordinator, the Centers for Medicare and Medicaid Services, the Food and Drug Administration, the National Institute for Standards and Technology, and the President's Council on Science and Technology, to name just a few. The new legislation has resulted in a deluge of requests for commentary and testimony on an array of draft guidance documents and proposed regulations, with some of the most notable being the meaningful use criteria, privacy and security, mobile applications, clinical decision support, metadata, and The Common Rule. In 2011, AMIA's PPC members collaborated with the Board, AMIA members, and AMIA staff to produce 17 public comments on proposed regulations, which has played an important role in improving them. AMIA's PPC also has a number of activities directed at developing broader policy statements. These include published summaries of the annual policy invitational meeting on special topics such as secondary use of data, unintended consequences, innovation, and clinical documentation.1–5 The educational activities of the PPC also include policy forums at AMIA meetings, online tutorials through the AMIA web site, and mentoring for aspiring policy leaders. In addition, we have created a series of issue briefs on key topics, and have generated a number of white papers intended for general audiences and the media. Some have been published in the Association's journal (J Am Med Inform Assoc) and others are available on the AMIA web site (http://www.amia.org/public-policy/positions-and-statements). Although the Board oversees advocacy initiatives in support of AMIA positions, all of AMIA's policy activities involve close collaboration with Washington Health Strategies Group, whose tracking of federal activities, Congressional hearings, and legislation in areas of interest to AMIA members and the informatics community provides AMIA with real-time information on national priorities. Washington Health Strategies Group works with AMIA staff and also provides regular policy updates for corporate members and hosts the annual Hill Day, in which members of the PPC and other AMIA members meet with Congressional members and staff to discuss current policy issues, such as the implementation challenges involved with HITECH. While AMIA's main focus has been on federal activities, the PPC recently spun off a Regional Informatics Action Group to assist those working on policy issues in states. The state-level implementation of regional extension centers and health information exchanges under HITECH, and health insurance exchanges under healthcare reform, could yield a vibrant ecosystem for innovative partnerships and information-sharing in the future. HITECH laid the groundwork for transformational change in the way healthcare will be delivered and experienced by clinicians and patients, and for expanding an information infrastructure through which clinical data can be reused for research, planning, and population health. But with implementation on such a large scale, there is a risk of perpetuating the same fragmentation that has led to medical errors and disparities in quality and care integration. We need to carefully study and learn from the early adopters about the implementation process, and share information that encourages innovation and quality improvement. While it may feel that the healthcare industry is traveling on an unknown path, AMIA can help to lead the way by promoting the trusted environment needed for HIT and exchange in the future. We need to continue to demonstrate how AMIA, a diverse community of practice with shared interests, can evolve better systems through collaborative efforts. In the long run, that's the best way for our field to deliver on the promise of HIT. The purpose of the Messages from AMIA section is to provide a forum for AMIA to inform and involve its current and potential members about the goals and the directions of the association. These messages, which reflect the directions and opinions of AMIA leaders only, are intended to inspire members and readers to connect with the association on strategic objectives and activities. See also http://www.amia.org/presidents-page. None. Not commissioned; not internally peer reviewed.

Physicians are often unaware of the results of tests pending at discharge (TPADs). The authors designed and implemented an automated system to notify the responsible inpatient physician of the finalized results of TPADs using secure, network email. The system coordinates a series of electronic events triggered by the discharge time stamp and sends an email to the identified discharging attending physician once finalized results are available. A carbon copy is sent to the primary care physicians in order to facilitate communication and the subsequent transfer of responsibility. Logic was incorporated to suppress selected tests and to limit notification volume. The system was activated for patients with TPADs discharged by randomly selected inpatient-attending physicians during a 6-month pilot. They received approximately 1.6 email notifications per discharged patient with TPADs. Eighty-four per cent of inpatient-attending physicians receiving automated email notifications stated that they were satisfied with the system in a brief survey (59% survey response rate). Automated email notification is a useful strategy for managing results of TPADs.

Little is known about physicians' perception of the ease or difficulty of implementing electronic health records (EHR). This study identified factors related to the perceived difficulty of implementing EHR. 163 physicians completed surveys before and after the implementation of EHR in an externally funded pilot program in three Massachusetts communities. Ordinal hierarchical logistic regression was used to identify baseline factors that correlated with physicians' report of difficulty with EHR implementation. Compared with physicians with ownership stake in their practices, physician employees were less likely to describe EHR implementation as difficult (adjusted OR 0.5, 95% CI 0.3 to 1.0). Physicians who perceived their staff to be innovative were also less likely to view EHR implementation as difficult (adjusted OR 0.4, 95% CI 0.2 to 0.8). Physicians who own their practice may need more external support for EHR implementation than those who do not. Innovative clinical support staff may ease the EHR implementation process and contribute to its success.

OBJECTIVE: To evaluate the impact of a real-time computerized decision support tool in the emergency department that guides medication dosing for the elderly on physician ordering behavior and on adverse drug events (ADEs). DESIGN: A prospective controlled trial was conducted over 26 weeks. The status of the decision support tool alternated OFF (7/17/06-8/29/06), ON (8/29/06-10/10/06), OFF (10/10/06-11/28/06), and ON (11/28/06-1/16/07) in consecutive blocks during the study period. In patients ≥65 who were ordered certain benzodiazepines, opiates, non-steroidals, or sedative-hypnotics, the computer application either adjusted the dosing or suggested a different medication. Physicians could accept or reject recommendations. MEASUREMENTS: The primary outcome compared medication ordering consistent with recommendations during ON versus OFF periods. Secondary outcomes included the admission rate, emergency department length of stay for discharged patients, 10-fold dosing orders, use of a second drug to reverse the original medication, and rate of ADEs using previously validated explicit chart review. RESULTS: 2398 orders were placed for 1407 patients over 1548 visits. The majority (49/53; 92.5%) of recommendations for alternate medications were declined. More orders were consistent with dosing recommendations during ON (403/1283; 31.4%) than OFF (256/1115; 23%) periods (p≤0.0001). 673 (43%) visits were reviewed for ADEs. The rate of ADEs was lower during ON (8/237; 3.4%) compared with OFF (31/436; 7.1%) periods (p=0.02). The remaining secondary outcomes showed no difference. LIMITATIONS: Single institution study, retrospective chart review for ADEs. CONCLUSION: Though overall agreement with recommendations was low, real-time computerized decision support resulted in greater acceptance of medication recommendations. Fewer ADEs were observed when computerized decision support was active.

BACKGROUND: Electronic health record (EHR) adoption is a national priority in the USA, and well-designed EHRs have the potential to improve quality and safety. However, physicians are reluctant to implement EHRs due to financial constraints, usability concerns, and apprehension about unintended consequences, including the introduction of medical errors related to EHR use. The goal of this study was to characterize and describe physicians' attitudes towards three consequences of EHR implementation: (1) the potential for EHRs to introduce new errors; (2) improvements in healthcare quality; and (3) changes in overall physician satisfaction. METHODS: Using data from a 2007 statewide survey of Massachusetts physicians, we conducted multivariate regression analysis to examine relationships between practice characteristics, perceptions of EHR-related errors, perceptions of healthcare quality, and overall physician satisfaction. RESULTS: 30% of physicians agreed that EHRs create new opportunities for error, but only 2% believed their EHR has created more errors than it prevented. With respect to perceptions of quality, there was no significant association between perceptions of EHR-associated errors and perceptions of EHR-associated changes in healthcare quality. Finally, physicians who believed that EHRs created new opportunities for error were less likely be satisfied with their practice situation (adjusted OR 0.49, p=0.001). CONCLUSIONS: Almost one third of physicians perceived that EHRs create new opportunities for error. This perception was associated with lower levels of physician satisfaction.

OBJECTIVE: To develop a set of high-severity, clinically significant drug-drug interactions (DDIs) for use in electronic health records (EHRs). METHODS: A panel of experts was convened with the goal of identifying critical DDIs that should be used for generating medication-related decision support alerts in all EHRs. Panelists included medication knowledge base vendors, EHR vendors, in-house knowledge base developers from academic medical centers, and both federal and private agencies involved in the regulation of medication use. Candidate DDIs were assessed by the panel based on the consequence of the interaction, severity levels assigned to them across various medication knowledge bases, availability of therapeutic alternatives, monitoring/management options, predisposing factors, and the probability of the interaction based on the strength of evidence available in the literature. RESULTS: Of 31 DDIs considered to be high risk, the panel approved a final list of 15 interactions. Panelists agreed that this list represented drugs that are contraindicated for concurrent use, though it does not necessarily represent a complete list of all such interacting drug pairs. For other drug interactions, severity may depend on additional factors, such as patient conditions or timing of co-administration. DISCUSSION: The panel provided recommendations on the creation, maintenance, and implementation of a central repository of high severity interactions. CONCLUSIONS: A set of highly clinically significant drug-drug interactions was identified, for which warnings should be generated in all EHRs. The panel highlighted the complexity of issues surrounding development and implementation of such a list.

BACKGROUND: Accurate clinical problem lists are critical for patient care, clinical decision support, population reporting, quality improvement, and research. However, problem lists are often incomplete or out of date. OBJECTIVE: To determine whether a clinical alerting system, which uses inference rules to notify providers of undocumented problems, improves problem list documentation. STUDY DESIGN AND METHODS: Inference rules for 17 conditions were constructed and an electronic health record-based intervention was evaluated to improve problem documentation. A cluster randomized trial was conducted of 11 participating clinics affiliated with a large academic medical center, totaling 28 primary care clinical areas, with 14 receiving the intervention and 14 as controls. The intervention was a clinical alert directed to the provider that suggested adding a problem to the electronic problem list based on inference rules. The primary outcome measure was acceptance of the alert. The number of study problems added in each arm as a pre-specified secondary outcome was also assessed. Data were collected during 6-month pre-intervention (11/2009-5/2010) and intervention (5/2010-11/2010) periods. RESULTS: 17,043 alerts were presented, of which 41.1% were accepted. In the intervention arm, providers documented significantly more study problems (adjusted OR=3.4, p<0.001), with an absolute difference of 6277 additional problems. In the intervention group, 70.4% of all study problems were added via the problem list alerts. Significant increases in problem notation were observed for 13 of 17 conditions. CONCLUSION: Problem inference alerts significantly increase notation of important patient problems in primary care, which in turn has the potential to facilitate quality improvement. TRIAL REGISTRATION: ClinicalTrials.gov: NCT01105923.

OBJECTIVE: To summarize the Canadian health information technology (HIT) policy experience and impart lessons learned to the US as it determines its policy in this area. DESIGN: Qualitative analysis of interviews with identified key stakeholders followed by an electronic survey. MEASUREMENTS: We conducted semi-structured interviews with 29 key Canadian HIT policy and opinion leaders and used a grounded theory approach to analyze the results. The informant sample was chosen to provide views from different stakeholder groups including national representatives and regional representatives from three Canadian provinces. RESULTS: Canadian informants believed that much of the current US direction is positive, especially regarding incentives and meaningful use, but that there are key opportunities for the US to emphasize direct engagement with providers, define a clear business case for them, sponsor large scale evaluations to assess HIT impact in a broad array of settings, determine standards but also enable access to resources needed for mid-course corrections of standards when issues are identified, and, finally, leverage implementation of digital imaging systems. LIMITATIONS: Not all stakeholder groups were included, such as providers or patients. In addition, as in all qualitative research, a selection bias could be present due to the relatively small sample size. CONCLUSIONS: Based on Canadian experience with HIT policy, stakeholders identified as lessons for the US the need to increase direct engagement with providers and the importance of defining the business case for HIT, which can be achieved through large scale evaluations, and of recognizing and leveraging successes as they emerge.

OBJECTIVE: To report the frequency, types, and causes of errors associated with outpatient computer-generated prescriptions, and to develop a framework to classify these errors to determine which strategies have greatest potential for preventing them. MATERIALS AND METHODS: This is a retrospective cohort study of 3850 computer-generated prescriptions received by a commercial outpatient pharmacy chain across three states over 4 weeks in 2008. A clinician panel reviewed the prescriptions using a previously described method to identify and classify medication errors. Primary outcomes were the incidence of medication errors; potential adverse drug events, defined as errors with potential for harm; and rate of prescribing errors by error type and by prescribing system. RESULTS: Of 3850 prescriptions, 452 (11.7%) contained 466 total errors, of which 163 (35.0%) were considered potential adverse drug events. Error rates varied by computerized prescribing system, from 5.1% to 37.5%. The most common error was omitted information (60.7% of all errors). DISCUSSION: About one in 10 computer-generated prescriptions included at least one error, of which a third had potential for harm. This is consistent with the literature on manual handwritten prescription error rates. The number, type, and severity of errors varied by computerized prescribing system, suggesting that some systems may be better at preventing errors than others. CONCLUSIONS: Implementing a computerized prescribing system without comprehensive functionality and processes in place to ensure meaningful system use does not decrease medication errors. The authors offer targeted recommendations on improving computerized prescribing systems to prevent errors.

BACKGROUND: The electronic exchange of health information among healthcare providers has the potential to produce enormous clinical benefits and financial savings, although realizing that potential will be challenging. The American Recovery and Reinvestment Act of 2009 will reward providers for 'meaningful use' of electronic health records, including participation in clinical data exchange, but the best ways to do so remain uncertain. METHODS: We analyzed patient visits in one community in which a high proportion of providers were using an electronic health record and participating in data exchange. Using claims data from one large private payer for individuals under age 65 years, we computed the number of visits to a provider which involved transitions in care from other providers as a percentage of total visits. We calculated this 'transition percentage' for individual providers and medical groups. RESULTS: On average, excluding radiology and pathology, approximately 51% of visits involved care transitions between individual providers in the community and 36%-41% involved transitions between medical groups. There was substantial variation in transition percentage across medical specialties, within specialties and across medical groups. Specialists tended to have higher transition percentages and smaller ranges within specialty than primary care physicians, who ranged from 32% to 95% (including transitions involving radiology and pathology). The transition percentages of pediatric practices were similar to those of adult primary care, except that many transitions occurred among pediatric physicians within a single medical group. CONCLUSIONS: Care transition patterns differed substantially by type of practice and should be considered in designing incentives to foster providers' meaningful use of health data exchange services.

Serious medication errors occur commonly in the period after hospital discharge. Medication reconciliation in the postdischarge ambulatory setting may be one way to reduce the frequency of these errors. The authors describe the design and implementation of a novel tool built into an ambulatory electronic medical record (EMR) to facilitate postdischarge medication reconciliation. The tool compares the preadmission medication list within the ambulatory EMR to the hospital discharge medication list, highlights all changes, and allows the EMR medication list to be easily updated. As might be expected for a novel tool intended for use in a minority of visits, use of the tool was low at first: 20% of applicable patient visits within 30 days of discharge. Clinician outreach, education, and a pop-up reminder succeeded in increasing use to 41% of applicable visits. Review of feedback identified several usability issues that will inform subsequent versions of the tool and provide generalizable lessons for how best to design medication reconciliation tools for this setting.

BACKGROUND: Clinical decision support systems can prevent knowledge-based prescription errors and improve patient outcomes. The clinical effectiveness of these systems, however, is substantially limited by poor user acceptance of presented warnings. To enhance alert acceptance it may be useful to quantify the impact of potential modulators of acceptance. METHODS: We built a logistic regression model to predict alert acceptance of drug-drug interaction (DDI) alerts in three different settings. Ten variables from the clinical and human factors literature were evaluated as potential modulators of provider alert acceptance. ORs were calculated for the impact of knowledge quality, alert display, textual information, prioritization, setting, patient age, dose-dependent toxicity, alert frequency, alert level, and required acknowledgment on acceptance of the DDI alert. RESULTS: 50,788 DDI alerts were analyzed. Providers accepted only 1.4% of non-interruptive alerts. For interruptive alerts, user acceptance positively correlated with frequency of the alert (OR 1.30, 95% CI 1.23 to 1.38), quality of display (4.75, 3.87 to 5.84), and alert level (1.74, 1.63 to 1.86). Alert acceptance was higher in inpatients (2.63, 2.32 to 2.97) and for drugs with dose-dependent toxicity (1.13, 1.07 to 1.21). The textual information influenced the mode of reaction and providers were more likely to modify the prescription if the message contained detailed advice on how to manage the DDI. CONCLUSION: We evaluated potential modulators of alert acceptance by assessing content and human factors issues, and quantified the impact of a number of specific factors which influence alert acceptance. This information may help improve clinical decision support systems design.

BACKGROUND: Accurate knowledge of a patient's medical problems is critical for clinical decision making, quality measurement, research, billing and clinical decision support. Common structured sources of problem information include the patient problem list and billing data; however, these sources are often inaccurate or incomplete. OBJECTIVE: To develop and validate methods of automatically inferring patient problems from clinical and billing data, and to provide a knowledge base for inferring problems. STUDY DESIGN AND METHODS: We identified 17 target conditions and designed and validated a set of rules for identifying patient problems based on medications, laboratory results, billing codes, and vital signs. A panel of physicians provided input on a preliminary set of rules. Based on this input, we tested candidate rules on a sample of 100,000 patient records to assess their performance compared to gold standard manual chart review. The physician panel selected a final rule for each condition, which was validated on an independent sample of 100,000 records to assess its accuracy. RESULTS: Seventeen rules were developed for inferring patient problems. Analysis using a validation set of 100,000 randomly selected patients showed high sensitivity (range: 62.8-100.0%) and positive predictive value (range: 79.8-99.6%) for most rules. Overall, the inference rules performed better than using either the problem list or billing data alone. CONCLUSION: We developed and validated a set of rules for inferring patient problems. These rules have a variety of applications, including clinical decision support, care improvement, augmentation of the problem list, and identification of patients for research cohorts.

OBJECTIVE: Clinical decision support (CDS) is a powerful tool for improving healthcare quality and ensuring patient safety; however, effective implementation of CDS requires effective clinical and technical governance structures. The authors sought to determine the range and variety of these governance structures and identify a set of recommended practices through observational study. DESIGN: Three site visits were conducted at institutions across the USA to learn about CDS capabilities and processes from clinical, technical, and organizational perspectives. Based on the results of these visits, written questionnaires were sent to the three institutions visited and two additional sites. Together, these five organizations encompass a variety of academic and community hospitals as well as small and large ambulatory practices. These organizations use both commercially available and internally developed clinical information systems. MEASUREMENTS: Characteristics of clinical information systems and CDS systems used at each site as well as governance structures and content management approaches were identified through extensive field interviews and follow-up surveys. RESULTS: Six recommended practices were identified in the area of governance, and four were identified in the area of content management. Key similarities and differences between the organizations studied were also highlighted. CONCLUSION: Each of the five sites studied contributed to the recommended practices presented in this paper for CDS governance. Since these strategies appear to be useful at a diverse range of institutions, they should be considered by any future implementers of decision support.

BACKGROUND: Medication-related decision support can reduce the frequency of preventable adverse drug events. However, the design of current medication alerts often results in alert fatigue and high over-ride rates, thus reducing any potential benefits. METHODS: The authors previously reviewed human-factors principles for relevance to medication-related decision support alerts. In this study, instrument items were developed for assessing the appropriate implementation of these human-factors principles in drug-drug interaction (DDI) alerts. User feedback regarding nine electronic medical records was considered during the development process. Content validity, construct validity through correlation analysis, and inter-rater reliability were assessed. RESULTS: The final version of the instrument included 26 items associated with nine human-factors principles. Content validation on three systems resulted in the addition of one principle (Corrective Actions) to the instrument and the elimination of eight items. Additionally, the wording of eight items was altered. Correlation analysis suggests a direct relationship between system age and performance of DDI alerts (p=0.0016). Inter-rater reliability indicated substantial agreement between raters (κ=0.764). CONCLUSION: The authors developed and gathered preliminary evidence for the validity of an instrument that measures the appropriate use of human-factors principles in the design and display of DDI alerts. Designers of DDI alerts may use the instrument to improve usability and increase user acceptance of medication alerts, and organizations selecting an electronic medical record may find the instrument helpful in meeting their clinicians' usability needs.

The objective of this review is to describe the implementation of human factors principles for the design of alerts in clinical information systems. First, we conduct a review of alarm systems to identify human factors principles that are employed in the design and implementation of alerts. Second, we review the medical informatics literature to provide examples of the implementation of human factors principles in current clinical information systems using alerts to provide medication decision support. Last, we suggest actionable recommendations for delivering effective clinical decision support using alerts. A review of studies from the medical informatics literature suggests that many basic human factors principles are not followed, possibly contributing to the lack of acceptance of alerts in clinical information systems. We evaluate the limitations of current alerting philosophies and provide recommendations for improving acceptance of alerts by incorporating human factors principles in their design.

OBJECTIVE: To assess physicians' attitudes toward health information exchange (HIE) and physicians' willingness to pay to participate in HIE. DESIGN: We conducted a cross-sectional mail survey of 1296 licensed physicians (77% response rate) in Massachusetts in 2007. MEASUREMENTS: Perceptions of the potential effects of HIE on healthcare costs, quality of care, clinicians' time, patients' privacy concerns, and willingness to pay for HIE. RESULTS: After excluding 253 physicians who did not see any outpatients, we analyzed 1043 responses. Overall, 70% indicated that HIE would reduce costs, while 86% said it would improve quality and 76% believed that it would save time. On the other hand, 16% reported being very concerned about HIE's effect on privacy, while 55.0% were somewhat concerned and 29% not at all concerned. Slightly more than half of the physicians (54%) said they would be willing to pay an unspecified monthly fee to participate in HIE, but only 37% said they would be willing to pay $150 per month for it. Primary care physicians and those in larger practices tended to have more positive attitudes toward HIE. CONCLUSIONS: Physicians perceive that HIE will have generally positive effects, though a considerable fraction harbor concerns about privacy. While physicians may be willing to participate in HIE, they are not consistently willing to pay to participate. HIE business models that require substantial physician subscription fees may face significant challenges.

The Massachusetts eHealth Collaborative (MAeHC) was formed to improve patient safety and quality of care by promoting the use of health information technology through community-based implementation of electronic health records (EHRs) and health information exchange. The Collaborative has recently implemented EHRs in a diverse set of competitively selected communities, encompassing nearly 500 physicians serving over 500,000 patients. Targeting both EHR implementation and health information exchange at the community level has identified numerous challenges and strategies for overcoming them. This article describes the formation and implementation phases of the Collaborative, focusing on barriers identified, lessons learned, and policy issues.

OBJECTIVE: Interruptive alerts within electronic applications can cause "alert fatigue" if they fire too frequently or are clinically reasonable only some of the time. We assessed the impact of non-interruptive, real-time medication laboratory alerts on provider lab test ordering. DESIGN: We enrolled 22 outpatient practices into a prospective, randomized, controlled trial. Clinics either used the existing system or received on-screen recommendations for baseline laboratory tests when prescribing new medications. Since the warnings were non-interruptive, providers did not have to act upon or acknowledge the notification to complete a medication request. MEASUREMENTS: Data were collected each time providers performed suggested laboratory testing within 14 days of a new prescription order. Findings were adjusted for patient and provider characteristics as well as patient clustering within clinics. RESULTS: Among 12 clinics with 191 providers in the control group and 10 clinics with 175 providers in the intervention group, there were 3673 total events where baseline lab tests would have been advised: 1988 events in the control group and 1685 in the intervention group. In the control group, baseline labs were requested for 771 (39%) of the medications. In the intervention group, baseline labs were ordered by clinicians in 689 (41%) of the cases. Overall, no significant association existed between the intervention and the rate of ordering appropriate baseline laboratory tests. CONCLUSION: We found that non-interruptive medication laboratory monitoring alerts were not effective in improving receipt of recommended baseline laboratory test monitoring for medications. Further work is necessary to optimize compliance with non-critical recommendations.

OBJECTIVE: Few data exist measuring the effect of differentiating drug-drug interaction (DDI) alerts in computerized provider order entry systems (CPOE) by level of severity ("tiering"). We sought to determine if rates of provider compliance with DDI alerts in the inpatient setting differed when a tiered presentation was implemented. DESIGN: We performed a retrospective analysis of alert log data on hospitalized patients at two academic medical centers during the period from 2/1/2004 through 2/1/2005. Both inpatient CPOE systems used the same DDI checking service, but one displayed alerts differentially by severity level (tiered presentation, including hard stops for the most severe alerts) while the other did not. Participants were adult inpatients who generated a DDI alert, and providers who wrote the orders. Alerts were presented during the order entry process, providing the clinician with the opportunity to change the patient's medication orders to avoid the interaction. MEASUREMENTS: Rate of compliance to alerts at a tiered site compared to a non-tiered site. RESULTS: We reviewed 71,350 alerts, of which 39,474 occurred at the non-tiered site and 31,876 at the tiered site. Compliance with DDI alerts was significantly higher at the site with tiered DDI alerts compared to the non-tiered site (29% vs. 10%, p < 0.001). At the tiered site, 100% of the most severe alerts were accepted, vs. only 34% at the non-tiered site; moderately severe alerts were also more likely to be accepted at the tiered site (29% vs. 10%). CONCLUSION: Tiered alerting by severity was associated with higher compliance rates of DDI alerts in the inpatient setting, and lack of tiering was associated with a high override rate of more severe alerts.

Alerts and prompts represent promising types of decision support in electronic prescribing to tackle inadequacies in prescribing. A systematic review was conducted to evaluate the efficacy of computerized drug alerts and prompts searching EMBASE, CINHAL, MEDLINE, and PsychINFO up to May 2007. Studies assessing the impact of electronic alerts and prompts on clinicians' prescribing behavior were selected and categorized by decision support type. Most alerts and prompts (23 out of 27) demonstrated benefit in improving prescribing behavior and/or reducing error rates. The impact appeared to vary based on the type of decision support. Some of these alerts (n = 5) reported a positive impact on clinical and health service management outcomes. For many categories of reminders, the number of studies was very small and few data were available from the outpatient setting. None of the studies evaluated features that might make alerts and prompts more effective. Details of an updated search run in Jan 2009 are included in the supplement section of this review.

Don E. Detmer has served as President and Chief Executive Officer of the American Medical Informatics Association (AMIA) for the past five years, helping to set a course for the organization and demonstrating remarkable leadership as AMIA has evolved into a vibrant and influential professional association. On the occasion of Dr. Detmer's retirement, we fondly reflect on his professional life and his many contributions to biomedical informatics and, more generally, to health care in the U.S. and globally.

OBJECTIVE Electronic health records (EHRs) have potential to improve quality and safety, but many physicians do not use these systems to full capacity. The objective of this study was to determine whether this usage gap is narrowing over time. DESIGN Follow-up mail survey of 1,144 physicians in Massachusetts who completed a 2005 survey. MEASUREMENTS Adoption of EHRs and availability and use of 10 EHR functions. RESULTS The response rate was 79.4%. In 2007, 35% of practices had EHRs, up from 23% in 2005. Among practices with EHRs, there was little change between 2005 and 2007 in the availability of nine of ten EHR features; the notable exception was electronic prescribing, reported as available in 44.7% of practices with EHRs in 2005 and 70.8% in 2007. Use of EHR functions changed inconsequentially, with more than one out of five physicians not using each available function regularly in both 2005 and 2007. Only electronic prescribing increased substantially: in 2005, 19.9% of physicians with this function available used it most or all the time, compared with 42.6% in 2007 (p < 0.001). CONCLUSIONS By 2007, more than one third of practices in Massachusetts reported having EHRs; the availability and use of electronic prescribing within these systems has increased. In contrast, physicians reported little change in the availability and use of other EHR functions. System refinements, certification efforts, and health policies, including standards development, should address the gaps in both EHR adoption and the use of key functions.

OBJECTIVE Electronic health records (EHRs) have the potential to advance the quality of care, but studies have shown mixed results. The authors sought to examine the extent of EHR usage and how the quality of care delivered in ambulatory care practices varied according to duration of EHR availability. METHODS The study linked two data sources: a statewide survey of physicians' adoption and use of EHR and claims data reflecting quality of care as indicated by physicians' performance on widely used quality measures. Using four years of measurement, we combined 18 quality measures into 6 clinical condition categories. While the survey of physicians was cross-sectional, respondents indicated the year in which they adopted EHR. In an analysis accounting for duration of EHR use, we examined the relationship between EHR adoption and quality of care. RESULTS The percent of physicians reporting adoption of EHR and availability of EHR core functions more than doubled between 2000 and 2005. Among EHR users in 2005, the average duration of EHR use was 4.8 years. For all 6 clinical conditions, there was no difference in performance between EHR users and non-users. In addition, for these 6 clinical conditions, there was no consistent pattern between length of time using an EHR and physicians performance on quality measures in both bivariate and multivariate analyses. CONCLUSIONS In this cross-sectional study, we found no association between duration of using an EHR and performance with respect to quality of care, although power was limited. Intensifying the use of key EHR features, such as clinical decision support, may be needed to realize quality improvement from EHRs. Future studies should examine the relationship between the extent to which physicians use key EHR functions and their performance on quality measures over time.

OBJECTIVE: Computerized monitors can effectively detect and potentially prevent adverse drug events (ADEs). Most monitors have been developed in large academic hospitals and are not readily usable in other settings. We assessed the ability of a commercial program to identify and prevent ADEs in a community hospital. DESIGN: and Measurement We prospectively evaluated the commercial application in a community-based hospital. We examined the frequency and types of alerts produced, how often they were associated with ADEs and potential ADEs, and the potential financial impact of monitoring for ADEs. RESULTS: Among 2,407 patients screened, the application generated 516 high priority alerts. We were able to review 266 alerts at the time they were generated and among these, 30 (11.3%) were considered substantially important to warrant contacting the physician caring for the patient. These 30 alerts were associated with 4 ADEs and 11 potential ADEs. In all 15 cases, the responsible physician was unaware of the event, leading to a change in clinical care in 14 cases. Overall, 23% of high priority alerts were associated with an ADE (95% confidence interval [CI] 12% to 34%) and another 15% were associated with a potential ADE (95% CI 6% to 24%). Active surveillance used approximately 1.5 hours of pharmacist time daily. CONCLUSIONS: A commercially available, computer-based ADE detection tool was effective at identifying ADEs. When used as part of an active surveillance program, it can have an impact on preventing or ameliorating ADEs.

Patients, policymakers, providers, payers, employers, and others have increasing interest in using personal health records (PHRs) to improve healthcare costs, quality, and efficiency. While organizations now invest millions of dollars in PHRs, the best PHR architectures, value propositions, and descriptions are not universally agreed upon. Despite widespread interest and activity, little PHR research has been done to date, and targeted research investment in PHRs appears inadequate. The authors reviewed the existing PHR specific literature (100 articles) and divided the articles into seven categories, of which four in particular--evaluation of PHR functions, adoption and attitudes of healthcare providers and patients towards PHRs, PHR related privacy and security, and PHR architecture--present important research opportunities. We also briefly discuss other research related to PHRs, PHR research funding sources, and PHR business models. We believe that additional PHR research can increase the likelihood that future PHR system deployments will beneficially impact healthcare costs, quality, and efficiency.

OBJECTIVE: Recommendations for routine laboratory monitoring to reduce the risk of adverse medication events are not consistently followed. We evaluated the impact of electronic reminders delivered to primary care physicians on rates of appropriate routine medication laboratory monitoring. DESIGN: We enrolled 303 primary care physicians caring for 1,922 patients across 20 ambulatory clinics that had at least one overdue routine laboratory test for a given medication between January and June 2004. Clinics were randomized so that physicians received either usual care or electronic reminders at the time of office visits focused on potassium, creatinine, liver function, thyroid function, and therapeutic drug levels. MEASUREMENTS: Primary outcomes were the receipt of recommended laboratory monitoring within 14 days following an outpatient clinic visit. The effect of the intervention was assessed for each reminder after adjusting for clustering within clinics, as well as patient and provider characteristics. RESULTS: Medication-laboratory monitoring non-compliance ranged from 1.6% (potassium monitoring with potassium-supplement use) to 6.3% (liver function monitoring with HMG CoA Reductase Inhibitor use). Rates of appropriate laboratory monitoring following an outpatient visit ranged from 14% (therapeutic drug levels) to 64% (potassium monitoring with potassium-sparing diuretic use). Reminders for appropriate laboratory monitoring had no impact on rates of receiving appropriate testing for creatinine, potassium, liver function, renal function, or therapeutic drug level monitoring. CONCLUSION: We identified high rates of appropriate laboratory monitoring, and electronic reminders did not significantly improve these monitoring rates. Future studies should focus on settings with lower baseline adherence rates and alternate drug-laboratory combinations.

Computerized physician order entry (CPOE) is an application that is used to electronically write physician orders either in the hospital or in the outpatient setting. It is used in about 15% of U.S. Hospitals and a smaller percentage of ambulatory clinics. It is linked with clinical decision support, which provides much of the value of implementing it. A number of studies have assessed the impact of CPOE with respect to a variety of parameters, including costs of care, medication safety, use of guidelines or protocols, and other measures of the effectiveness or quality of care. Most of these studies have been undertaken at CPOE exemplar sites with homegrown clinical information systems. With the increasing implementation of commercial CPOE systems in various settings of care has come evidence that some implementation approaches may not achieve previously published results or may actually cause new errors or even harm. This has lead to new initiatives to evaluate CPOE systems, which have been undertaken by both vendors and other groups who evaluate vendors, focused on CPOE vendor capabilities and effective approaches to implementation that can achieve benefits seen in published studies. In addition, an electronic health record (EHR) vendor certification process is ongoing under the province of the Certification Commission for Health Information Technology (CCHIT) (which includes CPOE) that will affect the purchase and use of these applications by hospitals and clinics and their participation in public and private health insurance programs. Large employers have also joined this focus by developing flight simulation tools to evaluate the capabilities of these CPOE systems once implemented, potentially linking the results of such programs to reimbursement through pay for performance programs. The increasing role of CPOE systems in health care has invited much more scrutiny about the effectiveness of these systems in actual practice which has the potential to improve their ultimate performance.

Incorporation of clinical decision support (CDS) capabilities is required to realize the greatest benefits from computerized provider order entry (CPOE) systems. Discussions at a conference on CDS in CPOE held in San Francisco, California, June 21-22, 2005 produced several papers in this issue of JAMIA. The first paper reviews CDS for electronic prescribing within CPOE systems; (1) the second describes current controversies regarding creation, maintenance, and uses of CPOE order sets for CDS; (2) and the third presents issues related to certification as a potential means of validating CPOE systems for widespread use. (3) This manuscript summarizes all of the discussions at the meeting and provides a pragmatically oriented view of how to implement CPOE with CDS.

While medications can improve patients' health, the process of prescribing them is complex and error prone, and medication errors cause many preventable injuries. Computer provider order entry (CPOE) with clinical decision support (CDS), can improve patient safety and lower medication-related costs. To realize the medication-related benefits of CDS within CPOE, one must overcome significant challenges. Healthcare organizations implementing CPOE must understand what classes of CDS their CPOE systems can support, assure that clinical knowledge underlying their CDS systems is reasonable, and appropriately represent electronic patient data. These issues often influence to what extent an institution will succeed with its CPOE implementation and achieve its desired goals. Medication-related decision support is probably best introduced into healthcare organizations in two stages, basic and advanced. Basic decision support includes drug-allergy checking, basic dosing guidance, formulary decision support, duplicate therapy checking, and drug-drug interaction checking. Advanced decision support includes dosing support for renal insufficiency and geriatric patients, guidance for medication-related laboratory testing, drug-pregnancy checking, and drug-disease contraindication checking. In this paper, the authors outline some of the challenges associated with both basic and advanced decision support and discuss how those challenges might be addressed. The authors conclude with summary recommendations for delivering effective medication-related clinical decision support addressed to healthcare organizations, application and knowledge base vendors, policy makers, and researchers.

BACKGROUND: Electronic health records (EHRs) have great potential to improve safety, quality, and efficiency in medicine. However, adoption has been slow, and a key concern has been that clinicians will require more time to complete their work using EHRs. Most previous studies addressing this issue have been done in primary care. OBJECTIVE: To assess the impact of using an EHR on specialists' time. DESIGN Prospective, before-after trial of the impact of an EHR on attending physician time in four specialty clinics at an integrated delivery system: cardiology, dermatology, endocrine, and pain. MEASUREMENTS: We used a time-motion method to measure physician time spent in one of 85 designated activities. RESULTS: Attending physicians were monitored before and after the switch from paper records to a web-based ambulatory EHR. Across all specialties, 15 physicians were observed treating 157 patients while still using paper-based records, and 15 physicians were observed treating 146 patients after adoption. Following EHR implementation, the average adjusted total time spent per patient across all specialties increased slightly but not significantly (Delta = 0.94 min., p = 0.83) from 28.8 (SE = 3.6) to 29.8 (SE = 3.6) min. CONCLUSION: These data suggest that implementation of an EHR had little effect on overall visit time in specialty clinics.

OBJECTIVES: There are limited data regarding implementing electronic health records (EHR) in underserved settings. We evaluated the implementation of an EHR within the Indian Health Service (IHS), a federally funded health system for Native Americans. DESIGN: We surveyed 223 primary care clinicians practicing at 26 IHS health centers that implemented an EHR between 2003 and 2005. METHODS: The survey instrument assessed clinician attitudes regarding EHR implementation, current utilization of individual EHR functions, and attitudes regarding the use of information technology to improve quality of care in underserved settings. We fit a multivariable logistic regression model to identify correlates of increased utilization of the EHR. RESULTS: The overall response rate was 56%. Of responding clinicians, 66% felt that the EHR implementation process was positive. One-third (35%) believed that the EHR improved overall quality of care, with many (39%) feeling that it decreased the quality of the patient-doctor interaction. One-third of clinicians (34%) reported consistent use of electronic reminders, and self-report that EHRs improve quality was strongly associated with increased utilization of the EHR (odds ratio 3.03, 95% confidence interval 1.05-8.8). The majority (87%) of clinicians felt that information technology could potentially improve quality of care in rural and underserved settings through the use of tools such as online information sources, telemedicine programs, and electronic health records. CONCLUSIONS: Clinicians support the use of information technology to improve quality in underserved settings, but many felt that it was not currently fulfilling its potential in the IHS, potentially due to limited use of key functions within the EHR.

OBJECTIVE: Despite emerging evidence that electronic health records (EHRs) can improve the efficiency and quality of medical care, most physicians in office practice in the United States do not currently use an EHR. We sought to measure the correlates of EHR adoption. DESIGN: Mailed survey to a stratified random sample of all medical practices in Massachusetts in 2005, with one physician per practice randomly selected for survey. MEASUREMENTS: EHR adoption rates. RESULTS: The response rate was 71% (1345/1884). Overall, while 45% of physicians were using an EHR, EHRs were present in only 23% of practices. In multivariate analysis, practice size was strongly correlated with EHR adoption; 52% of practices with 7 or more physicians had an EHR, as compared with 14% of solo practices (adjusted odds ratio, 3.66; 95% confidence interval, 2.28-5.87). Hospital-based practices (adjusted odds ratio, 2.44; 95% confidence interval, 1.53-3.91) and practices that teach medical students or residents (adjusted odds ratio, 2.30; 95% confidence interval, 1.60-3.31) were more likely to have an EHR. The most frequently cited barriers to adoption were start-up financial costs (84%), ongoing financial costs (82%), and loss of productivity (81%). CONCLUSIONS: While almost half of physicians in Massachusetts are using an EHR, fewer than one in four practices in Massachusetts have adopted EHRs. Adoption rates are lower in smaller practices, those not affiliated with hospitals, and those that do not teach medical students or residents. Interventions to expand EHR use must address both financial and non-financial barriers, especially among smaller practices.

Massachusetts payers and providers have encouraged clinician usage of e-Prescribing technology to improve patient safety, enhance office practice efficiencies, and reduce medical costs. This report describes three early pilot e-Prescribing projects as case studies. These projects identified the e-Prescribing needs of clinicians, illustrated key issues that made implementation difficult, and clarified the impact of various types of functionality. The authors identified ten key barriers: (1) previous negative technology experiences, (2) initial and long-term cost, (3) lost productivity, (4) competing priorities, (5) change management issues, (6) interoperability limitations, (7) information technology (IT) requirements, (8) standards limitations, (9) waiting for an "all-in-one solution," and (10) confusion about competing product offerings including hospital/Integrated Delivery System (IDN)-sponsored projects. In Massachusetts, regional projects have helped to address these barriers, and e-Prescribing activities are accelerating rapidly within the state.

OBJECTIVE: Computerized physician order entry with clinical decision support has been shown to improve medication safety in adult inpatients, but few data are available regarding its usefulness in the long-term care setting. The objective of this study was to examine opportunities for improving medication safety in that clinical setting by determining the proportion of medication orders that would generate a warning message to the prescriber via a computerized clinical decision support system and assessing the extent to which these alerts would affect prescribers' actions. DESIGN: The study was set within a randomized controlled trial of computerized clinical decision support conducted in the long-stay units of a large, academically-affiliated long-term care facility. In March 2002, a computer-based clinical decision support system (CDSS) was added to an existing computerized physician order entry (CPOE) system. Over a subsequent one-year study period, prescribers ordering drugs for residents on three resident-care units of the facility were presented with alerts; these alerts were not displayed to prescribers in the four control units. MEASUREMENTS: We assessed the frequency of drug orders associated with various categories of alerts across all participating units of the facility. To assess the impact of actually receiving an alert on prescriber behavior during drug ordering, we calculated separately for the intervention and control units the proportion of the alerts, within each category, that were followed by an appropriate action and estimated the relative risk of an appropriate action in the intervention units compared to the control units. RESULTS: During the 12 months of the study, there were 445 residents on the participating units of the facility, contributing 3,726 resident-months of observation time. During this period, 47,997 medication orders were entered through the CPOE system-approximately 9 medication orders per resident per month. 9,414 alerts were triggered (2.5 alerts per resident-month). The alert categories most often triggered were related to risks of central nervous system side-effects such as over-sedation (20%). Alerts for risk of drug-associated constipation (13%) or renal insufficiency/electrolyte imbalance (12%) were also common. Twelve percent of the alerts were related to orders for warfarin. Overall, prescribers who received alerts were only slightly more likely to take an appropriate action (relative risk 1.11, 95% confidence interval 1.00, 1.22). Alerts related to orders for warfarin or central nervous system side effects were most likely to engender an appropriate action, such as ordering a recommended laboratory test or canceling an ordered drug. CONCLUSION: Long-term care facilities must implement new system-level approaches with the potential to improve medication safety for their residents. The number of medication orders that triggered a warning message in this study suggests that CPOE with a clinical decision support system may represent one such tool. However, the relatively low rate of response to these alerts suggests that further refinements to such systems are required, and that their impact on medication errors and adverse drug events must be carefully assessed.

OBJECTIVE: Although computerized physician order entry (CPOE) may decrease errors and improve quality, hospital adoption has been slow. The high costs and limited data on financial benefits of CPOE systems are a major barrier to adoption. The authors assessed the costs and financial benefits of the CPOE system at Brigham and Women's Hospital over ten years. DESIGN: Cost and benefit estimates of a hospital CPOE system at Brigham and Women's Hospital (BWH), a 720-adult bed, tertiary care, academic hospital in Boston. MEASUREMENTS: Institutional experts provided data about the costs of the CPOE system. Benefits were determined from published studies of the BWH CPOE system, interviews with hospital experts, and relevant internal documents. Net overall savings to the institution and operating budget savings were determined. All data are presented as value figures represented in 2002 dollars. RESULTS: Between 1993 and 2002, the BWH spent $11.8 million to develop, implement, and operate CPOE. Over ten years, the system saved BWH $28.5 million for cumulative net savings of $16.7 million and net operating budget savings of $9.5 million given the institutional 80% prospective reimbursement rate. The CPOE system elements that resulted in the greatest cumulative savings were renal dosing guidance, nursing time utilization, specific drug guidance, and adverse drug event prevention. The CPOE system at BWH has resulted in substantial savings, including operating budget savings, to the institution over ten years. CONCLUSION: Other hospitals may be able to save money and improve patient safety by investing in CPOE systems.

OBJECTIVE: To maximize effectiveness, clinical decision-support systems must have access to accurate diagnostic and prescribing information. We measured the accuracy of electronic claims diagnoses and electronic antibiotic prescribing for acute respiratory infections (ARIs) and urinary tract infections (UTIs) in primary care. DESIGN: A retrospective, cross-sectional study of randomly selected visits to nine clinics in the Brigham and Women's Practice-Based Research Network between 2000 and 2003 with a principal claims diagnosis of an ARI or UTI (N = 827). MEASUREMENTS: We compared electronic billing diagnoses and electronic antibiotic prescribing to the gold standard of blinded chart review. RESULTS: Claims-derived, electronic ARI diagnoses had a sensitivity of 98%, specificity of 96%, and positive predictive value of 96%. Claims-derived, electronic UTI diagnoses had a sensitivity of 100%, specificity of 87%, and positive predictive value of 85%. According to the visit note, physicians prescribed antibiotics in 45% of ARI visits and 73% of UTI visits. Electronic antibiotic prescribing had a sensitivity of 43%, specificity of 93%, positive predictive value of 90%, and simple agreement of 64%. The sensitivity of electronic antibiotic prescribing increased over time from 22% in 2000 to 58% in 2003 (p for trend < 0.0001). CONCLUSION: Claims-derived, electronic diagnoses for ARIs and UTIs appear accurate. Although closing, a large gap persists between antibiotic prescribing documented in the visit note and the use of electronic antibiotic prescribing. Barriers to electronic antibiotic prescribing in primary care must be addressed to leverage the potential that computerized decision-support systems offer in reducing costs, improving quality, and improving patient safety.

OBJECTIVE: Infobuttons are message-based content search and retrieval functions embedded within other applications that dynamically return information relevant to the clinical task at hand. The objective of this study was to determine whether infobuttons effectively answer providers' questions about medications or affect patient care decisions. DESIGN: The authors implemented and evaluated a medication infobutton application called KnowledgeLink. Health care providers at 18 outpatient clinics were randomized to one of two versions of KnowledgeLink, one that linked to information from Micromedex (Thomson Micromedex, Greenwood Village, Co) and the other to material from SkolarMD (Wolters Kluwer Health, Palo Alto, CA). MEASUREMENTS: Data were collected about the frequency of use and demographics of users, patients, and drugs that were queried. Users were periodically surveyed with short questionnaires and then with a more extensive survey at the end of one year. RESULTS: During the first year, KnowledgeLink was used 7,972 times by 359 users to look up information about 1,723 medications for 4,961 patients. Clinicians used KnowledgeLink twice a month on average, and during an average of 1.2% of patient encounters. KnowledgeLink was used by a wide variety of medical staff, not just physicians and nurse practitioners. The frequency of usage and the questions asked varied with user role (primary care physician, specialist physician, nurse practitioner). Although the median KnowledgeLink session was brief (21 seconds), KnowledgeLink answered users' queries 84% of the time, and altered patient care decisions 15% of the time. Users rated KnowledgeLink favorably on multiple scales, recommended extending KnowledgeLink to other content domains, and suggested enhancing the interface to allow refinement of the query and selection of the target resource. CONCLUSION: An infobutton can satisfy information needs about medications. Although used infrequently and for brief sessions, KnowledgeLink was positively received, answered most users' questions, and had a significant impact on medical decision making. The next steps would be to broaden the domains that KnowledgeLink covers to more specifically tailor results to the user type, to provide options when queries are not immediately answered, and to implement KnowledgeLink within other electronic clinical applications.

OBJECTIVE: Clinicians are increasingly using handheld computers (HC) during patient care. We sought to assess the role of HC-based clinical reference software in medical practice by conducting a survey and assessing actual usage behavior. DESIGN: During a 2-week period in February 2005, 3600 users of a HC-based clinical reference application were asked by e-mail to complete a survey and permit analysis of their usage patterns. The software includes a pharmacopeia, an infectious disease reference, a medical diagnostic and therapeutic reference and transmits medical alerts and other notifications during HC synchronizations. Software usage data were captured during HC synchronization for the 4 weeks prior to survey completion. MEASUREMENTS: Survey responses and software usage data. RESULTS: The survey response rate was 42% (n = 1501). Physicians reported using the clinical reference software for a mean of 4 years and 39% reported using the software during more than half of patient encounters. Physicians who synchronized their HC during the data collection period (n = 1249; 83%) used the pharmacopeia for unique drug lookups a mean of 6.3 times per day (SD 12.4). The majority of users (61%) believed that in the prior 4 weeks, use of the clinical reference prevented adverse drug events or medication errors 3 or more times. Physicians also believed that alerts and other notifications improved patient care if they were public health warnings (e.g. about influenza), new immunization guidelines or drug alert warnings (e.g. rofecoxib withdrawal). CONCLUSION: Current adopters of HC-based medical references use these tools frequently, and found them to improve patient care and be valuable in learning of recent alerts and warnings.

Computerized drug prescribing alerts can improve patient safety, but are often overridden because of poor specificity and alert overload. Our objective was to improve clinician acceptance of drug alerts by designing a selective set of drug alerts for the ambulatory care setting and minimizing workflow disruptions by designating only critical to high-severity alerts to be interruptive to clinician workflow. The alerts were presented to clinicians using computerized prescribing within an electronic medical record in 31 Boston-area practices. There were 18,115 drug alerts generated during our six-month study period. Of these, 12,933 (71%) were noninterruptive and 5,182 (29%) interruptive. Of the 5,182 interruptive alerts, 67% were accepted. Reasons for overrides varied for each drug alert category and provided potentially useful information for future alert improvement. These data suggest that it is possible to design computerized prescribing decision support with high rates of alert recommendation acceptance by clinicians.

Recently there has been a remarkable upsurge in activity surrounding the adoption of personal health record (PHR) systems for patients and consumers. The biomedical literature does not yet adequately describe the potential capabilities and utility of PHR systems. In addition, the lack of a proven business case for widespread deployment hinders PHR adoption. In a 2005 working symposium, the American Medical Informatics Association's College of Medical Informatics discussed the issues surrounding personal health record systems and developed recommendations for PHR-promoting activities. Personal health record systems are more than just static repositories for patient data; they combine data, knowledge, and software tools, which help patients to become active participants in their own care. When PHRs are integrated with electronic health record systems, they provide greater benefits than would stand-alone systems for consumers. This paper summarizes the College Symposium discussions on PHR systems and provides definitions, system characteristics, technical architectures, benefits, barriers to adoption, and strategies for increasing adoption.

After the first session of the American College of Medical Informatics 2004 retreat, during which the history of electronic health records was reviewed, the second session served as a forum for discussion about the state of the art of EHR adoption. Adoption and diffusion rates for both inpatient and outpatient EHRs are low for a myriad of reasons ranging from personal physician concerns about workflow to broad environmental issues. Initial recommendations for addressing these issues include providing communication and education to both providers and consumers and alignment of incentives for clinicians.

The state of Massachusetts has significant early experience in planning for and implementing interoperability networks for exchange of clinical and financial data. Members of our evolving data-sharing organizations gained valuable experience that is of potential benefit to others regarding the governance, policies, and technologies underpinning regional health information organizations. We describe the history, roles, and evolution of organizations and their plans for and success with pilot projects.

OBJECTIVE: The aim of this study was to evaluate the impact of an integrated patient-specific electronic clinical reminder system on diabetes and coronary artery disease (CAD) care and to assess physician attitudes toward this reminder system. DESIGN: We enrolled 194 primary care physicians caring for 4549 patients with diabetes and 2199 patients with CAD at 20 ambulatory clinics. Clinics were randomized so that physicians received either evidence-based electronic reminders within their patients' electronic medical record or usual care. There were five reminders for diabetes care and four reminders for CAD care. MEASUREMENTS: The primary outcome was receipt of recommended care for diabetes and CAD. We created a summary outcome to assess the odds of increased compliance with overall diabetes care (based on five measures) and overall CAD care (based on four measures). We surveyed physicians to assess attitudes toward the reminder system. RESULTS: Baseline adherence rates to all quality measures were low. While electronic reminders increased the odds of recommended diabetes care (odds ratio [OR] 1.30, 95% confidence interval [CI] 1.01-1.67) and CAD (OR 1.25, 95% CI 1.01-1.55), the impact of individual reminders was variable. A total of three of nine reminders effectively increased rates of recommended care for diabetes or CAD. The majority of physicians (76%) thought that reminders improved quality of care. CONCLUSION: An integrated electronic reminder system resulted in variable improvement in care for diabetes and CAD. These improvements were often limited and quality gaps persist.

Recent reports have identified medical errors as a significant cause of morbidity and mortality among patients. A variety of approaches have been implemented to identify errors and their causes. These approaches include retrospective reporting and investigation of errors and adverse events and prospective analyses for identifying hazardous situations. The above approaches, along with other sources, contribute to data that are used to analyze patient safety risks. A variety of data structures and terminologies have been created to represent the information contained in these sources of patient safety data. Whereas many representations may be well suited to the particular safety application for which they were developed, such application-specific and often organization-specific representations limit the sharability of patient safety data. The result is that aggregation and comparison of safety data across organizations, practice domains, and applications is difficult at best. A common reference data model and a broadly applicable terminology for patient safety data are needed to aggregate safety data at the regional and national level and conduct large-scale studies of patient safety risks and interventions.

OBJECTIVE: To examine various strategies for the identification of adverse drug events (ADEs) among older persons in the ambulatory clinical setting. DESIGN: A cohort study of Medicare enrollees (n = 31,757 per month) receiving medical care from a large multispecialty group practice during a 12-month observation period (July 1, 1999 through June 30, 2000). MEASUREMENTS: Possible drug-related incidents occurring in the ambulatory clinical setting were detected using signals from multiple sources. RESULTS: During the tracking period, there were 1,523 identified ADEs, of which 421 (28%) were considered preventable. Across all sources, 23,917 signals were found; 12,791 (53%) were potential incidents that led to review of a patient's medical record and 2,266 (9%) were presented to physician reviewers. Although the positive predictive value (PPV) for reports from providers was high compared with other sources (54%), only 11% of the ADEs and 6% of the preventable ADEs were identified through this source. PPVs for other sources ranged from a low of 4% for administrative incident reports to a high of 12% for free-text review of electronic notes. Computer-generated signals were the source for 31% of the ADEs and 37% of the preventable ADEs. Electronic notes were the source for 39% of the ADEs and 29% of the preventable ADEs. There was little overlap in the ADEs identified across all sources. CONCLUSION: Our findings emphasize the limitations of voluntary reporting by health care providers as the principal means for detection of ADEs and suggest that multiple strategies are required to detect ADEs in geriatric ambulatory patients.

OBJECTIVE: The aim of this study was to determine characteristics of drug allergy alert overrides, assess how often they lead to preventable adverse drug events (ADEs), and suggest methods for improving the allergy-alerting system. DESIGN: Chart review was performed on a stratified random subset of all allergy alerts occurring during a 3-month period (August through October 2002) at a large academic hospital. MEASUREMENTS: Factors that were measured were drug/allergy combinations that triggered alerts, frequency of specific override reasons, characteristics of ADEs, and completeness of allergy documentation. RESULTS: A total of 6,182 (80%) of 7,761 alerts were overridden in 1,150 patients. In this sample, only 10% of alerts were triggered by an exact match between the drug ordered and allergy listed. Physicians' most common reasons for overriding alerts were "Aware/Will monitor" (55%), "Patient does not have this allergy/tolerates" (33%), and "Patient taking already" (10%). In a stratified random subset of 320 patients (28% of 1,150) on chart review, 19 (6%) experienced ADEs attributed to the overridden drug; of these, 9 (47%) were serious. None of the ADEs was considered preventable, because the overrides were deemed clinically justifiable. The degree of completeness of patients' allergy lists was highly variable and generally low in both paper charts and the CPOE system. CONCLUSION: Overrides of drug-allergy alerts were common and about 1 in 20 resulted in ADEs, but all of the overrides resulting in ADEs appeared clinically justifiable. The high rate of alert overrides was attributable to frequent nonexact match alerts and infrequent updating of allergy lists. Based on these findings, we have made specific recommendations for increasing the specificity of alerting and thereby improving the clinical utility of the drug allergy alerting system.

BACKGROUND: Improving the safety, quality, and efficiency of health care will require immediate and ubiquitous access to complete patient information and decision support provided through a National Health Information Infrastructure (NHII). METHODS: To help define the action steps needed to achieve an NHII, the U.S. Department of Health and Human Services sponsored a national consensus conference in July 2003. RESULTS: Attendees favored a public-private coordination group to guide NHII activities, provide education, share resources, and monitor relevant metrics to mark progress. They identified financial incentives, health information standards, and overcoming a few important legal obstacles as key NHII enablers. Community and regional implementation projects, including consumer access to a personal health record, were seen as necessary to demonstrate comprehensive functional systems that can serve as models for the entire nation. Finally, the participants identified the need for increased funding for research on the impact of health information technology on patient safety and quality of care. Individuals, organizations, and federal agencies are using these consensus recommendations to guide NHII efforts.

Delivery of excellent primary care-central to overall medical care-demands that providers have the necessary information when they give care. This paper, developed by the National Alliance for Primary Care Informatics, a collaborative group sponsored by a number of primary care societies, argues that providers' and patients' information and decision support needs can be satisfied only if primary care providers use electronic medical records (EMRs). Although robust EMRs are now available, only about 5% of U.S. primary care providers use them. Recently, with only modest investments, Australia, New Zealand, and England have achieved major breakthroughs in implementing EMRs in primary care. Substantial benefits realizable through routine use of electronic medical records include improved quality, safety, and efficiency, along with increased ability to conduct education and research. Nevertheless, barriers to adoption exist and must be overcome. Implementing specific policies can accelerate utilization of EMRs in the U.S.

CONTEXT: Although patient safety is a major problem, most health care organizations rely on spontaneous reporting, which detects only a small minority of adverse events. As a result, problems with safety have remained hidden. Chart review can detect adverse events in research settings, but it is too expensive for routine use. Information technology techniques can detect some adverse events in a timely and cost-effective way, in some cases early enough to prevent patient harm. OBJECTIVE: To review methodologies of detecting adverse events using information technology, reports of studies that used these techniques to detect adverse events, and study results for specific types of adverse events. DESIGN: Structured review. METHODOLOGY: English-language studies that reported using information technology to detect adverse events were identified using standard techniques. Only studies that contained original data were included. MAIN OUTCOME MEASURES: Adverse events, with specific focus on nosocomial infections, adverse drug events, and injurious falls. RESULTS: Tools such as event monitoring and natural language processing can inexpensively detect certain types of adverse events in clinical databases. These approaches already work well for some types of adverse events, including adverse drug events and nosocomial infections, and are in routine use in a few hospitals. In addition, it appears likely that these techniques will be adaptable in ways that allow detection of a broad array of adverse events, especially as more medical information becomes computerized. CONCLUSION: Computerized detection of adverse events will soon be practical on a widespread basis.

In health care today, most adverse events are detected using spontaneous reporting, which identifies only a small number of adverse events.1 This is probably the major reason that problems with patient safety have been overlooked until recently. However, information technology can be used in a variety of ways to detect adverse events continuously and relatively inexpensively. In an accompanying paper,2 we review the methodologies for detecting adverse events using information technology and the evidence regarding their efficacy. This editorial presents some of what we believe are future possibilities in this domain and discusses policy issues regarding the development of strategies that may result in wider use of such tools. Spontaneous reporting is attractive because it is inexpensive compared with other approaches for detecting adverse events. Events detected via this route can be useful for quality improvement. However, because reported events represent only a tiny fraction of all adverse events that occur, absolute rates of spontaneous reporting or changes in them are not particularly useful, except to assess safety culture or whether strategies to improve reporting have worked.3 In contrast, a variety of information tchnology approaches can be used to identify a large proportion of all adverse events that occur,4 and this proportion can be expected to increase as more electronic data become available and tools are refined. Although the data regarding automated detection of some types of adverse events are now substantial,5–9 many research and development issues remain to be addressed. For nosocomial infections, adverse drug events and falls, there is a major need for studies that compare different approaches to detection and identify methods that will improve the positive predictive value (which is generally low) for individual signals. This is important because the major cost of such detection strategies is the time of the personnel who respond to the signals. Another key research area involves the definition of approaches that will allow exportation of such detection modules to hospitals in general and to small rural and community hospitals in particular. Finally, tools that allow detection of a wide array of adverse events are needed. Although claims data can provide limited information, especially for inpatients, they do not include sufficient detail to identify a large proportion of adverse events.10 A key benefit of electronic medical records may be that it will be possible to search them using computerized detection tools. Such approaches appear promising based on early data,11 but they need much more evaluation with respect to performance and generalizability. Finally, standards regarding definitions and representation of adverse events would be useful, as would better tools for classifying what went wrong in preventable events. Patient safety is extraordinarily important to the public, but the policy issues around adverse event detection and malpractice are nettlesome. Unfortunately, given the current structures of health care in the U.S., there are strong incentives for organizations to turn a blind eye to adverse events. In particular, serious, preventable adverse events typically must be reported to the state, and such events often lead to multiple visits from the department of public health or end up in the press, with adverse consequences for the institution. Thus, adverse events have negative connotations to many, and our current system offers few incentives to organizations to look for them aggressively. In particular, those who ultimately must approve resources for monitoring systems (chief executive officers and chief operating officers) can avoid investing in them, especially since there are so many competing demands for funds. As a result, financial incentives or regulation may be needed to achieve widespread adoption of routine automated monitoring for adverse events. Several years ago the Centers for Medicare and Medicaid Services (the former Health Care Financing Admin-istration) published draft regulations in the Federal Register that would have mandated computerized monitoring of adverse drug events in inpatients.12 These regulations had a number of unrelated problems, and are undergoing revision. We believe that setting up incentives for hospitals to monitor both adverse drug events and nosocomial infections using computerized detection would be desirable now. Eventually, health care systems should look routinely for adverse events using computerized detection approaches both inside and outside of hospitals, but they will not take on this burden without incentives. Incentives may come in the form of carrots (e.g., higher reimbursement for compliant organizations) or as sticks (e.g., making such monitoring a condition of participation). Another enabler is for the government to make monitoring tools available; the Center for Medicare and Medicaid Services is currently considering this possibility. A related issue that must be addressed is how individuals and organizations that identify and improve their systems using error and adverse event detection should be treated. Although this issue is complicated, the current “blame-and-shame” approach is highly counterproductive. In aviation, nonpunitive ap-proaches have been highly effective in determining the causes of adverse events and developing strategies and interventions for prevention.13 If good techniques for identifying adverse events are developed and used broadly, it will be possible to use such information to improve safety in an ongoing way and, in particular, to use it to assess the impact of systemic changes. We believe that achieving widespread adoption of these techniques may require regulation, because such screening requires resources and organizations are justifiably fearful that uncovering problems may increase litigation risk. However, if legislation or regulations were enacted to provide better protection for health care organizations, substantial improvement in patient safety may result.

While evidence-based medicine has increasingly broad-based support in health care, it remains difficult to get physicians to actually practice it. Across most domains in medicine, practice has lagged behind knowledge by at least several years. The authors believe that the key tools for closing this gap will be information systems that provide decision support to users at the time they make decisions, which should result in improved quality of care. Furthermore, providers make many errors, and clinical decision support can be useful for finding and preventing such errors. Over the last eight years the authors have implemented and studied the impact of decision support across a broad array of domains and have found a number of common elements important to success. The goal of this report is to discuss these lessons learned in the interest of informing the efforts of others working to make the practice of evidence-based medicine a reality.

We are well aware of the status of open-source electronic health records (EHRs), and are glad that Kantor et al. have written about this as these deserve more attention from AMIA members. Space did not allow us to comment about all issues relating to the electronic health record, but this is an important one. Kantor et al. eloquently elaborate the case for open-source EHRs, which have many potential advantages. The cost issue in particular is especially important for small practices, which are cash-poor, and groups of five or fewer account for most of the primary care providers in the United States. However, it is important to note, as we did in the report, that the United States is not leading in implementation of EHRs in the outpatient setting—we are far behind the rest of the industrialized world. Furthermore, none of these other countries have utilized an open-source approach—instead, they established standards for the EHR—for the underlying data types, and for a reference information model. In countries such as England and Australia, a small number of vendors (typically 2 to 4) then established a major market share. The United States has recently made major progress in this area with adoption of the CHI standards, and a collaborative effort between the Institute of Medicine and HL7 to begin establishing what the content of the electronic health record should be. Furthermore, even at a cost of $10,000 to 20,000 per provider, EHRs appear to be cost-saving.1 Unfortunately, because of the structuring of our reimbursement system, many of the savings do not accrue to those who make the capital outlay—the providers—but instead to purchasers of health care, and payers, arguing for changes in the reimbursement structure like those proposed in General Electric's Bridges to Excellence Program.2 The open-source approach has worked best for tools, and there is no example that we are aware of in which something as complex as an EHR has succeeded. Linux, for example, is an operating system. Furthermore, there are concerns about versioning, interfaces, maintenance and improvement, and governance with decision support with an open-source approach. Nonetheless, we believe that the open-source approach to EHRs is an attractive one, especially for small practices, and, in particular, applaud the efforts of the American Academy of Family Practitioners. However, we suggest that parallel initiatives with both traditional and open source initiatives should be pursued. For open-source approaches, studies and reports in the peer-reviewed literature are especially important. Success with open source will require substantial investment, and given the uncertain returns it is still not entirely clear how to make the model work or who will make the investment (the government is one option, but major governmental investment in a single record for the United States would be surprising). We believe that given the successes in the rest of the world with other approaches, the unproven nature of this one, and our capitalistic society, it would not be prudent to place all the nation's eggs in this basket.

OBJECTIVE: To describe advanced clinical information systems in the context in which they have been implemented and are being used. DESIGN: Case series of five U.S. hospitals, including inpatient, ambulatory and emergency units. Descriptive study with data collected from interviews, observations, and document analysis. MEASUREMENTS: The use of computerized results, notes, orders, and event monitors and the type of decision support; data capture mechanisms and data form; impact on clinician satisfaction and clinical processes and outcomes; and the organizational factors associated with successful implementation. RESULTS: All sites have implemented a wide range of clinical information systems with extensive decision support. The systems had been well accepted by clinicians and have improved clinical processes. Successful implementation required leadership and long-term commitment, a focus on improving clinical processes, and gaining clinician involvement and maintaining productivity. CONCLUSION: Despite differences in approach there are many similarities between sites in the clinical information systems in use and the factors important to successful implementation. The experience of these sites may provide a valuable guide for others who are yet to start, or are just beginning, the implementation of clinical information systems.

There is scant published experience with implementing complex, multistep computerized practice guidelines for the long-term management of chronic diseases. We have implemented a system for creating, maintaining, and navigating computer-based clinical algorithms integrated with our electronic medical record. This article describes our progress and reports on lessons learned that might guide future work in this field. We discuss issues and obstacles related to choosing and adapting a guideline for electronic implementation, representing and executing the guideline as a computerized algorithm, and integrating it into the clinical workflow of outpatient care. Although obstacles were encountered at each of these steps, the most difficult were related to workflow integration.

OBJECTIVE: Detecting adverse events is pivotal for measuring and improving medical safety, yet current techniques discourage routine screening. The authors hypothesized that discharge summaries would include information on adverse events, and they developed and evaluated an electronic method for screening medical discharge summaries for adverse events. DESIGN: A cohort study including 424 randomly selected admissions to the medical services of an academic medical center was conducted between January and July 2000. The authors developed a computerized screening tool that searched free-text discharge summaries for trigger words representing possible adverse events. MEASUREMENTS: All discharge summaries with a trigger word present underwent chart review by two independent physician reviewers. The presence of adverse events was assessed using structured implicit judgment. A random sample of discharge summaries without trigger words also was reviewed. RESULTS: Fifty-nine percent (251 of 424) of the discharge summaries contained trigger words. Based on discharge summary review, 44.8% (327 of 730) of the alerted trigger words indicated a possible adverse event. After medical record review, the tool detected 131 adverse events. The sensitivity and specificity of the screening tool were 69% and 48%, respectively. The positive predictive value of the tool was 52%. CONCLUSION: Medical discharge summaries contain information regarding adverse events. Electronic screening of discharge summaries for adverse events using keyword searches is feasible but thus far has poor specificity. Nonetheless, computerized clinical narrative screening methods could potentially offer researchers and quality managers a means to routinely detect adverse events.

Few data exist about the impact of computerized prescribing systems on outpatient medication errors (MEs) and adverse drug events (ADEs). We compared the rates of MEs and ADEs in handwritten sites versus sites with basic computerized prescribing. These systems reduced ME rates but did not significantly reduce ADE rates. Failure to monitor accounted for a large percentage of preventable ADEs. More advanced computerized prescribing systems with decision support and monitoring functions may be necessary to reduce outpatient ADE rates.

Detecting and preventing adverse medical events (AEs) is essential for improving medical quality. While electronic approaches for detecting and preventing adverse drug events have been developed, AEs, which include the entire range of events and are thus more diverse, have been harder to detect. Prior studies have detected AEs through structured chart reviews. While this approach is effective, it is costly and time consuming. Thus, we developed a computerized discharge abstract screening tool to detect AEs. Our initial sample consisted of 424 randomly selected patients discharged from the medical services of the Brigham and Women's Hospital between January 1 to June 30, 2000. We developed a set of alert signals based on screening criteria used in the Harvard Medical Practice Study 3 that ultimately including 94 trigger words. Individual trigger words were then identified using text-based searches of the hospital course section of electronically stored discharge summaries. Discharge summaries generating an alert were classified as “screened positive discharge summaries“ and were reviewed to determine the context in which the trigger word had been used and whether an AE appeared likely based on the discharge summary. All screened positive discharge summaries underwent chart review by two independent physician reviewers. The presence of an AE was assessed using structured implicit judgement. A random 25% of screened negative discharge summaries were also reviewed. The positive predictive values for the electronic tool was determined by dividing the number of admissions with discharge summary trigger words and an AE by the total number of screened positive discharge summaries. Time spent reviewing discharge abstracts was recorded. Nine hundred and fifty-three alerts were detected, and after adjusting for repeated signals within the same discharge summary, a total of 733 unique alerts were generated in 251/424 (59%) patients. In 131 screened positive discharge summaries the patient had experienced an AEs based on chart review (kappa statistic = 0.77). Sixty AE occurred within the 173 patients without screened positive discharge summaries. The sensitivity and specificity of the screening tool were 69% and 48% respectively. The positive predictive value of the tool was 52%. The most common category of AE detected was adverse drug events representing 52% of the detected events. The time required to review the screened discharge abstracts was 18 hours. By using computerized screening of discharge abstracts we were able to identify AE's in 31% of the patients sampled. The tool performed reasonable well, however removing individual trigger words with low positive predictive values and other improvements could also improve sensitivity. Using an electronic screening tool, we were able to screen 424 charts in 18 hours. Using Harvard Medical Practice methodology this same initial sample would have required approximately 70 hours. Electronically screening discharge summaries for adverse events appears to be an efficient and feasible means of detecting AE within hospitalized medical patients. Reprinted from the Proceedings of the 2001 AMIA Annual Symposium, with permission.

Previous studies have found that many adverse drug events (ADEs) in inpatients can be detected or prevented by alerting physicians to measured physiologic parameters such as an elevated creatinine or hyperkalemia. 1 , 2 In developing a decision support system for an outpatient Electronic Medical Record, we have begun to retrospectively study associations between drugs and labs that could trigger an alert to physicians. It is unknown whether such a drug-lab monitoring system is useful in identifying or preventing ADEs in outpatients. We developed a list of drug-lab triggers using previously published lists and established contraindications for specific drugs. Lab and dose criteria were set using published data when possible and expert opinion when the data was unavailable. For each trigger, we searched the electronic medical records of outpatient clinics in the Partners Health System between 1/1998 and 2/2001. Only the records of patients who were prescribed the drug of interest and had a documented lab of interest were eligible. For each patient record, we determined if the trigger conditions were met at least once during the study period. The proportion of eligible patients who satisfied the trigger criteria at least once is reported in Table 1 . Observed frequencies were calculated by dividing the number of patients who fulfilled all lab and dose criteria by the total number of patients prescribed the drug and had the relevant lab recorded. The most frequent association was between a high allopurinol dose and renal insufficiency occurring in 8.9% of patients prescribed allopurinol. Drug-lab Triggers and Observed Frequency in Outpatients Drug-lab Triggers and Observed Frequency in Outpatients Drug-lab triggers have potential to alert physicians to impending and actual adverse drug events. Events will need to be reviewed in order to calculate a positive predictive value for each trigger. Additionally, confirmation of the utility of triggers will require prospective study of patient outcomes associated with a positive trigger. Reprinted from the Proceedings of the 2001 AMIA Annual Symposium, with permission.

The authors developed a novel feature in their clinical information systems, which allows clinicians to request notification about laboratory results. Clinicians who are expecting a particular laboratory result for a particular patient can request a report of the result via an alphanumeric pager as soon as the result is filed into the patient database. This feature has gained popularity and is heavily used in both inpatient and outpatient settings, at a rate of about 2,300 times per month. This event-monitor-based feature illustrates one way that information technology can be applied to improve communication in health care.

OBJECTIVE: Problems involving drug knowledge are one of the most common causes of serious medication errors. Although the information that clinicians need is often available somewhere, retrieving it expeditiously has been problematic. At the same time, clinicians are faced with an ever-expanding pharmacology knowledge base. Recently, point-of-care technology has become more widely available and more practical with the advent of handheld, or palmtop, computing. Therefore, the authors evaluated the clinical contribution of a drug database developed for the handheld computer. ePocrates Rx (formerly known as qRx; ePocrates, San Carlos, California) is a comprehensive drug information guide that is downloadable free from the Internet and designed for the Palm OS platform align="right". DESIGN: A seven-day online survey of 3,000 randomly selected ePocrates Rx users was conducted during March 2000. MEASUREMENTS: User technology experience, product evaluation and usage patterns, and the effects of the drug reference database on information-seeking behavior, practice efficiency, decision making, and patient care. RESULTS: The survey response rate was 32 percent (n=946). Physicians reported that ePocrates Rx saves time during information retrieval, is easily incorporated into their usual workflow, and improves drug-related decision making. They also felt that it reduced the rate of preventable adverse drug events. CONCLUSIONS: Self-reported perceptions by responding clinicians endorse improved access to drug information and improved practice efficiency associated with the use of handheld devices. The clinical and practical value of using these devices in clinical settings will clearly grow further as wireless communication becomes more ubiquitous and as more applications become available.

BACKGROUND: Increasing data suggest that error in medicine is frequent and results in substantial harm. The recent Institute of Medicine report (LT Kohn, JM Corrigan, MS Donaldson, eds: To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press, 1999) described the magnitude of the problem, and the public interest in this issue, which was already large, has grown. GOAL: The goal of this white paper is to describe how the frequency and consequences of errors in medical care can be reduced (although in some instances they are potentiated) by the use of information technology in the provision of care, and to make general and specific recommendations regarding error reduction through the use of information technology. RESULTS: General recommendations are to implement clinical decision support judiciously; to consider consequent actions when designing systems; to test existing systems to ensure they actually catch errors that injure patients; to promote adoption of standards for data and systems; to develop systems that communicate with each other; to use systems in new ways; to measure and prevent adverse consequences; to make existing quality structures meaningful; and to improve regulation and remove disincentives for vendors to provide clinical decision support. Specific recommendations are to implement provider order entry systems, especially computerized prescribing; to implement bar-coding for medications, blood, devices, and patients; and to utilize modern electronic systems to communicate key pieces of asynchronous data such as markedly abnormal laboratory values. CONCLUSIONS: Appropriate increases in the use of information technology in health care- especially the introduction of clinical decision support and better linkages in and among systems, resulting in process simplification-could result in substantial improvement in patient safety.

OBJECTIVE: To evaluate the use of a computer program to identify adverse drug events (ADEs) in the ambulatory setting and to evaluate the relative contribution of four computer search methods for identifying ADEs, including diagnosis codes, allergy rules, computer event monitoring rules, and text searching. DESIGN: Retrospective analysis of one year of data from an electronic medical record, including records for 23,064 patients with a primary care physician, of whom 15,665 actually came for care. MEASUREMENT: Presence of an ADE; sensitivity and specificity of computer searches for ADE. RESULTS: The computer program identified 25,056 incidents, which were associated with an estimated 864 (95 percent confidence interval [CI], 750-978) ADES. Thus, the ADE rate was 5.5 (CI, 5.2-5.9) per 100 patients coming for care. Furthermore, in 79 (CI, 68-89) ADEs, the patient required hospitalization, resulting in an estimated rate of 3.4 (CI, 2.7-4.3) admissions per 1,000 patients. The sensitivity of the search methods for identifying ADEs was estimated to be 58 (CI, 18-98) percent, and the estimated specificity was 88 (CI, 87-88) percent. The positive predictive value was 7.5 (CI, 6.5-8.5) percent, and the negative predictive value was 99.2 (CI, 95.5-99.98) percent. Compared with age and gender-matched controls with no positive screen, patients with ADEs had twice as many outpatient visits and were taking nearly three times as many drugs. Antihypertensives, ACE-inhibitors, antibiotics, and diuretics were associated with 56 (CI, 47-65) percent of ADES. Among ADEs, 23 (CI, 16-32) percent were life-threatening or serious, and 38 (CI, 29-47) percent were judged preventable. CONCLUSION: Computerized search programs can detect ADEs, and free-text searches were especially useful. Adverse drug events were frequent, and admissions were not rare, although most hospitals today do not identify them. Thus, such detection programs demonstrate "value-added" for the electronic record and may be useful for directing and assessing the impact of quality improvement efforts.

BACKGROUND: Medication errors are common, and while most such errors have little potential for harm they cause substantial extra work in hospitals. A small proportion do have the potential to cause injury, and some cause preventable adverse drug events. OBJECTIVE: To evaluate the impact of computerized physician order entry (POE) with decision support in reducing the number of medication errors. DESIGN: Prospective time series analysis, with four periods. SETTING AND PARTICIPANTS: All patients admitted to three medical units were studied for seven to ten-week periods in four different years. The baseline period was before implementation of POE, and the remaining three were after. Sophistication of POE increased with each successive period. INTERVENTION: Physician order entry with decision support features such as drug allergy and drug-drug interaction warnings. MAIN OUTCOME MEASURE: Medication errors, excluding missed dose errors. RESULTS: During the study, the non-missed-dose medication error rate fell 81 percent, from 142 per 1,000 patient-days in the baseline period to 26.6 per 1,000 patient-days in the final period (P < 0.0001). Non-intercepted serious medication errors (those with the potential to cause injury) fell 86 percent from baseline to period 3, the final period (P = 0.0003). Large differences were seen for all main types of medication errors: dose errors, frequency errors, route errors, substitution errors, and allergies. For example, in the baseline period there were ten allergy errors, but only two in the following three periods combined (P < 0.0001). CONCLUSIONS: Computerized POE substantially decreased the rate of non-missed-dose medication errors. A major reduction in errors was achieved with the initial version of the system, and further reductions were found with addition of decision support features.

OBJECTIVE: To evaluate the effect of an automatic alerting system on the time until treatment is ordered for patients with critical laboratory results. DESIGN: Prospective randomized controlled trial. INTERVENTION: A computer system to detect critical conditions and automatically notify the responsible physician via the hospital's paging system. PATIENTS: Medical and surgical inpatients at a large academic medical center. One two-month study period for each service. MAIN OUTCOMES: Interval from when a critical result was available for review until an appropriate treatment was ordered. Secondary outcomes were the time until the critical condition resolved and the frequency of adverse events. METHODS: The alerting system looked for 12 conditions involving laboratory results and medications. For intervention patients, the covering physician was automatically notified about the presence of the results. For control patients, no automatic notification was made. Chart review was performed to determine the outcomes. RESULTS: After exclusions, 192 alerting situations (94 interventions, 98 controls) were analyzed. The intervention group had a 38 percent shorter median time interval (1.0 hours vs. 1.6 hours, P = 0.003; mean, 4.1 vs. 4.6 hours, P = 0.003) until an appropriate treatment was ordered. The time until the alerting condition resolved was less in the intervention group (median, 8.4 hours vs. 8.9 hours, P = 0.11; mean, 14.4 hours vs. 20.2 hours, P = 0.11), although these results did not achieve statistical significance. The impact of the intervention was more pronounced for alerts that did not meet the laboratory's critical reporting criteria. There was no significant difference between the two groups in the number of adverse events. CONCLUSION: An automatic alerting system reduced the time until an appropriate treatment was ordered for patients who had critical laboratory results. Information technologies that facilitate the transmission of important patient data can potentially improve the quality of care.

BACKGROUND: Adverse drug events (ADEs) are both common and costly. Most hospitals identify ADEs using spontaneous reporting, but this approach lacks sensitivity; chart review identifies more events but is expensive. Computer-based approaches to ADE identification appear promising, but they have not been directly compared with chart review and they are not widely used. OBJECTIVES: To develop a computer-based ADE monitor, and to compare the rate and type of ADEs found with the monitor with those discovered by chart review and by stimulated voluntary report. DESIGN: Prospective cohort study in one tertiary-care hospital. PARTICIPANTS: All patients admitted to nine medical and surgical units in a tertiary-care hospital over an eight-month period. MAIN OUTCOME MEASURE: Adverse drug events identified by the computer-based monitor, by chart review, and by stimulated voluntary report. METHODS: A computer-based monitoring program identified alerts, which were situations suggesting that an ADE might be present (e.g., an order for an antidote such as naloxone). A trained reviewer then examined patients' hospital records to determine whether an ADE had occurred. The results of the computer-based monitoring strategy were compared with two other ADE detection strategies: intensive chart review and stimulated voluntary report by nurses and pharmacists. The monitor and the chart review strategies were independent, and the reviewers were blinded. RESULTS: The computer monitoring strategy identified 2,620 alerts, of which 275 were determined to be ADEs. The chart review found 398 ADEs, whereas voluntary report detected 23. Of the 617 ADEs detected by at least one method, 76 ADEs were detected by both computer monitor and chart review. The computer monitor identified 45 percent; chart review, 65 percent; and voluntary report, 4 percent. The ADEs identified by computer monitor were more likely to be classified as "severe" than were those identified by chart review (51 versus 42 percent, p = .04). The positive predictive value of computer-generated alerts was 16 percent during the first eight weeks of the study; rule modifications increased this to 23 percent in the final eight weeks. The computer strategy required 11 person-hours per week to execute, whereas chart review required 55 person-hours per week and voluntary report strategy required 5. CONCLUSIONS: The computer-based monitor identified fewer ADEs than did chart review but many more ADEs than did stimulated voluntary report. The overlap among the ADEs identified using different methods was small, suggesting that the incidence of ADEs may be higher than previously reported and that different detection methods capture different events. The computer-based monitoring system represents an efficient approach for measuring ADE frequency and gauging the effectiveness of ADE prevention programs.

OBJECTIVE: The purpose of the study is to determine how frequently critical laboratory results (CLRs) occur and how rapidly they are acted upon. A CLR was defined as a result that met either the critical reporting criteria used by the laboratory at Brigham and Women's Hospital or other, more complex criteria. DESIGN: This is a retrospective cohort study in a large academic tertiary-care hospital. MEASUREMENTS: The proportion of chemistry and hematology results obtained in a 13-day period that met the hospital laboratory's critical reporting criteria were calculated. The charts of a stratified random sample of patients with CLRs due to sodium, potassium, and glucose were reviewed to determine the time interval until an appropriate treatment was ordered and the time interval until the critical condition was resolved. RESULTS: In 13 days, 1938 of 201,037 laboratory results (0.96%, or 0.44 per patient-day) met the hospital's critical reporting criteria. In the chart review, 222 CLRs were included in the stratified random sample, and 99 of these met the inclusion criteria. Among these 99 CLRs, the median time interval until an appropriate treatment was ordered was 2.5 hours. This interval was 1.8 hours when the CLR met the laboratory's criteria and a phone call was made, and 2.8 hours when the CLR met more complex criteria not requiring a phone call (p = 0.07). For 27 (27%) of the CLRs, an appropriate treatment was ordered only after five or more hours. The median time until the condition resolved was 14.3 hours: 12.0 hours for CLRs that met the hospital's criteria and 20.9 hours for the CLRs that met the more complex criteria (p = 0.006). CONCLUSION: Although CLRs meeting the hospital's criteria were reported promptly by the laboratory, treatment delays were still common. Results that did not meet the hospital's critical criteria but still represented serious clinical situations were more often associated with treatment delays. Difficulty communicating critical results directly to the responsible caregiver is the likely cause of some delays in treatment. New communications methods, including computer-based technologies, should be explored and tested for their potential to reduce treatment delays and improve clinical care.

BACKGROUND: Vancomycin-resistant enterococci represent an increasingly important cause of nosocomial infections. Minimizing vancomycin use represents a key strategy in preventing the spread of these infections. OBJECTIVE: To determine whether a structured ordering intervention using computerized physician order entry that requires use of a guideline could reduce intravenous vancomycin use. DESIGN: Randomized controlled trial assessing frequency and duration of vancomycin therapy by physicians. PARTICIPANTS AND SETTING: Three hundred ninety-six physicians and 1,798 patients in a tertiary-care teaching hospital. INTERVENTION: Computer screen displaying, at the time of physician order entry, an adaptation of the Centers for Disease Control and Prevention guidelines for appropriate vancomycin use. MAIN OUTCOME MEASURES: The frequency of initiation and renewal of vancomycin therapy as well the duration of therapy prescribed on a per prescriber basis. RESULTS: Compared with the control group, intervention physicians wrote 32 percent fewer orders (11.3 versus 16.7 orders per physician; P = 0.04) and had 28 percent fewer patients for whom they either initiated or renewed an order for vancomycin (7.4 versus 10.3 orders per physician; P = 0.02). In addition, the duration of vancomycin therapy attributable to physicians in the intervention group was 36 percent lower than the duration of therapy prescribed by control physicians (26.5 versus 41.2 days; P = 0.05). Analysis of pharmacy data confirmed a decrease in the overall hospital use of intravenous vancomycin during the study period. CONCLUSION: Implementation of a computerized guideline using physician order entry decreased vancomycin use. Computerized guidelines represent a promising tool for changing prescribing practices.

OBJECTIVE: Inappropriate utilization of diagnostic testing has been well documented. The purpose of this study was to measure the impact of presenting real time, evidence-based critiques about the appropriateness of abdominal radiograph (KUB) orders on physician decision making. DESIGN: Prospective trial where evidence-based critiques were presented to ordering clinicians in two kinds of situations: (1) a KUB was likely to have a low probability of providing useful information, or (2) an alternative view(s) was more appropriate given the clinical circumstance. There were two phases of the trial: Phase 1 was a 9-week period where evidence-based critiques were presented at the time of ordering a KUB, followed by Phase 2, a 19-week period in which orderers were randomized to receive critiques either amended to include both institutional data regarding the utility of the critiques and stronger messages about the lack of utility of the study, or the same critiques as presented in Phase 1, depending upon indication. Based upon the radiologist's report of their interpretation of the exams, the results of the examinations were scored as positive, equivocal, or negative using structured criteria. RESULTS: 299 KUBs in Phase 1 and 385 KUBs in Phase 2 received at least one critique. Cancellation rates of low yield films were low, and were similar in Phase 1 and 2, 8/258 (3%) vs. 10/283 (4%). Compliance with the recommendation for alternative view(s) was higher: 19/104 (38%) in Phase 1 vs. 96/176 (55%) in Phase 2 (p = 0.006). The results differentiated low-yield from non-low-yield films: 5% of low-yield films vs. 20% of non-low-yield films were positive in Phase 2 (p < 0.0001). Surgical physicians were less likely to cancel (p = 0.07) or to change to the suggested view(s) (p < 0.0001) than medical physicians or nurses. CONCLUSIONS: The intervention identified clinical situations in which KUBs appeared to have a low clinical yield. In response to evidence-based critiques, providers were reluctant to cancel their order, but were more willing to change to different views. To reduce the number of inappropriate radiographic films, stronger incentives or interventions may be required.

OBJECTIVES: To evaluate user satisfaction, correlates of satisfaction, and self-reported usage patterns regarding physician order entry (POE) in one hospital. DESIGN: Surveys were sent to physician and nurse POE users from medical and surgical services. RESULTS: The users were generally satisfied with POE (mean = 5.07 on a 1 to 7 scale). The physicians were more satisfied than the nurses, and the medical staff were more satisfied than the surgical staff; satisfaction levels were acceptable (more than 3.50) even in the less satisfied groups. Satisfaction was highly correlated with perceptions about POE's effects on productivity, ease of use, and speed. POE features directed at improving the quality of care were less strongly correlated with satisfaction. The physicians valued POE's off-floor accessibility most, and the nurses valued legibility and accuracy of POE orders most. Some features, such as off-floor ordering, were perceived to be highly useful and reported to be frequently used by the physicians; while other features, such as "quick mode'' ordering and personal order sets, received little self-reported use. CONCLUSIONS: Survey of POE users showed that satisfaction with POE was good. Satisfaction was more correlated with perceptions about POE's effect on productivity than with POE's effect on quality of care. Physicians and nurses constitute two very different types of users, underscoring the importance of involving both physicians and nonphysicians in POE development. The results suggest that development efforts should focus on improving system speed, adding on-line help, and emphasizing quality benefits of POE.

STUDY OBJECTIVE: To evaluate the potential ability of computerized information systems (ISs) to identify and prevent adverse events in medical patients. DESIGN: Clinical descriptions of all 133 adverse events identified through chart review for a cohort of 3,138 medical patients were evaluated by two reviewers. MEASUREMENTS: For each adverse event, three hierarchical levels of IS sophistication were considered: Level 1--demographics, results for all diagnostic tests, and current medications would be available on-line; Level 2--all orders would be entered on-line by physicians; and Level 3--additional clinical data, such as automated problem lists, would be available on-line. Potential for event identification and potential for event prevention were scored by each reviewer according to two distinct sets of event monitors. RESULTS: Of all the adverse events, 53% were judged identifiable using Level 1 information, 58% were judged identifiable using Level 2 information, and 89% were judged identifiable using Level 3 information. The highest-yield event monitors for identifying adverse events were "panic" laboratory results, unexpected transfer to an intensive care unit, and hospital-incurred trauma. With information from Levels 1, 2, and 3, 5%, 13%, and 23% of the adverse events, respectively, were judged preventable. For preventing these adverse events, guided-dose algorithms, drug-laboratory checks, and drug-patient characteristic checks held the most potential.

Gil Kuperman, MD, PhD, David W. Bates, MD, MSc, Reed M. Gardner, PhD, William W. Stead, MD; Standardized Coding of the Medical Problem List, Journal of the Amer