OBJECTIVE: This study aims to compare the effectiveness of 2 ambient AI scribe technologies in reducing physician burnout, improving workflow satisfaction, and enhancing documentation efficiency through a randomized crossover trial. MATERIALS AND METHODS: An open-label randomized crossover trial involving 160 outpatient clinicians was conducted at a tertiary academic medical center. Volunteers were randomized to 2 groups of 80 with 2 crossover periods. We assessed workflow satisfaction (1-7 scale), burnout (Copenhagen Burnout Index), and efficiency metrics (eg, electronic health record time outside scheduled hours, documentation time, etc.). Data was analyzed using Wilcoxon signed-rank tests and generalized linear mixed models. RESULTS: Surveys from 136 respondents were analyzed. Clinicians reported greater improvements in satisfaction with product B (2.51 points on a 7-point scale) compared to product A (1.91 points; mean difference: 0.60, 95% CI: 0.32-0.90). Both tools reduced personal and work burnout scores, but differences between tools were not meaningful. Product B demonstrated greater reductions in average minutes-in-notes per day compared to product A (B - A = -3.19 minutes; 95% CI -4.87 to -1.50). No meaningful differences were observed in pajama time or patient-related burnout. DISCUSSION: Both tools improved workflow satisfaction and reduced burnout, with product B showing superior performance in satisfaction and documentation time. However, efficiency metrics like pajama time were largely unaffected, potentially due to participant selection bias and the study period's timing. CONCLUSION: Product B yielded greater satisfaction and time savings compared to product A, though both tools effectively reduced physician burnout and improved workflow satisfaction.

OBJECTIVES: Large language models (LLMs) are increasingly utilized in healthcare, transforming medical practice through advanced language processing capabilities. However, the evaluation of LLMs predominantly relies on human qualitative assessment, which is time-consuming, resource-intensive, and may be subject to variability and bias. There is a pressing need for quantitative metrics to enable scalable, objective, and efficient evaluation. MATERIALS AND METHODS: We propose a unified evaluation framework that bridges qualitative and quantitative methods to assess LLM performance in healthcare settings. This framework maps evaluation aspects-such as linguistic quality, efficiency, content integrity, trustworthiness, and usefulness-to both qualitative assessments and quantitative metrics. We apply our approach to empirically evaluate the Epic In-Basket feature, which uses LLM to generate patient message replies. RESULTS: The empirical evaluation demonstrates that while Artificial Intelligence (AI)-generated replies exhibit high fluency, clarity, and minimal toxicity, they face challenges with coherence and completeness. Clinicians' manual decision to use AI-generated drafts correlates strongly with quantitative metrics, suggesting that quantitative metrics have the potential to reduce human effort in the evaluation process and make it more scalable. DISCUSSION: Our study highlights the potential of a unified evaluation framework that integrates qualitative and quantitative methods, enabling scalable and systematic assessments of LLMs in healthcare. Automated metrics streamline evaluation and monitoring processes, but their effective use depends on alignment with human judgment, particularly for aspects requiring contextual interpretation. As LLM applications expand, refining evaluation strategies and fostering interdisciplinary collaboration will be critical to maintaining high standards of accuracy, ethics, and regulatory compliance. CONCLUSION: Our unified evaluation framework bridges the gap between qualitative human assessments and automated quantitative metrics, enhancing the reliability and scalability of LLM evaluations in healthcare. While automated quantitative evaluations are not ready to fully replace qualitative human evaluations, they can be used to enhance the process and, with relevant benchmarks derived from the unified framework proposed here, they can be applied to LLM monitoring and evaluation of updated versions of the original technology evaluated using qualitative human standards.

IMPORTANCE: The US healthcare system faces significant challenges, including clinician burnout, operational inefficiencies, and concerns about patient safety. Artificial intelligence (AI), particularly generative AI, has the potential to address these challenges, but its adoption, effectiveness, and barriers to implementation are not well understood. OBJECTIVE: To evaluate the current state of AI adoption in US healthcare systems, assess successes and barriers to implementation during the early generative AI era. DESIGN, SETTING, AND PARTICIPANTS: This cross-sectional survey was conducted in Fall 2024, and included 67 health systems members of the Scottsdale Institute, a collaborative of US non-profit healthcare organizations. Forty-three health systems completed the survey (64% response rate). Respondents provided data on the deployment status and perceived success of 37 AI use cases across 10 categories. MAIN OUTCOMES AND MEASURES: The primary outcomes were the extent of AI use case development, piloting, or deployment, the degree of reported success for AI use cases, and the most significant barriers to adoption. RESULTS: Across the 43 responding health systems, AI adoption and perceptions of success varied significantly. Ambient Notes, a generative AI tool for clinical documentation, was the only use case with 100% of respondents reporting adoption activities, and 53% reported a high degree of success with using AI for Clinical Documentation. Imaging and radiology emerged as the most widely deployed clinical AI use case, with 90% of organizations reporting at least partial deployment, although successes with diagnostic use cases were limited. Similarly, many organizations have deployed AI for clinical risk stratification such as early sepsis detection, but only 38% report high success in this area. Immature AI tools were identified a significant barrier to adoption, cited by 77% of respondents, followed by financial concerns (47%) and regulatory uncertainty (40%). CONCLUSIONS AND RELEVANCE: Ambient Notes is rapidly advancing in US healthcare systems and demonstrating early success. Other AI use cases show varying degrees of adoption and success, constrained by barriers such as immature AI tools, financial concerns, and regulatory uncertainty. Addressing these challenges through robust evaluations, shared strategies, and governance models will be essential to ensure effective integration and adoption of AI into healthcare practice.

OBJECTIVE: The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion. MATERIALS AND METHODS: Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution. RESULTS: An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center. DISCUSSION: By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution. CONCLUSIONS: We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

Artificial intelligence/machine learning models are being rapidly developed and used in clinical practice. However, many models are deployed without a clear understanding of clinical or operational impact and frequently lack monitoring plans that can detect potential safety signals. There is a lack of consensus in establishing governance to deploy, pilot, and monitor algorithms within operational healthcare delivery workflows. Here, we describe a governance framework that combines current regulatory best practices and lifecycle management of predictive models being used for clinical care. Since January 2021, we have successfully added models to our governance portfolio and are currently managing 52 models.

Burnout among healthcare providers has been increasingly recognized as a significant problem.1 The National Academy of Medicine has defined burnout as “a syndrome characterized by high emotional exhaustion, high depersonalization (ie, cynicism), and a low sense of personal accomplishment from work.”2 The Agency for Healthcare Research and Quality similarly defines Burnout as a long-term stress reaction marked by emotional exhaustion, depersonalization, and a lack of sense of personal accomplishment.3 Clinician burnout is both costly and has been associated with reduced job satisfaction, quality and safety of care, and patient health outcomes.4 Burnout is common—affecting between 35% and 54% of U.S. nurses and physicians and between 45% and 60% for medical students and residents.2 Research to date has identified a number of contributing factors as associated with burnout.5 Among these, health information technologies (HITs) are often implicated. Electronic health record (EHR) systems, for example, are often seen as cumbersome to use, failing to fulfill the promise of improved healthcare delivery, and little more than a means of meeting regulatory and billing requirements.6 However, there remains considerable debate in the informatics community as to the actual role health information technologies play in the problem of clinician burnout.7 Existing research suggests that technologies may be confounded with other important causes, including regulatory mandates, clinical volumes, increasing hyperspecialization among healthcare providers, and a mismatch between the incentives driving system designers and purchasers and those driving providers. Regardless of the role information technology plays in clinician burnout, innovative solutions to prevent or mitigate burnout are urgently needed. In this special focus issue of Journal of the American Medical Informatics Association, we target articles evaluating the role that health information technologies have in causing and mitigating burnout, identify confounding factors, and consider informatics and policy-based solutions. This special focus issue is an outgrowth of the 2020 American College of Medical Informatics Symposium, “Clinician Burnout: Is it Informatics’ Fault, and What Can We Do About It?” In parallel with this special issue, the American College of Medical Informatics (ACMI) Symposium also led to the 25x5 initiative,8 to reduce the burden imposed by clinical documentation on healthcare providers in the United States to 25% of its current level within 5 years. The 25x5 initiative in turn resulted in a National Library of Medicine–funded 6-week symposium that concluded in February 2021. Follow-on work will lay out concrete steps that can be taken to reduce burden, create a community of like-minded stakeholders, and will work with key organizations and associations to guide this change. Taken together, the 2020 ACMI symposium, the 25x5 initiative, and this special issue comprise concrete steps the informatics community is taking to address the problem of clinician burnout. This special issue includes 24 articles across the variety of JAMIA formats: Research and Applications (n = 6),9–14 Brief Communications (n = 4),7,15–17 Reviews (n = 5),18–22 and Perspectives (n = 9).23–31 In the following paragraphs, we summarize selected papers reflecting 3 key themes: (1) understanding the relationship between HIT and clinician burnout, (2) emerging HIT approaches to mitigate clinician burnout, and (3) future directions. Several articles in this special focus issue anchor our understanding of the relationship between HIT and clinician burnout. Two review articles, one by Yan et al21 and another by Nguyen et al19 both identified documentation burden, high inbox message volumes, and negative perceptions of EHR functionality and usability as key EHR-related factors most consistently associated with objective measures of provider burnout in the extant literature. Both review articles also identified time spent on EHR after work hours—often called “pajama time”—but not total time spent on EHR, as being associated with burnout. Findings from these review articles point to opportunities for HIT systems to identify clinicians at increased risk of burnout. Baxter et al16 demonstrated that 3 leading EHR vendors currently provide “off-the-shelf” metrics to measure provider activity on the EHR through log data. While further work is needed to harmonize the metrics’ definitions so that meaningful cross-vendor comparisons can be made, these measures are now routinely available to healthcare organizations and informatics researchers. Two articles in this special issue offer practical insights on how these metrics could be used to identify the subset of clinicians at elevated risk of burnout and to target burnout mitigation interventions. Eschenroeder et al15 analyzed data from the KLAS Arch Collaborative and found that physicians who spend 6 or more hours per week performing after-hours charting were more likely to report burnout. Similarly, Peccoralo et al14 found using survey data from a single institution that faculty members who used EHR for more than 90 minutes a day after hours or who spent more than 60 minutes a day performing clerical tasks were more likely to report burnout. Taken together, these findings suggest that risk of burnout for full-time clinicians may rise significantly if they spend more than 60 to 90 minutes per day on the EHR after hours. Articles in this special issue also highlight opportunities to leverage HIT to address the pervasive problem of clinician burnout. Several contributions build on the evidence base for approaches that healthcare organizations could adopt. Lourie et al12 reported that personalized customization and training sessions across 14 specialty and 31 primary care ambulatory care practices led to improved self-reported efficiency and burnout perception. Simpson et al17 found that a 2-week EHR optimization sprint consisting of EHR changes and one-on-one training sessions led to an improvement in clinicians’ satisfaction toward the EHR in a single-specialty practice but did not impact measures of emotional burnout. A qualitative study conducted by Tran et al13 found that medical scribes are commonly used to offload 7 categories of clinical or clerical tasks as a way to a alleviate burnout attributable to the use of HIT. While these approaches require dedicated resources, these articles should help healthcare organizations build the case for these investments. As suboptimal EHR usability has often been cited as a significant contributor to clinician burnout in the United States, the editors of this special focus issue invited key EHR vendors and usability experts to elucidate current approaches to and opportunities for vendors to improve EHR usability. Leading EHR vendors were invited to respond to a semi-structured written survey on how they meet or exceed the 2015 EHR usability (or user-centered design) requirements issued by the ONC (https://www.healthit.gov/test-method/safety-enhanced-design#ccg). Anonymized responses from 4 major vendors (Supplementary Appendix) were sent to usability experts to comment on the strengths and weaknesses adopted by the EHR industry, and to suggest improvement opportunities. When compared to research from 2015 on the usability of EHR products, Hettinger et al23 noted that vendors have increased their adoption and maturity of user-centered design practices. This observation highlighted the ongoing efforts from U.S. federal policy makers, as summarized by Gettinger et al25 to promote HIT usability by implementing usability standards and funding research to examine the efficacy of these policies. However, much work remains. Hettinger et al highlighted the usability reality gap between EHR as designed by vendors and EHR as implemented by each healthcare organization, citing the paucity of the workforce trained to optimally configure and usability or safety test local configurations as a key driver of this gap. Carayon and Salwei24 further pointed out that the path for reducing clinician burnout through improving EHR usability requires a continuous approach, as vendors and their clients need to work together to turn their focus away from technology embedded in work-as-imagined toward sociotechnical systems supporting work-as-done. EHR vendors should also recognize that they can and should partner with informatics innovators to advance EHR usability and mitigate clinician burnout. In a block-randomized study, Semanik et al11 found that problem-oriented summaries of clinical data, built directly into a vendor EHR, allowed clinicians across three academic medical centers to retrieve data faster and with fewer errors. With the use of this tool, clinicians also reported a reduced cognitive load and increased satisfaction. This article by Semanik et al demonstrates how EHR vendors could support efforts to mitigate burnout by spreading and sustaining usability innovations coming from an individual customer across their customer base. So where does the topic of informatics and clinician burnout go from here? Perspectives articles from several ACMI members offer new lenses through which to view, understand, and address HIT-associated clinician burnout. Williams30 contended that moral injury associated with EHR, as defined by EHR use that leads clinicians to transgress deeply held moral beliefs and expectations, may be a hidden contributor to clinician burnout. Weir et al31 further postulated that burnout may be linked to drivers of intrinsic motivation, and that goal-based decision making, sense making, and agency or autonomy should be considered in the design of future technological interventions to mitigate clinician burnout. From a methodological perspective, significant opportunities remain. Moy et al20 pointed out in their scoping review that standard and validated measures of documentation burden are still lacking, which in turn forms a barrier to the rigorous study of documentation burden. Moy et al further called for efforts to operationalize the concept of documentation burden and develop best practices for measurement. Kannampallil et al26 proposed a conceptual framework that would allow the informatics community to build on EHR activity measures evaluated by Baxter et al16 and use technology to assess holistically clinicians’ workload, cognitive burden, and well-being. The editors of this JAMIA special issue recognize that this body of work is but a snapshot of a rapidly growing and evolving topic. New technologies such as ambient voice speech to text, internet of things, natural language processing and machine learning–driven data visualization, and Fast Healthcare Interoperability Resources, as highlighted by Dymek et al28 and Gettinger and Zayas-Cabán,25 may yet open up opportunities to support more meaningful clinician-patient interactions and more efficient workflows. The policy landscape is also constantly changing, as evidenced by the recent simplification in documentation requirements initiated by the Centers for Medicare and Medicaid Services Burden Reduction efforts intended to place “Patients Over Paperwork.”32 As a target for multidisciplinary scientific inquiry, the subject of HIT-associated clinician burnout must continue to evolve through future empirical studies. Its current evidence base remains modest, at best. We therefore encourage readers of this special issue to participate in and accelerate the ongoing work in this area. STR, KZ, and EGP presented the special issue proposal to JAMIA; STR, KZ, and EGP fulfilled Associate Editor duties; EGP fulfilled EIC duties; EGP contributed to instrument design, data collection from vendors and commissioning of invited perspectives articles; and EGP, STR, and KZ contributed to drafting and finalization of the editorial. Supplementary material is available at Journal of the American Medical Informatics Association online. The authors have no relevant conflicts of interest to declare.

OBJECTIVE: While electronic health record (EHR) systems store copious amounts of patient data, aggregating those data across patients can be challenging. Visual analytic tools that integrate with EHR systems allow clinicians to gain better insight and understanding into clinical care and management. We report on our experience building Tableau-based visualizations and integrating them into our EHR system. MATERIALS AND METHODS: Visual analytic tools were created as part of 12 clinician-initiated quality improvement projects. We built the visual analytic tools in Tableau and linked it within our EPIC environment. We identified 5 visual themes that spanned the various projects. To illustrate these themes, we choose 1 exemplary project which aimed to improve obstetric operating room efficiency. RESULTS: Across our 12 projects, we identified 5 visual themes that are integral to project success: scheduling & optimization (in 11/12 projects); provider assessment (10/12); executive assessment (8/12); patient outcomes (7/12); and control and goal charts (2/12). DISCUSSION: Many visualizations share common themes. Identification of these themes has allowed our internal team to be more efficient and directed in developing visualizations for future projects. CONCLUSION: Organizing visual analytics into themes can allow informatics teams to more efficiently provide visual products to clinical collaborators.

The novel coronavirus disease-19 (COVID-19) pandemic has altered our economy, society, and healthcare system. While this crisis has presented the U.S. healthcare delivery system with unprecedented challenges, the pandemic has catalyzed rapid adoption of telehealth, or the entire spectrum of activities used to deliver care at a distance. Using examples reported by U.S. healthcare organizations, including ours, we describe the role that telehealth has played in transforming healthcare delivery during the 3 phases of the U.S. COVID-19 pandemic: (1) stay-at-home outpatient care, (2) initial COVID-19 hospital surge, and (3) postpandemic recovery. Within each of these 3 phases, we examine how people, process, and technology work together to support a successful telehealth transformation. Whether healthcare enterprises are ready or not, the new reality is that virtual care has arrived.

Objective: As available data increases, so does the opportunity to develop risk scores on more refined patient populations. In this paper we assessed the ability to derive a risk score for a patient no-showing to a clinic visit. Methods: Using data from 2 264 235 outpatient appointments we assessed the performance of models built across 14 different specialties and 55 clinics. We used regularized logistic regression models to fit and assess models built on the health system, specialty, and clinic levels. We evaluated fits based on their discrimination and calibration. Results: Overall, the results suggest that a relatively robust risk score for patient no-shows could be derived with an average C-statistic of 0.83 across clinic level models and strong calibration. Moreover, the clinic specific models, even with lower training set sizes, often performed better than the more general models. Examination of the individual models showed that risk factors had different degrees of predictability across the different specialties. Implementation of optimal modeling strategies would lead to capturing an additional 4819 no-shows per-year. Conclusion: Overall, this work highlights both the opportunity for and the importance of leveraging the available electronic health record data to develop more refined risk models.

BACKGROUND AND OBJECTIVE: Physician awareness of the results of tests pending at discharge (TPADs) is poor. We developed an automated system that notifies responsible physicians of TPAD results via secure, network email. We sought to evaluate the impact of this system on self-reported awareness of TPAD results by responsible physicians, a necessary intermediary step to improve management of TPAD results. METHODS: We conducted a cluster-randomized controlled trial at a major hospital affiliated with an integrated healthcare delivery network in Boston, Massachusetts. Adult patients with TPADs who were discharged from inpatient general medicine and cardiology services were assigned to the intervention or usual care arm if their inpatient attending physician and primary care physician (PCP) were both randomized to the same study arm. Patients of physicians randomized to discordant study arms were excluded. We surveyed these physicians 72 h after all TPAD results were finalized. The primary outcome was awareness of TPAD results by attending physicians. Secondary outcomes included awareness of TPAD results by PCPs, awareness of actionable TPAD results, and provider satisfaction. RESULTS: We analyzed data on 441 patients. We sent 441 surveys to attending physicians and 353 surveys to PCPs and received 275 and 152 responses from 83 different attending physicians and 112 different PCPs, respectively (attending physician survey response rate of 63%). Intervention attending physicians and PCPs were significantly more aware of TPAD results (76% vs 38%, adjusted/clustered OR 6.30 (95% CI 3.02 to 13.16), p<0.001; 57% vs 33%, adjusted/clustered OR 3.08 (95% CI 1.43 to 6.66), p=0.004, respectively). Intervention attending physicians tended to be more aware of actionable TPAD results (59% vs 29%, adjusted/clustered OR 4.25 (0.65, 27.85), p=0.13). One hundred and eighteen (85%) and 43 (63%) intervention attending physician and PCP survey respondents, respectively, were satisfied with this intervention. CONCLUSIONS: Automated email notification represents a promising strategy for managing TPAD results, potentially mitigating an unresolved patient safety concern. CLINICAL TRIAL REGISTRATION: ClinicalTrials.gov (NCT01153451).

Usability testing is increasingly being recognized as a way to increase the usability and safety of health information technology (HIT). Medical simulation centers can serve as testing environments for HIT usability studies. We integrated the quality assurance version of our emergency department (ED) electronic health record (EHR) into our medical simulation center and piloted a clinical care scenario in which emergency medicine resident physicians evaluated a simulated ED patient and documented electronically using the ED EHR. Meticulous planning and close collaboration with expert simulation staff was important for designing test scenarios, pilot testing, and running the sessions. Similarly, working with information systems teams was important for integration of the EHR. Electronic tools are needed to facilitate entry of fictitious clinical results while the simulation scenario is unfolding. EHRs can be successfully integrated into existing simulation centers, which may provide realistic environments for usability testing, training, and evaluation of human-computer interactions.

INTRODUCTION: Electronic prescribing systems have often been promoted as a tool for reducing medication errors and adverse drug events. Recent evidence has revealed that adoption of electronic prescribing systems can lead to unintended consequences such as the introduction of new errors. The purpose of this study is to identify and characterize the unrealized potential and residual consequences of electronic prescribing on pharmacy workflow in an outpatient pharmacy. METHODS: A multidisciplinary team conducted direct observations of workflow in an independent pharmacy and semi-structured interviews with pharmacy staff members about their perceptions of the unrealized potential and residual consequences of electronic prescribing systems. We used qualitative methods to iteratively analyze text data using a grounded theory approach, and derive a list of major themes and subthemes related to the unrealized potential and residual consequences of electronic prescribing. RESULTS: We identified the following five themes: Communication, workflow disruption, cost, technology, and opportunity for new errors. These contained 26 unique subthemes representing different facets of our observations and the pharmacy staff's perceptions of the unrealized potential and residual consequences of electronic prescribing. DISCUSSION: We offer targeted solutions to improve electronic prescribing systems by addressing the unrealized potential and residual consequences that we identified. These recommendations may be applied not only to improve staff perceptions of electronic prescribing systems but also to improve the design and/or selection of these systems in order to optimize communication and workflow within pharmacies while minimizing both cost and the potential for the introduction of new errors.

Physicians are often unaware of the results of tests pending at discharge (TPADs). The authors designed and implemented an automated system to notify the responsible inpatient physician of the finalized results of TPADs using secure, network email. The system coordinates a series of electronic events triggered by the discharge time stamp and sends an email to the identified discharging attending physician once finalized results are available. A carbon copy is sent to the primary care physicians in order to facilitate communication and the subsequent transfer of responsibility. Logic was incorporated to suppress selected tests and to limit notification volume. The system was activated for patients with TPADs discharged by randomly selected inpatient-attending physicians during a 6-month pilot. They received approximately 1.6 email notifications per discharged patient with TPADs. Eighty-four per cent of inpatient-attending physicians receiving automated email notifications stated that they were satisfied with the system in a brief survey (59% survey response rate). Automated email notification is a useful strategy for managing results of TPADs.

BACKGROUND: Electronic health record (EHR) adoption is a national priority in the USA, and well-designed EHRs have the potential to improve quality and safety. However, physicians are reluctant to implement EHRs due to financial constraints, usability concerns, and apprehension about unintended consequences, including the introduction of medical errors related to EHR use. The goal of this study was to characterize and describe physicians' attitudes towards three consequences of EHR implementation: (1) the potential for EHRs to introduce new errors; (2) improvements in healthcare quality; and (3) changes in overall physician satisfaction. METHODS: Using data from a 2007 statewide survey of Massachusetts physicians, we conducted multivariate regression analysis to examine relationships between practice characteristics, perceptions of EHR-related errors, perceptions of healthcare quality, and overall physician satisfaction. RESULTS: 30% of physicians agreed that EHRs create new opportunities for error, but only 2% believed their EHR has created more errors than it prevented. With respect to perceptions of quality, there was no significant association between perceptions of EHR-associated errors and perceptions of EHR-associated changes in healthcare quality. Finally, physicians who believed that EHRs created new opportunities for error were less likely be satisfied with their practice situation (adjusted OR 0.49, p=0.001). CONCLUSIONS: Almost one third of physicians perceived that EHRs create new opportunities for error. This perception was associated with lower levels of physician satisfaction.

OBJECTIVE: To determine the effects of a personal health record (PHR)-linked medications module on medication accuracy and safety. DESIGN: From September 2005 to March 2007, we conducted an on-treatment sub-study within a cluster-randomized trial involving 11 primary care practices that used the same PHR. Intervention practices received access to a medications module prompting patients to review their documented medications and identify discrepancies, generating 'eJournals' that enabled rapid updating of medication lists during subsequent clinical visits. MEASUREMENTS: A sample of 267 patients who submitted medications eJournals was contacted by phone 3 weeks after an eligible visit and compared with a matched sample of 274 patients in control practices that received a different PHR-linked intervention. Two blinded physician adjudicators determined unexplained discrepancies between documented and patient-reported medication regimens. The primary outcome was proportion of medications per patient with unexplained discrepancies. RESULTS: Among 121,046 patients in eligible practices, 3979 participated in the main trial and 541 participated in the sub-study. The proportion of medications per patient with unexplained discrepancies was 42% in the intervention arm and 51% in the control arm (adjusted OR 0.71, 95% CI 0.54 to 0.94, p=0.01). The number of unexplained discrepancies per patient with potential for severe harm was 0.03 in the intervention arm and 0.08 in the control arm (adjusted RR 0.31, 95% CI 0.10 to 0.92, p=0.04). CONCLUSIONS: When used, concordance between documented and patient-reported medication regimens and reduction in potentially harmful medication discrepancies can be improved with a PHR medication review tool linked to the provider's medical record. TRIAL REGISTRATION NUMBER: This study was registered at ClinicalTrials.gov (NCT00251875).

Actionable reminders (electronic reminders linked to computerized order entry) might improve care by facilitating direct ordering of recommended tests. The authors implemented four enhanced actionable reminders targeting performance of annual mammography, one-time bone-density screening, and diabetic testing. There was no difference in rates of appropriate testing between the four intervention and four matched, control primary care clinics for screening mammography (OR 0.81, 95% CI 0.64 to 1.02), bone-density exams (OR 1.29, 95% CI 0.82 to 2.02), HbA1c monitoring (OR 0.91, 95% CI 0.58 to 1.42) and LDL cholesterol monitoring (OR 1.40, 95% CI 0.76 to 2.59). Of the survey respondents, 79% almost never used the system or were unaware of the functionality. In the 9/228 (3.9%) cases with indirect evidence of mammography reminder use, there was a significantly lower proportion with test performance. Our actionable reminders did not improve receipt of overdue testing, potentially due to limitations of workflow integration.

OBJECTIVE: To report the frequency, types, and causes of errors associated with outpatient computer-generated prescriptions, and to develop a framework to classify these errors to determine which strategies have greatest potential for preventing them. MATERIALS AND METHODS: This is a retrospective cohort study of 3850 computer-generated prescriptions received by a commercial outpatient pharmacy chain across three states over 4 weeks in 2008. A clinician panel reviewed the prescriptions using a previously described method to identify and classify medication errors. Primary outcomes were the incidence of medication errors; potential adverse drug events, defined as errors with potential for harm; and rate of prescribing errors by error type and by prescribing system. RESULTS: Of 3850 prescriptions, 452 (11.7%) contained 466 total errors, of which 163 (35.0%) were considered potential adverse drug events. Error rates varied by computerized prescribing system, from 5.1% to 37.5%. The most common error was omitted information (60.7% of all errors). DISCUSSION: About one in 10 computer-generated prescriptions included at least one error, of which a third had potential for harm. This is consistent with the literature on manual handwritten prescription error rates. The number, type, and severity of errors varied by computerized prescribing system, suggesting that some systems may be better at preventing errors than others. CONCLUSIONS: Implementing a computerized prescribing system without comprehensive functionality and processes in place to ensure meaningful system use does not decrease medication errors. The authors offer targeted recommendations on improving computerized prescribing systems to prevent errors.

Electronic health records (EHRs) and EHR-connected patient portals offer patient-provider collaboration tools for visit-based care. During a randomized controlled trial, primary care patients completed pre-visit electronic journals (eJournals) containing EHR-based medication, allergies, and diabetes (study arm 1) or health maintenance, personal history, and family history (study arm 2) topics to share with their provider. Assessment with surveys and usage data showed that among 2027 patients invited to complete an eJournal, 70.3% submitted one and 71.1% of submitters had one opened by their provider. Surveyed patients reported they felt more prepared for the visit (55.9%) and their provider had more accurate information about them (58.0%). More arm 1 versus arm 2 providers reported that eJournals were visit-time neutral (100% vs 53%; p<0.013), helpful to patients in visit preparation (66% vs 20%; p=0.082), and would recommend them to colleagues (78% vs 22%; p=0.0143). eJournal integration into practice warrants further study.

Technology has great potential to reduce medication errors in hospitals. This case report describes barriers to, and facilitators of, the implementation of a pharmacy bar code scanning system to reduce medication dispensing errors at a large academic medical center. Ten pharmacy staff were interviewed about their experiences during the implementation. Interview notes were iteratively reviewed to identify common themes. The authors identified three main barriers to pharmacy bar code scanning system implementation: process (training requirements and process flow issues), technology (hardware, software, and the role of vendors), and resistance (communication issues, changing roles, and negative perceptions about technology). The authors also identified strategies to overcome these barriers. Adequate training, continuous improvement, and adaptation of workflow to address one's own needs mitigated process barriers. Ongoing vendor involvement, acknowledgment of technology limitations, and attempts to address them were crucial in overcoming technology barriers. Staff resistance was addressed through clear communication, identifying champions, emphasizing new information provided by the system, and facilitating collaboration.

The National Resource Center for Health Information Technology (NRC) was formed in the fall of 2004 as part of the Agency for Healthcare Research and Quality (AHRQ) health IT portfolio to support its grantees. One of the core functions of the NRC was to assist grantees in their evaluation efforts of Health IT. This manuscript highlights some common challenges experienced by health IT project teams at nonacademic institutions, including inappropriately scoped and resourced evaluation efforts, inappropriate choice of metrics, inadequate planning for data collection and analysis, and lack of consideration of qualitative methodologies. Many of these challenges can be avoided or overcome. The strategies adopted by various AHRQ grantees and the lessons learned from their projects should become part of the toolset for current and future implementers of health IT as the nation moves rapidly towards its widespread adoption.

OBJECTIVE Electronic health records (EHRs) have potential to improve quality and safety, but many physicians do not use these systems to full capacity. The objective of this study was to determine whether this usage gap is narrowing over time. DESIGN Follow-up mail survey of 1,144 physicians in Massachusetts who completed a 2005 survey. MEASUREMENTS Adoption of EHRs and availability and use of 10 EHR functions. RESULTS The response rate was 79.4%. In 2007, 35% of practices had EHRs, up from 23% in 2005. Among practices with EHRs, there was little change between 2005 and 2007 in the availability of nine of ten EHR features; the notable exception was electronic prescribing, reported as available in 44.7% of practices with EHRs in 2005 and 70.8% in 2007. Use of EHR functions changed inconsequentially, with more than one out of five physicians not using each available function regularly in both 2005 and 2007. Only electronic prescribing increased substantially: in 2005, 19.9% of physicians with this function available used it most or all the time, compared with 42.6% in 2007 (p < 0.001). CONCLUSIONS By 2007, more than one third of practices in Massachusetts reported having EHRs; the availability and use of electronic prescribing within these systems has increased. In contrast, physicians reported little change in the availability and use of other EHR functions. System refinements, certification efforts, and health policies, including standards development, should address the gaps in both EHR adoption and the use of key functions.

This Viewpoint paper has grown out of a presentation at the American College of Medical Informatics 2007 Winter Symposium, the resulting discussion, and several activities that have coalesced around an issue that most informaticians accept as true but is not commonly considered during the implementation of Electronic Health Records (EHR) outside of academia or research institutions. Successful EHR implementation is facilitated and sometimes determined by formative evaluation, usually focusing on process rather than outcomes. With greater federal funding for the implementation of electronic health record systems in health care organizations unfamiliar with research protocols, the need for formative evaluation assistance is growing. Such assistance, in the form of tools and protocols necessary to do formative evaluation and resulting in successful EHR implementations, should be provided by practicing medical informaticians.

BACKGROUND: Computerized provider order entry (CPOE) systems can help hospitals improve health care quality, but they can also introduce new problems. The extent to which hospitals experience unintended consequences of CPOE, which include more than errors, has not been quantified in prior research. OBJECTIVE: To discover the extent and importance of unintended adverse consequences related to CPOE implementation in U.S. hospitals. DESIGN, SETTING, AND PARTICIPANTS: Building on a prior qualitative study involving fieldwork at five hospitals, we developed and then administered a telephone survey concerning the extent and importance of CPOE-related unintended adverse consequences to representatives from 176 hospitals in the U.S. that have CPOE. MEASUREMENTS: Self report by key informants of the extent and level of importance to the overall function of the hospital of eight types of unintended adverse consequences experienced by sites with inpatient CPOE. RESULTS We found that hospitals experienced all eight types of unintended adverse consequences, although respondents identified several they considered more important than others. Those related to new work/more work, workflow, system demands, communication, emotions, and dependence on the technology were ranked as most severe, with at least 72% of respondents ranking them as moderately to very important. Hospital representatives are less sure about shifts in the power structure and CPOE as a new source of errors. There is no relation between kinds of unintended consequences and number of years CPOE has been used. Despite the relatively short length of time most hospitals have had CPOE (median five years), it is highly infused, or embedded, within work practice at most of these sites. CONCLUSIONS: The unintended consequences of CPOE are widespread and important to those knowledgeable about CPOE in hospitals. They can be positive, negative, or both, depending on one's perspective, and they continue to exist over the duration of use. Aggressive detection and management of adverse unintended consequences is vital for CPOE success.

OBJECTIVE: Despite emerging evidence that electronic health records (EHRs) can improve the efficiency and quality of medical care, most physicians in office practice in the United States do not currently use an EHR. We sought to measure the correlates of EHR adoption. DESIGN: Mailed survey to a stratified random sample of all medical practices in Massachusetts in 2005, with one physician per practice randomly selected for survey. MEASUREMENTS: EHR adoption rates. RESULTS: The response rate was 71% (1345/1884). Overall, while 45% of physicians were using an EHR, EHRs were present in only 23% of practices. In multivariate analysis, practice size was strongly correlated with EHR adoption; 52% of practices with 7 or more physicians had an EHR, as compared with 14% of solo practices (adjusted odds ratio, 3.66; 95% confidence interval, 2.28-5.87). Hospital-based practices (adjusted odds ratio, 2.44; 95% confidence interval, 1.53-3.91) and practices that teach medical students or residents (adjusted odds ratio, 2.30; 95% confidence interval, 1.60-3.31) were more likely to have an EHR. The most frequently cited barriers to adoption were start-up financial costs (84%), ongoing financial costs (82%), and loss of productivity (81%). CONCLUSIONS: While almost half of physicians in Massachusetts are using an EHR, fewer than one in four practices in Massachusetts have adopted EHRs. Adoption rates are lower in smaller practices, those not affiliated with hospitals, and those that do not teach medical students or residents. Interventions to expand EHR use must address both financial and non-financial barriers, especially among smaller practices.

Confusion about patients' medication regimens during the hospital admission and discharge process accounts for many preventable and serious medication errors. Many organizations have begun to redesign their clinical processes to address this patient safety concern. Partners HealthCare, an integrated delivery network in Boston, Massachusetts, has answered this interdisciplinary challenge by leveraging its multiple outpatient electronic medical records (EMR) and inpatient computerized provider order entry (CPOE) systems to facilitate the process of medication reconciliation. This manuscript describes the design of a novel application and the associated services that aggregate medication data from EMR and CPOE systems so that clinicians can efficiently generate an accurate pre-admission medication list. Information collected with the use of this application subsequently supports the writing of admission and discharge orders by physicians, performance of admission assessment by nurses, and reconciliation of inpatient orders by pharmacists. Results from early pilot testing suggest that this new medication reconciliation process is well accepted by clinicians and has significant potential to prevent medication errors during transitions of care.

The authors developed a novel feature in their clinical information systems, which allows clinicians to request notification about laboratory results. Clinicians who are expecting a particular laboratory result for a particular patient can request a report of the result via an alphanumeric pager as soon as the result is filed into the patient database. This feature has gained popularity and is heavily used in both inpatient and outpatient settings, at a rate of about 2,300 times per month. This event-monitor-based feature illustrates one way that information technology can be applied to improve communication in health care.