OBJECTIVES: To quantify utilization and impact on documentation time of a large language model-powered ambient artificial intelligence (AI) scribe. MATERIALS AND METHODS: This prospective quality improvement study was conducted at a large academic medical center with 45 physicians from 8 ambulatory disciplines over 3 months. Utilization and documentation times were derived from electronic health record (EHR) use measures. RESULTS: The ambient AI scribe was utilized in 9629 of 17 428 encounters (55.25%) with significant interuser heterogeneity. Compared to baseline, median time per note reduced significantly by 0.57 minutes. Median daily documentation, afterhours, and total EHR time also decreased significantly by 6.89, 5.17, and 19.95 minutes/day, respectively. DISCUSSION: An early pilot of an ambient AI scribe demonstrated robust utilization and reduced time spent on documentation and in the EHR. There was notable individual-level heterogeneity. CONCLUSION: Large language model-powered ambient AI scribes may reduce documentation burden. Further studies are needed to identify which users benefit most from current technology and how future iterations can support a broader audience.

OBJECTIVE: This study evaluates the pilot implementation of ambient AI scribe technology to assess physician perspectives on usability and the impact on physician burden and burnout. MATERIALS AND METHODS: This prospective quality improvement study was conducted at Stanford Health Care with 48 physicians over a 3-month period. Outcome measures included burden, burnout, usability, and perceived time savings. RESULTS: Paired survey analysis (n = 38) revealed large statistically significant reductions in task load (-24.42, p <.001) and burnout (-1.94, p <.001), and moderate statistically significant improvements in usability scores (+10.9, p <.001). Post-survey responses (n = 46) indicated favorable utility with improved perceptions of efficiency, documentation quality, and ease of use. DISCUSSION: In one of the first pilot implementations of ambient AI scribe technology, improvements in physician task load, burnout, and usability were demonstrated. CONCLUSION: Ambient AI scribes like DAX Copilot may enhance clinical workflows. Further research is needed to optimize widespread implementation and evaluate long-term impacts.

The ability of large language models (LLMs) to follow natural language instructions with human-level fluency suggests many opportunities in healthcare to reduce administrative burden and improve quality of care. However, evaluating LLMs on realistic text generation tasks for healthcare remains challenging. Existing question answering datasets for electronic health record (EHR) data fail to capture the complexity of information needs and documentation burdens experienced by clinicians. To address these challenges, we introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes clinician-written reference responses for 303 instructions, and provides 276 longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality of each LLM response. We found high error rates, ranging from 35% (GPT-4) to 68% (MPT-7B-Instruct), and 8.3% drop in accuracy moving from 32k to 2k context lengths for GPT-4. Finally, we report correlations between clinician rankings and automated natural language generation metrics as a way to rank LLMs without human review. MedAlign is provided under a research data use agreement to enable LLM evaluations on tasks aligned with clinician needs and preferences.

OBJECTIVE: To assess the efficacy of interruptive electronic alerts in improving adherence to the American Board of Internal Medicine's Choosing Wisely recommendations to reduce unnecessary laboratory testing. MATERIALS AND METHODS: We administered 5 cluster randomized controlled trials simultaneously, using electronic medical record alerts regarding prostate-specific antigen (PSA) testing, acute sinusitis treatment, vitamin D testing, carotid artery ultrasound screening, and human papillomavirus testing. For each alert, we assigned 5 outpatient clinics to an interruptive alert and 5 were observed as a control. Primary and secondary outcomes were the number of postalert orders per 100 patients at each clinic and number of triggered alerts divided by orders, respectively. Post hoc analysis evaluated whether physicians experiencing interruptive alerts reduced their alert-triggering behaviors. RESULTS: Median postalert orders per 100 patients did not differ significantly between treatment and control groups; absolute median differences ranging from 0.04 to 0.40 for PSA testing. Median alerts per 100 orders did not differ significantly between treatment and control groups; absolute median differences ranged from 0.004 to 0.03. In post hoc analysis, providers receiving alerts regarding PSA testing in men were significantly less likely to trigger additional PSA alerts than those in the control sites (Incidence Rate Ratio 0.12, 95% CI [0.03-0.52]). DISCUSSION: Interruptive point-of-care alerts did not yield detectable changes in the overall rate of undesired orders or the order-to-alert ratio between active and silent sites. Complementary behavioral or educational interventions are likely needed to improve efforts to curb medical overuse. CONCLUSION: Implementation of interruptive alerts at the time of ordering was not associated with improved adherence to 5 Choosing Wisely guidelines. TRIAL REGISTRATION: NCT02709772.

OBJECTIVE: The COVID-19 pandemic changed clinician electronic health record (EHR) work in a multitude of ways. To evaluate how, we measure ambulatory clinician EHR use in the United States throughout the COVID-19 pandemic. MATERIALS AND METHODS: We use EHR meta-data from ambulatory care clinicians in 366 health systems using the Epic EHR system in the United States from December 2019 to December 2020. We used descriptive statistics for clinician EHR use including active-use time across clinical activities, time after-hours, and messages received. Multivariable regression to evaluate total and after-hours EHR work adjusting for daily volume and organizational characteristics, and to evaluate the association between messages and EHR time. RESULTS: Clinician time spent in the EHR per day dropped at the onset of the pandemic but had recovered to higher than prepandemic levels by July 2020. Time spent actively working in the EHR after-hours showed similar trends. These differences persisted in multivariable models. In-Basket messages received increased compared with prepandemic levels, with the largest increase coming from messages from patients, which increased to 157% of the prepandemic average. Each additional patient message was associated with a 2.32-min increase in EHR time per day (P < .001). DISCUSSION: Clinicians spent more total and after-hours time in the EHR in the latter half of 2020 compared with the prepandemic period. This was partially driven by increased time in Clinical Review and In-Basket messaging. CONCLUSIONS: Reimbursement models and workflows for the post-COVID era should account for these demands on clinician time that occur outside the traditional visit.

Journal of the American Medical Informatics Association, ocab268, https://doi.org/10.1093/jamia/ocab268 In the originally published version of this manuscript, the affiliation of Chris Longhurst was incorrect. Dr Longhurst’s affiliation should read, “Department of Medicine, UC San Diego Health, La Jolla, California, USA”, instead of, “Center for Clinical Informatics and Improvement Research, University of California San Francisco, San Francisco, California, USA.” This error has been corrected.

Electronic health record (EHR) log data have shown promise in measuring physician time spent on clinical activities, contributing to deeper understanding and further optimization of the clinical environment. In this article, we propose 7 core measures of EHR use that reflect multiple dimensions of practice efficiency: total EHR time, work outside of work, time on documentation, time on prescriptions, inbox time, teamwork for orders, and an aspirational measure for the amount of undivided attention patients receive from their physicians during an encounter, undivided attention. We also illustrate sample use cases for these measures for multiple stakeholders. Finally, standardization of EHR log data measure specifications, as outlined here, will foster cross-study synthesis and comparative research.

OBJECTIVE: To reduce pathogen exposure, conserve personal protective equipment, and facilitate health care personnel work participation in the setting of the COVID-19 pandemic, three affiliated institutions rapidly and independently deployed inpatient telemedicine programs during March 2020. We describe key features and early learnings of these programs in the hospital setting. METHODS: Relevant clinical and operational leadership from an academic medical center, pediatric teaching hospital, and safety net county health system met to share learnings shortly after deploying inpatient telemedicine. A summative analysis of their learnings was re-circulated for approval. RESULTS: All three institutions faced pressure to urgently standup new telemedicine systems while still maintaining secure information exchange. Differences across patient demographics and technological capabilities led to variation in solution design, though key technical considerations were similar. Rapid deployment in each system relied on readily available consumer-grade technology, given the existing familiarity to patients and clinicians and minimal infrastructure investment. Preliminary data from the academic medical center over one month suggested positive adoption with 631 inpatient video calls lasting an average (standard deviation) of 16.5 minutes (19.6) based on inclusion criteria. DISCUSSION: The threat of an imminent surge of COVID-19 patients drove three institutions to rapidly develop inpatient telemedicine solutions. Concurrently, federal and state regulators temporarily relaxed restrictions that would have previously limited these efforts. Strategic direction from executive leadership, leveraging off-the-shelf hardware, vendor engagement, and clinical workflow integration facilitated rapid deployment. CONCLUSION: The rapid deployment of inpatient telemedicine is feasible across diverse settings as a response to the COVID-19 pandemic.

Journal of the American Medical Informatics Association, doi: 10.1093/jamia/ocaa077 The authors would like to issue a correction to our manuscript titled “Rapid Deployment of Inpatient Telemedicine In Response to COVID-19 Across Three Health Systems”. Despite how common it has been to use the terms “master” and “slave” to describe the configuration of hardware, we should have used more culturally appropriate descriptors. We wish to foster a more inclusive scientific community, and thus these terms in Table 1 and the section titled “Implementation of inpatient telemedicine at Stanford Health Care” have been updated to read “hub” and “spoke” instead.

BACKGROUND: Provider organizations increasingly have the ability to exchange patient health information electronically. Organizational health information exchange (HIE) policy decisions can impact the extent to which external information is readily available to providers, but this relationship has not been well studied. OBJECTIVE: Our objective was to examine the relationship between electronic exchange of patient health information across organizations and organizational HIE policy decisions. We focused on 2 key decisions: whether to automatically search for information from other organizations and whether to require HIE-specific patient consent. METHODS: We conducted a retrospective time series analysis of the effect of automatic querying and the patient consent requirement on the monthly volume of clinical summaries exchanged. We could not assess degree of use or usefulness of summaries, organizational decision-making processes, or generalizability to other vendors. RESULTS: Between 2013 and 2015, clinical summary exchange volume increased by 1349% across 11 organizations. Nine of the 11 systems were set up to enable auto-querying, and auto-querying was associated with a significant increase in the monthly rate of exchange (P = .006 for change in trend). Seven of the 11 organizations did not require patient consent specifically for HIE, and these organizations experienced a greater increase in volume of exchange over time compared to organizations that required consent. CONCLUSIONS: Automatic querying and limited consent requirements are organizational HIE policy decisions that impact the volume of exchange, and ultimately the information available to providers to support optimal care. Future efforts to ensure effective HIE may need to explicitly address these factors.