The ability of large language models (LLMs) to follow natural language instructions with human-level fluency suggests many opportunities in healthcare to reduce administrative burden and improve quality of care. However, evaluating LLMs on realistic text generation tasks for healthcare remains challenging. Existing question answering datasets for electronic health record (EHR) data fail to capture the complexity of information needs and documentation burdens experienced by clinicians. To address these challenges, we introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes clinician-written reference responses for 303 instructions, and provides 276 longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality of each LLM response. We found high error rates, ranging from 35% (GPT-4) to 68% (MPT-7B-Instruct), and 8.3% drop in accuracy moving from 32k to 2k context lengths for GPT-4. Finally, we report correlations between clinician rankings and automated natural language generation metrics as a way to rank LLMs without human review. MedAlign is provided under a research data use agreement to enable LLM evaluations on tasks aligned with clinician needs and preferences.

OBJECTIVE: Artificial intelligence (AI) and machine learning (ML) enabled healthcare is now feasible for many health systems, yet little is known about effective strategies of system architecture and governance mechanisms for implementation. Our objective was to identify the different computational and organizational setups that early-adopter health systems have utilized to integrate AI/ML clinical decision support (AI-CDS) and scrutinize their trade-offs. MATERIALS AND METHODS: We conducted structured interviews with health systems with AI deployment experience about their organizational and computational setups for deploying AI-CDS at point of care. RESULTS: We contacted 34 health systems and interviewed 20 healthcare sites (58% response rate). Twelve (60%) sites used the native electronic health record vendor configuration for model development and deployment, making it the most common shared infrastructure. Nine (45%) sites used alternative computational configurations which varied significantly. Organizational configurations for managing AI-CDS were distinguished by how they identified model needs, built and implemented models, and were separable into 3 major types: Decentralized translation (n = 10, 50%), IT Department led (n = 2, 10%), and AI in Healthcare (AIHC) Team (n = 8, 40%). DISCUSSION: No singular computational configuration enables all current use cases for AI-CDS. Health systems need to consider their desired applications for AI-CDS and whether investment in extending the off-the-shelf infrastructure is needed. Each organizational setup confers trade-offs for health systems planning strategies to implement AI-CDS. CONCLUSION: Health systems will be able to use this framework to understand strengths and weaknesses of alternative organizational and computational setups when designing their strategy for artificial intelligence.

Using a risk stratification model to guide clinical practice often requires the choice of a cutoff-called the decision threshold-on the model's output to trigger a subsequent action such as an electronic alert. Choosing this cutoff is not always straightforward. We propose a flexible approach that leverages the collective information in treatment decisions made in real life to learn reference decision thresholds from physician practice. Using the example of prescribing a statin for primary prevention of cardiovascular disease based on 10-year risk calculated by the 2013 pooled cohort equations, we demonstrate the feasibility of using real-world data to learn the implicit decision threshold that reflects existing physician behavior. Learning a decision threshold in this manner allows for evaluation of a proposed operating point against the threshold reflective of the community standard of care. Furthermore, this approach can be used to monitor and audit model-guided clinical decision making following model deployment.

OBJECTIVE: To reduce pathogen exposure, conserve personal protective equipment, and facilitate health care personnel work participation in the setting of the COVID-19 pandemic, three affiliated institutions rapidly and independently deployed inpatient telemedicine programs during March 2020. We describe key features and early learnings of these programs in the hospital setting. METHODS: Relevant clinical and operational leadership from an academic medical center, pediatric teaching hospital, and safety net county health system met to share learnings shortly after deploying inpatient telemedicine. A summative analysis of their learnings was re-circulated for approval. RESULTS: All three institutions faced pressure to urgently standup new telemedicine systems while still maintaining secure information exchange. Differences across patient demographics and technological capabilities led to variation in solution design, though key technical considerations were similar. Rapid deployment in each system relied on readily available consumer-grade technology, given the existing familiarity to patients and clinicians and minimal infrastructure investment. Preliminary data from the academic medical center over one month suggested positive adoption with 631 inpatient video calls lasting an average (standard deviation) of 16.5 minutes (19.6) based on inclusion criteria. DISCUSSION: The threat of an imminent surge of COVID-19 patients drove three institutions to rapidly develop inpatient telemedicine solutions. Concurrently, federal and state regulators temporarily relaxed restrictions that would have previously limited these efforts. Strategic direction from executive leadership, leveraging off-the-shelf hardware, vendor engagement, and clinical workflow integration facilitated rapid deployment. CONCLUSION: The rapid deployment of inpatient telemedicine is feasible across diverse settings as a response to the COVID-19 pandemic.

Journal of the American Medical Informatics Association, doi: 10.1093/jamia/ocaa077 The authors would like to issue a correction to our manuscript titled “Rapid Deployment of Inpatient Telemedicine In Response to COVID-19 Across Three Health Systems”. Despite how common it has been to use the terms “master” and “slave” to describe the configuration of hardware, we should have used more culturally appropriate descriptors. We wish to foster a more inclusive scientific community, and thus these terms in Table 1 and the section titled “Implementation of inpatient telemedicine at Stanford Health Care” have been updated to read “hub” and “spoke” instead.