EDBT 2026 Demo / reviewers in the wild / expert
A Jay Holmgren
dblp:200/4306
· DBLP profile ↗
36ranked-venue papers
16as first author
25since 2021 · last 2025
0000-0002-7939-6831ORCID · corroborated
Domains — the database's venue-derived domains; a paper can count in several
Applied, interdisciplinary, general and emerging computing · 36 · 16 first-author · 25 since 2021
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2025 | Trending in the right direction: critical access hospitals increased adoption of advanced electronic health record functions from 2018 to 2023abstractOBJECTIVES: We analyzed trends in adoption of advanced patient engagement and clinical data analytics functionalities among critical access hospitals (CAHs) and non-CAHs to assess how historical gaps have changed. MATERIALS AND METHODS: We used 2014, 2018, and 2023 data from the American Hospital Association Annual Survey IT Supplement to measure differences in adoption rates (ie, the "adoption gap") of patient engagement and clinical data analytics functionalities across CAHs and non-CAHs. We measured changes over time in CAH and non-CAH adoption of 6 "core" clinical data analytics functionalities, 5 "core" patient engagement functionalities, 5 new patient engagement functionalities, and 3 bulk data export use cases. We constructed 2 composite measures for core functionalities and analyzed adoption for other functionalities individually. RESULTS: Core functionality adoption increased from 21% of CAHs in 2014 to 56% in 2023 for clinical data analytics and 18% to 49% for patient engagement. The CAH adoption gap in both domains narrowed from 2018 to 2023 (both P < .01). More than 90% of all hospitals had adopted viewing and downloading electronic data and clinical notes by 2023. The largest CAH adoption gaps in 2023 were for Fast Healthcare Interoperability Resources (FHIR) bulk export use cases (eg, analytics and reporting: 63% of CAHs, 81% of non-CAHs, P < .001). DISCUSSION: Adoption of advanced electronic health record functionalities has increased for CAHs and non-CAHs, and some adoption gaps have been closed since 2018. However, CAHs may continue to struggle with clinical data analytics and FHIR-based functionalities. CONCLUSION: Some crucial patient engagement functionalities have reached near-universal adoption; however, policymakers should consider programs to support CAHs in closing remaining adoption gaps. Nate C. Apathy, A Jay Holmgren, Paige Nong, Julia Adler-Milstein, Jordan Everson |
J. Am. Medical Informatics Assoc. | 2 |
| 2025 | Differences in physician electronic health record use by telemedicine intensity: evidence from 2 academic medical centersabstractOBJECTIVE: Evaluate the association between telemedicine intensity and ambulatory physician electronic health record (EHR) use following the COVID-19 pandemic. MATERIALS AND METHODS: This retrospective study included ambulatory physicians in 11 specialties at 2 large academic medical centers (Washington University in St Louis [WashU], University of California San Francisco [UCSF]). EHR use measures, including time-based and frequency-based, were analyzed in the post-COVID-19 period (March 1, 2021, through March 7, 2022). Multivariable regression models with 2-way fixed effects were used to assess the association between telemedicine intensity and EHR use. RESULTS: Fully telemedicine physician-weeks were associated with higher EHR (hours per 8 patient scheduled hours; β = 3.2 at WashU, β = 1.4 at UCSF; P < .001) and documentation time (β = 2.7 at WashU, β = 1.4 at UCSF; P < .001). Several differences in discrete EHR-based tasks were observed: fully telemedicine physician-days were associated with lesser ordering, and there were mixed patterns for information seeking and clinical communication tasks. DISCUSSION: Expanded use of telemedicine was associated with significant changes in physician EHR use post-COVID-19 onset. Increased EHR time may suggest a shift in workload, whereas decreased ordering may suggest constraints in virtual care, such as ability to perform physical examination and the reliance on patient-reported symptoms. Institutional differences usage patterns suggest that telemedicine's impact is context-specific and provides opportunities for understanding how to optimize EHRs to support telemedicine. CONCLUSION: Telemedicine shifts physician EHR. Supporting physicians through optimized EHR tools, tailored workflows, and team-based interventions is essential for sustainable virtual care delivery without exacerbating EHR burden. Robert Thombley, Elise Eiden, Sunny S. Lou, Julia Adler-Milstein, Thomas George Kannampallil, A Jay Holmgren |
J. Am. Medical Informatics Assoc. | 7 |
| 2025 | A SNAPpy use of large language models: using large language models to classify treatment plans in pediatric acute otitis mediaabstractBACKGROUND AND SIGNIFICANCE: Acute otitis media (AOM) is a leading cause of pediatric antibiotic overuse. Safety Net Antibiotic Prescriptions (SNAPs) are recommended for antibiotic stewardship but are difficult to identify due to lack of structured documentation. OBJECTIVE: This study validates the accuracy of Versa, a GPT-4o based HIPAA-compliant large language model (LLM), to classify AOM treatment plans from physician notes. METHODS: A retrospective cross-sectional study analyzed pediatric AOM encounters. Multiple prompting strategies were used to classify treatment plans and validated against a representative sample of manual reviews by 2 pediatricians. A locally fine-tuned model, Clinical-Longformer was also trained and tested against Versa and human review. RESULTS: In total, 5707 encounters were included; 374 reviewed manually. Zero-shot accuracy was 97.8%; few-shot accuracy was 85%. Clinical-Longformer achieved 93.3% accuracy. CONCLUSION: Versa effectively identifies AOM treatment plans, providing a cost-efficient quality improvement tracking tool for prescription practice patterns in pediatric antibiotic stewardship efforts. Jessica J. Pourian, Ben Michaels, Anh Vo, A Jay Holmgren, Augusto Garcia-Agundez, Valerie Flaherman |
J. Am. Medical Informatics Assoc. | 4 |
| 2025 | Interoperability of health-related social needs data at US hospitalsabstractOBJECTIVE: To measure hospital engagement in interoperable exchange of health-related social needs (HRSN) data. MATERIALS AND METHODS: This study combined national data from the 2022 American Hospital Association (AHA) Annual Survey, AHA IT Supplement, and the Centers for Medicare and Medicaid Services Impact File. Multivariable logistic regression was used to identify hospital characteristics associated with receiving HRSN data from external organizations. RESULTS: Of 2502 hospitals, 61.4% reported electronically receiving HRSN data from external sources, most commonly from health information exchange organizations. Hospitals participating in accountable care organizations or patient-centered medical homes and hospitals using Epic or Cerner electronic health records (EHRs) were more likely to receive external HRSN data. In contrast, for-profit hospitals and public hospitals were less likely to participate in HRSN data exchange. DISCUSSION: Hospital ownership, participation in value-based care models, and EHR vendor capabilities are important drivers in advancing HRSN data exchange. CONCLUSION: Additional policy and technological support may be needed to enhance HRSN data interoperability. Sahil Sandhu, Michael Liu, Laura M. Gottlieb, A Jay Holmgren, Lisa S. Rotenstein, Matthew S. Pantell |
J. Am. Medical Informatics Assoc. | 4 |
| 2025 | The number of patient scheduled hours resulting in a 40-hour work week by physician specialty and setting: a cross-sectional study using electronic health record event log dataabstractOBJECTIVE: To quantify how many patient scheduled hours would result in a 40-h work week (PSH40) for ambulatory physicians and to determine how PSH40 varies by specialty and practice type. METHODS: We calculated PSH40 for 186 188 ambulatory physicians across 395 organizations from November 2021 through April 2022 stratified by specialty. RESULTS: Median PSH40 for the sample was 33.2 h (IQR: 28.7-36.5). PSH40 was lowest in infectious disease (26.2, IQR: 21.6-31.1), geriatrics (27.2, IQR: 21.5-32.0) and hematology (28.6, IQR: 23.6-32.6) and highest in plastic surgery (35.7, IQR: 32.8-37.7), pain medicine (35.8, IQR: 32.6-37.9) and sports medicine (36.0, IQR: 33.3-38.1). DISCUSSION: Health system leaders and physicians will benefit from data driven and transparent discussions about work hour expectations. The PSH40 measure can also be used to quantify the impact of variations in the clinical care environment on the in-person ambulatory patient care time available to physicians. CONCLUSIONS: PSH40 is a novel measure that can be generated from vendor-derived metrics and used by operational leaders to inform work expectations. It can also support research into the impact of changes in the care environment on physicians' workload and capacity. Christine A. Sinsky, Lisa S. Rotenstein, A Jay Holmgren, Nate C. Apathy |
J. Am. Medical Informatics Assoc. | 3 |
| 2025 | New indices to track interoperability among US hospitalsabstractOBJECTIVES: To develop indices of US hospital interoperability to capture the current state and assess progress over time. MATERIALS AND METHODS: A Technical Expert Panel (TEP) informed selection of items from the American Hospital Association Health IT Supplement survey, which were aggregated into interoperability concepts (components) and then further combined into indices. Indices were refined through psychometric analysis and additional TEP input. Final indices included a "Core Index" measuring adoption of foundational interoperability capabilities, a "Pathfinder Index" representing adoption of advanced interoperability technologies and auxiliary exchange activities, and a "Friction Index" quantifying barriers. The first 2 indices were scored from 0 (no interoperability) to 100 (full interoperability); the Friction Index was scored 0 (no friction) to 100 (maximum friction). We calculated indices annually from 2021 to 2023, stratifying by hospital characteristics. RESULTS: Items within components created reliable and meaningful measures, and associations between components within indices followed the TEP's expectations. Weighted mean scores for the Core (2023), Pathfinder (2022), and Friction (2023) Indices were 61, 57, and 30, respectively. Hospitals with 500+ beds (large), not designated as critical access, in metropolitan areas, and using market leading electronic health records had statistically significant higher mean scores on all indices. Index values also improved modestly over time. DISCUSSION: Hospitals performed best on the Core Index. Given recent policy and programmatic initiatives, we anticipate continued improvement across all indices. CONCLUSION: Ongoing index tracking can inform policy impact evaluations and highlight persistent interoperability disparities across hospitals. Catherine E. Strawley, Julia Adler-Milstein, A Jay Holmgren, Jordan Everson |
J. Am. Medical Informatics Assoc. | 3 |
| 2025 | Imputation of missing aggregate EHR audit log data across individual and multiple organizations
Nate C. Apathy, A Jay Holmgren, Edward R. Melnick, Robert A. McDougal |
J. Biomed. Informatics | 3 |
| 2024 | Consistency is key: documentation distribution and efficiency in primary careabstractOBJECTIVES: We analyzed the degree to which daily documentation patterns in primary care varied and whether specific patterns, consistency over time, and deviations from clinicians' usual patterns were associated with note-writing efficiency. MATERIALS AND METHODS: We used electronic health record (EHR) active use data from the Oracle Cerner Advance platform capturing hourly active documentation time for 498 physicians and advance practice clinicians (eg, nurse practitioners) for 65 152 clinic days. We used k-means clustering to identify distinct daily patterns of active documentation time and analyzed the relationship between these patterns and active documentation time per note. We determined each primary care clinician's (PCC) modal documentation pattern and analyzed how consistency and deviations were related to documentation efficiency. RESULTS: We identified 8 distinct daily documentation patterns; the 3 most common patterns accounted for 80.6% of PCC-days and differed primarily in average volume of documentation time (78.1 minutes per day; 35.4 minutes per day; 144.6 minutes per day); associations with note efficiency were mixed. PCCs with >80% of days attributable to a single pattern demonstrated significantly more efficient documentation than PCCs with lower consistency; for high-consistency PCCs, days that deviated from their usual patterns were associated with less efficient documentation. DISCUSSION: We found substantial variation in efficiency across daily documentation patterns, suggesting that PCC-level factors like EHR facility and consistency may be more important than when documentation occurs. There were substantial efficiency returns to consistency, and deviations from consistent patterns were costly. CONCLUSION: Organizational leaders aiming to reduce documentation burden should pay specific attention to the ability for PCCs to execute consistent documentation patterns day-to-day. Nate C. Apathy, Joshua Biro, A Jay Holmgren |
J. Am. Medical Informatics Assoc. | 3 |
| 2024 | Impact of response bias in three surveys on primary care providers' experiences with electronic health recordsabstractOBJECTIVE: To identify impacts of different survey methodologies assessing primary care physicians' (PCPs') experiences with electronic health records (EHRs), we compared three surveys: the 2022 Continuous Certification Questionnaire (CCQ) from the American Board of Family Medicine, the 2022 University of California San Francisco (UCSF) Physician Health IT Survey, and the 2021 National Electronic Health Records Survey (NEHRS). MATERIALS AND METHODS: We evaluated differences between survey pairs using Rao-Scott corrected chi-square tests, which account for weighting. RESULTS: CCQ received 3991 responses from PCPs (100% response rate), UCSF received 1375 (3.6% response rate), and NEHRS received 858 (18.2% response rate). Substantial, statistically significant differences in demographics were detected across the surveys. CCQ respondents were younger and more likely to work in a health system; NEHRS respondents were more likely to work in private practice; and UCSF respondents disproportionately practiced in larger academic settings. Many EHR experience indicators were similar between CCQ and NEHRS, but CCQ respondents reported higher documentation burden. DISCUSSION: The UCSF approach is unlikely to supply reliable data. Significant demographic differences between CCQ and NEHRS raise response bias concerns, and while there were similarities in some reported EHR experiences, there were important, significant differences. CONCLUSION: Federal EHR policy monitoring and maintenance require reliable data. This test of existing and alternative sources suggest that diversified data sources are necessary to understand physicians' experiences with EHRs and interoperability. Comprehensive surveys administered by specialty boards have the potential to contribute to these efforts, since they are likely to be free of response bias. Nathaniel Hendrix, Natalya Maisel, Jordan Everson, Vaishali Patel 0001, Andrew W. Bazemore, Lisa S. Rotenstein, A Jay Holmgren, Alexander H. Krist, Julia Adler-Milstein, Robert L. Phillips |
J. Am. Medical Informatics Assoc. | 7 |
| 2023 | Behavioral "nudges" in the electronic health record to reduce waste and misuse: 3 interventionsabstractElectronic health records (EHRs) offer decision support in the form of alerts, which are often though not always interruptive. These alerts, though sometimes effective, can come at the cost of high cognitive burden and workflow disruption. Less well studied is the design of the EHR itself-the ordering provider's "choice architecture"-which "nudges" users toward alternatives, sometimes unintentionally toward waste and misuse, but ideally intentionally toward better practice. We studied 3 different workflows at our institution where the existing choice architecture was potentially nudging providers toward erroneous decisions, waste, and misuse in the form of inappropriate laboratory work, incorrectly specified computerized tomographic imaging, and excessive benzodiazepine dosing for imaging-related sedation. We changed the architecture to nudge providers toward better practice and found that the 3 nudges were successful to varying degrees in reducing erroneous decision-making and mitigating waste and misuse. Carrie K. Grouse, Maggie W. Waung, A Jay Holmgren, John Mongan, Aaron B. Neinstein, S. Andrew Josephson, Raman R. Khanna |
J. Am. Medical Informatics Assoc. | 3 |
| 2023 | COVID exacerbated the gender disparity in physician electronic health record inbox burdenabstractThe COVID-19 pandemic was associated with significant changes to the delivery of ambulatory care, including a dramatic increase in patient messages to physicians. While asynchronous messaging is a valuable communication modality for patients, a greater volume of patient messages is associated with burnout and decreased well-being for physicians. Given that women physicians experienced greater electronic health record (EHR) burden and received more patient messages pre-pandemic, there is concern that COVID may have exacerbated this disparity. Using EHR audit log data of ambulatory physicians at an academic medical center, we used a difference-in-differences framework to evaluate the impact of the pandemic on patient message volume and compare differences between men and women physicians. We found patient message volume increased post-COVID for all physicians, and women physicians saw an additional increase compared to men. Our results contribute to the growing evidence of different communication expectations for women physicians that contribute to the gender disparity in EHR burden. Lisa S. Rotenstein, A Jay Holmgren |
J. Am. Medical Informatics Assoc. | 2 |
| 2022 | Documentation dynamics: note composition, burden, and physician efficiency
Nate C. Apathy, Lisa S. Rotenstein, David W. Bates, A Jay Holmgren |
AMIA | 4 |
| 2022 | Variation in Use of Electronic Health Records within and Across Organizations
Dori A. Cross, Nate C. Apathy, A Jay Holmgren |
AMIA | 3 |
| 2022 | Impact of Patient Access to EHR Notes on Clinician EHR Documentation
A Jay Holmgren, Nate C. Apathy |
AMIA | 1 |
| 2022 | Adoption of Advanced IT Functions by Critical Access Hospitals as a Function of Proximity to a Regional Extension Center
Masha Kuznetsova, Nate C. Apathy, A Jay Holmgren |
AMIA | 3 |
| 2022 | Ctrl-C: A Cross-Sectional Study of the EHR Usage Patterns of US Oncology Clinicians
Sumi Sinha, A Jay Holmgren, Julian Hong, Lisa S. Rotenstein |
AMIA | 2 |
| 2022 | Assessing the impact of patient access to clinical notes on clinician EHR documentationabstractRecent policy changes have required health care delivery organizations provide patients electronic access to their clinical notes free of charge. There is concern that this could have an unintended consequence of increased electronic health record (EHR) work as clinicians may feel the need to adapt their documentation practices in light of their notes being accessible to patients, potentially exacerbating EHR-induced clinician burnout. Using a national, longitudinal data set consisting of all ambulatory care physicians and advance practice providers using an Epic Systems EHR, we used an interrupted time-series analysis to evaluate the immediate impact of the policy change on clinician note length and time spent documenting in the EHR. We found no evidence of a change in note length or time spent writing notes following the implementation of the policy, suggesting patient access to clinical notes did not increase documentation workload for clinicians. A Jay Holmgren, Nate C. Apathy |
J. Am. Medical Informatics Assoc. | 1 |
| 2022 | Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record useabstractOBJECTIVE: The COVID-19 pandemic changed clinician electronic health record (EHR) work in a multitude of ways. To evaluate how, we measure ambulatory clinician EHR use in the United States throughout the COVID-19 pandemic. MATERIALS AND METHODS: We use EHR meta-data from ambulatory care clinicians in 366 health systems using the Epic EHR system in the United States from December 2019 to December 2020. We used descriptive statistics for clinician EHR use including active-use time across clinical activities, time after-hours, and messages received. Multivariable regression to evaluate total and after-hours EHR work adjusting for daily volume and organizational characteristics, and to evaluate the association between messages and EHR time. RESULTS: Clinician time spent in the EHR per day dropped at the onset of the pandemic but had recovered to higher than prepandemic levels by July 2020. Time spent actively working in the EHR after-hours showed similar trends. These differences persisted in multivariable models. In-Basket messages received increased compared with prepandemic levels, with the largest increase coming from messages from patients, which increased to 157% of the prepandemic average. Each additional patient message was associated with a 2.32-min increase in EHR time per day (P < .001). DISCUSSION: Clinicians spent more total and after-hours time in the EHR in the latter half of 2020 compared with the prepandemic period. This was partially driven by increased time in Clinical Review and In-Basket messaging. CONCLUSIONS: Reimbursement models and workflows for the post-COVID era should account for these demands on clinician time that occur outside the traditional visit. A Jay Holmgren, N. Lance Downing, Mitchell Tang, Christopher D. Sharp, Christopher A. Longhurst, Robert S. Huckman |
J. Am. Medical Informatics Assoc. | 1 |
| 2022 | Corrigendum to: Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record useabstractJournal of the American Medical Informatics Association, ocab268, https://doi.org/10.1093/jamia/ocab268 In the originally published version of this manuscript, the affiliation of Chris Longhurst was incorrect. Dr Longhurst’s affiliation should read, “Department of Medicine, UC San Diego Health, La Jolla, California, USA”, instead of, “Center for Clinical Informatics and Improvement Research, University of California San Francisco, San Francisco, California, USA.” This error has been corrected. A Jay Holmgren, N. Lance Downing, Mitchell Tang, Christopher D. Sharp, Christopher A. Longhurst, Robert S. Huckman |
J. Am. Medical Informatics Assoc. | 1 |
| 2022 | Association between state-level malpractice environment and clinician electronic health record (EHR) timeabstractOBJECTIVE: Clinicians spend significant time working in the electronic health record (EHR). The US is an outlier in EHR time, suggesting that EHR-related work may be driven in part by the legal environment and threat of malpractice. To assess this, we evaluate the association between state-level malpractice climate and clinician time spent in the EHR. MATERIALS AND METHODS: We use EHR metadata from 351 ambulatory care health systems in the United States using Epic from January-August 2019 combined with state-level data on malpractice incidence and payouts. We used descriptive statistics to measure variation in clinician EHR time, including total EHR time, documentation time per day, and after-hours EHR time per day. Multi-variable regression evaluated the association between clinicians in high malpractice states and EHR use. RESULTS: We found no association between location in a state in the top-quartile of malpractice payouts and time spent in the EHR per day, time spent in the EHR outside of scheduled hours, or time spent documenting per day, except for a subgroup of the clinicians in the highest malpractice specialties, where there was a small increase in EHR time per day (B = 6.08 min, P < 0.001) and time spent documenting notes (B = 2.77 min, P < 0.001). DISCUSSION: State-level differences in malpractice incidence are unlikely to be a significant driver of EHR work for most clinicians. CONCLUSION: Policymakers seeking to address EHR documentation burden should examine burden driven by other socio-technical demands on clinician time, such as billing or quality measurement. A Jay Holmgren, Lisa S. Rotenstein, N. Lance Downing, David W. Bates, Kevin A. Schulman |
J. Am. Medical Informatics Assoc. | 1 |
| 2021 | Spillover Effects from the HITECH Act on Innovation in Medical Informatics
Nate C. Apathy, A Jay Holmgren, Shane M. Greenstein |
AMIA | 2 |
| 2021 | The Impact of Quality Feedback and Reporting on Hospital EHR Medication Safety Improvement
A Jay Holmgren, David W. Bates |
AMIA | 1 |
| 2021 | Assessing the Impact of COVID-19 on Clinician Electronic Health Record Use
A Jay Holmgren, N. Lance Downing, Mitchell Tang, Christopher D. Sharp, Christopher A. Longhurst, Robert S. Huckman |
AMIA | 1 |
| 2021 | A decade post-HITECH: Critical access hospitals have electronic health records but struggle to keep up with other advanced functionsabstractOBJECTIVE: Despite broad electronic health record (EHR) adoption in U.S. hospitals, there is concern that an "advanced use" digital divide exists between critical access hospitals (CAHs) and non-CAHs. We measured EHR adoption and advanced use over time to analyzed changes in the divide. MATERIALS AND METHODS: We used 2008 to 2018 American Hospital Association Information Technology survey data to update national EHR adoption statistics. We stratified EHR adoption by CAH status and measured advanced use for both patient engagement (PE) and clinical data analytics (CDA) domains. We used a linear probability regression for each domain with year-CAH interactions to measure temporal changes in the relationship between CAH status and advanced use. RESULTS: In 2018, 98.3% of hospitals had adopted EHRs; there were no differences by CAH status. A total of 58.7% and 55.6% of hospitals adopted advanced PE and CDA functions, respectively. In both domains, CAHs were less likely to be advanced users: 46.6% demonstrated advanced use for PE and 32.0% for CDA. Since 2015, the advanced use divide has persisted for PE and widened for CDA. DISCUSSION: EHR adoption among hospitals is essentially ubiquitous; however, CAHs still lag behind in advanced use functions critical to improving care quality. This may be rooted in different advanced use needs among CAH patients and lack of access to technical expertise. CONCLUSIONS: The advanced use divide prevents CAH patients from benefitting from a fully digitized healthcare system. To close the widening gap in CDA, policymakers should consider partnering with vendors to develop implementation guides and standards for functions like dashboards and high-risk patient identification algorithms to better support CAH adoption. Nate C. Apathy, A Jay Holmgren, Julia Adler-Milstein |
J. Am. Medical Informatics Assoc. | 2 |
| 2021 | Assessing hospital electronic health record vendor performance across publicly reported quality measuresabstractOBJECTIVE: Little is known regarding variation among electronic health record (EHR) vendors in quality performance. This issue is compounded by selection effects in which high-quality hospitals coalesce to a subset of market leading vendors. We measured hospital performance, stratified by EHR vendor, across 4 quality metrics. MATERIALS AND METHODS: We used data on 1272 hospitals in 2018 across 4 quality measures: Leapfrog Computerized Provider Order Entry/EHR Evaluation, Centers for Medicare and Medicaid Services Hospital Compare Star Ratings, Hospital-Acquired Condition (HAC) score, and Hospital Readmission Reduction Program (HRRP) ratio. We examined score distributions and used multivariable regression to evaluate the association between vendor and score, recovering partial R2 to assess the proportion of quality variation explained by vendor. RESULTS: We found significant variation across and within EHR vendors. The largest vendor, vendor A, had the highest mean score on the Leapfrog Computerized Provider Order Entry/EHR Evaluation and HRRP ratio, vendor G had the highest Hospital Compare score, and vendor F had the highest HAC score. In adjusted models, no vendor was significantly associated with higher performance on more than 2 measures. EHR vendor explained between 1.2% (HAC) and 7.6 (HRRP) of the variation in quality performance. DISCUSSION: No EHR vendor was associated with higher quality across all measures, and the 2 largest vendors were not associated with the highest scores. Only a small fraction of quality variation was explained by EHR vendor choice. CONCLUSIONS: Top performance on quality measures can be achieved with any EHR vendor; much of quality performance is driven by the hospital and how it uses the EHR. A Jay Holmgren, Masha Kuznetsova, David C. Classen, David W. Bates |
J. Am. Medical Informatics Assoc. | 1 |
| 2020 | Learning by Doing: Resident Physicians' Use of Electronic Health Records
A Jay Holmgren, Brenessa Lindeman, Eric W. Ford |
AMIA | 1 |
| 2020 | The tradeoffs between safety and alert fatigue: Data from a national evaluation of hospital medication-related clinical decision supportabstractOBJECTIVE: The study sought to evaluate the overall performance of hospitals that used the Computerized Physician Order Entry Evaluation Tool in both 2017 and 2018, along with their performance against fatal orders and nuisance orders. MATERIALS AND METHODS: We evaluated 1599 hospitals that took the test in both 2017 and 2018 by using their overall percentage scores on the test, along with the percentage of fatal orders appropriately alerted on, and the percentage of nuisance orders incorrectly alerted on. RESULTS: Hospitals showed overall improvement; the mean score in 2017 was 58.1%, and this increased to 66.2% in 2018. Fatal order performance improved slightly from 78.8% to 83.0% (P < .001), though there was almost no change in nuisance order performance (89.0% to 89.7%; P = .43). Hospitals alerting on one or more nuisance orders had a 3-percentage-point increase in their overall score. DISCUSSION: Despite the improvement of overall scores in 2017 and 2018, there was little improvement in fatal order performance, suggesting that hospitals are not targeting the deadliest orders first. Nuisance order performance showed almost no improvement, and some hospitals may be achieving higher scores by overalerting, suggesting that the thresholds for which alerts are fired from are too low. CONCLUSIONS: Although hospitals improved overall from 2017 to 2018, there is still important room for improvement for both fatal and nuisance orders. Hospitals that incorrectly alerted on one or more nuisance orders had slightly higher overall performance, suggesting that some hospitals may be achieving higher scores at the cost of overalerting, which has the potential to cause clinician burnout and even worsen safety. Zoe Co, A Jay Holmgren, David C. Classen, Lisa P. Newmark, Diane L. Seger, Melissa Danforth, David W. Bates |
J. Am. Medical Informatics Assoc. | 2 |
| 2020 | Barriers to hospital electronic public health reporting and implications for the COVID-19 pandemicabstractWe sought to identify barriers to hospital reporting of electronic surveillance data to local, state, and federal public health agencies and the impact on areas projected to be overwhelmed by the COVID-19 pandemic. Using 2018 American Hospital Association data, we identified barriers to surveillance data reporting and combined this with data on the projected impact of the COVID-19 pandemic on hospital capacity at the hospital referral region level. Our results find the most common barrier was public health agencies lacked the capacity to electronically receive data, with 41.2% of all hospitals reporting it. We also identified 31 hospital referral regions in the top quartile of projected bed capacity needed for COVID-19 patients in which over half of hospitals in the area reported that the relevant public health agency was unable to receive electronic data. Public health agencies' inability to receive electronic data is the most prominent hospital-reported barrier to effective syndromic surveillance. This reflects the policy commitment of investing in information technology for hospitals without a concomitant investment in IT infrastructure for state and local public health agencies. A Jay Holmgren, Nate C. Apathy, Julia Adler-Milstein |
J. Am. Medical Informatics Assoc. | 1 |
| 2020 | Barriers to hospital electronic public health reporting and implications for the COVID-19 pandemic: the authors' replyabstractWe appreciate the productive discussion about our article, Barriers to Hospital Electronic Public Health Reporting and Implications for the COVID-19 Pandemic, which serves to advance efforts to strengthen information sharing between hospitals and public health agencies. The letter by Staes et al raises important considerations and contributes to a useful dialogue regarding the current state and barriers to hospital–public health agency electronic data sharing as well as opportunities to increase knowledge of extant public health capabilities and foster more comprehensive utilization of those capabilities. We agree that the current level of interoperability between hospitals and public health agencies is not at an ideal level, even under normal circumstances, and critical infrastructure gaps have been laid bare as a result of the COVID-19 pandemic.1 Our findings should not be interpreted as evidence of fault or sole responsibility on either the hospital or public health side for barriers that impede effective information sharing. Indeed, we are careful to focus our discussion on the likely roots of current challenges, which we trace to federal incentive programs that have focused almost entirely on health care delivery organizations, rather than the public health agencies and other community partners that play a critical role in emergency preparedness, disease monitoring, and efforts to improve population health. Instead, we believe the value in our findings comes in prompting better coordination between public health and clinical partners to address current shortcomings and ensure that robust electronic data exchange is meeting the needs for both clinical and public health organizations. The first step towards this coordination is better understanding of how the other side views the issues. We suspect that, when citing barriers to public health receipt of data, hospitals are not referring specifically to the pure technical capability (which the letter indicates exists at a broad level). As with any interoperability effort, functional interoperability requires the technological capability to send and receive data alongside the nontechnical factors such as data governance, incentives to share electronically, a clear onboarding and testing process, and more. Surveys like the AHA IT Supplement shed some light on where the sticky points may lie but isn’t able to home in on and separate one from another. Given these constraints, the key insight from our study is that 1 side of the exchange—namely, the Chief Information Officers or Chief Medical Information Officers who typically respond to the AHA IT Supplement—perceive some aspect of public health agency ability to receive data as a barrier to effective electronic exchange. Regardless of whether that barrier is technical in nature or related to a socio-technical process such as data governance, public health agencies should be aware that nearly 40% of potential exchange partner hospitals view their ability to receive data electronically as a barrier to effective exchange. Awareness of this perceived barrier—especially if it is inconsistent with barriers perceived by public health agencies—is a critical first step towards resolving outstanding issues and clarifying any misunderstandings. We suggest that 1 possible strategy going forward is for public health agencies and hospitals to publicly list their electronic exchange partners, similar to how health information exchange organizations publicly list participants.2 This may help both clinical and public health organizations better understand who is successfully sharing data, enable both parties to engage in peer learning and best practice dissemination, serve as an accountability mechanism for all parties, and allow researchers to differentiate between stated ability to send and receive data electronically and actual connectivity in practice. Secondarily, national surveys of public health agency informatics infrastructure and capabilities should seek to capture more detailed data than they have historically, which has thus far prevented insight into such basic questions as regional variation in capabilities, much less the geographic or proportional scope of connectivity for a given public health agency.3,4 We applaud public health agencies’ hard work on building electronic case reporting capabilities through platforms such as AIMS. However, it’s important to note that the AIMS service is primarily facilitating data exchange between public health laboratories and public health agencies, not from hospitals or other clinical exchange partners. This underscores the complex nature of interoperability for public health surveillance, which frequently involves local and state agencies establishing and maintaining bidirectional interoperability with multiple exchange partners of many types. Most importantly, we wholeheartedly agree with Staes et al that increasing support for public health agencies to build a more robust informatics infrastructure is a critical policy goal to ensure accurate, reliable data exchange. A Jay Holmgren, Nate C. Apathy, Julia Adler-Milstein |
J. Am. Medical Informatics Assoc. | 1 |
| 2019 | Building Safer EHRs: Hospital Medication Order Safety Performance
A Jay Holmgren, Zoe Co, Lisa P. Newmark, Melissa Danforth, David C. Classen, David W. Bates |
AMIA | 1 |
| 2018 | Tracking National Progress in Hospital Interoperability
A Jay Holmgren, Julia Adler-Milstein |
AMIA | 1 |
| 2018 | Are all certified EHRs created equal? Assessing the relationship between EHR vendor and hospital meaningful use performanceabstractObjective: The federal electronic health record (EHR) certification process was intended to ensure a baseline level of system quality and the ability to support meaningful use criteria. We sought to assess whether there was variation across EHR vendors in the degree to which hospitals using products from those vendors were able to achieve high levels of performance on meaningful use criteria. Materials and Methods: We created a cross-sectional national hospital sample from the Office of the National Coordinator for Health Information Technology EHR Products Used for Meaningful Use Attestation public use file and the Centers for Medicare & Medicaid Services Medicare EHR Incentive Program Eligible Hospitals public use file. We used regression models to assess the relationship between vendor and hospital performance on 6 Stage 2 Meaningful Use criteria, controlling for hospital characteristics. We also calculated how much variation in performance is explained by vendor choice. Results: We found significant associations between specific vendor and level of hospital performance for all 6 meaningful use criteria. Epic was associated with significantly higher performance on 5 of the 6 criteria; relationships for other vendors were mixed, with some associated with significantly worse performance on multiple criteria. EHR vendor choice accounted for between 7% and 34% of performance variation across the 6 criteria. Discussion: A nontrivial proportion of variation in hospital meaningful use performance is explained by vendor choice, and certain vendors are more often associated with better meaningful use performance than others. Our results suggest that policy-makers should improve the certification process by including more "real-world" scenario testing and provider feedback or ratings to reduce this variation. Hospitals can use these results to guide interactions with vendors. Conclusion: Vendor choice accounts for a meaningful proportion of variation in hospital meaningful use performance, and specific vendors are consistently associated with higher or lower performance across criteria. A Jay Holmgren, Julia Adler-Milstein, Jeffrey McCullough |
J. Am. Medical Informatics Assoc. | 1 |
| 2018 | Assessing the impact of health system organizational structure on hospital electronic data sharingabstractObjective: Horizontal consolidation in the hospital industry has gained momentum in the United States despite concerns over rising costs and lower quality. Hospital systems frequently point to potential gains in interoperability and electronic exchange of patient information as consolidation benefits. We sought to assess whether hospitals in different health system structures varied in their interoperable data sharing. Materials and methods: We created a cross-sectional national hospital sample from the 2014 AHA Annual Survey and 2015 IT Supplement. We combined the existing taxonomy of health system organizational forms and the ONC's functionality-based, technology-agnostic definition of interoperability. We used logistic regression models to assess the relationship between health systems' organizational forms and interoperability engagement, controlling for hospital characteristics. Results: We found that interoperability engagement varied greatly across hospitals in different health system structures, with facilities in more centralized health systems more likely to be interoperable. Hospitals in one system type, featuring centralized insurance product development but diverse service offerings across member organizations, had significantly higher odds of being engaged in interoperable data sharing in our multivariate regression results. Discussion: The heterogeneity in health system interoperability engagement indicates that incentives to share data vary greatly across organizational strategies and structures. Our results suggest that horizontal consolidation in the hospital industry may not bring significant gains in interoperability progress unless that consolidation takes a specific business alignment form. Conclusion: Policymakers should be wary of claims that horizontal consolidation will lead to interoperability gains. Future research should explore the specific mechanisms that lead to greater interoperability in certain health system organizational structures. A Jay Holmgren, Eric W. Ford |
J. Am. Medical Informatics Assoc. | 1 |
| 2017 | Advanced Use of EHRs in US Hospitals and the Emergence of a Digital "Use" Divide
Julia Adler-Milstein, A Jay Holmgren |
AMIA | 2 |
| 2017 | Electronic health record adoption in US hospitals: the emergence of a digital "advanced use" divideabstractOBJECTIVE: While most hospitals have adopted electronic health records (EHRs), we know little about whether hospitals use EHRs in advanced ways that are critical to improving outcomes, and whether hospitals with fewer resources - small, rural, safety-net - are keeping up. MATERIALS AND METHODS: Using 2008-2015 American Hospital Association Information Technology Supplement survey data, we measured "basic" and "comprehensive" EHR adoption among hospitals to provide the latest national numbers. We then used new supplement questions to assess advanced use of EHRs and EHR data for performance measurement and patient engagement functions. To assess a digital "advanced use" divide, we ran logistic regression models to identify hospital characteristics associated with high adoption in each advanced use domain. RESULTS: We found that 80.5% of hospitals adopted at least a basic EHR system, a 5.3 percentage point increase from 2014. Only 37.5% of hospitals adopted at least 8 (of 10) EHR data for performance measurement functions, and 41.7% of hospitals adopted at least 8 (of 10) patient engagement functions. Critical access hospitals were less likely to have adopted at least 8 performance measurement functions (odds ratio [OR] = 0.58; P < .001) and at least 8 patient engagement functions (OR = 0.68; P = 0.02). DISCUSSION: While the Health Information Technology for Economic and Clinical Health Act resulted in widespread hospital EHR adoption, use of advanced EHR functions lags and a digital divide appears to be emerging, with critical-access hospitals in particular lagging behind. This is concerning, because EHR-enabled performance measurement and patient engagement are key contributors to improving hospital performance. CONCLUSION: Hospital EHR adoption is widespread and many hospitals are using EHRs to support performance measurement and patient engagement. However, this is not happening across all hospitals. Julia Adler-Milstein, A Jay Holmgren, Peter Kralovec, Chantal Worzala, Talisha Searcy, Vaishali Patel 0001 |
J. Am. Medical Informatics Assoc. | 2 |
| 2016 | U.S. Hospitals Engagement in Core Domains of Interoperability
A Jay Holmgren, Vaishali Patel 0001, Dustin Charles, Julia Adler-Milstein |
AMIA | 1 |