OBJECTIVE: Surgical outcome prediction is challenging but necessary for postoperative management. Current machine learning models utilize pre- and post-op data, excluding intraoperative information in surgical notes. Current models also usually predict binary outcomes even when surgeries have multiple outcomes that require different postoperative management. This study addresses these gaps by incorporating intraoperative information into multimodal models for multiclass glaucoma surgery outcome prediction. MATERIALS AND METHODS: We developed and evaluated multimodal deep learning models for multiclass glaucoma trabeculectomy surgery outcomes using both structured EHR data and free-text operative notes. We compare those to baseline models that use structured EHR data exclusively, or neural network models that leverage only operative notes. RESULTS: The multimodal neural network had the highest performance with a macro AUROC of 0.750 and F1 score of 0.583. It outperformed the baseline machine learning model with structured EHR data alone (macro AUROC of 0.712 and F1 score of 0.486). Additionally, the multimodal model achieved the highest recall (0.692) for hypotony surgical failure, while the surgical success group had the highest precision (0.884) and F1 score (0.775). DISCUSSION: This study shows that operative notes are an important source of predictive information. The multimodal predictive model combining perioperative notes and structured pre- and post-op EHR data outperformed other models. Multiclass surgical outcome prediction can provide valuable insights for clinical decision-making. CONCLUSIONS: Our results show the potential of deep learning models to enhance clinical decision-making for postoperative management. They can be applied to other specialties to improve surgical outcome predictions.

INTRODUCTION: Research on how people interact with electronic health records (EHRs) increasingly involves the analysis of metadata on EHR use. These metadata can be recorded unobtrusively and capture EHR use at a scale unattainable through direct observation or self-reports. However, there is substantial variation in how metadata on EHR use are recorded, analyzed and described, limiting understanding, replication, and synthesis across studies. RECOMMENDATIONS: In this perspective, we provide guidance to those working with EHR use metadata by describing 4 common types, how they are recorded, and how they can be aggregated into higher-level measures of EHR use. We also describe guidelines for reporting analyses of EHR use metadata-or measures of EHR use derived from them-to foster clarity, standardization, and reproducibility in this emerging and critical area of research.

Electronic health record (EHR) log data capture clinical workflows and are a rich source of information to understand variation in practice patterns. Variation in how EHRs are used to document and support care delivery is associated with clinical and operational outcomes, including measures of provider well-being and burnout. Standardized measures that describe EHR use would facilitate generalizability and cross-institution, cross-vendor research. Here, we describe the current state of outpatient EHR use measures offered by various EHR vendors, guided by our prior conceptual work that proposed seven core measures to describe EHR use. We evaluate these measures and other reporting options provided by vendors for maturity and similarity to previously proposed standardized measures. Working toward improved standardization of EHR use measures can enable and accelerate high-impact research on physician burnout and job satisfaction as well as organizational efficiency and patient health.

Recent changes to billing policy have reduced documentation requirements for outpatient notes, providing an opportunity to rethink documentation workflows. While many providers use templates to write notes-whether to insert short phrases or draft entire notes-we know surprisingly little about how these templates are used in practice. In this retrospective cross-sectional study, we observed the templates that primary providers and other members of the care team used to write the provider progress note for 2.5 million outpatient visits across 52 specialties at an academic health center between 2018 and 2020. Templates were used to document 89% of visits, with a median of 2 used per visit. Only 17% of the 100 230 unique templates were ever used by more than one person and most providers had their own full-note templates. These findings suggest template use is frequent but fragmented, complicating template revision and maintenance. Reframing template use as a form of computer programming suggests ways to maintain the benefits of personalization while leveraging standardization to reduce documentation burden.

As healthcare providers have transitioned from paper to electronic health records they have gained access to increasingly sophisticated documentation aids such as custom note templates. However, little is known about how providers use these aids. To address this gap, we examine how 48 ophthalmologists and their staff create and use content-importing phrases — a customizable and composable form of note template — to document office visits across two years. In this case study, we find 1) content-importing phrases were used to document the vast majority of visits (95%), 2) most content imported by these phrases was structured data imported by data-links rather than boilerplate text, and 3) providers primarily used phrases they had created while staff largely used phrases created by other people. We conclude by discussing how framing clinical documentation as end-user programming can inform the design of electronic health records and other documentation systems mixing data and narrative text.

OBJECTIVE: To systematically review published literature and identify consistency and variation in the aims, measures, and methods of studies using electronic health record (EHR) audit logs to observe clinical activities. MATERIALS AND METHODS: In July 2019, we searched PubMed for articles using EHR audit logs to study clinical activities. We coded and clustered the aims, measures, and methods of each article into recurring categories. We likewise extracted and summarized the methods used to validate measures derived from audit logs and limitations discussed of using audit logs for research. RESULTS: Eighty-five articles met inclusion criteria. Study aims included examining EHR use, care team dynamics, and clinical workflows. Studies employed 6 key audit log measures: counts of actions captured by audit logs (eg, problem list viewed), counts of higher-level activities imputed by researchers (eg, chart review), activity durations, activity sequences, activity clusters, and EHR user networks. Methods used to preprocess audit logs varied, including how authors filtered extraneous actions, mapped actions to higher-level activities, and interpreted repeated actions or gaps in activity. Nineteen studies validated results (22%), but only 9 (11%) through direct observation, demonstrating varying levels of measure accuracy. DISCUSSION: While originally designed to aid access control, EHR audit logs have been used to observe diverse clinical activities. However, most studies lack sufficient discussion of measure definition, calculation, and validation to support replication, comparison, and cross-study synthesis. CONCLUSION: EHR audit logs have potential to scale observational research but the complexity of audit log measures necessitates greater methodological transparency and validated standards.

Electronic health record (EHR) log data have shown promise in measuring physician time spent on clinical activities, contributing to deeper understanding and further optimization of the clinical environment. In this article, we propose 7 core measures of EHR use that reflect multiple dimensions of practice efficiency: total EHR time, work outside of work, time on documentation, time on prescriptions, inbox time, teamwork for orders, and an aspirational measure for the amount of undivided attention patients receive from their physicians during an encounter, undivided attention. We also illustrate sample use cases for these measures for multiple stakeholders. Finally, standardization of EHR log data measure specifications, as outlined here, will foster cross-study synthesis and comparative research.

Response: Drs Black and Klubert raise 3 points in their letter about the limitations of using electronic health record (EHR) timestamp data for workflow timing and modeling1: (1) Our methodologies may not generalize to other specialties beyond ophthalmology. We agree that different workflows in other medical specialties may not be accurately represented by EHR timestamps. We note, however, that a key part of our methodology involves careful analysis and mapping of those individual workflows to EHR timestamps before using them. We are pleased that our methods appear to work among 27 different providers within a single ophthalmology department, and hope other researchers will be interested in applying them in other settings. (2) Our methodologies do not time the granular details of the patient encounter. We agree that this is correct, but note that the goal of this publication was not to measure specific EHR activities during a patient encounter. Instead, our goals were to estimate clinical exam times and documentation time, and to identify trainee involvement in the encounter. We are currently extending our research to develop methodologies for measuring more granular activities. (3) Our methodologies reflect provider EHR use, which may not match actual provider activity during the patient encounter. We completely agree with this limitation and noted it in the paper. Again, we emphasize that the purpose of our study was studying the use of EHR timestamps in workflow research regardless of whether the EHR activity took place during the encounter or after it. We agree with Drs Klubert and Black’s statement that improvement in EHR user interfaces and clinical workflow is warranted.2–4 Overall, we have found that the large volume of readily available EHR data is a distinct benefit for workflow studies, if used carefully. While imperfect, it can be used for simulation models testing changes to staffing, clinic workflows, clinic resources, etc., since they evaluate relative rather than absolute comparisons. Similarly, EHR timing data can be used to compare the relative difference in clinic times after changes are implemented, assuming that EHR use patterns remained the same. We appreciate Drs Klubert and Black’s anecdotes of their challenges with EHR timestamps in their letter, and encourage them to publish their findings in the future so we can compare our specific results with theirs.

Objective: Outpatient clinics lack guidance for tackling modern efficiency and productivity demands. Workflow studies require large amounts of timing data that are prohibitively expensive to collect through observation or tracking devices. Electronic health records (EHRs) contain a vast amount of timing data - timestamps collected during regular use - that can be mapped to workflow steps. This study validates using EHR timestamp data to predict outpatient ophthalmology clinic workflow timings at Oregon Health and Science University and demonstrates their usefulness in 3 different studies. Materials and Methods: Four outpatient ophthalmology clinics were observed to determine their workflows and to time each workflow step. EHR timestamps were mapped to the workflow steps and validated against the observed timings. Results: The EHR timestamp analysis produced times that were within 3 min of the observed times for >80% of the appointments. EHR use patterns affected the accuracy of using EHR timestamps to predict workflow times. Discussion: EHR timestamps provided a reasonable approximation of workflow and can be used for workflow studies. They can be used to create simulation models, analyze EHR use, and quantify the impact of trainees on workflow. Conclusion: The secondary use of EHR timestamp data is a valuable resource for clinical workflow studies. Sample timestamp data files and algorithms for processing them are provided and can be used as a template for more studies in other clinical specialties and settings.

Porting applications to new high performance parallel and distributed computing platforms is a challenging task. Since writing parallel code by hand is time consuming and costly, porting codes would ideally be automated by using some parallelization tools and compilers. In this paper, we compare the performance of three parallelization tools and compilers based on the NAS Parallel Benchmark and a CFD application, ARC3D, on the SGI Origin2000 multiprocessor. The tools and compilers compared include: 1) CAPTools: an interactive computer aided parallelization toolkit, 2) Portland Group's HPF compiler, and 3) the MIPSPro FORTRAN compiler available on the Origin2000, with support for shared memory multiprocessing directives and MP runtime library. The tools and compilers are evaluated in four areas: 1) required user interaction, 2) limitations, 3) portability and 4) performance. Based on these results, a discussion on the feasibility of computer-aided parallelization of aerospace applications is presented along with suggestions for future work.

The use of parallel processors has made it possible to execute large scale applications such as finite element analysis. Generally, the speedup is limited by the interprocessor communication used by message-passing multicomputers. The major question addressed by users of message-passing machines is the identification, of the most efficient type of communication scheme for the. particular application. Practical considerations such as the following are frequently neglected: the range of message sizes for which a communication scheme is appropriate, the impact on the subdomain size, the. memory requirements versus the actual memory of the machine, and how the optimal communication method changes as a fixed size problem is scaled to a larger number of processors. We address these practical issues for 2-D finite, element problems executed on the Intel Delta machine.