OBJECTIVE: Clinical staff often help clinicians review and respond to messages from patients. This study aimed to characterize primary care staff members' experiences with inbox work. MATERIALS AND METHODS: In this qualitative study, we conducted direct observations and focus groups with clinical staff at 4 academic primary care clinics. We used inductive thematic analysis to code the resulting notes and transcripts for themes in staff members' experience with inbox work. RESULTS: Nine medical assistants and 3 nurses participated in the study. Staff described inbox work as fragmented, feeling like an assembly line, requiring frequent communication with other team members to clarify and manage tasks, and requiring navigation of expectations that varied between patients, clinicians, and clinics. Staff described some messages as being more difficult to manage due to how requests were posed, challenges with subsequent communication, and mismatches between data from different sources. Staff also described how tools that structured or automated message management aided inbox work. DISCUSSION: Staff addressed routine messages by following known protocols and appreciated tools that structured their inbox work. However, staff also regularly encountered messages with information that conflicted with clinic records or that contained multiple, redundant, or vague requests. Addressing these messages required additional work to clarify information (ie, data work) and manage resulting tasks (ie, articulation work). CONCLUSION: Clinic workflows and health information technology should support not only the readily standardized work of addressing routine messages but also the more varied work of preparing messages to be addressed in the first place.

INTRODUCTION: Research on how people interact with electronic health records (EHRs) increasingly involves the analysis of metadata on EHR use. These metadata can be recorded unobtrusively and capture EHR use at a scale unattainable through direct observation or self-reports. However, there is substantial variation in how metadata on EHR use are recorded, analyzed and described, limiting understanding, replication, and synthesis across studies. RECOMMENDATIONS: In this perspective, we provide guidance to those working with EHR use metadata by describing 4 common types, how they are recorded, and how they can be aggregated into higher-level measures of EHR use. We also describe guidelines for reporting analyses of EHR use metadata-or measures of EHR use derived from them-to foster clarity, standardization, and reproducibility in this emerging and critical area of research.

OBJECTIVE: The aim of this article is to compare the aims, measures, methods, limitations, and scope of studies that employ vendor-derived and investigator-derived measures of electronic health record (EHR) use, and to assess measure consistency across studies. MATERIALS AND METHODS: We searched PubMed for articles published between July 2019 and December 2021 that employed measures of EHR use derived from EHR event logs. We coded the aims, measures, methods, limitations, and scope of each article and compared articles employing vendor-derived and investigator-derived measures. RESULTS: One hundred and two articles met inclusion criteria; 40 employed vendor-derived measures, 61 employed investigator-derived measures, and 1 employed both. Studies employing vendor-derived measures were more likely than those employing investigator-derived measures to observe EHR use only in ambulatory settings (83% vs 48%, P = .002) and only by physicians or advanced practice providers (100% vs 54% of studies, P < .001). Studies employing vendor-derived measures were also more likely to measure durations of EHR use (P < .001 for 6 different activities), but definitions of measures such as time outside scheduled hours varied widely. Eight articles reported measure validation. The reported limitations of vendor-derived measures included measure transparency and availability for certain clinical settings and roles. DISCUSSION: Vendor-derived measures are increasingly used to study EHR use, but only by certain clinical roles. Although poorly validated and variously defined, both vendor- and investigator-derived measures of EHR time are widely reported. CONCLUSION: The number of studies using event logs to observe EHR use continues to grow, but with inconsistent measure definitions and significant differences between studies that employ vendor-derived and investigator-derived measures.

Recent changes to billing policy have reduced documentation requirements for outpatient notes, providing an opportunity to rethink documentation workflows. While many providers use templates to write notes-whether to insert short phrases or draft entire notes-we know surprisingly little about how these templates are used in practice. In this retrospective cross-sectional study, we observed the templates that primary providers and other members of the care team used to write the provider progress note for 2.5 million outpatient visits across 52 specialties at an academic health center between 2018 and 2020. Templates were used to document 89% of visits, with a median of 2 used per visit. Only 17% of the 100 230 unique templates were ever used by more than one person and most providers had their own full-note templates. These findings suggest template use is frequent but fragmented, complicating template revision and maintenance. Reframing template use as a form of computer programming suggests ways to maintain the benefits of personalization while leveraging standardization to reduce documentation burden.

As healthcare providers have transitioned from paper to electronic health records they have gained access to increasingly sophisticated documentation aids such as custom note templates. However, little is known about how providers use these aids. To address this gap, we examine how 48 ophthalmologists and their staff create and use content-importing phrases — a customizable and composable form of note template — to document office visits across two years. In this case study, we find 1) content-importing phrases were used to document the vast majority of visits (95%), 2) most content imported by these phrases was structured data imported by data-links rather than boilerplate text, and 3) providers primarily used phrases they had created while staff largely used phrases created by other people. We conclude by discussing how framing clinical documentation as end-user programming can inform the design of electronic health records and other documentation systems mixing data and narrative text.

OBJECTIVE: To systematically review published literature and identify consistency and variation in the aims, measures, and methods of studies using electronic health record (EHR) audit logs to observe clinical activities. MATERIALS AND METHODS: In July 2019, we searched PubMed for articles using EHR audit logs to study clinical activities. We coded and clustered the aims, measures, and methods of each article into recurring categories. We likewise extracted and summarized the methods used to validate measures derived from audit logs and limitations discussed of using audit logs for research. RESULTS: Eighty-five articles met inclusion criteria. Study aims included examining EHR use, care team dynamics, and clinical workflows. Studies employed 6 key audit log measures: counts of actions captured by audit logs (eg, problem list viewed), counts of higher-level activities imputed by researchers (eg, chart review), activity durations, activity sequences, activity clusters, and EHR user networks. Methods used to preprocess audit logs varied, including how authors filtered extraneous actions, mapped actions to higher-level activities, and interpreted repeated actions or gaps in activity. Nineteen studies validated results (22%), but only 9 (11%) through direct observation, demonstrating varying levels of measure accuracy. DISCUSSION: While originally designed to aid access control, EHR audit logs have been used to observe diverse clinical activities. However, most studies lack sufficient discussion of measure definition, calculation, and validation to support replication, comparison, and cross-study synthesis. CONCLUSION: EHR audit logs have potential to scale observational research but the complexity of audit log measures necessitates greater methodological transparency and validated standards.

Electronic health record (EHR) log data have shown promise in measuring physician time spent on clinical activities, contributing to deeper understanding and further optimization of the clinical environment. In this article, we propose 7 core measures of EHR use that reflect multiple dimensions of practice efficiency: total EHR time, work outside of work, time on documentation, time on prescriptions, inbox time, teamwork for orders, and an aspirational measure for the amount of undivided attention patients receive from their physicians during an encounter, undivided attention. We also illustrate sample use cases for these measures for multiple stakeholders. Finally, standardization of EHR log data measure specifications, as outlined here, will foster cross-study synthesis and comparative research.

Author(s): Rule, Adam; Birmingham, Amanda; Zuniga, Cristal; Altintas, Ilkay; Huang, Shih-Cheng; Knight, Rob; Moshiri, Niema; Nguyen, Mai H; Rosenthal, Sara Brin; Pérez, Fernando; Rose, Peter W | Editor(s): Lewitter, Fran

Computational notebooks combine code, visualizations, and text in a single document. Researchers, data analysts, and even journalists are rapidly adopting this new medium. We present three studies of how they are using notebooks to document and share exploratory data analyses. In the first, we analyzed over 1 million computational notebooks on GitHub, finding that one in four had no explanatory text but consisted entirely of visualizations or code. In a second study, we examined over 200 academic computational notebooks, finding that although the vast majority described methods, only a minority discussed reasoning or results. In a third study, we interviewed 15 academic data analysts, finding that most considered computational notebooks personal, exploratory, and messy. Importantly, they typically used other media to share analyses. These studies demonstrate a tension between exploration and explanation in constructing and sharing computational notebooks. We conclude with opportunities to encourage explanation in computational media without hindering exploration.

Computational notebooks aim to support collaborative data analysis by combining code, visualizations, and text in a single easily shared document. Yet, as notebooks evolve and grow they often become difficult to navigate or understand, discouraging sharing and reuse. We present the design and evaluation of a Jupyter Notebook extension providing facilities for annotated cell folding. Through a lab study and multi-week deployment we find cell folding aids notebook navigation and comprehension, not only by the original author, but also by collaborators viewing the notebook in a meeting or revising it on their own. However, in some cases cell folding encouraged collaborators to overlook folded sections or spend longer reviewing a notebook before editing it. These findings extend our understanding of code folding's trade-offs to a new medium and demonstrate its benefits for everyday collaboration. We conclude by discussing how dynamic reorganization can support sharing and reuse of computational notebooks.

Resuming suspended activities is fundamental to knowledge work. As activities grow in complexity or fade from memory, they become increasingly difficult to cognitively resume. Motivated by the efficiency of images for cueing autobiographical memory, we conducted 2 studies of how visual histories of computer-mediated activity might aid users in restoring the mental context of suspended activities. In the first exploratory study we demonstrate that users provided a visual history consisting of small thumbnail images of their desktop can reconstruct detailed mental contexts of earlier computer work. In the second observational study we examine methods and cues that web developers, scientific writers, and graphic designers currently use to restore mental context. We find that participants from all three professions engage in situated sensemaking, reconstructing context by generating stories to explain cues in their documents with implicit meaning, such as sudden changes in writing quality or the shape of blocks of code. We identify characteristics of each activity that influence the information, artifacts, and strategies participants used to reconstruct context. We conclude that visual histories can help users reconstruct mental context but that challenges remain in designing lightweight image-based cues that help users reconstruct context for a range of activities.

Creative projects can span weeks, months, or even years. Working on these timescales can be difficult due to the need to restore context - a task's physical, digital, and mental resources - after each break. Prior research on using computers to restore context has focused on digital context, reopening collections of documents or visualizing interactions with a single program. My research explores how visualizing system-wide activity can help people restore the mental context of creative work.

The use of autonomous robots in organizations is expected to increase steadily over the next few decades. Although some empirical work exists that examines how people collaborate with robots, little is known about how to best design interfaces to support operators in understanding aspects of the task or tasks at hand. This paper presents a design investigation to understand how interfaces should be designed to support multi-user, multi-robot teams. Through contextual inquiry, concept generation, and concept evaluation, we determine what operators should see, and with what salience different types of information should be presented. We present our findings through a series of design questions that development teams can use to help define interaction and design interfaces for these systems.