OBJECTIVES: We sought to (1) characterize the process of diagnosing pneumonia in an emergency department (ED) and (2) examine clinician reactions to a clinician-facing diagnostic discordance feedback tool. MATERIALS AND METHODS: We designed a diagnostic feedback tool, using electronic health record data from ED clinicians' patients to establish concordance or discordance between ED diagnosis, radiology reports, and hospital discharge diagnosis for pneumonia. We conducted semistructured interviews with 11 ED clinicians about pneumonia diagnosis and reactions to the feedback tool. We administered surveys measuring individual differences in mindset beliefs, comfort with feedback, and feedback tool usability. We qualitatively analyzed interview transcripts and descriptively analyzed survey data. RESULTS: Thematic results revealed: (1) the diagnostic process for pneumonia in the ED is characterized by diagnostic uncertainty and may be secondary to goals to treat and dispose the patient; (2) clinician diagnostic self-evaluation is a fragmented, inconsistent process of case review and follow-up that a feedback tool could fill; (3) the feedback tool was described favorably, with task and normative feedback harnessing clinician values of high-quality patient care and personal excellence; and (4) strong reactions to diagnostic feedback varied from implicit trust to profound skepticism about the validity of the concordance metric. Survey results suggested a relationship between clinicians' individual differences in learning and failure beliefs, feedback experience, and usability ratings. DISCUSSION AND CONCLUSION: Clinicians value feedback on pneumonia diagnoses. Our results highlight the importance of feedback about diagnostic performance and suggest directions for considering individual differences in feedback tool design and implementation.

OBJECTIVES: To design an interface to support communication of machine learning (ML)-based prognosis for patients with advanced solid tumors, incorporating oncologists' needs and feedback throughout design. MATERIALS AND METHODS: Using an interdisciplinary user-centered design approach, we performed 5 rounds of iterative design to refine an interface, involving expert review based on usability heuristics, input from a color-blind adult, and 13 individual semi-structured interviews with oncologists. Individual interviews included patient vignettes and a series of interfaces populated with representative patient data and predicted survival for each treatment decision point when a new line of therapy (LoT) was being considered. Ongoing feedback informed design decisions, and directed qualitative content analysis of interview transcripts was used to evaluate usability and identify enhancement requirements. RESULTS: Design processes resulted in an interface with 7 sections, each addressing user-focused questions, supporting oncologists to "tell a story" as they discuss prognosis during a clinical encounter. The iteratively enhanced interface both triggered and reflected design decisions relevant when attempting to communicate ML-based prognosis, and exposed misassumptions. Clinicians requested enhancements that emphasized interpretability over explainability. Qualitative findings confirmed that previously identified issues were resolved and clarified necessary enhancements (eg, use months not days) and concerns about usability and trust (eg, address LoT received elsewhere). Appropriate use should be in the context of a conversation with an oncologist. CONCLUSION: User-centered design, ongoing clinical input, and a visualization to communicate ML-related outcomes are important elements for designing any decision support tool enabled by artificial intelligence, particularly when communicating prognosis risk.

OBJECTIVES: (1) Characterize persistent hazards and inefficiencies in inpatient medication administration; (2) Explore cognitive attributes of medication administration tasks; and (3) Discuss strategies to reduce medication administration technology-related hazards. MATERIALS AND METHODS: Interviews were conducted with 32 nurses practicing at 2 urban, eastern and western US health systems. Qualitative analysis using inductive and deductive coding included consensus discussion, iterative review, and coding structure revision. We abstracted hazards and inefficiencies through the lens of risks to patient safety and the cognitive perception-action cycle (PAC). RESULTS: Persistent safety hazards and inefficiencies related to MAT organized around the PAC cycle included: (1) Compatibility constraints create information silos; (2) Missing action cues; (3) Intermittent communication flow between safety monitoring systems and nurses; (4) Occlusion of important alerts by other, less helpful alerts; (5) Dispersed information: Information required for tasks is not collocated; (6) Inconsistent data organization: Mismatch of the display and the user's mental model; (7) Hidden medication administration technologies (MAT) limitations: Inaccurate beliefs about MAT functionality contribute to overreliance on the technology; (8) Software rigidity causes workarounds; (9) Cumbersome dependencies between technology and the physical environment; and (10) Technology breakdowns require adaptive actions. DISCUSSION: Errors might persist in medication administration despite successful Bar Code Medication Administration and Electronic Medication Administration Record deployment for reducing errors. Opportunities to improve MAT require a deeper understanding of high-level reasoning in medication administration, including control over the information space, collaboration tools, and decision support. CONCLUSION: Future medication administration technology should consider a deeper understanding of nursing knowledge work for medication administration.

OBJECTIVE: The US Preventive Services Task Force (USPSTF) requires the estimation of lifetime pack-years to determine lung cancer screening eligibility. Leading electronic health record (EHR) vendors calculate pack-years using only the most recently recorded smoking data. The objective was to characterize EHR smoking data issues and to propose an approach to addressing these issues using longitudinal smoking data. MATERIALS AND METHODS: In this cross-sectional study, we evaluated 16 874 current or former smokers who met USPSTF age criteria for screening (50-80 years old), had no prior lung cancer diagnosis, and were seen in 2020 at an academic health system using the Epic® EHR. We described and quantified issues in the smoking data. We then estimated how many additional potentially eligible patients could be identified using longitudinal data. The approach was verified through manual review of records from 100 subjects. RESULTS: Over 80% of evaluated records had inaccuracies, including missing packs-per-day or years-smoked (42.7%), outdated data (25.1%), missing years-quit (17.4%), and a recent change in packs-per-day resulting in inaccurate lifetime pack-years estimation (16.9%). Addressing these issues by using longitudinal data enabled the identification of 49.4% more patients potentially eligible for lung cancer screening (P < .001). DISCUSSION: Missing, outdated, and inaccurate smoking data in the EHR are important barriers to effective lung cancer screening. Data collection and analysis strategies that reflect changes in smoking habits over time could improve the identification of patients eligible for screening. CONCLUSION: The use of longitudinal EHR smoking data could improve lung cancer screening.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

The psychology of motivation can help us understand the impact of electronic health records (EHRs) on clinician burnout both directly and indirectly. Informatics approaches to EHR usability tend to focus on the extrinsic motivation associated with successful completion of clearly defined tasks in clinical workflows. Intrinsic motivation, which includes the need for autonomy, sense-making, creativity, connectedness, and mastery is not well supported by current designs and workflows. This piece examines existing research on the importance of 3 psychological drives in relation to healthcare technology: goal-based decision-making, sense-making, and agency/autonomy. Because these motives are ubiquitous, foundational to human functioning, automatic, and unconscious, they may be overlooked in technological interventions. The results are increased cognitive load, emotional distress, and unfulfilling workplace environments. Ultimately, we hope to stimulate new research on EHR design focused on expanding functionality to support intrinsic motivation, which, in turn, would decrease burnout and improve care.

OBJECTIVE: The study sought to evaluate a novel electronic health record (EHR) add-on application for chronic disease management that uses an integrated display to decrease user cognitive load, improve efficiency, and support clinical decision making. MATERIALS AND METHODS: We designed a chronic disease management application using the technology framework known as SMART on FHIR (Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources). We used mixed methods to obtain user feedback on a prototype to support ambulatory providers managing chronic obstructive pulmonary disease. Each participant managed 2 patient scenarios using the regular EHR with and without access to our prototype in block-randomized order. The primary outcome was the percentage of expert-recommended ideal care tasks completed. Timing, keyboard and mouse use, and participant surveys were also collected. User experiences were captured using a retrospective think-aloud interview analyzed by concept coding. RESULTS: With our prototype, the 13 participants completed more recommended care (81% vs 48%; P < .001) and recommended tasks per minute (0.8 vs 0.6; P = .03) over longer sessions (7.0 minutes vs 5.4 minutes; P = .006). Keystrokes per task were lower with the prototype (6 vs 18; P < .001). Qualitative themes elicited included the desire for reliable presentation of information which matches participants' mental models of disease and for intuitive navigation in order to decrease cognitive load. DISCUSSION: Participants completed more recommended care by taking more time when using our prototype. Interviews identified a tension between using the inefficient but familiar EHR vs learning to use our novel prototype. Concept coding of user feedback generated actionable insights. CONCLUSIONS: Mixed methods can support the design and evaluation of SMART on FHIR EHR add-on applications by enhancing understanding of the user experience.

If biomedical informatics is a science, we believe it best prospers by the careful application of the scientific method to testable hypotheses. From time to time this may require the use of simplified models to prove cause and effect or the lack thereof. Our experiment applied a controlled model of a clinical encounter and focused on the question of whether the use of an electronic health records system (EHR) on a computer is the cause of decrements in communications. As appropriate for a controlled experiment, cognitive load, except for use of the EHR, was balanced across both study arms. Our findings suggest that EHR use per se is not a problem but an advantage for residents , communications-wise.1 However, as Hauser and Zeng2 suggest, in practice the number and types of tasks that EHRs ask users to perform is far higher than typical with paper charts. These tasks are valuable but add to the cognitive load of the users; thus, in practice the cognitive load from using an EHR system might be far higher than in our controlled study. This does not invalidate the findings of our study or render it “simplistic.” Rather, it suggests that the next plausible hypothesis to test is whether higher user cognitive loads due to specific interface designs and/or additions of new tasks or interruptions are the cause of perceived problems with examination room communications. This hypothesis does have implications for strategies on how to address the issue of the perceived negative effects of computer usage in the examination room. If the problem is not the computer per se but the user’s cognitive load, then strategies such as LEVEL (Let the patient Look-on; Eye-contact; Value the computer; Explain actions; Log off) that focus on integrating the computer into the interview are not enough.3,4 They will be successful only to the degree that they slow care down and thus secondarily reduce cognitive load. Moreover, to answer Hauser and Zeng’s question about “why (would) a physician greets a patient more warmly, when walking into a room with a laptop,” the answer is, “Yes, if it takes less effort to come to understand a patient’s prior history and symptoms, it may be easier to remember to be social in complex environments.” There is an optimistic note in this—better designs for EHRs that reduce cognitive burden for providers may allow patients to have a more pleasant and person-centric experience. The science of informatics requires both carefully controlled experiments and real-world observational studies. To dismiss the structured experiment merely because it is structured is to dismiss an important part of the science of informatics.

BACKGROUND: The effects of electronic health records (EHRs) on doctor-patient communication are unclear. OBJECTIVE: To evaluate the effects of EHR use compared with paper chart use, on novice physicians' communication skills. DESIGN: Within-subjects randomized controlled trial using observed structured clinical examination methods to assess the impact of use of an EHR on communication. SETTING: A large academic internal medicine training program. POPULATION: First-year internal medicine residents. INTERVENTION: Residents interviewed, diagnosed, and initiated treatment of simulated patients using a paper chart or an EHR on a laptop computer. Video recordings of interviews were rated by three trained observers using the Four Habits scale. RESULTS: Thirty-two residents completed the study and had data available for review (61.5% of those enrolled in the residency program). In most skill areas in the Four Habits model, residents performed at least as well using the EHR and were statistically better in six of 23 skills areas (p<0.05). The overall average communication score was better when using an EHR: mean difference 0.254 (95% CI 0.05 to 0.45), p = 0.012, Cohen's d of 0.47 (a moderate effect). Residents scoring poorly (>3 average score) with paper methods (n = 8) had clinically important improvement when using the EHR. LIMITATIONS: This study was conducted in first-year residents in a training environment using simulated patients at a single institution. CONCLUSIONS: Use of an EHR on a laptop computer appears to improve the ability of first-year residents to communicate with patients relative to using a paper chart.