OBJECTIVES: To assess the feasibility and implementation, usability, acceptability and efficacy of virtual reality (VR), and augmented reality (AR) smartphone applications for upskilling care home workers in hand hygiene and to explore underlying learning mechanisms. MATERIALS AND METHODS: Care homes in Northwest England were recruited. We took a mixed-methods and pre-test and post-test approach by analyzing uptake and completion rates of AR, immersive VR or non-immersive VR training, validated and bespoke questionnaires, observations, videos, and interviews. Quantitative data were analyzed descriptively. Qualitative data were analyzed using a combined inductive and deductive approach. RESULTS: Forty-eight care staff completed AR training (n = 19), immersive VR training (n = 21), or non-immersive VR training (n = 8). The immersive VR and AR training had good usability with System Usability Scale scores of 84.40 and 77.89 (of 100), respectively. They had high acceptability, with 95% of staff supporting further use. The non-immersive VR training had borderline poor usability, scoring 67.19 and only 63% would support further use. There was minimal improved knowledge, with an average of 6% increase to the knowledge questionnaire. Average hand hygiene technique scores increased from 4.77 (of 11) to 7.23 after the training. Repeated practice, task realism, feedback and reminding, and interactivity were important learning mechanisms triggered by AR/VR. Feasibility and implementation considerations included managerial support, physical space, providing support, screen size, lagging Internet, and fitting the headset. CONCLUSIONS: AR and immersive VR apps are feasible, usable, and acceptable for delivering training. Future work should explore whether they are more effective than previous training and ensure equity in training opportunities.

OBJECTIVE: Outpatient no-shows have important implications for costs and the quality of care. Predictive models of no-shows could be used to target intervention delivery to reduce no-shows. We reviewed the effectiveness of predictive model-based interventions on outpatient no-shows, intervention costs, acceptability, and equity. MATERIALS AND METHODS: Rapid systematic review of randomized controlled trials (RCTs) and non-RCTs. We searched Medline, Cochrane CENTRAL, Embase, IEEE Xplore, and Clinical Trial Registries on March 30, 2022 (updated on July 8, 2022). Two reviewers extracted outcome data and assessed the risk of bias using ROB 2, ROBINS-I, and confidence in the evidence using GRADE. We calculated risk ratios (RRs) for the relationship between the intervention and no-show rates (primary outcome), compared with usual appointment scheduling. Meta-analysis was not possible due to heterogeneity. RESULTS: We included 7 RCTs and 1 non-RCT, in dermatology (n = 2), outpatient primary care (n = 2), endoscopy, oncology, mental health, pneumology, and an magnetic resonance imaging clinic. There was high certainty evidence that predictive model-based text message reminders reduced no-shows (1 RCT, median RR 0.91, interquartile range [IQR] 0.90, 0.92). There was moderate certainty evidence that predictive model-based phone call reminders (3 RCTs, median RR 0.61, IQR 0.49, 0.68) and patient navigators reduced no-shows (1 RCT, RR 0.55, 95% confidence interval 0.46, 0.67). The effect of predictive model-based overbooking was uncertain. Limited information was reported on cost-effectiveness, acceptability, and equity. DISCUSSION AND CONCLUSIONS: Predictive modeling plus text message reminders, phone call reminders, and patient navigator calls are probably effective at reducing no-shows. Further research is needed on the comparative effectiveness of predictive model-based interventions addressed to patients at high risk of no-shows versus nontargeted interventions addressed to all patients.

OBJECTIVE: The study sought to outline how a clinical risk prediction model for identifying patients at risk of infection is perceived by home care nurses, and to inform how the output of the model could be integrated into a clinical workflow. MATERIALS AND METHODS: This was a qualitative study using semi-structured interviews with 50 home care nurses. Interviews explored nurses' perceptions of clinical risk prediction models, their experiences using them in practice, and what elements are important for the implementation of a clinical risk prediction model focusing on infection. Interviews were audio-taped and transcribed, with data evaluated using thematic analysis. RESULTS: Two themes were derived from the data: (1) informing nursing practice, which outlined how a clinical risk prediction model could inform nurse clinical judgment and be used to modify their care plan interventions, and (2) operationalizing the score, which summarized how the clinical risk prediction model could be incorporated in home care settings. DISCUSSION: The findings indicate that home care nurses would find a clinical risk prediction model for infection useful, as long as it provided both context around the reasons why a patient was deemed to be at high risk and provided some guidance for action. CONCLUSIONS: It is important to evaluate the potential feasibility and acceptability of a clinical risk prediction model, to inform the intervention design and implementation strategy. The results of this study can provide guidance for the development of the clinical risk prediction tool as an intervention for integration in home care settings.

OBJECTIVE: The study sought to identify smartphone apps that support hand hygiene practice and to assess their content, technical and functional features, and quality. A secondary objective was to make design and research recommendations for future apps. MATERIALS AND METHODS: We searched the UK Google Play and Apple App stores for hand hygiene smartphone apps aimed at adults. Information regarding content, technical and functional features was extracted and summarized. Two raters evaluated each app, using the IMS Institute for Healthcare Informatics functionality score and the Mobile App Rating Scale (MARS). RESULTS: A total of 668 apps were identified, with 90 meeting the inclusion criteria. Most (96%) were free to download. The majority (78%) intended to educate or inform or remind users to hand wash (69%), using behavior change techniques such as personalization and prompting practice. Only 20% and 4% named a best practice guideline or had expert involvement in development, respectively. Innovative means of engagement were used in 42% (eg, virtual or augmented reality or geolocation-based reminders). Apps included an average of 2.4 out of 10 of the IMS functionality criteria (range, 0-8). The mean MARS score was 3.2 ± 0.5 out of 5, and 68% had a minimum acceptability score of 3. Two had been tested or trialed. CONCLUSIONS: Although many hand hygiene apps exist, few provide content on best practice. Many did not meet the minimum acceptability criterion for quality or were formally trialed or tested. Research should assess the feasibility and effectiveness of hand hygiene apps (especially within healthcare settings), including when and how they "work." We recommend that future apps to support hand hygiene practice are developed with infection prevention and control experts and align with best practice. Robust research is needed to determine which innovative methods of engagement create "sticky" apps.

OBJECTIVE: This article reports results from a systematic literature review related to the evaluation of data visualizations and visual analytics technologies within the health informatics domain. The review aims to (1) characterize the variety of evaluation methods used within the health informatics community and (2) identify best practices. METHODS: A systematic literature review was conducted following PRISMA guidelines. PubMed searches were conducted in February 2017 using search terms representing key concepts of interest: health care settings, visualization, and evaluation. References were also screened for eligibility. Data were extracted from included studies and analyzed using a PICOS framework: Participants, Interventions, Comparators, Outcomes, and Study Design. RESULTS: After screening, 76 publications met the review criteria. Publications varied across all PICOS dimensions. The most common audience was healthcare providers (n = 43), and the most common data gathering methods were direct observation (n = 30) and surveys (n = 27). About half of the publications focused on static, concentrated views of data with visuals (n = 36). Evaluations were heterogeneous regarding setting and measurements used. DISCUSSION: When evaluating data visualizations and visual analytics technologies, a variety of approaches have been used. Usability measures were used most often in early (prototype) implementations, whereas clinical outcomes were most common in evaluations of operationally-deployed systems. These findings suggest opportunities for both (1) expanding evaluation practices, and (2) innovation with respect to evaluation methods for data visualizations and visual analytics technologies across health settings. CONCLUSION: Evaluation approaches are varied. New studies should adopt commonly reported metrics, context-appropriate study designs, and phased evaluation strategies.

Objective: To explore home care nurses' numeracy and graph literacy and their relationship to comprehension of visualized data. Materials and Methods: A multifactorial experimental design using online survey software. Nurses were recruited from 2 Medicare-certified home health agencies. Numeracy and graph literacy were measured using validated scales. Nurses were randomized to 1 of 4 experimental conditions. Each condition displayed data for 1 of 4 quality indicators, in 1 of 4 different visualized formats (bar graph, line graph, spider graph, table). A mixed linear model measured the impact of numeracy, graph literacy, and display format on data understanding. Results: In all, 195 nurses took part in the study. They were slightly more numerate and graph literate than the general population. Overall, nurses understood information presented in bar graphs most easily (88% correct), followed by tables (81% correct), line graphs (77% correct), and spider graphs (41% correct). Individuals with low numeracy and low graph literacy had poorer comprehension of information displayed across all formats. High graph literacy appeared to enhance comprehension of data regardless of numeracy capabilities. Discussion and Conclusion: Clinical dashboards are increasingly used to provide information to clinicians in visualized format, under the assumption that visual display reduces cognitive workload. Results of this study suggest that nurses' comprehension of visualized information is influenced by their numeracy, graph literacy, and the display format of the data. Individual differences in numeracy and graph literacy skills need to be taken into account when designing dashboard technology.

In a recent JAMIA article, Otte-Trojel et al. 1 present a realist review of patient portals. We commend the authors for using this approach to synthesizing evidence, which is a divergence from traditional systematic review methodology. We believe realist approaches have much to offer the medical informatics community, providing a means to not only determine if health IT interventions provide benefit in terms of outcomes, but to understand why and in what contexts such benefits may occur. However, we feel it is important to address some concerns we have regarding the way in which the authors used realist methods in their review. Our intention is to encourage the authors to expand on this work and to clarify for readers some of the key concepts of realist reviews and how they differ from traditional systematic reviews. In this, we respond to the call of realist evaluators for collective scrutiny of each other’s work to drive the method forward.2 Realist reviews identify theories of how an intervention works, for whom, and in what circumstances, and then test and refine those theories through consideration of primary studies.3 For realists, interventions themselves do not produce outcomes. Rather, interventions offer resources; outcomes depend on how recipients respond to those resources, which will vary according to the context. Realist theories, referred to as Context Mechanism Outcome configurations, explain how different contexts trigger particular mechanisms (the reasoning and responses of recipients) which, in turn, give rise to a particular pattern of outcomes. An important initial stage in a realist review is “theory elicitation,” where reviewers explore the literature with the explicit purpose of identifying theories.4 Otte-Trojel et al. 1 undertook an exploratory review to “identify ways in which patient portals may contribute to health service delivery and patient outcomes.” In reporting the results of this initial review, the authors describe what could be considered a mixture of resources that patient portals might offer (patient access to information and services, patient decision-support) and possible outcomes (coordination of care around the patient; interpersonal continuity of care; health services efficiency; and service convenience to patients and caregivers). However, nothing has been reported about how patients might respond to those resources or how their responses might vary according to the context. Looking at the reference list, it seems the authors drew primarily on journal articles. We suggest that a broader search might have assisted in identifying theories; while journal articles can provide some insight, stakeholders’ theories about how patient portals work are likely to be found in editorials, websites of healthcare providers and patient portal vendors, medical informatics mailing lists, and patient information websites. In a realist review, it is only once the theories have been identified that identification of primary studies takes place. Searching should be purposive and iterative, driven not by the intervention but by the theories.4 For example, if one of the theories suggests that giving patients access to their health record will increase their understanding of their condition and thereby enable them to take a more active role in their care, a relevant search would not only look for primary studies on patient portals but also other interventions that seek to engage patients in their care by increasing their knowledge of their condition. Rather than taking this approach, the search strategy employed by Otte-Trojel et al. 1 is closer to that of a traditional systematic review, with search terms that describe the intervention. Similarly, the choice of outcomes to focus on should be driven where possible by the theories, rather than being based on an existing review as Otte-Trojel et al. 1 have done. In the results section of the paper, the authors describe four mechanisms. We would suggest that the authors’ descriptions of mechanisms focus on resources that the intervention provides, rather than the response of recipients. For example, the mechanism “interpersonal continuity of care” describes how patient portals allow patients to communicate asynchronously with a preferred provider but does not explain what would motivate a patient to do so. In describing context, the authors refer only to organisational context, stating that context at the service unit level and patient-provider level was rarely described in detail. We appreciate that studies do vary in the extent to which context is described. However, an important aspect of context is at the individual level in terms of nature and severity of the patient’s condition. While not identified as a context by the authors, they implicitly acknowledge this as a context when discussing outcomes, pointing to the emphasis in the studies on chronic disease patients and the modest outcomes for patients whose condition is already well controlled. Finally, we feel it is important to acknowledge that different study designs make different contributions to theory testing. From our reading of the paper, Otte-Trojel et al. 1 appear to have treated all studies as potentially providing evidence on contexts, mechanisms, and outcomes. Randomized controlled trials (RCTs) provide information on outcome patterns and, by examining differences in, for example, intervention delivery or patient population, some pointers to likely contextual differences might also be identified. However, RCTs seldom provide information about mechanisms as RCTs are concerned with identifying regularity between a particular intervention and a particular outcome, not with understanding how the intervention changed the reasoning and behavior of recipients. To understand how recipients respond to an intervention, it is necessary to look at qualitative studies, which explore these responses in detail. Realist reviewers would not typically look to qualitative studies for evidence on outcome patterns because such studies rarely explore outcomes and, where they do, small numbers and lack of standardised measurement make it difficult to draw reliable conclusions. We feel Otte-Trojel et al’s 1 findings would have produced more sharply defined Context Mechanism Outcome configurations if they had engaged in a process of knitting together different forms of evidence from different study types as we describe above. Contexts, mechanisms, and outcomes do not just fall out of the primary studies so the realist reviewer has to shuttle between theory and data, integrating the data in imaginative rather than mechanistic ways.5 None.

OBJECTIVES: To evaluate the impact of electronic health record (EHR) implementation on nursing care processes and outcomes. DESIGN: Interrupted time series analysis, 2003-2009. SETTING: A large US not-for-profit integrated health care organization. PARTICIPANTS: 29 hospitals in Northern and Southern California. INTERVENTION: An integrated EHR including computerized physician order entry, nursing documentation, risk assessment tools, and documentation tools. MAIN OUTCOME MEASURES: Percentage of patients with completed risk assessments for hospital acquired pressure ulcers (HAPUs) and falls (process measures) and rates of HAPU and falls (outcome measures). RESULTS: EHR implementation was significantly associated with an increase in documentation rates for HAPU risk (coefficient 2.21, 95% CI 0.67 to 3.75); the increase for fall risk was not statistically significant (0.36; -3.58 to 4.30). EHR implementation was associated with a 13% decrease in HAPU rates (coefficient -0.76, 95% CI -1.37 to -0.16) but no decrease in fall rates (-0.091; -0.29 to 0.11). Irrespective of EHR implementation, HAPU rates decreased significantly over time (-0.16; -0.20 to -0.13), while fall rates did not (0.0052; -0.01 to 0.02). Hospital region was a significant predictor of variation for both HAPU (0.72; 0.30 to 1.14) and fall rates (0.57; 0.41 to 0.72). CONCLUSIONS: The introduction of an integrated EHR was associated with a reduction in the number of HAPUs but not in patient fall rates. Other factors, such as changes over time and hospital region, were also associated with variation in outcomes. The findings suggest that EHR impact on nursing care processes and outcomes is dependent on a number of factors that should be further explored.