OBJECTIVES: To examine changes in technology-related errors (TREs), their manifestations and underlying mechanisms at 3 time points after the implementation of computerized provider order entry (CPOE) in an electronic health record; and evaluate the clinical decision support (CDS) available to mitigate the TREs at 5-years post-CPOE. MATERIALS AND METHODS: Prescribing errors (n = 1315) of moderate, major, or serious potential harm identified through review of 35 322 orders at 3 time points (immediately, 1-year, and 4-years post-CPOE) were assessed to identify TREs at a tertiary pediatric hospital. TREs were coded using the Technology-Related Error Mechanism classification. TRE rates, percentage of prescribing errors that were TREs, and mechanism rates were compared over time. Each TRE was tested in the CPOE 5-years post-implementation to assess the availability of CDS to mitigate the error. RESULTS: TREs accounted for 32.5% (n = 428) of prescribing errors; an adjusted rate of 1.49 TREs/100 orders (95% confidence interval [CI]: 1.06, 1.92). At 1-year post-CPOE, the rate of TREs was 40% lower than immediately post (incident rate ratio [IRR]: 0.60; 95% CI: 0.41, 0.89). However, at 4-years post, the TRE rate was not significantly different to baseline (IRR: 0.80; 95% CI: 0.59, 1.08). "New workflows required by the CPOE" was the most frequent TRE mechanism at all time points. CDS was available to mitigate 32.7% of TREs. DISCUSSION: In a pediatric setting, TREs persisted 4-years post-CPOE with no difference in the rate compared to immediately post-CPOE. CONCLUSION: Greater attention is required to address TREs to enhance the safety benefits of systems.

OBJECTIVES: Falls pose a significant challenge in residential aged care facilities (RACFs). Existing falls prediction tools perform poorly and fail to capture evolving risk factors. We aimed to develop and internally validate dynamic fall risk prediction models and create point-based scoring systems for residents with and without dementia. MATERIALS AND METHODS: A longitudinal cohort study using electronic data from 27 RACFs in Sydney, Australia. The study included 5492 permanent residents, with a 70%-30% split for training and validation. The outcome measure was the incidence of falls. We tracked residents for 60 months, using monthly landmarks with 1-month prediction windows. We employed landmarking dynamic prediction for model development, a time-dependent area under receiver operating characteristics curve (AUROCC) for model evaluations, and a regression coefficient approach to create point-based scoring systems. RESULTS: The model identified 15 independent predictors of falls in dementia and 12 in nondementia cohorts. Falls history was the key predictor of subsequent falls in both dementia (HR 4.75, 95% CI, 4.45-5.06) and nondementia cohorts (HR 4.20, 95% CI, 3.87-4.57). The AUROCC across landmarks ranged from 0.67 to 0.87 for dementia and from 0.66 to 0.86 for nondementia cohorts but generally remained between 0.75 and 0.85 in both cohorts. The total point risk score ranged from -2 to 57 for dementia and 0 to 52 for nondementia cohorts. DISCUSSION: Our novel risk prediction models and scoring systems provide timely person-centered information for continuous monitoring of fall risk in RACFs. CONCLUSION: Embedding these tools within electronic health records could facilitate the implementation of targeted proactive interventions to prevent falls.

OBJECTIVES: To describe the application of nudges within electronic health records (EHRs) and their effects on inpatient care delivery, and identify design features that support effective decision-making without the use of interruptive alerts. MATERIALS AND METHODS: We searched Medline, Embase, and PsychInfo (in January 2022) for randomized controlled trials, interrupted time-series and before-after studies reporting effects of nudge interventions embedded in hospital EHRs to improve care. Nudge interventions were identified at full-text review, using a pre-existing classification. Interventions using interruptive alerts were excluded. Risk of bias was assessed using the ROBINS-I tool (Risk of Bias in Non-randomized Studies of Interventions) for non-randomized studies or the Cochrane Effective Practice and Organization of Care Group methodology for randomized trials. Study results were summarized narratively. RESULTS: We included 18 studies evaluating 24 EHR nudges. An improvement in care delivery was reported for 79.2% (n = 19; 95% CI, 59.5-90.8) of nudges. Nudges applied were from 5 of 9 possible nudge categories: change choice defaults (n = 9), make information visible (n = 6), change range or composition of options (n = 5), provide reminders (n = 2), and change option-related effort (n = 2). Only one study had a low risk of bias. Nudges targeted ordering of medications, laboratory tests, imaging, and appropriateness of care. Few studies evaluated long-term effects. DISCUSSION: Nudges in EHRs can improve care delivery. Future work could explore a wider range of nudges and evaluate long-term effects. CONCLUSION: Nudges can be implemented in EHRs to improve care delivery within current system capabilities; however, as with all digital interventions, careful consideration of the sociotechnical system is crucial to enhance their effectiveness.

OBJECTIVE: To conduct a systematic review and meta-analysis to assess: 1) changes in medication error rates and associated patient harm following electronic medication system (EMS) implementation; and 2) evidence of system-related medication errors facilitated by the use of an EMS. MATERIALS AND METHODS: We searched Medline, Scopus, Embase, and CINAHL for studies published between January 2005 and March 2019, comparing medication errors rates with or without assessments of related harm (actual or potential) before and after EMS implementation. EMS was defined as a computer-based system enabling the prescribing, supply, and/or administration of medicines. Study quality was assessed. RESULTS: There was substantial heterogeneity in outcomes of the 18 included studies. Only 2 were strong quality. Meta-analysis of 5 studies reporting change in actual harm post-EMS showed no reduced risk (RR: 1.22, 95% CI: 0.18-8.38, P = .8) and meta-analysis of 3 studies reporting change in administration errors found a significant reduction in error rates (RR: 0.77, 95% CI: 0.72-0.83, P = .004). Of 10 studies of prescribing error rates, 9 reported a reduction but variable denominators precluded meta-analysis. Twelve studies provided specific examples of system-related medication errors; 5 quantified their occurrence. DISCUSSION AND CONCLUSION: Despite the wide-scale adoption of EMS in hospitals around the world, the quality of evidence about their effectiveness in medication error and associated harm reduction is variable. Some confidence can be placed in the ability of systems to reduce prescribing error rates. However, much is still unknown about mechanisms which may be most effective in improving medication safety and design features which facilitate new error risks.

OBJECTIVE: The management and follow-up of diagnostic test results is a major patient safety concern. The aim of this qualitative study was to explore how clinicians manage test results on an everyday basis (work-as-done) in a health information technology-enabled emergency department setting. The objectives were to identify (1) variations in work-as-done in test results management and (2) the strategies clinicians use to ensure optimal management of diagnostic test results. MATERIALS AND METHODS: Qualitative interviews (n = 26) and field observations were conducted across 3 Australian emergency departments. Interview data coded for results management (ie, tracking, acknowledgment, and follow-up), and artifacts, were reviewed to identify variations in descriptions of work-as-done. Thematic analysis was performed to identify common themes. RESULTS: Despite using the same test result management application, there were variations in how the system was used. We identified 5 themes relating to electronic test results management: (1) tracking test results, (2) use and understanding of system functionality, (3) visibility of result actions and acknowledgment, (4) results inbox use, and (5) challenges associated with the absence of an inbox for results notifications for advanced practice nurses. DISCUSSION: Our findings highlight that variations in work-as-done can function to overcome perceived impediments to managing test results in a HIT-enabled environment and thus identify potential risks in the process. By illuminating work-as-done, we identified strategies clinicians use to enhance test result management including paper-based manual processes, cognitive reminders, and adaptive use of electronic medical record functionality. CONCLUSIONS: Test results tracking and follow-up is a priority area in need of health information technology development and training to improve team-based collaboration/communication of results follow-up and diagnostic safety.

OBJECTIVE: To investigate the impact of health information technology (IT) systems on clinicians' work practices and patient engagement in the management and follow-up of test results. MATERIALS AND METHODS: A search for studies reporting health IT systems and clinician test results management was conducted in the following databases: MEDLINE, EMBASE, CINAHL, Web of Science, ScienceDirect, ProQuest, and Scopus from January 1999 to June 2018. Test results follow-up was defined as provider follow-up of results for tests that were sent to the laboratory and radiology services for processing or analysis. RESULTS: There are some findings from controlled studies showing that health IT can improve the proportion of tests followed-up (15 percentage point change) and increase physician awareness of test results that require action (24-28 percentage point change). Taken as whole, however, the evidence of the impact of health IT on test result management and follow-up is not strong. DISCUSSION: The development of safe and effective test results management IT systems should pivot on several axes. These axes include 1) patient-centerd engagement (involving shared, timely, and meaningful information); 2) diagnostic processes (that involve the integration of multiple people and different clinical settings across the health care spectrum); and 3) organizational communications (the myriad of multi- transactional processes requiring feedback, iteration, and confirmation) that contribute to the patient care process. CONCLUSION: Existing evidence indicates that health IT in and of itself does not (and most likely cannot) provide a complete solution to issues related to test results management and follow-up.

Quantifying alert override has been the focus of much research in health informatics, with override rate traditionally viewed as a surrogate inverse indicator for alert effectiveness. However, relying on alert override to assess computerized alerts assumes that alerts are being read and determined to be irrelevant by users. Our research suggests that this is unlikely to be the case when users are experiencing alert overload. We propose that over time, alert override becomes habitual. The override response is activated by environmental cues and repeated automatically, with limited conscious intention. In this paper we outline this new perspective on understanding alert override. We present evidence consistent with the notion of alert override as a habitual behavior and discuss implications of this novel perspective for future research on alert override, a common and persistent problem accompanying decision support system implementation.

OBJECTIVE: To conduct a systematic review and meta-analysis of the impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay (LOS), and mortality in intensive care units (ICUs). METHODS: We searched for English-language literature published between January 2000 and January 2016 using Medline, Embase, and CINAHL. Titles and abstracts of 586 unique citations were screened. Studies were included if they: (1) reported results for an ICU population; (2) evaluated the impact of CPOE or the addition of CDSSs to an existing CPOE system; (3) reported quantitative data on medication errors, ICU LOS, hospital LOS, ICU mortality, and/or hospital mortality; and (4) used a randomized controlled trial or quasi-experimental study design. RESULTS: Twenty studies met our inclusion criteria. The transition from paper-based ordering to commercial CPOE systems in ICUs was associated with an 85% reduction in medication prescribing error rates and a 12% reduction in ICU mortality rates. Overall meta-analyses of LOS and hospital mortality did not demonstrate a significant change. DISCUSSION AND CONCLUSION: Critical care settings, both adult and pediatric, involve unique complexities, making them vulnerable to medication errors and adverse patient outcomes. The currently limited evidence base requires research that has sufficient statistical power to identify the true effect of CPOE implementation. There is also a critical need to understand the nature of errors arising post-CPOE and how the addition of CDSSs can be used to provide greater benefit to delivering safe and effective patient care.

OBJECTIVE: To conduct a cost-effectiveness analysis of a hospital electronic medication management system (eMMS). METHODS: We compared costs and benefits of paper-based prescribing with a commercial eMMS (CSC MedChart) on one cardiology ward in a major 326-bed teaching hospital, assuming a 15-year time horizon and a health system perspective. The eMMS implementation and operating costs were obtained from the study site. We used data on eMMS effectiveness in reducing potential adverse drug events (ADEs), and potential ADEs intercepted, based on review of 1 202 patient charts before (n = 801) and after (n = 401) eMMS. These were combined with published estimates of actual ADEs and their costs. RESULTS: The rate of potential ADEs following eMMS fell from 0.17 per admission to 0.05; a reduction of 71%. The annualized eMMS implementation, maintenance, and operating costs for the cardiology ward were A$61 741 (US$55 296). The estimated reduction in ADEs post eMMS was approximately 80 actual ADEs per year. The reduced costs associated with these ADEs were more than sufficient to offset the costs of the eMMS. Estimated savings resulting from eMMS implementation were A$63-66 (US$56-59) per admission (A$97 740-$102 000 per annum for this ward). Sensitivity analyses demonstrated results were robust when both eMMS effectiveness and costs of actual ADEs were varied substantially. CONCLUSION: The eMMS within this setting was more effective and less expensive than paper-based prescribing. Comparison with the few previous full economic evaluations available suggests a marked improvement in the cost-effectiveness of eMMS, largely driven by increased effectiveness of contemporary eMMs in reducing medication errors.

This study describes the implementation and impact of an electronic test result acknowledgement (RA) system in the Mater Mothers' Hospital in Brisbane, Australia. The Verdi application electronically records clinicians' acknowledgement of the review of results. Hospital data (August 2011-August 2012) were extracted to measure clinicians' acknowledgement practices. There were 27,354 inpatient test results for 6855 patients. All test results were acknowledged. 60% (95% CI 59% to 61%) of laboratory and 44% (95% CI 40% to 48%) of imaging results were acknowledged within 24 h. The median time between report availability and acknowledgement was 18.1 h for laboratory and 1 day 18 h for imaging results. The median time from when a result was first viewed to its acknowledgement was 7 min for laboratory and 1 min for imaging results. The longest recorded time to acknowledgement was 38 days. Electronic RA provides a safety net to enhance test result management.

In the recent JAMIA article ‘Measuring value for money: a scoping review on economic evaluation of health information systems (HIS),’ Bassi and Lau1 identified 42 studies conducted from different perspectives within public, private and mixed healthcare systems and assessed 33 of them as good quality. The importance of establishing value for money of large HIS investments is indisputable. Numerous assessments of economic aspects of different HIS implementations have been conducted; however, there is a paucity of reviews classifying and summarizing their results. Bassi and Lau's review makes a useful contribution by introducing a new methodological framework, which classifies HIS studies with an ‘economic component’ and assesses their quality. In our view, the framework has some limitations that may confuse readers and result in misguided conclusions. Here we briefly discuss these limitations and we hope this will stimulate further discussion. One of the central challenges is that many HIS studies, including those identified by Bassi and Lau, contain some information on benefits and/or costs, but such studies were often not designed to conduct an economic evaluation or assess economic efficiency as defined by the economics discipline. The stated objectives and the correspondent methods (eg, econometric modelling) may be sufficient to quantify ‘cost savings' associated with investing in HIS, but cannot show that the investment is economically efficient (ie, provides better value than the alternative use of resources). Readers may incorrectly interpret results of such studies as evidence of economic efficiency. It would be helpful to establish what is required to demonstrate ‘value for money’ of HIS, and how to differentiate between studies that set out, and achieve, this objective from those studies that serve other, also useful, purposes. For example, in an earlier systematic review Shekelle and colleagues2 identified 256 studies on costs and benefits of health information technology and used a multidimensional classification which, among other criteria, included study objectives and analytic techniques employed by the reviewed studies—namely, ‘predictive analysis’, ‘hypothesis-testing’, and ‘meta-analysis’. In the absence of a conventional framework for categorizing HIS economic studies and assessing their quality, Bassi and Lau used the health economics framework of Drummond and colleagues3 and supplemented it with the usual criteria for assessing academic papers. Drummond and colleagues developed their framework over many years by numerous discussions, consultations, and meetings with academic experts, clinicians, and editors of leading medical journals.4 This widely accepted and extensively used framework includes a classification of health economic evaluation studies and a checklist for assessing their quality. The classification is based on mutually exclusive categories and distinguishes full economic evaluations (cost-effectiveness analysis (CEA), cost-utility (CUA), and cost-benefit (CBA) analyses) from partial economic evaluations (eg, cost analyses or cost-outcome descriptions). Full economic evaluation is a comparative analysis of alternative options/programmes that involves identification, measurement, and valuation of both costs and outcomes, and establishes the difference in costs in relation to difference in outcomes in an incremental fashion. Results of economic analyses are expressed as an incremental cost-effectiveness ratio, net monetary benefit, or net health benefit.3 Conducted from a societal perspective (ie, including all costs and consequences, regardless of who bears them), health economic evaluations are consistent with the economic theory of maximizing social welfare, because by allocating limited resources to interventions with the lowest cost per unit of health gain, decision-makers can increase the total health and well-being of the society. Drummond's economic evaluation study checklist includes quality assessment criteria (eg, whether the costs and outcomes for each alternative option were well defined, whether the perspective of the analysis was stated, whether discounting of costs and outcomes for a specified time frame was conducted). Bassi and Lau selected five of these criteria (perspective of the analysis, a specified time frame, presence of an alternative option for comparison, assessment of costs and outcomes) to identify some of the reviewed studies as ‘economic analyses’. Their sixth criterion stated that costs and outcomes should be compared for each option. Thus this sixth criterion called for an average cost-effectiveness ratio rather than an incremental cost-effectiveness ratio. However, an average cost-effectiveness ratio is likely to underestimate the cost for each additional unit of outcome compared with an incremental cost-effectiveness ratio.3 Using their definition of economic analysis, the authors identified only one CUA, two CEAs and six CBAs, while most studies (21 studies) were classified as either ‘input cost analysis' or ‘cost-related outcome analysis’. Although the latter term resembles Drummond's ‘cost-outcome description’, it is rather ambiguous as some papers could be classified as both an ‘input cost analysis' and ‘cost-related outcome analysis’. Some of the high-quality ‘hypothesis-testing’ papers that investigated associations between adoption of HIS and potential cost savings fell into this category. However, as argued above, no amount of cost savings provides evidence of economic efficiency unless the alternative use of resources and associated outcomes is considered. The difficulties experienced by Bassi and Lau in adequately identifying and classifying HIS studies with an economic component, suggest that Drummond's framework may apply only to studies that were conducted to demonstrate economic efficiency by explicitly comparing two options in an incremental fashion. This is not to say that other HIS economic studies are unimportant. Full economic evaluation is not the only method of estimating costs and benefits. Society may be concerned with improving technical and productive efficiency from the resources that have already been allocated; however, to avoid confusion such studies should be differentiated accordingly. This letter is written with the intention of stimulating further discussion about frameworks for categorizing HIS economic studies and assessing their quality. The framework needs to be sensitive to the stated objectives of HIS economic papers and specific enough to minimize ambiguity in allocating studies into categories. Reaching greater agreement about a framework is important for both assessing current evidence and guiding future research. EG conceived the idea of initiating discussion and wrote the first draft of the letter. Both authors contributed to the subsequent drafts, writing of the final version, and preparation of the manuscript. None. Not commissioned; externally peer reviewed.

OBJECTIVES: To compare the manifestations, mechanisms, and rates of system-related errors associated with two electronic prescribing systems (e-PS). To determine if the rate of system-related prescribing errors is greater than the rate of errors prevented. METHODS: Audit of 629 inpatient admissions at two hospitals in Sydney, Australia using the CSC MedChart and Cerner Millennium e-PS. System related errors were classified by manifestation (eg, wrong dose), mechanism, and severity. A mechanism typology comprised errors made: selecting items from drop-down menus; constructing orders; editing orders; or failing to complete new e-PS tasks. Proportions and rates of errors by manifestation, mechanism, and e-PS were calculated. RESULTS: 42.4% (n=493) of 1164 prescribing errors were system-related (78/100 admissions). This result did not differ by e-PS (MedChart 42.6% (95% CI 39.1 to 46.1); Cerner 41.9% (37.1 to 46.8)). For 13.4% (n=66) of system-related errors there was evidence that the error was detected prior to study audit. 27.4% (n=135) of system-related errors manifested as timing errors and 22.5% (n=111) wrong drug strength errors. Selection errors accounted for 43.4% (34.2/100 admissions), editing errors 21.1% (16.5/100 admissions), and failure to complete new e-PS tasks 32.0% (32.0/100 admissions). MedChart generated more selection errors (OR=4.17; p=0.00002) but fewer new task failures (OR=0.37; p=0.003) relative to the Cerner e-PS. The two systems prevented significantly more errors than they generated (220/100 admissions (95% CI 180 to 261) vs 78 (95% CI 66 to 91)). CONCLUSIONS: System-related errors are frequent, yet few are detected. e-PS require new tasks of prescribers, creating additional cognitive load and error opportunities. Dual classification, by manifestation and mechanism, allowed identification of design features which increase risk and potential solutions. e-PS designs with fewer drop-down menu selections may reduce error risk.

OBJECTIVE: To quantify and compare the time doctors and nurses spent on direct patient care, medication-related tasks, and interactions before and after electronic medication management system (eMMS) introduction. METHODS: Controlled pre-post, time and motion study of 129 doctors and nurses for 633.2 h on four wards in a 400-bed hospital in Sydney, Australia. We measured changes in proportions of time on tasks and interactions by period, intervention/control group, and profession. RESULTS: eMMS was associated with no significant change in proportions of time spent on direct care or medication-related tasks relative to control wards. In the post-period control ward, doctors spent 19.7% (2 h/10 h shift) of their time on direct care and 7.4% (44.4 min/10 h shift) on medication tasks, compared to intervention ward doctors (25.7% (2.6 h/shift; p=0.08) and 8.5% (51 min/shift; p=0.40), respectively). Control ward nurses in the post-period spent 22.1% (1.9 h/8.5 h shift) of their time on direct care and 23.7% on medication tasks compared to intervention ward nurses (26.1% (2.2 h/shift; p=0.23) and 22.6% (1.9 h/shift; p=0.28), respectively). We found intervention ward doctors spent less time alone (p=0.0003) and more time with other doctors (p=0.003) and patients (p=0.009). Nurses on the intervention wards spent less time with doctors following eMMS introduction (p=0.0001). CONCLUSIONS: eMMS introduction did not result in redistribution of time away from direct care or towards medication tasks. Work patterns observed on these intervention wards were associated with previously reported significant reductions in prescribing error rates relative to the control wards.

Consistency or efficiency? A dilemma for designersIn this issue of the journal, Russ et al 1 present a discussion of our paper. 2In our study, we found that prescribers were not utilizing all eprescribing system functions, despite the functions' potential to improve efficiency of work, and a consequence of this was the generation of clinically unnecessary alerts. 2 In their response, 'When 'technically preventable' alerts occur, the design-not the prescriber-has failed,' 1 Russ et al suggest that we have diagnosed the problem correctly (a discordance between prescribers

OBJECTIVES: To determine the frequency with which computerized alerts occur and the proportion triggered as a result of prescribers not utilizing e-prescribing system functions. METHODS: An audit of electronic inpatient medication charts at a teaching hospital in Sydney, Australia, was conducted to identify alerts fired, to categorize the system functions used by prescribers, and to assess if use of short-cut system functions could have prevented the alerts. RESULTS: Of the 2209 active orders reviewed, 600 (27.2%) triggered at least one alert. Therapeutic duplication alerts were the most frequent (n=572). One third of these (20.2% of all alerts) was 'technically preventable' and would not have fired if prescribers had used a short-cut system function to prescribe. Under-utilized system functions included the option to 'MODIFY' existing orders and use of the 'AND' function for concurrent orders. Pregnancy alerts, set for women aged between 12 and 55 years, were triggered for 43% of drugs ordered for this group. CONCLUSION: Developers of decision support systems should test the extent to which technically preventable alerts may arise when prescribers fail to use system functions as designed. Designs which aim to improve the efficiency of the prescribing process but which do not align with the cognitive processes of users may fail to achieve this desired outcome and produce unexpected consequences such as triggering unnecessary alerts and user frustration. Ongoing user training to support effective use of e-prescribing system functions and modifications to the mechanisms underlying alert generation are needed to ensure that prescribers are presented with fewer but more meaningful alerts.

OBJECTIVE: To assess evidence of the impact of Picture Archiving and Communication Systems (PACS) on clinicians' work practices in the intensive care unit (ICU). METHODS: We searched Medline, Pre-Medline, CINAHL, Embase, and the SPIE Digital Library databases for English-language publications between 1980 and September 2010 using Medical Subject Headings terms and keywords. RESULTS: Eleven studies from the USA and UK were included. All studies measured aspects of time associated with the introduction of PACS, namely the availability of images, the time a physician took to review an image, and changes in viewing patterns. Seven studies examined the impact on clinical decision-making, with the majority measuring the time to image-based clinical action. The effect of PACS on communication modes was reported in five studies. DISCUSSION: PACS can impact on clinician work practices in three main areas. Most of the evidence suggests an improvement in the efficiency of work practices. Quick image availability can impact on work associated with clinical decision-making, although the results were inconsistent. PACS can change communication practices, particularly between the ICU and radiology; however, the evidence base is insufficient to draw firm conclusions in this area. CONCLUSION: The potential for PACS to impact positively on clinician work practices in the ICU and improve patient care is great. However, the evidence base is limited and does not reflect aspects of contemporary PACS technology. Performance measures developed in previous studies remain relevant, with much left to investigate to understand how PACS can support new and improved ways of delivering care in the intensive care setting.

OBJECTIVE: To assess whether a low level of decision support within a hospital computerized provider order entry system has an observable influence on the medication ordering process on ward-rounds and to assess prescribers' views of the decision support features. METHODS: 14 specialty teams (46 doctors) were shadowed by the investigator while on their ward-rounds and 16 prescribers from these teams were interviewed. RESULTS: Senior doctors were highly influential in prescribing decisions during ward-rounds but rarely used the computerized provider order entry system. Junior doctors entered the majority of medication orders into the system, nearly always ignored computerized alerts and never raised their occurrence with other doctors on ward-rounds. Interviews with doctors revealed that some decision support features were valued but most were not perceived to be useful. DISCUSSION AND CONCLUSION: The computerized alerts failed to target the doctors who were making the prescribing decisions on ward-rounds. Senior doctors were the decision makers, yet the junior doctors who used the system received the alerts. As a result, the alert information was generally ignored and not incorporated into the decision-making processes on ward-rounds. The greatest value of decision support in this setting may be in non-ward-round situations where senior doctors are less influential. Identifying how prescribing systems are used during different clinical activities can guide the design of decision support that effectively supports users in different situations. If confirmed, the findings reported here present a specific focus and user group for designers of medication decision support.

OBJECTIVE: To model how individual violations in routine clinical processes cumulatively contribute to the risk of adverse events in hospital using an agent-based simulation framework. DESIGN: An agent-based simulation was designed to model the cascade of common violations that contribute to the risk of adverse events in routine clinical processes. Clinicians and the information systems that support them were represented as a group of interacting agents using data from direct observations. The model was calibrated using data from 101 patient transfers observed in a hospital and results were validated for one of two scenarios (a misidentification scenario and an infection control scenario). Repeated simulations using the calibrated model were undertaken to create a distribution of possible process outcomes. The likelihood of end-of-chain risk is the main outcome measure, reported for each of the two scenarios. RESULTS: The simulations demonstrate end-of-chain risks of 8% and 24% for the misidentification and infection control scenarios, respectively. Over 95% of the simulations in both scenarios are unique, indicating that the in-patient transfer process diverges from prescribed work practices in a variety of ways. CONCLUSIONS: The simulation allowed us to model the risk of adverse events in a clinical process, by generating the variety of possible work subject to violations, a novel prospective risk analysis method. The in-patient transfer process has a high proportion of unique trajectories, implying that risk mitigation may benefit from focusing on reducing complexity rather than augmenting the process with further rule-based protocols.

BACKGROUND: Computerized provider order entry (CPOE) systems have been strongly promoted as a means to improve the quality and efficiency of healthcare. METHODS: This systematic review aimed to assess the evidence of the impact of CPOE on medical-imaging services and patient outcomes. RESULTS: Fourteen studies met the inclusion criteria, most of which (10/14) used a pre-/postintervention comparison design. Eight studies demonstrated benefits, such as decreased test utilization, associated with decision-support systems promoting adherence to test ordering guidelines. Three studies evaluating medical-imaging ordering and reporting times showed statistically significant decreases in turnaround times. CONCLUSIONS: The findings reveal the potential for CPOE to contribute to significant efficiency and effectiveness gains in imaging services. The diversity and scope of the research evidence can be strengthened through increased attention to the circumstances and mechanisms that contribute to the success (or otherwise) of CPOE and its contribution to the enhancement of patient care delivery.

The substantial growth in mobile handheld technologies has heralded the opportunity to provide physicians with access to information, resources, and people at the right time and place. But is this technology delivering the benefits to workflow and patient care promised by increased mobility? The authors conducted a systematic review to examine evidence regarding the impact of mobile handheld technology on hospital physicians' work practices and patient care, focusing on quantification of the espoused virtues of mobile technologies. The authors identified thirteen studies that demonstrated the ability of personal digital assistants (PDAs) to positively impact on areas of rapid response, error prevention, and data management and accessibility. The use of PDAs demonstrates the greatest benefits in contexts where time is a critical factor and a rapid response crucial. However, the extent to which these devices improved outcomes and workflow efficiencies because of their mobility was largely absent from the literature. The paucity of evidence calls for much needed future research that asks explicit questions about the impact the mobility of devices has on work practices and outcomes.

Previous reviews have examined evidence of the impact of CPOE on medication errors, but have used highly variable definitions of "error". We attempted to answer a very focused question, namely, what evidence exists that CPOE systems reduce prescribing errors among hospital inpatients? We identified 13 papers (reporting 12 studies) published between 1998 and 2007. Nine demonstrated a significant reduction in prescribing error rates for all or some drug types. Few studies examined changes in error severity, but minor errors were most often reported as decreasing. Several studies reported increases in the rate of duplicate orders and failures to discontinue drugs, often attributed to inappropriate selection from a dropdown menu or to an inability to view all active medication orders concurrently. The evidence-base reporting the effectiveness of CPOE to reduce prescribing errors is not compelling and is limited by modest study sample sizes and designs. Future studies should include larger samples including multiple sites, controlled study designs, and standardized error and severity reporting. The role of decision support in minimizing severe prescribing error rates also requires investigation.

OBJECTIVE: This paper presents a multiple perspectives model of clinical information system implementation, the CONTEXTual Implementation Model (CIM). Although other implementation models have been developed, few are grounded in data and others fail to take adequate account of the clinical environment and users' requirements. DESIGN: The CIM arose from qualitative data collected from four clinical units in two large Australian teaching hospitals. The aim of the study was to explore physicians' test management work practices associated with the compulsory use of a hospital-wide, mandatory computerized provider order entry (CPOE) system.(1) The dataset consisted of non-participatory observations of physicians using CPOE (n=55 sessions) and interviews with health professionals (n=28) about test management work practices. Data were analyzed by two researchers independently using an iterative grounded approach. RESULTS: A core underlying theme of 'contextual differences' emerged which explained physicians' use of the CPOE system in the sites. The CIM focuses attention on diversity at three contextual levels: the organizational level; the clinical or departmental level, and the individual level. Within each of these levels there are dimensions for consideration (for example, organizational culture, leadership and diverse ways of working) which affect physicians' attitudes to, and use of, CPOE. CONCLUSION: The CIM provides a contextual differences perspective which can be used to facilitate the implementation of clinical information systems. Developing a clinical information system implementation model serves as a framework to guide future implementations to ensure their safe and efficient use and also improve the likelihood of uptake by physicians.

OBJECTIVE: To test whether the use of an evidence retrieval system that uses clinically targeted meta-search filters can enhance the rate at which clinicians make correct decisions, reduce the effort involved in locating evidence, and provide an intuitive match between clinical tasks and search filters. DESIGN: A laboratory experiment under controlled conditions asked 75 clinicians to answer eight randomly sequenced clinical questions, using one of two randomly assigned search engines. The first search engine Quick Clinical (QC) was equipped with meta-search filters (the combined use of meta-search and search filters) designed to answer typical clinical questions e.g., treatment, diagnosis, and the second 'library model' system (LM) offered free access to an identical evidence set with no filter support. MEASUREMENTS: Changes in clinical decision making were measured by the proportion of correct post-search answers provided to questions, the time taken to answer questions, and the number of searches and links to documents followed in a search session. The intuitive match between meta-search filters and clinical tasks was measured by the proportion and distribution of filters selected for individual clinical questions. RESULTS: Clinicians in the two groups performed equally well pre-search. Post search answers improved overall by 21%, with 52.2% of answers correct with QC and 54.7% with LM (chi(2) = 0.33, df = 1, p > 0.05). Users of QC obtained a significantly greater percentage of their correct answers within the first two minutes of searching compared to LM users (QC 58.2%; LM 32.9%; chi(2) = 19.203, df = 1, p < 0.001). There was a statistical difference for QC and LM survival curves, which plotted overall time to answer questions, irrespective of answer (Wilcoxon, p = 0.019) and for the average time to provide a correct answer (Wilcoxon, p = 0.006). The QC system users conducted significantly fewer searches per scenario (m = 3.0 SD = 1.15 versus m = 5.5 SD1.97, t = 6.63, df = 72, p = 0.0001). Clinicians using the QC system followed fewer document links than did those who used LM (respectively 3.9 links SD = 1.20 versus 4.7 links SD = 1.79, t = 2.13, df = 72, p = 0.0368). In 6 of the 8 questions, two meta-search filters accounted for 89% or more of clinicians' first choice, suggesting the choice of filter intuitively matched the clinical decision task at hand. CONCLUSIONS: Meta-search filters result in clinicians arriving at answers more quickly than unconstrained searches across information sources, and appear to increase the rate with which correct decisions are made. In time restricted clinical settings meta-search filters may thus improve overall decision accuracy, as fewer searches that could otherwise lead to a correct answer are abandoned. Meta-search filters appear to be intuitive to use, suggesting that the simplicity of the user model would fit very well into clinical settings.

OBJECTIVE: Few research designs look at the deep structure of complex social systems. We report the design and implementation of a multimethod evaluation model to assess the impact of computerized order entry systems on both the technical and social systems within a health care organization. DESIGN: We designed a multimethod evaluation model informed by sociotechnical theory and an appreciation of the nature of wicked problems. We mobilized this model to assess the impact of an electronic medication management system via a three-year program of research at a major academic hospital. MEASUREMENTS: Model components include measurements relating to three dimensions of system impact: safety and quality, organizational culture, and work and communication patterns. RESULTS: Application of the evaluation model required the development and testing of purpose-built measurement tools such as software to collect multidimensional work measurement data. The model applied established research methods including medication error audits and social network analysis. Design features of these tools and techniques are described, along with the practical challenges of their implementation. The distinctiveness of doing research within a unique paradigm of complex systems, explicating the wickedness and the dimensionality of sociotechnical theory, is articulated. CONCLUSION: Designing an effective evaluation model requires a deep understanding of the nature and complexity of the problems that information technology interventions in health care are trying to address. Adopting a sociotechnical perspective for model generation improves our ability to develop evaluation models that are adaptive and sensitive to the characteristics of wicked problems and provides a strong theoretical basis from which to analyze and interpret findings.

OBJECTIVE: To explore physicians' work practices in relation to their long-term use of a computerized physician order entry system (CPOE). DESIGN: A cross-sectional qualitative study was conducted in four clinical units in two large Australian teaching hospitals. One hospital had used CPOE for over 10 years to order all clinical laboratory and radiology tests and view test results and the other had used the computerized viewing facility of the system for over seven years with tests ordered manually. Data were collected by non-participatory observations of physicians (55 sessions) and 28 interviews. MEASUREMENTS: Content analysis of the observation field notes, reflections on observations and interview transcripts were conducted by two researchers independently. A thematic grounded theory approach was used to derive key themes that would explain physicians work practices associated with CPOE use. RESULTS: Three themes relating to physicians' established use of CPOE were identified: (1) the effect of the hospital and clinical environment; (2) changes to work practices; and (3) physicians' management of information. Physicians' test management work practices using CPOE were related to diversity between: the hospitals; the clinical units' environment, and the users of the system. CONCLUSION: Hospitals need to understand and analyze physicians' test management work practices prior to and during the implementation of CPOE to accommodate their diverse ways of working with computerized information systems. In the current mixed media environment, physicians' use of manual and computerized information systems for sourcing and recording information impacts on efficiency and patient safety.

OBJECTIVE: To assess the impact of clinicians' use of an online information retrieval system on their performance in answering clinical questions. DESIGN: Pre-/post-intervention experimental design. MEASUREMENTS: In a computer laboratory, 75 clinicians (26 hospital-based doctors, 18 family practitioners, and 31 clinical nurse consultants) provided 600 answers to eight clinical scenarios before and after the use of an online information retrieval system. We examined the proportion of correct answers pre- and post-intervention, direction of change in answers, and differences between professional groups. RESULTS: System use resulted in a 21% improvement in clinicians' answers, from 29% (95% confidence interval [CI] 25.4-32.6) correct pre- to 50% (95% CI 46.0-54.0) post-system use. In 33% (95% CI 29.1-36.9) answers were changed from incorrect to correct. In 21% (95% CI 17.1-23.9) correct pre-test answers were supported by evidence found using the system, and in 7% (95% CI 4.9-9.1) correct pre-test answers were changed incorrectly. For 40% (35.4-43.6) of scenarios, incorrect pre-test answers were not rectified following system use. Despite significant differences in professional groups' pre-test scores [family practitioners: 41% (95% CI 33.0-49.0), hospital doctors: 35% (95% CI 28.5-41.2), and clinical nurse consultants: 17% (95% CI 12.3-21.7; chi(2) = 29.0, df = 2, p < 0.01)], there was no difference in post-test scores. (chi(2) = 2.6, df = 2, p = 0.73). CONCLUSIONS: The use of an online information retrieval system was associated with a significant improvement in the quality of answers provided by clinicians to typical clinical problems. In a small proportion of cases, use of the system produced errors. While there was variation in the performance of clinical groups when answering questions unaided, performance did not differ significantly following system use. Online information retrieval systems can be an effective tool in improving the accuracy of clinicians' answers to clinical questions.

OBJECTIVES: To determine clinicians' (doctors', nurses', and allied health professionals') "actual" and "reported" use of a point-of-care online information retrieval system; and to make an assessment of the extent to which use is related to direct patient care by testing two hypotheses: hypothesis 1: clinicians use online evidence primarily to support clinical decisions relating to direct patient care; and hypothesis 2: clinicians use online evidence predominantly for research and continuing education. DESIGN: Web-log analysis of the Clinical Information Access Program (CIAP), an online, 24-hour, point-of-care information retrieval system available to 55,000 clinicians in public hospitals in New South Wales, Australia. A statewide mail survey of 5,511 clinicians. MEASUREMENTS: Rates of online evidence searching per 100 clinicians for the state and for the 81 individual hospitals studied; reported use of CIAP by clinicians through a self-administered questionnaire; and correlations between evidence searches and patient admissions. RESULTS: Monthly rates of 48.5 "search sessions" per 100 clinicians and 231.6 text hits to single-source databases per 100 clinicians (n = 619,545); 63% of clinicians reported that they were aware of CIAP and 75% of those had used it. Eighty-eight percent of users reported CIAP had the potential to improve patient care and 41% reported direct experience of this. Clinicians' use of CIAP on each day of the week was highly positively correlated with patient admissions (r = 0.99, p < 0.001). This was also true for all ten randomly selected hospitals. CONCLUSION: Clinicians' online evidence use increases with patient admissions, supporting the hypothesis that clinicians' use of evidence is related to direct patient care. Patterns of evidence use and clinicians' self-reports also support this hypothesis.

OBJECTIVES: To investigate the association between clinical team functioning and diffusion (awareness, use, and impact) of a 24-hour online evidence retrieval system. To examine the relationships between clinical team characteristics and the adoption of the online evidence system. DESIGN: 18 clinical teams, consisting of 180 clinicians from three Australian hospitals, were identified and studied. Teams were categorized as small ( 15). MEASUREMENTS: Clinical team functioning was assessed using the Team Climate Inventory (TCI). Awareness, use, and impact of an online evidence retrieval system were measured using a self-administered questionnaire. The relationships between TCI scores and awareness, use, and impact were examined using t-tests and one-way ANOVAs. Chi square analyses were used to examine differences between small and large teams. RESULTS were interpreted within a diffusion of innovations framework. RESULTS: Clinical team functioning was not related to awareness or use of the online evidence retrieval system. However, clinical team functioning was significantly associated with the impact of online evidence in terms of reported experience of improved patient care following system use. Clinicians in small teams ( 15) teams. CONCLUSIONS: Team functioning had the greatest impact on the fourth stage of innovation diffusion, the effective use of online evidence for clinical care. This supports Rogers' diffusion of innovation theory, to the effect that different types of communication about an innovation are important at different stages in the diffusion process. Members of small teams were more aware of the system than members of large teams. Team functioning is amenable to improvement through interventions. The findings suggest that the role of team climate in the diffusion of information systems is a promising area for future research.