BACKGROUND: Problem lists represent an integral component of high-quality care. However, they are often inaccurate and incomplete. We studied the effects of alerts integrated into the inpatient and outpatient computerized provider order entry systems to assist in adding problems to the problem list when ordering medications that lacked a corresponding indication. METHODS: We analyzed medication orders from 2 healthcare systems that used an innovative indication alert. We collected data at site 1 between December 2018 and January 2020, and at site 2 between May and June 2021. We reviewed random samples of 100 charts from each site that had problems added in response to the alert. Outcomes were: (1) alert yield, the proportion of triggered alerts that led to a problem added and (2) problem accuracy, the proportion of problems placed that were accurate by chart review. RESULTS: Alerts were triggered 131 134, and 6178 times at sites 1 and 2, respectively, resulting in a yield of 109 055 (83.2%) and 2874 (46.5%), P< .001. Orders were abandoned, for example, not completed, in 11.1% and 9.6% of orders, respectively, P<.001. Of the 100 sample problems, reviewers deemed 88% ± 3% and 91% ± 3% to be accurate, respectively, P = .65, with a mean of 90% ± 2%. CONCLUSIONS: Indication alerts triggered by medication orders initiated in the absence of a justifying diagnosis were useful for populating problem lists, with yields of 83.2% and 46.5% at 2 healthcare systems. Problems were placed with a reasonable level of accuracy, with 90% ± 2% of problems deemed accurate based on chart review.

Diabetes intensive care unit (ICU) patients are at increased risk of complications leading to in-hospital mortality. Assessing the likelihood of death is a challenging and time-consuming task due to a large number of influencing factors. Healthcare providers are interested in the detection of ICU patients at higher risk, such that risk factors can possibly be mitigated. While such severity scoring methods exist, they are commonly based on a snapshot of the health conditions of a patient during the ICU stay and do not specifically consider a patient's prior medical history. In this paper, a process mining/deep learning architecture is proposed to improve established severity scoring methods by incorporating the medical history of diabetes patients. First, health records of past hospital encounters are converted to event logs suitable for process mining. The event logs are then used to discover a process model that describes the past hospital encounters of patients. An adaptation of Decay Replay Mining is proposed to combine medical and demographic information with established severity scores to predict the in-hospital mortality of diabetes ICU patients. Significant performance improvements are demonstrated compared to established risk severity scoring methods and machine learning approaches using the Medical Information Mart for Intensive Care III dataset.

OBJECTIVE: We utilized a computerized order entry system-integrated function referred to as "void" to identify erroneous orders (ie, a "void" order). Using voided orders, we aimed to (1) identify the nature and characteristics of medication ordering errors, (2) investigate the risk factors associated with medication ordering errors, and (3) explore potential strategies to mitigate these risk factors. MATERIALS AND METHODS: We collected data on voided orders using clinician interviews and surveys within 24 hours of the voided order and using chart reviews. Interviews were informed by the human factors-based SEIPS (Systems Engineering Initiative for Patient Safety) model to characterize the work systems-based risk factors contributing to ordering errors; chart reviews were used to establish whether a voided order was a true medication ordering error and ascertain its impact on patient safety. RESULTS: During the 16-month study period (August 25, 2017, to December 31, 2018), 1074 medication orders were voided; 842 voided orders were true medication errors (positive predictive value = 78.3 ± 1.2%). A total of 22% (n = 190) of the medication ordering errors reached the patient, with at least a single administration, without causing patient harm. Interviews were conducted on 355 voided orders (33% response). Errors were not uniquely associated with a single risk factor, but the causal contributors of medication ordering errors were multifactorial, arising from a combination of technological-, cognitive-, environmental-, social-, and organizational-level factors. CONCLUSIONS: The void function offers a practical, standardized method to create a rich database of medication ordering errors. We highlight implications for utilizing the void function for future research, practice and learning opportunities.

OBJECTIVE: The study sought to determine availability and use of structured override reasons for drug-drug interaction (DDI) alerts in electronic health records. MATERIALS AND METHODS: We collected data on DDI alerts and override reasons from 10 clinical sites across the United States using a variety of electronic health records. We used a multistage iterative card sort method to categorize the override reasons from all sites and identified best practices. RESULTS: Our methodology established 177 unique override reasons across the 10 sites. The number of coded override reasons at each site ranged from 3 to 100. Many sites offered override reasons not relevant to DDIs. Twelve categories of override reasons were identified. Three categories accounted for 78% of all overrides: "will monitor or take precautions," "not clinically significant," and "benefit outweighs risk." DISCUSSION: We found wide variability in override reasons between sites and many opportunities to improve alerts. Some override reasons were irrelevant to DDIs. Many override reasons attested to a future action (eg, decreasing a dose or ordering monitoring tests), which requires an additional step after the alert is overridden, unless the alert is made actionable. Some override reasons deferred to another party, although override reasons often are not visible to other users. Many override reasons stated that the alert was inaccurate, suggesting that specificity of alerts could be improved. CONCLUSIONS: Organizations should improve the options available to providers who choose to override DDI alerts. DDI alerting systems should be actionable and alerts should be tailored to the patient and drug pairs.

To reduce the risk of wrong-patient errors, safety experts recommend allowing only one patient chart to be open at a time. Due to the lack of empirical evidence, the number of allowable open charts is often based on anecdotal evidence or institutional preference, and hence varies across institutions. Using an interrupted time series analysis of intercepted wrong-patient medication orders in an emergency department during 2010-2016 (83.6 intercepted wrong-patient events per 100 000 orders), we found no significant decrease in the number of intercepted wrong-patient medication orders during the transition from a maximum of 4 open charts to a maximum of 2 (b = -0.19, P = .33) and no significant increase during the transition from a maximum of 2 open charts to a maximum of 4 (b = 0.08, P = .67). These results have implications regarding decisions about allowable open charts in the emergency department in relation to the impact on workflow and efficiency.

Objective: To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods: We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results: We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion: Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion: CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

To reduce the risk of wrong-patient errors, safety experts recommend limiting the number of patient records providers can open at once in electronic health records (EHRs). However, it is unknown whether health care organizations follow this recommendation or what rationales drive their decisions. To address this gap, we conducted an electronic survey via 2 national listservs. Among 167 inpatient and outpatient study facilities using EHR systems designed to open multiple records at once, 44.3% were configured to allow ≥3 records open at once (unrestricted), 38.3% allowed only 1 record open (restricted), and 17.4% allowed 2 records open (hedged). Decision-making centered on efforts to balance safety and efficiency, but there was disagreement among organizations about how to achieve that balance. Results demonstrate no consensus on the number of records to be allowed open at once in EHRs. Rigorous studies are needed to determine the optimal number of records that balances safety and efficiency.

Objective: To examine medication errors potentially related to computerized prescriber order entry (CPOE) and refine a previously published taxonomy to classify them. Materials and Methods: We reviewed all patient safety medication reports that occurred in the medication ordering phase from 6 sites participating in a United States Food and Drug Administration-sponsored project examining CPOE safety. Two pharmacists independently reviewed each report to confirm whether the error occurred in the ordering/prescribing phase and was related to CPOE. For those related to CPOE, we assessed whether CPOE facilitated (actively contributed to) the error or failed to prevent the error (did not directly cause it, but optimal systems could have potentially prevented it). A previously developed taxonomy was iteratively refined to classify the reports. Results: Of 2522 medication error reports, 1308 (51.9%) were related to CPOE. Of these, CPOE facilitated the error in 171 (13.1%) and potentially could have prevented the error in 1137 (86.9%). The most frequent categories of "what happened to the patient" were delays in medication reaching the patient, potentially receiving duplicate drugs, or receiving a higher dose than indicated. The most frequent categories for "what happened in CPOE" included orders not routed to or received at the intended location, wrong dose ordered, and duplicate orders. Variations were seen in the format, categorization, and quality of reports, resulting in error causation being assignable in only 403 instances (31%). Discussion and Conclusion: Errors related to CPOE commonly involved transmission errors, erroneous dosing, and duplicate orders. More standardized safety reporting using a common taxonomy could help health care systems and vendors learn and implement prevention strategies.

OBJECTIVE: Medication order voiding allows clinicians to indicate that an existing order was placed in error. We explored whether the order voiding function could be used to record and study medication ordering errors. MATERIALS AND METHODS: We examined medication orders from an academic medical center for a 6-year period (2006-2011; n = 5 804 150). We categorized orders based on status (void, not void) and clinician-provided reasons for voiding. We used multivariable logistic regression to investigate the association between order voiding and clinician, patient, and order characteristics. We conducted chart reviews on a random sample of voided orders ( n = 198) to investigate the rate of medication ordering errors among voided orders, and the accuracy of clinician-provided reasons for voiding. RESULTS: We found that 0.49% of all orders were voided. Order voiding was associated with clinician type (physician, pharmacist, nurse, student, other) and order type (inpatient, prescription, home medications by history). An estimated 70 ± 10% of voided orders were due to medication ordering errors. Clinician-provided reasons for voiding were reasonably predictive of the actual cause of error for duplicate orders (72%), but not for other reasons. DISCUSSION AND CONCLUSION: Medication safety initiatives require availability of error data to create repositories for learning and training. The voiding function is available in several electronic health record systems, so order voiding could provide a low-effort mechanism for self-reporting of medication ordering errors. Additional clinician training could help increase the quality of such reporting.

OBJECTIVE: To design and implement a tool that creates a secure, privacy preserving linkage of electronic health record (EHR) data across multiple sites in a large metropolitan area in the United States (Chicago, IL), for use in clinical research. METHODS: The authors developed and distributed a software application that performs standardized data cleaning, preprocessing, and hashing of patient identifiers to remove all protected health information. The application creates seeded hash code combinations of patient identifiers using a Health Insurance Portability and Accountability Act compliant SHA-512 algorithm that minimizes re-identification risk. The authors subsequently linked individual records using a central honest broker with an algorithm that assigns weights to hash combinations in order to generate high specificity matches. RESULTS: The software application successfully linked and de-duplicated 7 million records across 6 institutions, resulting in a cohort of 5 million unique records. Using a manually reconciled set of 11 292 patients as a gold standard, the software achieved a sensitivity of 96% and a specificity of 100%, with a majority of the missed matches accounted for by patients with both a missing social security number and last name change. Using 3 disease examples, it is demonstrated that the software can reduce duplication of patient records across sites by as much as 28%. CONCLUSIONS: Software that standardizes the assignment of a unique seeded hash identifier merged through an agreed upon third-party honest broker can enable large-scale secure linkage of EHR data for epidemiologic and public health research. The software algorithm can improve future epidemiologic research by providing more comprehensive data given that patients may make use of multiple healthcare systems.

OBJECTIVE: To determine whether indication-based computer order entry alerts intercept wrong-patient medication errors. MATERIALS AND METHODS: At an academic medical center serving inpatients and outpatients, we developed and implemented a clinical decision support system to prompt clinicians for indications when certain medications were ordered without an appropriately coded indication on the problem list. Among all the alerts that fired, we identified every instance when a medication order was started but not completed and, within a fixed time interval, the same prescriber placed an order for the same medication for a different patient. We closely reviewed each of these instances to determine whether they were likely to have been intercepted errors. RESULTS: Over a 6-year period 127 320 alerts fired, which resulted in 32 intercepted wrong-patient errors, an interception rate of 0.25 per 1000 alerts. Neither the location of the prescriber nor the type of prescriber affected the interception rate. No intercepted errors were for patients with the same last name, but in 59% of the intercepted errors the prescriber had both patients' charts open when the first order was initiated. DISCUSSION: Indication alerts linked to the problem list have previously been shown to improve problem list completion. This analysis demonstrates another benefit, the interception of wrong-patient medication errors. CONCLUSIONS: Indication-based alerts yielded a wrong-patient medication error interception rate of 0.25 per 1000 alerts. These alerts could be implemented independently or in combination with other strategies to decrease wrong-patient medication errors.

To investigate whether clinical decision support that automates the matching of ordered drugs to problems (clinical diagnoses) on the problem list can enhance the maintenance of both medication and problem lists in the electronic medical record, we designed a clinical decision support system to match ordered drugs on the medication list and ongoing problems on the problem list. We evaluated the capability and performance of this clinical decision support system in medication-problem matching using physician expert chart audits to match ordered drugs to ongoing clinical problems. A clinical decision support system was shown to be useful in improving medication-problem matches in 140 randomly selected audited patient encounters in three inpatient units. Enhanced maintenance of both the medication and problem lists can permit the exploitation of advanced decision support strategies that yield higher patient safety.

BACKGROUND: Automated clinical decision support has shown promise in reducing medication errors; however, clinicians often do not comply with alerts. Because renal insufficiency is a common source of medication errors, the authors studied a trial of alerts designed to reduce inpatient administration of medications contraindicated due to renal insufficiency. METHODS: A minimum safe creatinine clearance was established for each inpatient formulary medication. Alerts recommending cancellation appeared when a medication order was initiated for a patient whose estimated creatinine clearance was less than the minimum safe creatinine clearance for the medication. Administration of medications in patients with creatinine clearances less than the medication's minimum safe clearance were studied for 14 months after, and four months before, alert implementation. In addition, the impact of patient age, gender, degree of renal dysfunction, time of day, and duration of housestaff training on the likelihood of housestaff compliance with the alerts was examined. RESULTS: The likelihood of a patient receiving at least one dose of contraindicated drug after the order was initiated decreased from 89% to 47% (p < 0.0001) after alert implementation. Analysis of the alerts seen by housestaff showed that alert compliance was higher in male patients (57% vs. 38%, p = 0.02), increased with the duration of housestaff training (p = 0.04), and increased in patients with worsening renal function (p = 0.007). CONCLUSION: Alerts were effective in decreasing the ordering and administration of drugs contraindicated due to renal insufficiency. Compliance with the alerts was higher in male patients, increased with the duration of housestaff training, and increased in patients with more severe renal dysfunction.

OBJECTIVE: Automated clinical decision support (CDS) has shown promise in improving safe medication use. The authors performed a trial of CDS, given both during computerized physician order entry (CPOE) and in response to new laboratory results, comparing the time courses of clinician behaviors related to digoxin use before and after implementation of the alerts. DESIGN: Alerts were implemented to notify of the potential risk from low electrolyte concentrations or unknown digoxin or electrolyte concentrations during CPOE. Alerts were also generated in response to newly reported hypokalemia and hypomagnesemia in patients given digoxin. MEASUREMENTS: Clinician responses to the alerts for six months were compared with responses to similar situations for six months prior to implementation. RESULTS: During CPOE, checking for unknown serum values increased after implementation compared with control at one hour: 19% vs. 6% for digoxin, 57% vs. 9% for potassium, and 40% vs. 12% for magnesium as well as at 24 hours (p < 0.01 for all comparisons). Electrolyte supplementation increased with newly reported hypokalemia and hypomagnesemia after implementation at one hour: 35% vs. 6% and 49% vs. 5% for potassium and magnesium, respectively, as well as at 24 hours (p < 0.01 for all comparisons). During CPOE, supplementation for hypokalemia was not improved, whereas supplementation for hypomagnesemia improved at one hour (p < 0.05). CONCLUSION: Overall, the alerts improved the safe use of digoxin. During CPOE, alerts associated with missing levels were effective. For hypokalemia and hypomagnesemia, the alerts given during CPOE were not as effective as those given at the time of newly reported low electrolytes.