BACKGROUND: Predictive models show promise in healthcare, but their successful deployment is challenging due to limited generalizability. Current external validation often focuses on model performance with restricted feature use from the original training data, lacking insights into their suitability at external sites. Our study introduces an innovative methodology for evaluating features during both the development phase and the validation, focusing on creating and validating predictive models for post-surgery patient outcomes with improved generalizability. METHODS: Electronic health records (EHRs) from 4 countries (United States, United Kingdom, Finland, and Korea) were mapped to the OMOP Common Data Model (CDM), 2008-2019. Machine learning (ML) models were developed to predict post-surgery prolonged opioid use (POU) risks using data collected 6 months before surgery. Both local and cross-site feature selection methods were applied in the development and external validation datasets. Models were developed using Observational Health Data Sciences and Informatics (OHDSI) tools and validated on separate patient cohorts. RESULTS: Model development included 41 929 patients, 14.6% with POU. The external validation included 31 932 (UK), 23 100 (US), 7295 (Korea), and 3934 (Finland) patients with POU of 44.2%, 22.0%, 15.8%, and 21.8%, respectively. The top-performing model, Lasso logistic regression, achieved an area under the receiver operating characteristic curve (AUROC) of 0.75 during local validation and 0.69 (SD = 0.02) (averaged) in external validation. Models trained with cross-site feature selection significantly outperformed those using only features from the development site through external validation (P < .05). CONCLUSIONS: Using EHRs across four countries mapped to the OMOP CDM, we developed generalizable predictive models for POU. Our approach demonstrates the significant impact of cross-site feature selection in improving model performance, underscoring the importance of incorporating diverse feature sets from various clinical settings to enhance the generalizability and utility of predictive healthcare models.

BACKGROUND: Treatment of surgical pain is a common reason for opioid prescriptions. Being able to predict which patients are at risk for opioid abuse, dependence, and overdose (opioid-related adverse outcomes [OR-AE]) could help physicians make safer prescription decisions. We aimed to develop a machine-learning algorithm to predict the risk of OR-AE following surgery using Medicaid data with external validation across states. METHODS: Five machine learning models were developed and validated across seven US states (90-10 data split). The model output was the risk of OR-AE 6-months following surgery. The models were evaluated using standard metrics and area under the receiver operating characteristic curve (AUC) was used for model comparison. We assessed calibration for the top performing model and generated bootstrap estimations for standard deviations. Decision curves were generated for the top-performing model and logistic regression. RESULTS: We evaluated 96,974 surgical patients aged 15 and 64. During the 6-month period following surgery, 10,464 (10.8%) patients had an OR-AE. Outcome rates were significantly higher for patients with depression (17.5%), diabetes (13.1%) or obesity (11.1%). The random forest model achieved the best predictive performance (AUC: 0.877; F1-score: 0.57; recall: 0.69; precision:0.48). An opioid disorder diagnosis prior to surgery was the most important feature for the model, which was well calibrated and had good discrimination. CONCLUSIONS: A machine learning models to predict risk of OR-AE following surgery performed well in external validation. This work could be used to assist pain management following surgery for Medicaid beneficiaries and supports a precision medicine approach to opioid prescribing.

OBJECTIVE: To assess the efficacy of interruptive electronic alerts in improving adherence to the American Board of Internal Medicine's Choosing Wisely recommendations to reduce unnecessary laboratory testing. MATERIALS AND METHODS: We administered 5 cluster randomized controlled trials simultaneously, using electronic medical record alerts regarding prostate-specific antigen (PSA) testing, acute sinusitis treatment, vitamin D testing, carotid artery ultrasound screening, and human papillomavirus testing. For each alert, we assigned 5 outpatient clinics to an interruptive alert and 5 were observed as a control. Primary and secondary outcomes were the number of postalert orders per 100 patients at each clinic and number of triggered alerts divided by orders, respectively. Post hoc analysis evaluated whether physicians experiencing interruptive alerts reduced their alert-triggering behaviors. RESULTS: Median postalert orders per 100 patients did not differ significantly between treatment and control groups; absolute median differences ranging from 0.04 to 0.40 for PSA testing. Median alerts per 100 orders did not differ significantly between treatment and control groups; absolute median differences ranged from 0.004 to 0.03. In post hoc analysis, providers receiving alerts regarding PSA testing in men were significantly less likely to trigger additional PSA alerts than those in the control sites (Incidence Rate Ratio 0.12, 95% CI [0.03-0.52]). DISCUSSION: Interruptive point-of-care alerts did not yield detectable changes in the overall rate of undesired orders or the order-to-alert ratio between active and silent sites. Complementary behavioral or educational interventions are likely needed to improve efforts to curb medical overuse. CONCLUSION: Implementation of interruptive alerts at the time of ordering was not associated with improved adherence to 5 Choosing Wisely guidelines. TRIAL REGISTRATION: NCT02709772.

OBJECTIVE: To assess usability and usefulness of a machine learning-based order recommender system applied to simulated clinical cases. MATERIALS AND METHODS: 43 physicians entered orders for 5 simulated clinical cases using a clinical order entry interface with or without access to a previously developed automated order recommender system. Cases were randomly allocated to the recommender system in a 3:2 ratio. A panel of clinicians scored whether the orders placed were clinically appropriate. Our primary outcome included the difference in clinical appropriateness scores. Secondary outcomes included total number of orders, case time, and survey responses. RESULTS: Clinical appropriateness scores per order were comparable for cases randomized to the order recommender system (mean difference -0.11 order per score, 95% CI: [-0.41, 0.20]). Physicians using the recommender placed more orders (median 16 vs 15 orders, incidence rate ratio 1.09, 95%CI: [1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Physicians used recommender suggestions in 98% of available cases. Approximately 95% of participants agreed the system would be useful for their workflows. DISCUSSION: User testing with a simulated electronic medical record interface can assess the value of machine learning and clinical decision support tools for clinician usability and acceptance before live deployments. CONCLUSIONS: Clinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface in a simulated setting. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.

OBJECTIVE: Build probabilistic topic model representations of hospital admissions processes and compare the ability of such models to predict clinical order patterns as compared to preconstructed order sets. MATERIALS AND METHODS: The authors evaluated the first 24 hours of structured electronic health record data for > 10 K inpatients. Drawing an analogy between structured items (e.g., clinical orders) to words in a text document, the authors performed latent Dirichlet allocation probabilistic topic modeling. These topic models use initial clinical information to predict clinical orders for a separate validation set of > 4 K patients. The authors evaluated these topic model-based predictions vs existing human-authored order sets by area under the receiver operating characteristic curve, precision, and recall for subsequent clinical orders. RESULTS: Existing order sets predict clinical orders used within 24 hours with area under the receiver operating characteristic curve 0.81, precision 16%, and recall 35%. This can be improved to 0.90, 24%, and 47% ( P < 10 -20 ) by using probabilistic topic models to summarize clinical data into up to 32 topics. Many of these latent topics yield natural clinical interpretations (e.g., "critical care," "pneumonia," "neurologic evaluation"). DISCUSSION: Existing order sets tend to provide nonspecific, process-oriented aid, with usability limitations impairing more precise, patient-focused support. Algorithmic summarization has the potential to breach this usability barrier by automatically inferring patient context, but with potential tradeoffs in interpretability. CONCLUSION: Probabilistic topic modeling provides an automated approach to detect thematic trends in patient care and generate decision support content. A potential use case finds related clinical orders for decision support.

OBJECTIVE: Computerized clinical reminders (CRs) were designed to reduce clinicians' reliance on their memory and to present evidence-based guidelines at point of care. However, the literature indicates that CR adoption and effectiveness has been variable. We examined the impact of four design modifications to CR software on learnability, efficiency, usability, and workload for intake nursing personnel in an outpatient clinic setting. These modifications were included in a redesign primarily to address barriers to effective CR use identified during a previous field study. DESIGN: In a simulation experiment, 16 nurses used prototypes of the current and redesigned system in a within-subject comparison for five simulated patient encounters. Prior to the experimental session, participants completed an exploration session, where "learnability" of the current and redesigned systems was assessed. MEASUREMENTS: Time, performance, and survey data were analyzed in conjunction with semi-structured debrief interview data. RESULTS: The redesign was found to significantly increase learnability for first-time users as measured by time to complete the first CR, efficiency as measured by task completion time for two of five patient scenarios, usability as determined by all three groupings of questions taken from a commonly used survey instrument, and two of six workload subscales of the NASA Task Load Index (TLX) survey: mental workload and frustration. CONCLUSION: Modest design modifications to existing CR software positively impacted variables that likely would increase the willingness for first-time nursing personnel to adopt and consistently use CRs.

OBJECTIVE: Evidence-based practices in preventive care and chronic disease management are inconsistently implemented. Computerized clinical reminders (CRs) can improve compliance with these practices in outpatient settings. However, since clinician adherence to CR recommendations is quite variable and declines over time, we conducted observations to determine barriers and facilitators to the effective use of CRs. DESIGN: We conducted an observational study of nurses and providers interacting with CRs in outpatient primary care clinics for two days in each of four geographically distributed Veterans Administration (VA) medical centers. MEASUREMENTS: Three observers recorded interactions of 35 nurses and 55 physicians and mid-level practitioners with the CRs, which function as part of an electronic medical record. Field notes were typed, coded in a spreadsheet, and then sorted into logical categories. We then integrated findings across observations into meaningful patterns and abstracted the data into themes, such as recurrent strategies. Several of these themes translated directly to barriers and facilitators to effective CR use. RESULTS: Optimally using the CR system for its intended purpose was impeded by (1) lack of coordination between nurses and providers; (2) using the reminders while not with the patient, impairing data acquisition and/or implementation of recommended actions; (3) workload; (4) lack of CR flexibility; and (5) poor interface usability. Facilitators included (1) limiting the number of reminders at a site; (2) strategic location of the computer workstations; (3) integration of reminders into workflow; and (4) the ability to document system problems and receive prompt administrator feedback. CONCLUSION: We identified barriers that might explain some of the variability in the use of CRs. Although these barriers may be difficult to overcome, some strategies may increase user acceptance and therefore the effectiveness of the CRs. These include explicitly assigning responsibility for each CR to nurses or providers, improving visibility of positive results from CRs in the electronic medical record, creating a feedback mechanism about CR use, and limiting the overall number of CRs.

OBJECTIVE: Substantial variations in adherence to guidelines for human immunodeficiency virus (HIV) care have been documented. To evaluate their effectiveness in improving quality of care, ten computerized clinical reminders (CRs) were implemented at two pilot and eight study sites. The aim of this study was to identify human factors barriers to the use of these CRs. DESIGN: Observational study was conducted of CRs in use at eight outpatient clinics for one day each and semistructured interviews were conducted with physicians, pharmacists, nurses, and case managers. MEASUREMENTS: Detailed handwritten field notes of interpretations and actions using the CRs and responses to interview questions were used for measurement. RESULTS: Barriers present at more than one site were (1) workload during patient visits (8 of 8 sites), (2) time to document when a CR was not clinically relevant (8 of 8 sites), (3) inapplicability of the CR due to context-specific reasons (9 of 26 patients), (4) limited training on how to use the CR software for rotating staff (5 of 8 sites) and permanent staff (3 of 8 sites), (5) perceived reduction of quality of provider-patient interaction (3 of 23 permanent staff), and (6) the decision to use paper forms to enable review of resident physician orders prior to order entry (2 of 8 sites). CONCLUSION: Six human factors barriers to the use of HIV CRs were identified. Reducing these barriers has the potential to increase use of the CRs and thereby improve the quality of HIV care.