Population health management (PHM) is an important approach to promote wellness and deliver health care to targeted individuals who meet criteria for preventive measures or treatment. A critical component for any PHM program is a data analytics platform that can target those eligible individuals. OBJECTIVE: The aim of this study was to design and implement a scalable standards-based clinical decision support (CDS) approach to identify patient cohorts for PHM and maximize opportunities for multi-site dissemination. MATERIALS AND METHODS: An architecture was established to support bidirectional data exchanges between heterogeneous electronic health record (EHR) data sources, PHM systems, and CDS components. HL7 Fast Healthcare Interoperability Resources and CDS Hooks were used to facilitate interoperability and dissemination. The approach was validated by deploying the platform at multiple sites to identify patients who meet the criteria for genetic evaluation of familial cancer. RESULTS: The Genetic Cancer Risk Detector (GARDE) platform was created and is comprised of four components: (1) an open-source CDS Hooks server for computing patient eligibility for PHM cohorts, (2) an open-source Population Coordinator that processes GARDE requests and communicates results to a PHM system, (3) an EHR Patient Data Repository, and (4) EHR PHM Tools to manage patients and perform outreach functions. Site-specific deployments were performed on onsite virtual machines and cloud-based Amazon Web Services. DISCUSSION: GARDE's component architecture establishes generalizable standards-based methods for computing PHM cohorts. Replicating deployments using one of the established deployment methods requires minimal local customization. Most of the deployment effort was related to obtaining site-specific information technology governance approvals.

OBJECTIVE: To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user's view. MATERIALS AND METHODS: We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate <0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1-there is no difference in precision among prediction models; and H2-the removal of any feature category does not change precision. RESULTS: A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P < 0.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (-0.147, P = 0.001). CONCLUSIONS: Machine learning potentially enables the intelligent filtering of medication alerts.

OBJECTIVE: HL7 SMART on FHIR apps have the potential to improve healthcare delivery and EHR usability, but providers must be aware of the apps and use them for these potential benefits to be realized. The HL7 CDS Hooks standard was developed in part for this purpose. The objective of this study was to determine if contextually relevant CDS Hooks prompts can increase utilization of a SMART on FHIR medical reference app (MDCalc for EHR). MATERIALS AND METHODS: We conducted a 7-month, provider-randomized trial with 70 providers in a single emergency department. The intervention was a collection of CDS Hooks prompts suggesting the use of 6 medical calculators in a SMART on FHIR medical reference app. The primary outcome was the percentage of provider-patient interactions in which the app was used to view a recommended calculator. Secondary outcomes were app usage stratified by individual calculators. RESULTS: Intervention group providers viewed a study calculator in the app in 6.0% of interactions compared to 2.6% in the control group (odds ratio = 2.45, 95% CI, 1.2-5.2, P value .02), an increase of 130%. App use was significantly greater for 2 of 6 calculators. DISCUSSION AND CONCLUSION: Contextually relevant CDS Hooks prompts led to a significant increase in SMART on FHIR app utilization. This demonstrates the potential of using CDS Hooks to guide appropriate use of SMART on FHIR apps and was a primary motivation for the development of the standard. Future research may evaluate potential impacts on clinical care decisions and outcomes.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.