OBJECTIVE: To guide research data services (RDS) teams in managing researcher variability (eg, differing deadlines, funding, expertise) when honest-brokering data, we present a framework based on operations management principles and a food service analogy. MATERIALS AND METHODS: Our framework describes 4 data service offerings with different levels of efficiency and service customization: vending machine, fast food, custom meal, and personal chef. We describe examples from 2 institutions. RESULTS: Vending machine and fast food are efficient but less customizable, making them better-suited for researchers with limited funding or time. Custom meal and personal chef are less efficient but more customized, making them well suited for better-resourced researchers. DISCUSSION: Efficiency and service tradeoffs should be balanced to align with demand and institutional goals. RDS teams can overcome such tradeoffs through uncompromised reduction or low-cost accommodation approaches. CONCLUSION: Our framework can be applied by RDS teams in their design and implementation of data services.

OBJECTIVES: To highlight the often overlooked role of user interface (UI) design in mitigating bias in artificial intelligence (AI)-based clinical decision support (CDS). MATERIALS AND METHODS: This perspective paper discusses the interdependency between AI-based algorithm development and UI design and proposes strategies for increasing the safety and efficacy of CDS. RESULTS: The role of design in biasing user behavior is well documented in behavioral economics and other disciplines. We offer an example of how UI designs play a role in how bias manifests in our machine learning-based CDS development. DISCUSSION: Much discussion on bias in AI revolves around data quality and algorithm design; less attention is given to how UI design can exacerbate or mitigate limitations of AI-based applications. CONCLUSION: This work highlights important considerations including the role of UI design in reinforcing/mitigating bias, human factors methods for identifying issues before an application is released, and risk communication strategies.

Healthcare systems are hampered by incomplete and fragmented patient health records. Record linkage is widely accepted as a solution to improve the quality and completeness of patient records. However, there does not exist a systematic approach for manually reviewing patient records to create gold standard record linkage data sets. We propose a robust framework for creating and evaluating manually reviewed gold standard data sets for measuring the performance of patient matching algorithms. Our 8-point approach covers data preprocessing, blocking, record adjudication, linkage evaluation, and reviewer characteristics. This framework can help record linkage method developers provide necessary transparency when creating and validating gold standard reference matching data sets. In turn, this transparency will support both the internal and external validity of recording linkage studies and improve the robustness of new record linkage strategies.

The OneFlorida Data Trust is a centralized research patient data repository created and managed by the OneFlorida Clinical Research Consortium ("OneFlorida"). It comprises structured electronic health record (EHR), administrative claims, tumor registry, death, and other data on 17.2 million individuals who received healthcare in Florida between January 2012 and the present. Ten healthcare systems in Miami, Orlando, Tampa, Jacksonville, Tallahassee, Gainesville, and rural areas of Florida contribute EHR data, covering the major metropolitan regions in Florida. Deduplication of patients is accomplished via privacy-preserving entity resolution (precision 0.97-0.99, recall 0.75), thereby linking patients' EHR, claims, and death data. Another unique feature is the establishment of mother-baby relationships via Florida vital statistics data. Research usage has been significant, including major studies launched in the National Patient-Centered Clinical Research Network ("PCORnet"), where OneFlorida is 1 of 9 clinical research networks. The Data Trust's robust, centralized, statewide data are a valuable and relatively unique research resource.

OBJECTIVE: This study aimed to understand the association between primary care physician (PCP) proficiency with the electronic health record (EHR) system and time spent interacting with the EHR. MATERIALS AND METHODS: We examined the use of EHR proficiency tools among PCPs at one large academic health system using EHR-derived measures of clinician EHR proficiency and efficiency. Our main predictors were the use of EHR proficiency tools and our outcomes focused on 4 measures assessing time spent in the EHR: (1) total time spent interacting with the EHR, (2) time spent outside scheduled clinical hours, (3) time spent documenting, and (4) time spent on inbox management. We conducted multivariable quantile regression models with fixed effects for physician-level factors and time in order to identify factors that were independently associated with time spent in the EHR. RESULTS: Across 441 primary care physicians, we found mixed associations between certain EHR proficiency behaviors and time spent in the EHR. Across EHR activities studied, QuickActions, SmartPhrases, and documentation length were positively associated with increased time spent in the EHR. Models also showed a greater amount of help from team members in note writing was associated with less time spent in the EHR and documenting. DISCUSSION: Examining the prevalence of EHR proficiency behaviors may suggest targeted areas for initial and ongoing EHR training. Although documentation behaviors are key areas for training, team-based models for documentation and inbox management require further study. CONCLUSIONS: A nuanced association exists between physician EHR proficiency and time spent in the EHR.

OBJECTIVE: Federated learning (FL) allows multiple distributed data holders to collaboratively learn a shared model without data sharing. However, individual health system data are heterogeneous. "Personalized" FL variations have been developed to counter data heterogeneity, but few have been evaluated using real-world healthcare data. The purpose of this study is to investigate the performance of a single-site versus a 3-client federated model using a previously described Coronavirus Disease 19 (COVID-19) diagnostic model. Additionally, to investigate the effect of system heterogeneity, we evaluate the performance of 4 FL variations. MATERIALS AND METHODS: We leverage a FL healthcare collaborative including data from 5 international healthcare systems (US and Europe) encompassing 42 hospitals. We implemented a COVID-19 computer vision diagnosis system using the Federated Averaging (FedAvg) algorithm implemented on Clara Train SDK 4.0. To study the effect of data heterogeneity, training data was pooled from 3 systems locally and federation was simulated. We compared a centralized/pooled model, versus FedAvg, and 3 personalized FL variations (FedProx, FedBN, and FedAMP). RESULTS: We observed comparable model performance with respect to internal validation (local model: AUROC 0.94 vs FedAvg: 0.95, P = .5) and improved model generalizability with the FedAvg model (P < .05). When investigating the effects of model heterogeneity, we observed poor performance with FedAvg on internal validation as compared to personalized FL algorithms. FedAvg did have improved generalizability compared to personalized FL algorithms. On average, FedBN had the best rank performance on internal and external validation. CONCLUSION: FedAvg can significantly improve the generalization of the model compared to other personalization FL algorithms; however, at the cost of poor internal validity. Personalized FL may offer an opportunity to develop both internal and externally validated algorithms.

OBJECTIVE: To assess the practice- and market-level factors associated with the amount of provider health information exchange (HIE) use. MATERIALS AND METHODS: Provider and practice-level data was drawn from the Meaningful Use Stage 2 Public Use Files from the Centers for Medicare and Medicaid Services, the Physician Compare National Downloadable File, and the Compendium of US Health Systems, among other sources. We analyzed the relationship between provider HIE use and practice and market factors using multivariable linear regression and compared primary care providers (PCPs) to non-PCPs. Provider volume of HIE use is measured as the percentage of referrals sent with electronic summaries of care (eSCR) reported by eligible providers attesting to the Meaningful Use electronic health record (EHR) incentive program in 2016. RESULTS: Providers used HIE in 49% of referrals; PCPs used HIE in fewer referrals (43%) than non-PCPs (57%). Provider use of products from EHR vendors was negatively related to HIE use, while use of Athenahealth and Greenway Health products were positively related to HIE use. Providers treating, on average, older patients and greater proportions of patients with diabetes used HIE for more referrals. Health system membership, market concentration, and state HIE consent policy were unrelated to provider HIE use. DISCUSSION: HIE use during referrals is low among office-based providers with the capability for exchange, especially PCPs. Practice-level factors were more commonly associated with greater levels of HIE use than market-level factors. CONCLUSION: This furthers the understanding that market forces, like competition, may be related to HIE adoption decisions but are less important for use once adoption has occurred.

Journal of the American Medical Informatics Association, doi: 10.1093/jamia/ocab024 The author name “Julia Adler-Milstein” was incorrectly given as “Julia Adler-Milstien”. This has been corrected online.

OBJECTIVE: To synthesize data quality (DQ) dimensions and assessment methods of real-world data, especially electronic health records, through a systematic scoping review and to assess the practice of DQ assessment in the national Patient-centered Clinical Research Network (PCORnet). MATERIALS AND METHODS: We started with 3 widely cited DQ literature-2 reviews from Chan et al (2010) and Weiskopf et al (2013a) and 1 DQ framework from Kahn et al (2016)-and expanded our review systematically to cover relevant articles published up to February 2020. We extracted DQ dimensions and assessment methods from these studies, mapped their relationships, and organized a synthesized summarization of existing DQ dimensions and assessment methods. We reviewed the data checks employed by the PCORnet and mapped them to the synthesized DQ dimensions and methods. RESULTS: We analyzed a total of 3 reviews, 20 DQ frameworks, and 226 DQ studies and extracted 14 DQ dimensions and 10 assessment methods. We found that completeness, concordance, and correctness/accuracy were commonly assessed. Element presence, validity check, and conformance were commonly used DQ assessment methods and were the main focuses of the PCORnet data checks. DISCUSSION: Definitions of DQ dimensions and methods were not consistent in the literature, and the DQ assessment practice was not evenly distributed (eg, usability and ease-of-use were rarely discussed). Challenges in DQ assessments, given the complex and heterogeneous nature of real-world data, exist. CONCLUSION: The practice of DQ assessment is still limited in scope. Future work is warranted to generate understandable, executable, and reusable DQ measures.

OBJECTIVE: In the context of patient broad consent for future research uses of their identifiable health record data, we compare the effectiveness of interactive trust-enhanced e-consent, interactive-only e-consent, and standard e-consent (no interactivity, no trust enhancement). MATERIALS AND METHODS: A randomized trial was conducted involving adult participants making a scheduled primary care visit. Participants were randomized into 1 of the 3 e-consent conditions. Primary outcomes were patient-reported satisfaction with and subjective understanding of the e-consent. Secondary outcomes were objective knowledge, perceived voluntariness, trust in medical researchers, consent decision, and time spent using the application. Outcomes were assessed immediately after use of the e-consent and at 1-week follow-up. RESULTS: Across all conditions, participants (N = 734) reported moderate-to-high satisfaction with consent (mean 4.3 of 5) and subjective understanding (79.1 of 100). Over 94% agreed to share their health record data. No statistically significant differences in outcomes were observed between conditions. Irrespective of condition, black participants and those with lower education reported lower satisfaction, subjective understanding, knowledge, perceived voluntariness, and trust in medical researchers, as well as spent more time consenting. CONCLUSIONS: A large majority of patients were willing to share their identifiable health records for research, and they reported positive consent experiences. However, incorporating optional additional information and messages designed to enhance trust in the research process did not improve consent experiences. To improve poorer consent experiences of racial and ethnic minority participants and those with lower education, other novel consent technologies and processes may be valuable. (An Interactive Patient-Centered Consent for Research Using Medical Records; NCT03063268).

OBJECTIVE: The purpose of this study was to assess patient perceptions of using an interactive electronic consent (e-consent) application when deciding whether or not to grant broad consent for research use of their identifiable electronic health record (EHR) information. MATERIALS AND METHODS: For this qualitative study, we conducted a series of 42 think-aloud interviews with 32 adults. Interview transcripts were coded and analyzed using a modified grounded theory approach. RESULTS: We identified themes related to patient preferences, reservations, and mixed attitudes toward consenting electronically; low- and high-information-seeking behavior; and an emphasis on reassuring information, such as data protections and prohibitions against sharing data with pharmaceutical companies. Participants expressed interest in the types of information contained in their EHRs, safeguards protecting EHR data, and specifics on studies that might use their EHR data. DISCUSSION: This study supports the potential value of interactive e-consent applications that allow patients to customize their consent experience. This study also highlights that some people have concerns about e-consent platforms and desire more detailed information about administrative processes and safeguards that protect EHR data used in research. CONCLUSION: This study contributes new insights on how e-consent applications could be designed to ensure that patients' information needs are met when seeking consent for research use of health record information. Also, this study offers a potential electronic approach to meeting the new Common Rule requirement that consent documents contain a "concise and focused" presentation of key information followed by more details.

Objective: Widespread health information exchange (HIE) is a national objective motivated by the promise of improved care and a reduction in costs. Previous reviews have found little rigorous evidence that HIE positively affects these anticipated benefits. However, early studies of HIE were methodologically limited. The purpose of the current study is to review the recent literature on the impact of HIE. Methods: We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to conduct our systematic review. PubMed and Scopus databases were used to identify empirical articles that evaluated HIE in the context of a health care outcome. Results: Our search strategy identified 24 articles that included 63 individual analyses. The majority of the studies were from the United States representing 9 states; and about 40% of the included analyses occurred in a handful of HIEs from the state of New York. Seven of the 24 studies used designs suitable for causal inference and all reported some beneficial effect from HIE; none reported adverse effects. Conclusions: The current systematic review found that studies with more rigorous designs all reported benefits from HIE. Such benefits include fewer duplicated procedures, reduced imaging, lower costs, and improved patient safety. We also found that studies evaluating community HIEs were more likely to find benefits than studies that evaluated enterprise HIEs or vendor-mediated exchanges. Overall, these finding bode well for the HIEs ability to deliver on anticipated improvements in care delivery and reduction in costs.

In this case report, the authors describe the implementation of a system for collecting patient-reported outcomes and integrating results in an electronic health record. The objective was to identify lessons learned in overcoming barriers to collecting and integrating patient-reported outcomes in an electronic health record. The authors analyzed qualitative data in 42 documents collected from system development meetings, written feedback from users, and clinical observations with practice staff, providers, and patients. Guided by the Unified Theory on the Adoption and Use of Information Technology, 5 emergent themes were identified. Two barriers emerged: (i) uncertain clinical benefit and (ii) time, work flow, and effort constraints. Three facilitators emerged: (iii) process automation, (iv) usable system interfaces, and (v) collecting patient-reported outcomes for the right patient at the right time. For electronic health record-integrated patient-reported outcomes to succeed as useful clinical tools, system designers must ensure the clinical relevance of the information being collected while minimizing provider, staff, and patient burden.

In today’s society, most people are both consumers of information technology and of health care. Virtually every person has consumed health care and will consume more as one ages. Moreover, 84% of US households own a computer, 1 and 64% of adults own a smartphone. 2 We carry pocket-sized devices that connect us to people around the world and vast stores of information. With these technologies, we manage our lives from mundane activities like reading, checking the weather, making to-do lists, and buying books and clothes, to more complex tasks such as learning, managing finances, shopping for houses, and maintaining ties with friends and family around the world. With such diverse and powerful technologies at our fingertips and myriad societal-level health care challenges in cost, quality, and outcome, it is tantalizing to imagine all of the ways that health information technologies (health IT) can be used to enhance people’s health and societies’ health care delivery. Patient-centered care respects and responds to individual differences in patient preferences, needs, and values. 3 To respond to such differences and achieve patient-centered care, patients and health care professionals must engage in constant communication. In recent years, researchers have examined a number of ostensibly patient-oriented technologies that could enhance such communication, including patient portals, personal health records (PHRs), and mobile health (mHealth) applications. Furthermore, it is not difficult to conceptualize pathways through which such information systems might improve communication between patients and clinicians, create more patient-centered care, and help achieve the triple aim of better experiences of care, better population health, and lower health care costs. 3 Yet, practically, these enticing tools and outcomes are far from reality. There is scant evidence that patients frequently or effectively access and use information systems that engage them and improve patient-centered care delivery. For example, patients generally have positive attitudes toward using patient portals, but studies have not shown portals to have positive impacts on patient empowerment, 4,5 health outcomes, or costs. 6,7 Also, racial and ethnic differences may impede widespread portal adoption and use, 6 and this threatens to compound already-existing disparities in health care access and communication. Another often-studied system type, the PHR, has been shown to infrequently contain patient-oriented features, which is also likely to limit patient-clinician communication. 6 Next, as smartphone adoption has increased, mHealth technologies have emerged as another set of tools that may enhance patient-clinician communication. Yet despite the existence of many applications, including hundreds for cancer alone, 8 we lack strong research evidence on how to design and use mHealth applications to consistently achieve patient-centered care. 3 Finally, the study of patient-facing systems to improve patient-centered care cannot be disentangled from the study of electronic health records (EHRs). EHRs are nearing ubiquity in the US health care system, meaning that patient engagement, communication, and attainment of patient-centered care is also inexorably tied to the design and use of EHRs. This special focus issue follows from the 2014 annual Workshop on Interactive Systems in Healthcare (WISH). The WISH workshop aims to promote deeper and more profound connections among the biomedical informatics, human-computer interaction, medical sociology, and anthropology communities. WISH 2014 focused on the challenge that information systems often fall short in adequately engaging patients and ensuring that clinical decisions are patient-centered. This may be attributed to a disconnect between system designers’ understanding of clinical work and care processes, a lack of clear protocols defining how patient-engaged technologies should be adopted and used, or an insufficient understanding of people’s information needs, preferences, and values. Therefore, the articles in this special focus issue reflect discipline-spanning research teams, methodologies, and perspectives while highlighting new approaches to designing, developing, and evaluating interactive information systems to support patient-centered care and patient engagement. We have organized the articles in this special focus issue into four themes: health IT for patient-centered heath care delivery and management, patient–provider interactions mediated by health IT, pervasive and mobile technologies to promote patient engagement, and designing for underserved patient populations. The first theme includes studies that focus on improving EHR and PHR effectiveness and patient-centered care outcome. The studies range from designing a scaffolding system to existing EHR, customizing a commercial EHR system, identifying strategies of using EHR during patient consultation, re-examining the role of EHR in primary clinical workflows, and designing an experiment to determine the PHR impact on patient engagement. For instance, to improve patient engagement and EHR effectiveness, researchers implemented a scaffolding system to include patient-reported outcomes integrated into the existing EHR and identified both facilitators (e.g., high degree of process automation, good interface usability, capability of targeting the right patients at the right time) and barriers (e.g., uncertain clinical benefits and constraints on time, workflows, and efforts). 9 Similarly, researchers from Texas Children’s Hospital customized a commercial EHR to include the design of a new work element for a cross-functional team to prioritize the outcome measurement in EHR optimization, which significantly improved the outcome status tracking and number of patients involved. 10 Moreover, researchers continue to investigate workflow issues 11 and the tensions of using EHR while interacting with patients face to face 12 in primary care settings, which helps inform commercial vendors about how to improve the design of EHR for accommodating the dynamic needs of frontline clinicians. Furthermore, an observational study on the use of PHR indicated significant improvement in the HbA1c levels of the active and super user groups while no other health outcomes improved. There was also no statistically significant improvement observed in patient engagement during the study. While the research context is limited in coronary artery disease patients, we hope this study could shed light on the current debates of PHR usefulness and effectiveness, and invite additional efforts in examining the effectiveness of patient-centered systems, including patient portals, on improving patient engagement and health outcomes. The second theme focuses on the design of health IT to improve the quality of patient–provider interaction. Studies included the development of a web-based toolkit to improve patient education and involvement in the care plan during hospitalization, 13 the design of a dashboard to facilitate data collection and efficient use of patient-reported outcomes, 14 and design recommendations for a web-based tool to satisfy the caregivers’ information needs in the context of inpatient pediatric hematopoietic stem cell transplant. 15 As there is a paucity of research in designing IT tools to support patients and caregivers in an inpatient setting, we hope that these studies will provide readers with valuable insights and practical design experiences for approaching the problem, including ways to engage patients, caregivers, and providers in the iterative user-centered design process. The third theme involves the use of pervasive and mobile technologies to promote patient engagement, such as. 16 Many mHealth technologies were found to be useful for patients to manage their care and improve health outcomes. However, certain patient factors must be considered when designing mHealth technologies, as they play a pivotal role in patient engagement. For example, patient factors like ethnicity, health literacy, and age were found to impact the use of mHealth applications for managing medication adherence 17 and service members’ background characteristics were reported to impact their engagement with an mHealth application for managing their post-trauma issues. 18 On the other hand, one size does not fit all. It is therefore important to customize mHealth tools for patients with special needs to ensure patient engagement. For instance, an mHealth tool designed for diabetic patients from economically disadvantaged communities and ethnic minorities was found to help patients self-monitor and reflect, 19 and a PHR application customized for post-cardiothoracic surgery patients was useful to support their medication management and tracking in a hospital setting. 20 The last theme that we identified is centered around designing for underserved patient populations, e.g. 21 It is well-known in the health informatics community that studying underserved patient populations is challenging. Thus, most previous research focused on a single case study. While a single case study offers valuable knowledge, cross-case analysis of diverse case studies offers exceptionally important insights to the success factors, barriers, and common patterns identified across multiple cases. 22 In addition, research targeted at underserved populations offers lessons particularly instrumental in the design of health IT to meet the specific needs of individual underserved populations. For example, a large-scale national program succeeded in promoting health and well-being in older adults using a suite of accessible computing, 23 a longitudinal participatory design approach supported the design of mHealth applications for overcoming perinatal depression of women from vulnerable populations after their pregnancies, 24 and the use of daily questionnaires helped to identify the association between service members’ background characteristics and their engagement with an mHealth application for managing their post-trauma issues/conditions. 18 In conclusion, researchers in this growing, vibrant health informatics community have been diligently exploring a range of relevant topics in the design, implementation, and evaluation of interactive, patient-centered health IT systems for enhancing patient engagement, as evidenced in the sample research included in this special focus issue. While challenges remain and future work abounds, we believe that our effort has led us a step closer to achieving a high level of health care quality and outcomes.

OBJECTIVE: In an effort to understand better the federal electronic health record (EHR) incentive programme's challenges, this study compared hospitals that did and did not receive meaningful use (MU) payments in the programme's first year based on the challenges they anticipated a year before. MATERIALS AND METHODS: This cross-sectional study used 2010 American Hospital Association survey data and 2011 Centers for Medicare and Medicaid Services data that identify hospitals receiving MU payments. Multivariate regression analysis assessed differences in 2010 anticipated challenges to MU for hospitals that were successful in earning 2011 MU payment compared to hospitals that intended to participate in the programme but were not yet successful. RESULTS: The study sample consisted of 2475 hospitals, 313 of which received MU payments in 2011. Controlling for standard hospital characteristics, hospitals that reported the computerized provider order entry (CPOE) MU criterion as a primary challenge were 18% less likely to receive a 2011 MU payment compared to hospitals that reported other criteria as primary challenges. DISCUSSION: CPOE was the main challenge among hospitals that failed to achieve MU in the first year of the programme. In order to maximize the incentive programme's effectiveness, policymakers, healthcare organizations, and EHR vendors may benefit from increased attention to hospitals' challenges with CPOE. CONCLUSION: As the EHR incentive programme matures, policymakers and other stakeholders should consider strategies that maintain the critical elements of MU while adequately supporting hospitals that desire to become MU but are impeded by specific technological, cultural, and organizational adoption and use challenges.