OBJECTIVES: Healthcare organizations, including Clinical and Translational Science Awards (CTSA) hubs funded by the National Institutes of Health, seek to enable secondary use of electronic health record (EHR) data through an enterprise data warehouse for research (EDW4R), but optimal approaches are unknown. In this qualitative study, our goal was to understand EDW4R impact, sustainability, demand management, and accessibility. MATERIALS AND METHODS: We engaged a convenience sample of informatics leaders from CTSA hubs (n = 21) for semi-structured interviews and completed a directed content analysis of interview transcripts. RESULTS: EDW4R have created institutional capacity for single- and multi-center studies, democratized access to EHR data for investigators from multiple disciplines, and enabled the learning health system. Bibliometrics have been challenging due to investigator non-compliance, but one hub's requirement to link all study protocols with funding records enabled quantifying an EDW4R's multi-million dollar impact. Sustainability of EDW4R has relied on multiple funding sources with a general shift away from the CTSA grant toward institutional and industry support. To address EDW4R demand, institutions have expanded staff, used different governance approaches, and provided investigator self-service tools. EDW4R accessibility can benefit from improved tools incorporating user-centered design, increased data literacy among scientists, expansion of informaticians in the workforce, and growth of team science. DISCUSSION: As investigator demand for EDW4R has increased, approaches to tracking impact, ensuring sustainability, and improving accessibility of EDW4R resources have varied. CONCLUSION: This study adds to understanding of how informatics leaders seek to support investigators using EDW4R across the CTSA consortium and potentially elsewhere.

OBJECTIVES: Collider bias is a common threat to internal validity in clinical research but is rarely mentioned in informatics education or literature. Conditioning on a collider, which is a variable that is the shared causal descendant of an exposure and outcome, may result in spurious associations between the exposure and outcome. Our objective is to introduce readers to collider bias and its corollaries in the retrospective analysis of electronic health record (EHR) data. TARGET AUDIENCE: Collider bias is likely to arise in the reuse of EHR data, due to data-generating mechanisms and the nature of healthcare access and utilization in the United States. Therefore, this tutorial is aimed at informaticians and other EHR data consumers without a background in epidemiological methods or causal inference. SCOPE: We focus specifically on problems that may arise from conditioning on forms of healthcare utilization, a common collider that is an implicit selection criterion when one reuses EHR data. Directed acyclic graphs (DAGs) are introduced as a tool for identifying potential sources of bias during study design and planning. References for additional resources on causal inference and DAG construction are provided.

OBJECTIVE: To improve problem list documentation and care quality. MATERIALS AND METHODS: We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. RESULTS: There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. DISCUSSION: The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. CONCLUSION: An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.

Scientific discovery depends on access to data and the knowledge this data makes possible.Research data sharing is increasingly recognized as a priority for organizations to support the successful conduct of research.The National Institutes of Health states, "data sharing enables researchers to rigorously test the validity of research findings, strengthen analyses through combined datasets, reuse hard-to-generate data, and explore new frontiers of discovery" [1].Conversely, in the absence of data sharing, there are increased risks related to the robustness, rigor, and replicability of results, and the potential of valuable data is diminished.For these reasons and more, institutional data sharing capacity is a critical topic for organizations to scrutinize, discuss, and advance.Advocacy and support for data sharing are often discussed with an emphasis on understanding and supporting the practices of individual investigators or scientific communities [2,3].However, a researcher's ability to successfully engage in and benefit from sound data sharing depends on their organizational setting and, specifically, the organization's data sharing capacity.For example, sharing data is easier and more equitable when organizational processes and procedures are established and documented, and research workforce members can access centralized training and infrastructure resources.The effect of how an organization approaches and supports data sharing extends beyond the success of its investigators.Institutions that share data can participate in innovative largescale initiatives and pursue new funding opportunities.Universities that contribute to creating

OBJECTIVE: Among National Institutes of Health Clinical and Translational Science Award (CTSA) hubs, effective approaches for enterprise data warehouses for research (EDW4R) development, maintenance, and sustainability remain unclear. The goal of this qualitative study was to understand CTSA EDW4R operations within the broader contexts of academic medical centers and technology. MATERIALS AND METHODS: We performed a directed content analysis of transcripts generated from semistructured interviews with informatics leaders from 20 CTSA hubs. RESULTS: Respondents referred to services provided by health system, university, and medical school information technology (IT) organizations as "enterprise information technology (IT)." Seventy-five percent of respondents stated that the team providing EDW4R service at their hub was separate from enterprise IT; strong relationships between EDW4R teams and enterprise IT were critical for success. Managing challenges of EDW4R staffing was made easier by executive leadership support. Data governance appeared to be a work in progress, as most hubs reported complex and incomplete processes, especially for commercial data sharing. Although nearly all hubs (n = 16) described use of cloud computing for specific projects, only 2 hubs reported using a cloud-based EDW4R. Respondents described EDW4R cloud migration facilitators, barriers, and opportunities. DISCUSSION: Descriptions of approaches to how EDW4R teams at CTSA hubs work with enterprise IT organizations, manage workforces, make decisions about data, and approach cloud computing provide insights for institutions seeking to leverage patient data for research. CONCLUSION: Identification of EDW4R best practices is challenging, and this study helps identify a breadth of viable options for CTSA hubs to consider when implementing EDW4R services.

OBJECTIVE: In response to COVID-19, the informatics community united to aggregate as much clinical data as possible to characterize this new disease and reduce its impact through collaborative analytics. The National COVID Cohort Collaborative (N3C) is now the largest publicly available HIPAA limited dataset in US history with over 6.4 million patients and is a testament to a partnership of over 100 organizations. MATERIALS AND METHODS: We developed a pipeline for ingesting, harmonizing, and centralizing data from 56 contributing data partners using 4 federated Common Data Models. N3C data quality (DQ) review involves both automated and manual procedures. In the process, several DQ heuristics were discovered in our centralized context, both within the pipeline and during downstream project-based analysis. Feedback to the sites led to many local and centralized DQ improvements. RESULTS: Beyond well-recognized DQ findings, we discovered 15 heuristics relating to source Common Data Model conformance, demographics, COVID tests, conditions, encounters, measurements, observations, coding completeness, and fitness for use. Of 56 sites, 37 sites (66%) demonstrated issues through these heuristics. These 37 sites demonstrated improvement after receiving feedback. DISCUSSION: We encountered site-to-site differences in DQ which would have been challenging to discover using federated checks alone. We have demonstrated that centralized DQ benchmarking reveals unique opportunities for DQ improvement that will support improved research analytics locally and in aggregate. CONCLUSION: By combining rapid, continual assessment of DQ with a large volume of multisite data, it is possible to support more nuanced scientific questions with the scale and rigor that they require.

OBJECTIVE: To assess and compare electronic health record (EHR) documentation of chronic disease in problem lists and encounter diagnosis records among Community Health Center (CHC) patients. MATERIALS AND METHODS: We assessed patient EHR data in a large clinical research network during 2012-2019. We included CHCs who provided outpatient, older adult primary care to patients age ≥45 years, with ≥2 office visits during the study. Our study sample included 1 180 290 patients from 545 CHCs across 22 states. We used diagnosis codes from 39 Chronic Condition Warehouse algorithms to identify chronic conditions from encounter diagnoses only and compared against problem list records. We measured correspondence including agreement, kappa, prevalence index, bias index, and prevalence-adjusted bias-adjusted kappa. RESULTS: Overlap of encounter diagnosis and problem list ascertainment was 59.4% among chronic conditions identified, with 12.2% of conditions identified only in encounters and 28.4% identified only in problem lists. Rates of coidentification varied by condition from 7.1% to 84.4%. Greatest agreement was found in diabetes (84.4%), HIV (78.1%), and hypertension (74.7%). Sixteen conditions had <50% agreement, including cancers and substance use disorders. Overlap for mental health conditions ranged from 47.4% for anxiety to 59.8% for depression. DISCUSSION: Agreement between the 2 sources varied substantially. Conditions requiring regular management in primary care settings may have a higher agreement than those diagnosed and treated in specialty care. CONCLUSION: Relying on EHR encounter data to identify chronic conditions without reference to patient problem lists may under-capture conditions among CHC patients in the United States.

Our goal is to summarize the collective experience of 15 organizations in dealing with uncoordinated efforts that result in unnecessary delays in understanding, predicting, preparing for, containing, and mitigating the COVID-19 pandemic in the US. Response efforts involve the collection and analysis of data corresponding to healthcare organizations, public health departments, socioeconomic indicators, as well as additional signals collected directly from individuals and communities. We focused on electronic health record (EHR) data, since EHRs can be leveraged and scaled to improve clinical care, research, and to inform public health decision-making. We outline the current challenges in the data ecosystem and the technology infrastructure that are relevant to COVID-19, as witnessed in our 15 institutions. The infrastructure includes registries and clinical data networks to support population-level analyses. We propose a specific set of strategic next steps to increase interoperability, overall organization, and efficiencies.

OBJECTIVE: Among National Institutes of Health Clinical and Translational Science Award (CTSA) hubs, adoption of electronic data warehouses for research (EDW4R) containing data from electronic health record systems is nearly ubiquitous. Although benefits of EDW4R include more effective, efficient support of scientists, little is known about how CTSA hubs have implemented EDW4R services. The goal of this qualitative study was to understand the ways in which CTSA hubs have operationalized EDW4R to support clinical and translational researchers. MATERIALS AND METHODS: After conducting semistructured interviews with informatics leaders from 20 CTSA hubs, we performed a directed content analysis of interview notes informed by naturalistic inquiry. RESULTS: We identified 12 themes: organization and data; oversight and governance; data access request process; data access modalities; data access for users with different skill sets; engagement, communication, and literacy; service management coordinated with enterprise information technology; service management coordinated within a CTSA hub; service management coordinated between informatics and biostatistics; funding approaches; performance metrics; and future trends and current technology challenges. DISCUSSION: This study is a step in developing an improved understanding and creating a common vocabulary about EDW4R operations across institutions. Findings indicate an opportunity for establishing best practices for EDW4R operations in academic medicine. Such guidance could reduce the costs associated with developing an EDW4R by establishing a clear roadmap and maturity path for institutions to follow. CONCLUSIONS: CTSA hubs described varying approaches to EDW4R operations that may assist other institutions in better serving investigators with electronic patient data.

OBJECTIVES: To identify the unmet information needs of clinical teams delivering care to patients with complex medical, social, and economic needs; and to propose principles for redesigning electronic health records (EHR) to address these needs. MATERIALS AND METHODS: In this observational study, we interviewed and observed care teams in 9 community health centers in Oregon and Washington to understand their use of the EHR when caring for patients with complex medical and socioeconomic needs. Data were analyzed using a comparative approach to identify EHR users' information needs, which were then used to produce EHR design principles. RESULTS: Analyses of > 300 hours of observations and 51 interviews identified 4 major categories of information needs related to: consistency of social determinants of health (SDH) documentation; SDH information prioritization and changes to this prioritization; initiation and follow-up of community resource referrals; and timely communication of SDH information. Within these categories were 10 unmet information needs to be addressed by EHR designers. We propose the following EHR design principles to address these needs: enhance the flexibility of EHR documentation workflows; expand the ability to exchange information within teams and between systems; balance innovation and standardization of health information technology systems; organize and simplify information displays; and prioritize and reduce information. CONCLUSION: Developing EHR tools that are simple, accessible, easy to use, and able to be updated by a range of professionals is critical. The identified information needs and design principles should inform developers and implementers working in community health centers and other settings where complex patients receive care.

Errors associated with medication documentation account for a substantial fraction of preventable medical errors. Hence, the Joint Commission has called for the adoption of reconciliation strategies at all United States healthcare institutions. Although studies suggest that reconciliation tools can reduce errors, it remains unclear how best to implement systems and processes that are reliable and sensitive to clinical workflow. The authors designed a primary care process that supported reconciliation without compromising clinic efficiency. This manuscript describes the design and implementation of Automated Patient History Intake Device (APHID): ambulatory check-in kiosks that allow patients to review the names, dosage, frequency, and pictures of their medications before their appointment. Medication lists are retrieved from the electronic health record and patient updates are captured and reviewed by providers during the clinic session. Results from the roll-in phase indicate the device is easy for patients to use and integrates well with clinic workflow.

OBJECTIVE: To understand information systems components important in supporting team-based care of chronic illness through a literature search. DESIGN: Systematic search of literature from 1996-2005 for evaluations of information systems used in the care of chronic illness. MEASUREMENTS: The relationship of design, quality, information systems components, setting, and other factors with process, quality outcomes, and health care costs was evaluated. RESULTS: In all, 109 articles were reviewed involving 112 information system descriptions. Chronic diseases targeted included diabetes (42.9% of reviewed articles), heart disease (36.6%), and mental illness (23.2%), among others. System users were primarily physicians, nurses, and patients. Sixty-seven percent of reviewed experiments had positive outcomes; 94% of uncontrolled, observational studies claimed positive results. Components closely correlated with positive experimental results were connection to an electronic medical record, computerized prompts, population management (including reports and feedback), specialized decision support, electronic scheduling, and personal health records. Barriers identified included costs, data privacy and security concerns, and failure to consider workflow. CONCLUSION The majority of published studies revealed a positive impact of specific health information technology components on chronic illness care. Implications for future research and system designs are discussed.