BACKGROUND: Computational phenotyping from electronic health records (EHRs) is essential for clinical research, decision support, and quality/population health assessment, but the proliferation of algorithms for the same conditions makes it difficult to identify which algorithm is most appropriate for reuse. OBJECTIVE: To develop a framework for assessing phenotyping algorithm fitness for purpose and reuse. FITNESS FOR PURPOSE: Phenotyping algorithms are fit for purpose when they identify the intended population with performance characteristics appropriate for the intended application. FITNESS FOR REUSE: Phenotyping algorithms are fit for reuse when the algorithm is implementable and generalizable-that is, it identifies the same intended population with similar performance characteristics when applied to a new setting. CONCLUSIONS: The PhenoFit framework provides a structured approach to evaluate and adapt phenotyping algorithms for new contexts increasing efficiency and consistency of identifying patient populations from EHRs.

OBJECTIVE: To characterize and demonstrate how to reduce the administrative burden experienced by patients when navigating medication affordability resources in the United States. MATERIALS AND METHODS: Informed by administrative burden theory, we conducted an environmental scan of medication affordability resources for atrial fibrillation, and four common comorbidities (diabetes, heart failure, hypertension, and lipid disorder). We systematically searched for resources (eg, patient assistance programs, savings cards and nonprofit support) and extracted information about types, eligibility criteria, needed documentation, and application processes. RESULTS: We identified 66 resources across 12 categories across the five conditions. The resources' varied eligibility criteria, application processes, and requirements for providing sensitive financial documents could introduce multiple administrative costs for patients. DISCUSSION: The volume and complexity of medication affordability resources and related application processes may create substantial administrative burden for patients that could prevent their use-especially when prescribed multiple medications. CONCLUSION: Medication affordability resource informatics tools that reduce administrative burden could advance equitable medication access.

Dear Editors, We recognize that inefficient and idiosyncratic workflows in health care contribute to a myriad of obstacles for all healthcare stakeholders, including misuse of resources, provider burnout, and increased burden on patients and their caregivers.1 Therefore, we were pleased to read Zayas-Cabán et al’s2 article, “Priorities to accelerate workflow automation in healthcare.” We applaud them for their work illuminating determinants, priorities, and associated strategies for workflow automation. We augment their findings by proposing a seventh priority to stand alongside their original 6—develop and promote infrastructure to facilitate findability, accessibility, interoperability, and reusability of workflows (Table 1). In our view, the automation of healthcare workflows depends on the deployment of reliable, valid, robust, and proven computable biomedical knowledge (CBK) artifacts. We define CBK artifacts as separately packaged software implementations of evidence-based procedural, logical, mathematical, and statistical algorithms.3 We hold this view at an equal level of importance as the need for high-quality, interoperable data and a deep understanding of workflows, as emphasized by Zayas-Cabán et al. Our perspective is that automation of workflows will depend on formalizing and explicitly representing current and desired (optimal) workflows so that they can be tracked and processed by computers in valuable ways. Moreover, to achieve powerful automation, infrastructure must be developed to coordinate and combine computable workflows or process models with AI models, computable guidelines, and other CBK artifacts.

Embedded pragmatic clinical trials (ePCTs) play a vital role in addressing current population health problems, and their use of electronic health record (EHR) systems promises efficiencies that will increase the speed and volume of relevant and generalizable research. However, as the number of ePCTs using EHR-derived data grows, so does the risk that research will become more vulnerable to biases due to differences in data capture and access to care for different subsets of the population, thereby propagating inequities in health and the healthcare system. We identify 3 challenges-incomplete and variable capture of data on social determinants of health, lack of representation of vulnerable populations that do not access or receive treatment, and data loss due to variable use of technology-that exacerbate bias when working with EHR data and offer recommendations and examples of ways to actively mitigate bias.

OBJECTIVE: The aim of this study was to analyze a publicly available sample of rule-based phenotype definitions to characterize and evaluate the variability of logical constructs used. MATERIALS AND METHODS: A sample of 33 preexisting phenotype definitions used in research that are represented using Fast Healthcare Interoperability Resources and Clinical Quality Language (CQL) was analyzed using automated analysis of the computable representation of the CQL libraries. RESULTS: Most of the phenotype definitions include narrative descriptions and flowcharts, while few provide pseudocode or executable artifacts. Most use 4 or fewer medical terminologies. The number of codes used ranges from 5 to 6865, and value sets from 1 to 19. We found that the most common expressions used were literal, data, and logical expressions. Aggregate and arithmetic expressions are the least common. Expression depth ranges from 4 to 27. DISCUSSION: Despite the range of conditions, we found that all of the phenotype definitions consisted of logical criteria, representing both clinical and operational logic, and tabular data, consisting of codes from standard terminologies and keywords for natural language processing. The total number and variety of expressions are low, which may be to simplify implementation, or authors may limit complexity due to data availability constraints. CONCLUSIONS: The phenotype definitions analyzed show significant variation in specific logical, arithmetic, and other operators but are all composed of the same high-level components, namely tabular data and logical expressions. A standard representation for phenotype definitions should support these formats and be modular to support localization and shared logic.

OBJECTIVES: The purpose of the study was to determine if association exists between evidence-based provider training and clinician proficiency in electronic health record (EHR) use and if so, which EHR use metrics and vendor-defined indices exhibited association. MATERIALS AND METHODS: We studied ambulatory clinicians' EHR use data published in the Epic Systems Signal report to assess proficiency between training participants (n = 133) and nonparticipants (n = 14). Data were collected in May 2019 and November 2019 on nonsurgeon clinicians from 6 primary care, 7 urgent care, and 27 specialty care clinics. EHR use training occurred from August 5 to August 15, 2019, prior to EHR upgrade and organizational instance alignment. Analytics performed were descriptive statistics, paired t-tests, multivariate correlations, and hierarchal multiple regression. RESULTS: For number of appointments per 30-day reporting period, trained clinicians sustained an average increase of 16 appointments (P < .05), whereas nontrained clinicians incurred a decrease of 8 appointments. Only the trained clinician group achieved postevent improvement in the vendor-defined Proficiency score with an effect size characterized as moderate to large (dCohen = 0.625). DISCUSSION: Controversies exist on the return of investment from formal EHR training for clinician users. Previously published literature has mostly focused on qualitative data indicators of EHR training success. The findings of our EHR use training study identified EHR use metrics and vendor-defined indices with the capacity for translation into productivity and generated revenue measurements. CONCLUSIONS: One EHR use metric and 1 vendor-defined index indicated improved proficiency among trained clinicians.

OBJECTIVE: We identified challenges and solutions to using electronic health record (EHR) systems for the design and conduct of pragmatic research. MATERIALS AND METHODS: Since 2012, the Health Care Systems Research Collaboratory has served as the resource coordinating center for 21 pragmatic clinical trial demonstration projects. The EHR Core working group invited these demonstration projects to complete a written semistructured survey and used an inductive approach to review responses and identify EHR-related challenges and suggested EHR enhancements. RESULTS: We received survey responses from 20 projects and identified 21 challenges that fell into 6 broad themes: (1) inadequate collection of patient-reported outcome data, (2) lack of structured data collection, (3) data standardization, (4) resources to support customization of EHRs, (5) difficulties aggregating data across sites, and (6) accessing EHR data. DISCUSSION: Based on these findings, we formulated 6 prerequisites for PCTs that would enable the conduct of pragmatic research: (1) integrate the collection of patient-centered data into EHR systems, (2) facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows, (3) support the creation of high-quality research data by using standards, (4) ensure adequate IT staff to support embedded research, (5) create aggregate, multidata type resources for multisite trials, and (6) create re-usable and automated queries. CONCLUSION: We are hopeful our collection of specific EHR challenges and research needs will drive health system leaders, policymakers, and EHR designers to support these suggestions to improve our national capacity for generating real-world evidence.

OBJECTIVE: The study sought to describe key features of clinical concepts and data required to implement clinical practice recommendations as clinical decision support (CDS) tools in electronic health record systems and to identify recommendation features that predict feasibility of implementation. MATERIALS AND METHODS: Using semistructured interviews, CDS implementers and clinician subject matter experts from 7 academic medical centers rated the feasibility of implementing 10 American College of Emergency Physicians Choosing Wisely Recommendations as electronic health record-embedded CDS and estimated the need for additional data collection. Ratings were combined with objective features of the guidelines to develop a predictive model for technical implementation feasibility. RESULTS: A linear mixed model showed that the need for new data collection was predictive of lower implementation feasibility. The number of clinical concepts in each recommendation, need for historical data, and ambiguity of clinical concepts were not predictive of implementation feasibility. CONCLUSIONS: The availability of data and need for additional data collection are essential to assess the feasibility of CDS implementation. Authors of practice recommendations and guidelines can enable organizations to more rapidly assess data availability and feasibility of implementation by including operational definitions for required data.

Pragmatic clinical trials often entail the use of electronic health record (EHR) and claims data, but bias and quality issues associated with these data can limit their fitness for research purposes particularly for study end points. Patient-reported health (PRH) data can be used to confirm or supplement EHR and claims data in pragmatic trials, but these data can bring their own biases. Moreover, PRH data can complicate analyses if they are discordant with other sources. Using experience in the design and conduct of multi-site pragmatic trials, we itemize the strengths and limitations of PRH data and identify situational criteria for determining when PRH data are appropriate or ideal to fill gaps in the evidence collected from EHRs. To provide guidance for the scientific rationale and appropriate use of patient-reported data in pragmatic clinical trials, we describe approaches for ascertaining and classifying study end points and addressing issues of incomplete data, data alignment, and concordance. We conclude by identifying areas that require more research.

Pragmatic clinical trials (PCTs) are research investigations embedded in health care settings designed to increase the efficiency of research and its relevance to clinical practice. The Health Care Systems Research Collaboratory, initiated by the National Institutes of Health Common Fund in 2010, is a pioneering cooperative aimed at identifying and overcoming operational challenges to pragmatic research. Drawing from our experience, we present 4 broad categories of informatics-related challenges: (1) using clinical data for research, (2) integrating data from heterogeneous systems, (3) using electronic health records to support intervention delivery or health system change, and (4) assessing and improving data capture to define study populations and outcomes. These challenges impact the validity, reliability, and integrity of PCTs. Achieving the full potential of PCTs and a learning health system will require meaningful partnerships between health system leadership and operations, and federally driven standards and policies to ensure that future electronic health record systems have the flexibility to support research.

OBJECTIVE: We assessed the sensitivity and specificity of 8 electronic health record (EHR)-based phenotypes for diabetes mellitus against gold-standard American Diabetes Association (ADA) diagnostic criteria via chart review by clinical experts. MATERIALS AND METHODS: We identified EHR-based diabetes phenotype definitions that were developed for various purposes by a variety of users, including academic medical centers, Medicare, the New York City Health Department, and pharmacy benefit managers. We applied these definitions to a sample of 173 503 patients with records in the Duke Health System Enterprise Data Warehouse and at least 1 visit over a 5-year period (2007-2011). Of these patients, 22 679 (13%) met the criteria of 1 or more of the selected diabetes phenotype definitions. A statistically balanced sample of these patients was selected for chart review by clinical experts to determine the presence or absence of type 2 diabetes in the sample. RESULTS: The sensitivity (62-94%) and specificity (95-99%) of EHR-based type 2 diabetes phenotypes (compared with the gold standard ADA criteria via chart review) varied depending on the component criteria and timing of observations and measurements. DISCUSSION AND CONCLUSIONS: Researchers using EHR-based phenotype definitions should clearly specify the characteristics that comprise the definition, variations of ADA criteria, and how different phenotype definitions and components impact the patient populations retrieved and the intended application. Careful attention to phenotype definitions is critical if the promise of leveraging EHR data to improve individual and population health is to be fulfilled.

Health information technology (HIT) standards are not new. Arguably, they date to the canonical list for causes of death in the London Bills of Mortality of 1528, which was later formalized during the middle of the 19th century into what we now recognize as the International Classifications of Diseases (ICD). Beginning in the middle of the 20th century, HIT standards evolved beyond vital statistics and began to capture data related to clinical morbidity, thereby facilitating nascent decision support, outcomes research, evidence generation, and health care quality improvement initiatives. Alas, with that expansion came a proliferation of competing and overlapping standards, giving substance to the critical aphorism that “the only nice thing about standards is that there are so many to choose from.” The emergence of large-scale computerization throughout health care in the last half century has further accelerated this divergence of standards, creating a veritable cacophony of noninteroperable medical record content, data exchange formalisms, and data silos. This special issue on data standards was prompted by a palpable maturation among HIT standards in clinical practice and biomedical research in just the last decade. There has been remarkable cooperation among HIT standards development organizations, including the new agreement to harmonize and coordinate overlapping content in Systematized Nomenclature of Medicine—Clinical Terms (SNOMED CT) and Logical Observation Identifiers Names and Codes (LOINC), and the historic cooperation between the SNOMED CT and ICD developers to create ICD11 on the semantic foundation of SNOMED CT. In parallel, there have also been unprecedented consolidation and harmonization of orthogonal standards into an emerging suite of specifications for health and biomedical observations such as the ONC Meaningful Use and the NIH Common Data Element efforts within the United States. While far from comprehensive or fully coherent, the current state of HIT standards is at a turning point, where we appear to be making more effective progress and practical applications than most would have predicted from the bad old days of just a few decades ago. The goal of this special focus issue of the JAMIA is to provide a forum for the latest evaluation of HIT standards in contexts of Meaningful Use, biomedical research, and big data. JAMIA editors solicited this focus issue with the hope that HIT standards and recent consolidation initiatives had indeed adequately matured so that the informatics community would respond with successful demonstrations for how particular standards can and will effectively support biomedical and population health applications. We thus broadly solicited scholarly contributions that would address evaluation, application, consolidation, or domain extensions of biomedical data standards within the framework of biomedical informatics, and we explicitly requested authors to provide supporting data, rigorous evaluation, or evidence of relevant consensus to support their work. We were not disappointed and enjoyed the opportunity to review many rich submissions for this highly competitive volume. We present in this issue the best of these submissions, collectively covering wide spectra of domains, applications, and approaches. The selected articles represent the full continuum of molecular, clinical, organizational, and population data, address a range of objectives from evaluation to application, and illustrate multiple approaches to HIT standards and interoperability from domain-specific to broad integration. Exemplifying the increasing coordination between multiple standards is the characterization of genetic data, as standardized by the Human Gene Nomenclature Committee with LOINC laboratory reports for genetic data.1 This represents the reuse of existing standards for genomic specification within an established framework for health data exchange and messaging. In work that also binds the basic science world to clinical practice, this history and success of the Human Proteome Organization Proteomics Standards Initiative details the broadly-based collaborations that have led to an increasingly mature and practical specification of proteomic findings.2 Similarly, poison control centers have demonstrated how they can collaborate with emergency departments by adopting their reference model for health information exchange to work within the HL7 Consolidated Clinical Document Architecture standard.3 This is an elegant demonstration of a community with an important clinical data exchange requirement choosing to embrace and enable an emerging mainstream mechanism, as opposed to the traditional solution of building yet another syntax to implement their reference model. On a more abstract level is the description of how the venerable Clinical Element Models can be systematically transformed into the openEHR archetype representations by invoking shared Web Ontology Language (OWL) formalisms.4 This demonstrates how the Clinical Information Modeling Initiative, through fostering community consensus on modeling syntax and languages, can stimulate informatics work that can harvest legacy content while strengthening standards harmonization and coherency. Demonstrating that not all standards are final or even fully robust, methods for discovering errors in large-scale terminologies, specifically SNOMED CT, substantially enhance the scale and completeness of quality assurance work within complex data standards.5 Such work has consistently demonstrated room for improvement in these large data specifications, and these standards together with their user communities are by far better for it. Correspondingly, a systemic review of the HL7/LOINC Document Ontology Role Axis highlights many shortcomings in representing clinical-role behaviors that are encountered in real-world resources.6 Such work, again, can and will feed back to the standards developers, making these components more robust within suites of HIT data standards. The pattern of empirical discovery is also evident in the systematic “bottom-up” evaluation of nursing documents across many standards developing organizations (SDO) contributors.7 Specifically, in the generation of eMeasures, clear and specific refinements of many clinical standards are needed and will, no doubt, come about because of these careful, reality-based evaluations. Continuing within the nursing domain, investigators demonstrate how collaboration across organizations working with established standards enables detailed and informatics messaging around hospital-acquired pressure ulcers.8 These efforts can directly address ONC challenges for the development of mobile applications to improve clinical care around this all too frequent complication of chronic care. Expanding from the specific to the strategic, a consensus community developed national action plans for collecting comparable nursing data to support secondary use as well as clinical care.9 By and large, nursing data is not yet systematically integrated into electronic health records, impoverishing both the care process and outcomes research. This national action plan promises to advance the evaluation and practice of nursing by supporting the generation of comparable nursing data for quality reporting and translational research. Domain-specific adaptations of existing data models and semantics obviously can extend to many other areas of health care. An evaluation of an oncology-specific implementation guide of the consolidated clinical data architecture standard is described as a case example, which completed the full HL7 ballot process.10 The paper also describes its clinical implementation by two organizations, which validated the overall strategy of domain-specific adaptations of established HIT standards. In a related domain-specific HL7 effort, the maturation of data elements for emergency departments is described.11 This process also involved the use and mapping of many related data standards. Meanwhile, the critical evaluation of semantics and value set binding to clinical models in the domain of heart failure demonstrates a pathway for semantically accurate interoperability in complex clinical domains.12 They were able to simplify that complexity into a framework of semantic patterns, enabling coherent access to heterogeneous data resources. Finally, the harmonization of distributed, heterogeneous clinical data into a shared specification based on HL7’s Virtual Medical Record specification illustrates how data standards can help integrate existing data, even if those data were not collected with fully specified or shared standards specifications.13 This paper shows how meta-standards can facilitate the integration of patient information from heterogeneous sources, a fundamental and all too common requirement for clinical decision support systems in the United States. Taken together, the papers presented in this special focus issue inform us about the state of HIT standards and their evolution. Clearly, they highlight that far more work remains, but more pertinently the overarching message is that collaboration, coordination, and convergence is occurring and may even be considered as the default effort. This is in vast contradiction to an earlier era when virtually every biomedical data standards development organization felt obligated to publish a competing “me too” standard, if only so that the products of their competitors would not succeed. We have all grown beyond that. HIT standards are now widely recognized not as commercial ventures in and of themselves but as critical public resources that can stimulate innovation and support applications that impact provider behaviors and patient outcomes. Efforts today are clearly focused on effective communication, semantic consistency, and interoperability. We can be sanguine about the likely state we may find ourselves in within the next decade. We have arrived at an era of real progress. Albeit slow, the maturation of practical, coherent, and interoperable biomedical data standards is undeniable and bodes well for clinical data interoperability.

Case report forms (CRFs) are used for structured-data collection in clinical research studies. Existing CRF-related standards encompass structural features of forms and data items, content standards, and specifications for using terminologies. This paper reviews existing standards and discusses their current limitations. Because clinical research is highly protocol-specific, forms-development processes are more easily standardized than is CRF content. Tools that support retrieval and reuse of existing items will enable standards adoption in clinical research applications. Such tools will depend upon formal relationships between items and terminological standards. Future standards adoption will depend upon standardized approaches for bridging generic structural standards and domain-specific content standards. Clinical research informatics can help define tools requirements in terms of workflow support for research activities, reconcile the perspectives of varied clinical research stakeholders, and coordinate standards efforts toward interoperability across healthcare and research data collection.

OBJECTIVE: To compare consistency of coding among professional SNOMED CT coders representing three commercial providers of coding services when coding clinical research concepts with SNOMED CT. DESIGN: A sample of clinical research questions from case report forms (CRFs) generated by the NIH-funded Rare Disease Clinical Research Network (RDCRN) were sent to three coding companies with instructions to code the core concepts using SNOMED CT. The sample consisted of 319 question/answer pairs from 15 separate studies. The companies were asked to select SNOMED CT concepts (in any form, including post-coordinated) that capture the core concept(s) reflected in the question. Also, they were asked to state their level of certainty, as well as how precise they felt their coding was. MEASUREMENTS: Basic frequencies were calculated to determine raw level agreement among the companies and other descriptive information. Krippendorff's alpha was used to determine a statistical measure of agreement among the coding companies for several measures (semantic, certainty, and precision). RESULTS: No significant level of agreement among the experts was found. CONCLUSION: There is little semantic agreement in coding of clinical research data items across coders from 3 professional coding services, even using a very liberal definition of agreement.

Current efforts to define and implement health data standards are driven by issues related to the quality, cost and continuity of care, patient safety concerns, and desires to speed clinical research findings to the bedside. The President's goal for national adoption of electronic medical records in the next decade, coupled with the current emphasis on translational research, underscore the urgent need for data standards in clinical research. This paper reviews the motivations and requirements for standardized clinical research data, and the current state of standards development and adoption--including gaps and overlaps--in relevant areas. Unresolved issues and informatics challenges related to the adoption of clinical research data and terminology standards are mentioned, as are the collaborations and activities the authors perceive as most likely to address them.

OBJECTIVE: To estimate the coverage provided by SNOMED CT for clinical research concepts represented by the items on case report forms (CRFs), as well as the semantic nature of those concepts relevant to post-coordination methods. DESIGN: Convenience samples from CRFs developed by rheumatologists conducting several longitudinal, observational studies of vasculitis were selected. A total of 17 CRFs were used as the basis of analysis for this study, from which a total set of 616 (unique) items were identified. Each unique data item was classified as either a clinical finding or procedure. The items were coded by the presence and nature of SNOMED CT coverage and classified into semantic types by 2 coders. MEASUREMENTS: Basic frequency analysis was conducted to determine levels of coverage provided by SNOMED CT. Estimates of coverage by various semantic characterizations were estimated. RESULTS: Most of the core clinical concepts (88%) from these clinical research data items were covered by SNOMED CT; however, far fewer of the concepts were fully covered (that is, where all aspects of the CRF item could be represented completely without post-coordination; 23%). In addition, a large majority of the concepts (83%) required post-coordination, either to clarify context (e.g., time) or to better capture complex clinical concepts (e.g., disease-related findings). For just over one third of the sampled CRF data items, both types of post-coordination were necessary to fully represent the meaning of the item. CONCLUSION: SNOMED CT appears well-suited for representing a variety of clinical concepts, yet is less suited for representing the full amount of information collected on CRFs.