OBJECTIVE: The annual American College of Medical Informatics (ACMI) symposium focused discussion on the national public health information systems (PHIS) infrastructure to support public health goals. The objective of this article is to present the strengths, weaknesses, threats, and opportunities (SWOT) identified by public health and informatics leaders in attendance. MATERIALS AND METHODS: The Symposium provided a venue for experts in biomedical informatics and public health to brainstorm, identify, and discuss top PHIS challenges. Two conceptual frameworks, SWOT and the Informatics Stack, guided discussion and were used to organize factors and themes identified through a qualitative approach. RESULTS: A total of 57 unique factors related to the current PHIS were identified, including 9 strengths, 22 weaknesses, 14 opportunities, and 14 threats, which were consolidated into 22 themes according to the Stack. Most themes (68%) clustered at the top of the Stack. Three overarching opportunities were especially prominent: (1) addressing the needs for sustainable funding, (2) leveraging existing infrastructure and processes for information exchange and system development that meets public health goals, and (3) preparing the public health workforce to benefit from available resources. DISCUSSION: The PHIS is unarguably overdue for a strategically designed, technology-enabled, information infrastructure for delivering day-to-day essential public health services and to respond effectively to public health emergencies. CONCLUSION: Most of the themes identified concerned context, people, and processes rather than technical elements. We recommend that public health leadership consider the possible actions and leverage informatics expertise as we collectively prepare for the future.

The COVID-19 pandemic exposed multiple weaknesses in the nation's public health system. Therefore, the American College of Medical Informatics selected "Rebuilding the Nation's Public Health Informatics Infrastructure" as the theme for its annual symposium. Experts in biomedical informatics and public health discussed strategies to strengthen the US public health information infrastructure through policy, education, research, and development. This article summarizes policy recommendations for the biomedical informatics community postpandemic. First, the nation must perceive the health data infrastructure to be a matter of national security. The nation must further invest significantly more in its health data infrastructure. Investments should include the education and training of the public health workforce as informaticians in this domain are currently limited. Finally, investments should strengthen and expand health data utilities that increasingly play a critical role in exchanging information across public health and healthcare organizations.

OBJECTIVES: To design an interface to support communication of machine learning (ML)-based prognosis for patients with advanced solid tumors, incorporating oncologists' needs and feedback throughout design. MATERIALS AND METHODS: Using an interdisciplinary user-centered design approach, we performed 5 rounds of iterative design to refine an interface, involving expert review based on usability heuristics, input from a color-blind adult, and 13 individual semi-structured interviews with oncologists. Individual interviews included patient vignettes and a series of interfaces populated with representative patient data and predicted survival for each treatment decision point when a new line of therapy (LoT) was being considered. Ongoing feedback informed design decisions, and directed qualitative content analysis of interview transcripts was used to evaluate usability and identify enhancement requirements. RESULTS: Design processes resulted in an interface with 7 sections, each addressing user-focused questions, supporting oncologists to "tell a story" as they discuss prognosis during a clinical encounter. The iteratively enhanced interface both triggered and reflected design decisions relevant when attempting to communicate ML-based prognosis, and exposed misassumptions. Clinicians requested enhancements that emphasized interpretability over explainability. Qualitative findings confirmed that previously identified issues were resolved and clarified necessary enhancements (eg, use months not days) and concerns about usability and trust (eg, address LoT received elsewhere). Appropriate use should be in the context of a conversation with an oncologist. CONCLUSION: User-centered design, ongoing clinical input, and a visualization to communicate ML-related outcomes are important elements for designing any decision support tool enabled by artificial intelligence, particularly when communicating prognosis risk.

OBJECTIVE: We evaluated nursing-related free-text communication orders to identify potential safety hazards and describe patterns and scope of care domains addressed that may reveal preventable workarounds and potential gaps in electronic health record (EHR) functionality. MATERIALS AND METHODS: A retrospective analysis of free-text EHR-based communication orders sent to or by nurses providing inpatient care at a major academic health system. Using built-in EHR tools and selection criteria, 13 193 orders were extracted, including 1373 unique orders. Using the Clinical Care Classification system standardized framework, we classified content by care domain and identified unique requests within each order. We reviewed each order for error-prone textual features based on standard patient safety guidance. We describe the distribution of domains, co-occurrence when 2 domains were present, and common patterns. RESULTS: The 1373 unique orders included a single request (65.3%), 2 requests related to 1 or 2 domains (19%), or 3 or more requests (15.7%). No orders included terms on the Joint Commission's "Do Not Use" list. However, 13.6% of unique orders, and 16.7% of those related to medications, included error-prone symbols or abbreviations according to Institute for Safe Medication Practices guidance. Order content spanned 20 different care components but physical regulation, fluid volume, nutritional, safety, and medication were most frequently identified as single or co-occurring topics. Patterns were heterogenous. DISCUSSION: Free-text communication orders reveal workarounds, responses to upstream workarounds, and design constraints that should be further investigated. Remediation strategies are needed to reduce safety hazards and workflow impediments. CONCLUSIONS: Analysis of free-text communication orders revealed opportunities for improvement.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

OBJECTIVE: The study sought to describe key features of clinical concepts and data required to implement clinical practice recommendations as clinical decision support (CDS) tools in electronic health record systems and to identify recommendation features that predict feasibility of implementation. MATERIALS AND METHODS: Using semistructured interviews, CDS implementers and clinician subject matter experts from 7 academic medical centers rated the feasibility of implementing 10 American College of Emergency Physicians Choosing Wisely Recommendations as electronic health record-embedded CDS and estimated the need for additional data collection. Ratings were combined with objective features of the guidelines to develop a predictive model for technical implementation feasibility. RESULTS: A linear mixed model showed that the need for new data collection was predictive of lower implementation feasibility. The number of clinical concepts in each recommendation, need for historical data, and ambiguity of clinical concepts were not predictive of implementation feasibility. CONCLUSIONS: The availability of data and need for additional data collection are essential to assess the feasibility of CDS implementation. Authors of practice recommendations and guidelines can enable organizations to more rapidly assess data availability and feasibility of implementation by including operational definitions for required data.

RE: Barriers to Hospital Electronic Public Health Reporting and Implications for the COVID-19 Pandemic, JAMIA, https://doi.org/10.1093/jamia/ocaa112 Dear Dr. Bakken (editor of JAMIA) and A Jay Holmgren, Nate C Apathy, and Julia Adler-Milstein (authors), We applaud efforts to address concerns about information sharing between hospitals and public health systems, particularly during the COVID-19 pandemic. However, we have concerns about the validity and usefulness of the findings reported by Holmgren, et al and the characterization of the ability of public health agencies (PHAs) to receive electronic data. First, the study findings do not match the situation “on the ground”: In 2018, PHAs were able to receive electronic data, particularly lab and immunization data. All relevant (50 states and 6 large cities, such as Los Angeles and New York City) PHAs were receiving laboratory data for electronic laboratory reporting (ELR) (Personal communication: Jason Hall, CDC, 2020) Similarly, 96% of these PHAs were receiving data for their immunization registries.1 In fact, these registries also provide data back to electronic health records and can report timely information about underimmunized populations at risk for outbreaks.2 The authors reference a lack of local health department capacity in the text and Figure 2; but in reality, it is typical for state or large city PHAs to receive electronic data from hospitals on behalf of local agencies. State or large city PHAs host the IT infrastructure for electronic surveillance and manage interfaces with clinical systems, while staff from smaller PHAs access the hosted system. Second, the authors overgeneralized the survey findings: The authors conflate syndromic surveillance with the larger concept of “electronic surveillance,” as if syndromic surveillance were the important data source for controlling the COVID-19 pandemic. Syndromic surveillance is useful for early detection and population-level monitoring, but it does not include patient-level data needed for case investigation and outbreak management. The authors have overinterpreted the responses to the 2018 American Hospital Association (AHA) Annual survey question, leading to possible bias. Hospital CEOs or their designees were asked: “What are some of the challenges your hospital has experienced when trying to submit health information to public health agencies to meet meaningful use (MU) requirements?”3 Four in 10 CEOs (41%) selected the barrier “Public health agencies lacked the capacity (eg, technical, staffing) to electronically receive data.”3 A hospital CEO or delegate may not be aware of their local and state PHA’s capacity to receive electronic health information, or even their own organization’s involvement with electronic reporting of lab results, immunizations, case reports or other data. In addition, the response option does not provide clarity as to which data, when, and in what form. The AHA survey is important, but results should be interpreted in context. We are concerned the publication could lead to incorrect assumptions at a time when clinical and public health systems need to communicate more than ever and may discourage health care providers, leaders, and health IT vendors from engaging with public health agencies to avail themselves of existing data exchange capabilities. Given the critical need for both ELR and case reporting to manage the COVID-19 outbreak, major efforts are underway to expand electronic case reporting (eCR) (https://cdc.gov/ecr) and reduce the burden for health systems.4 As of July 6, 2020, the APHL Informatics Messaging Service (AIMS), a national resource for ELR and eCR reporting, had received 803 239 COVID-19 case reports from over 2000 facilities in 20 health care organizations which were shared with PHAs from 47 jurisdictions; and all but 2 state PHAs can receive eCR messages from the AIMS platform, and enhancements are underway to automatically integrate data into surveillance systems (Personal communication: Laura Conn, CDC, 2020). Public health authorities describe reluctance from providers and health systems to implement electronic reporting on the grounds that implementation is too burdensome. More engagement is needed. There is no question that inadequate resources have been a limiting factor for public health agencies to receive data from health systems. This problem is exacerbated by the many-to-one (hospitals-to-public health agency) nature of population health activities, the variable nature of hospital data contributions, and the resources required to onboard and manage interfaces with multiple health systems. We encourage clinical partners to work with public health agencies to improve surveillance of both clinical and public health outcomes and leverage information exchange to benefit communities.5 We recommend increasing support for public health agencies to enhance their ability to exchange (both receive and send) information while health care systems receive support to send data. CJS wrote the first draft. All authors provided input and revisions. All authors approved final submission. We thank Erin Holt for her input on this letter. None declared.

INTRODUCTION: Identity information is often used to link records within or among information systems in public health and clinical settings. The quality and stability of birth certificate identifiers impacts both the success of linkage efforts and the value of birth certificate registries for identity resolution. OBJECTIVE: Our objectives were to describe: (1) the frequency and cause of changes to birth certificate identifiers as children age, and (2) the frequency of events (ie, adoptions, paternities, amendments) that may trigger changes and their impact on names. METHODS: We obtained two de-identified datasets from the Utah birth certificate registry: (1) change history from 2000 to 2012, and (2) occurrences for adoptions, paternities, and amendments among births in 1987 and 2000. We conducted cohort analyses for births in 1987 and 2000, examining the number, reason, and extent of changes over time. We conducted cross-sectional analyses to assess the patterns of changes between 2000 and 2012. RESULTS: In a cohort of 48 350 individuals born in 2000 in Utah, 3164 (6.5%) experienced a change in identifiers prior to their 13th birthday, with most changes occurring before 2 years of age. Cross-sectional analysis showed that identifiers are stable for individuals over 5 years of age, but patterns of changes fluctuate considerably over time, potentially due to policy and social factors. CONCLUSIONS: Identities represented in birth certificates change over time. Specific events that cause changes to birth certificates also fluctuate over time. Understanding these changes can help in the development of automated strategies to improve identity resolution.

OBJECTIVE: To develop expeditiously a pragmatic, modular, and extensible software framework for understanding and improving healthcare value (costs relative to outcomes). MATERIALS AND METHODS: In 2012, a multidisciplinary team was assembled by the leadership of the University of Utah Health Sciences Center and charged with rapidly developing a pragmatic and actionable analytics framework for understanding and enhancing healthcare value. Based on an analysis of relevant prior work, a value analytics framework known as Value Driven Outcomes (VDO) was developed using an agile methodology. Evaluation consisted of measurement against project objectives, including implementation timeliness, system performance, completeness, accuracy, extensibility, adoption, satisfaction, and the ability to support value improvement. RESULTS: A modular, extensible framework was developed to allocate clinical care costs to individual patient encounters. For example, labor costs in a hospital unit are allocated to patients based on the hours they spent in the unit; actual medication acquisition costs are allocated to patients based on utilization; and radiology costs are allocated based on the minutes required for study performance. Relevant process and outcome measures are also available. A visualization layer facilitates the identification of value improvement opportunities, such as high-volume, high-cost case types with high variability in costs across providers. Initial implementation was completed within 6 months, and all project objectives were fulfilled. The framework has been improved iteratively and is now a foundational tool for delivering high-value care. CONCLUSIONS: The framework described can be expeditiously implemented to provide a pragmatic, modular, and extensible approach to understanding and improving healthcare value.

OBJECTIVE: To understand how the source of information affects different adverse event (AE) surveillance methods. DESIGN: Retrospective analysis of inpatient adverse drug events (ADEs) and hospital-associated infections (HAIs) detected by either a computerized surveillance system (CSS) or manual chart review (MCR). MEASUREMENT: Descriptive analysis of events detected using the two methods by type of AE, type of information about the AE, and sources of the information. RESULTS: CSS detected more HAIs than MCR (92% vs 34%); however, a similar number of ADEs was detected by both systems (52% vs 51%). The agreement between systems was greater for HAIs than ADEs (26% vs 3%). The CSS missed events that did not have information in coded format or that were described only in physician narratives. The MCR detected events missed by CSS using information in physician narratives. Discharge summaries were more likely to contain information about AEs than any other type of physician narrative, followed by emergency department reports for HAIs and general consult notes for ADEs. Some ADEs found by MCR were detected by CSS but not verified by a clinician. LIMITATIONS: Inability to distinguish between CSS false positives and suspected AEs for cases in which the clinician did not document their assessment in the CSS. CONCLUSION: The effect that information source has on different surveillance methods depends on the type of AE. Integrating information from physician narratives with CSS using natural language processing would improve the detection of ADEs more than HAIs.

Clinicians are required to report selected conditions to public health authorities within a stipulated amount of time. The current reporting process is mostly paper-based and inefficient and may lead to delays in case investigation. As electronic medical records become more prevalent, electronic case reporting is becoming increasingly feasible. However, there is no existing standard for the electronic transmission of case reports from healthcare to public health entities. We identified the major requirements of electronic case reports and verified that the requirements support the work processes of the local health departments. We propose an extendable standards-based model to electronically transmit case information and associated laboratory information from healthcare to public health entities. The HL7 v2.5 message model is currently being implemented to transmit electronic case reports from Intermountain Healthcare to the Utah Department of Health.

We are grateful for the careful review of our article by Wilcke et al.1 We would like to reaffirm our belief that SNOMED CT is an extremely useful terminology and the best current choice for the representation of clinical findings generally, and specifically for the representation of findings in public health case reports. The broad use of SNOMED CT will lead to the understanding of important modeling issues as well as improved SNOMED CT content. We appreciate the respondents' clarification that ‘porcine enteric adenomatosis’ is a species-independent disorder. However, the fact that we held this erroneous opinion is understandable. As ‘Porcine’ is in the name of the term, and without further explanation, the ‘true’ meaning is difficult to discern. The name of the term is ambiguous, if not outright misleading. This situation points to the underlying difficulty of making it easy for users to understand the meaning of concepts in reference terminologies. Great emphasis has been placed on the accuracy and internal consistency of reference terminologies, with less emphasis on what is needed to make the content easily understood by clinical users and researchers. Human readable definitions would be a big help. We agree with the respondents that, ‘Information regarding species, whether it be human or non-human can and should be identified by the information model, not the terminology.’ We are aware of the ongoing efforts to form a working relationship between creators of information models and terminology developers so that models and terminology can be created in unison, thereby removing ambiguity in data representation. We applaud and support these efforts. We absolutely agree that, ‘Balancing the needs of practical applications with the importance of the purity and accuracy of the terminology is a very difficult task….’ We would actually state that it is impossible. We would assert that hierarchies are created for a purpose. The purpose could be to support automated maintenance of the terminology, to create definitional relationships, or to allow appropriate inferences. If the different users of the terminology have different purposes, it is not always possible to meet all needs with a single hierarchy, no matter how much care is taken. Even with the clarifications offered by the respondents, the fact remains that the hierarchical relationships represented in SNOMED CT do not ideally support public health reporting use cases. The desired case roll-up behavior is not supported by the unmodified SNOMED CT hierarchies and concepts. While we asked for the development of a hierarchy that is exclusively for human conditions, a better statement of the need would have been the development of a hierarchy that is valid for public health inferencing about human disease. The problem is not with SNOMED CT per se. The root of the problem is that all purposes cannot be met by a single hierarchy. We need to support many purpose-specific hierarchies. To this end, we support the respondents' suggestion for increasing creation of subsets. We look forward to learning more about how subsets can be created and exchanged to meet specific clinical needs. None. This study was conducted with the approval of the University of Utah. Not commissioned; not externally peer reviewed.

Authors evaluated the impact of computerized alerts on the quality of outpatient laboratory monitoring for transplant patients. For 356 outpatient liver transplant patients managed at LDS Hospital, Salt Lake City, this observational study compared traditional laboratory result reporting, using faxes and printouts, to computerized alerts implemented in 2004. Study alerts within the electronic health record notified clinicians of new results and overdue new orders for creatinine tests and immunosuppression drug levels. After implementing alerts, completeness of reporting increased from 66 to >99 %, as did positive predictive value that a report included new information (from 46 to >99 %). Timeliness of reporting and clinicians' responses improved after implementing alerts (p <0.001): median times for clinicians to receive and complete actions decreased to 9 hours from 33 hours using the prior traditional reporting system. Computerized alerts led to more efficient, complete, and timely management of laboratory information.

Laboratory results provide necessary information for the management of ambulatory patients. To realize the benefits of an electronic health record (EHR) and coded laboratory data (e.g., decision support and improved data access and display), results from laboratories that are external to the health care enterprise need to be integrated with internal results. We describe the development and clinical impact of integrating external results into the EHR at Intermountain Health Care (IHC). During 2004, over 14,000 external laboratory results for 128 liver transplant patients were added to the EHR. The results were used to generate computerized alerts that assisted clinicians with managing laboratory tests in the ambulatory setting. The external results were sent from 85 different facilities and can now be viewed in the EHR integrated with IHC results. We encountered regulatory, logistic, economic, and data quality issues that should be of interest to others developing similar applications.

OBJECTIVE: To develop a model to store information in an electronic medical record (EMR) for the management of transplant patients. The model for storing donor information must be designed to allow clinicians to access donor information from the transplant recipient's record and to allow donor data to be stored without needlessly proliferating new Logical Observation Identifier Names and Codes (LOINC) codes for already-coded laboratory tests. DESIGN: Information required to manage transplant patients requires the use of a donor's medical information while caring for the transplant patient. Three strategies were considered: (1) link the transplant patient's EMR to the donor's EMR; (2) use pre-coordinated observation identifiers (i.e., LOINC codes with *(wedge)DONOR specified in the system axes) to identify donor data stored in the transplant patient's EMR; and (3) use an information model that allows donor information to be stored in the transplant patient's record by allowing the "source" of the data (donor) and the "name" of the result (e.g., blood type) to be post-coordinated in the transplant patient's EMR. RESULTS: We selected the third strategy and implemented a flexible post-coordinated information model. There was no need to create new LOINC codes for already-coded laboratory tests. The model required that the data structure in the EMR allow for the storage of the "subject" of the test. CONCLUSION: The selected strategy met our design requirements and provided an extendable information model to store donor data. This model can be used whenever it is necessary to refer to one patient's data from another patient's EMR.