OBJECTIVE: We discuss implications of potential ascertainment biases for studies examining diabetes risk following SARS-CoV-2 infection using electronic health records (EHRs). We quantitatively explore sensitivity of results to misclassification of COVID-19 status using data from the U.S.-based Diabetes in Children, Adolescents and Young Adults (DiCAYA) Network on children (≤17 years) and young adults (18-44 years). MATERIALS AND METHODS: In our retrospective case study from the DiCAYA Network, SARS-CoV-2 was identified using labs and diagnoses from June 1, 2020 to December 31, 2021. Patients were followed through December 31, 2022 for new diabetes diagnoses. Sites examined incident diabetes by COVID-19 status using Cox proportional hazards models. Results were pooled in meta-analyses. A bias analysis examined potential impact of COVID-19 misclassification scenarios on results, guided by hypotheses that sensitivity would be <50% and would be higher among those who developed diabetes. RESULTS: Prevalence of documented COVID-19 was low overall and variable across sites (children: 4.4%-7.7%, young adults: 6.2%-22.7%). Individuals with documented COVID-19 were at higher risk of incident diabetes compared to those with no documented infection, but results were heterogeneous across sites. Findings were highly sensitive to COVID-19 misclassification assumptions. Observed results could be biased away from the null under several differential misclassification scenarios. DISCUSSION: Although EHR-based documentation of COVID-19 was associated with incident diabetes, COVID-19 phenotypes likely had low sensitivity, with considerable variation across sites. Misclassification assumptions strongly impacted interpretation of results. CONCLUSION: Given the potential for low phenotype sensitivity and misclassification, caution is warranted when interpreting analyses of COVID-19 and incident diabetes using clinical or administrative databases.

OBJECTIVE: Modernizing and strengthening the US public health data and information infrastructure requires a strong public health informatics (PHI) workforce. The study objectives were to characterize existing PHI specialists and assess informatics-related training needs. MATERIALS AND METHODS: To examine the PHI workforce, the 2021 Public Health Workforce Interests and Needs Survey (PH WINS), a nationally representative survey with 44 732 governmental public health (PH) respondents was analyzed. The survey included data from 47 state health agencies-central office, 29 large local health departments (Big Cities Health Coalition members), and 259 other local/regional health departments. Analysis focused on "public health informatics specialist" (PHI), "information system manager/information technology specialist" (IT/IS), "public health science" (PHS), and "clinical and laboratory" (CL) roles. RESULTS: PHI specialists account for less than 2% of the governmental PH workforce. A majority were female (68%), White (55%), and close to half in 31-50 age category (49%). Most (74%) were in non-supervisory roles and <1% in managerial/executive roles, with less than one-third (29%) earning >$75 000 salary. Skill gaps on informatics-related tasks included: identify appropriate data/information sources; collect valid data for decision making; participate in quality improvement processes; identify evidence-based approaches. The PHI specialists reported lower skill gaps in data/informatics areas when compared to other public health roles (PHS and CL), and this was consistent across state/local settings. DISCUSSION: Given the scale of work needed for modernization of information systems, PH agencies need more individuals in informatics roles. To attract PHI specialists, better salaries, clear PHI job classifications and permanent PHI workers are needed, which requires sustained investments from federal and state governments. CONCLUSION: Efforts to train PHI specialists, recruit and retain them in the governmental public health workforce, and address hiring issues in public health agencies are essential next steps to transform the US public health enterprise.

OBJECTIVE: To evaluate the real-world performance of the SMART/HL7 Bulk Fast Health Interoperability Resources (FHIR) Access Application Programming Interface (API), developed to enable push button access to electronic health record data on large populations, and required under the 21st Century Cures Act Rule. MATERIALS AND METHODS: We used an open-source Bulk FHIR Testing Suite at 5 healthcare sites from April to September 2023, including 4 hospitals using electronic health records (EHRs) certified for interoperability, and 1 Health Information Exchange (HIE) using a custom, standards-compliant API build. We measured export speeds, data sizes, and completeness across 6 types of FHIR. RESULTS: Among the certified platforms, Oracle Cerner led in speed, managing 5-16 million resources at over 8000 resources/min. Three Epic sites exported a FHIR data subset, achieving 1-12 million resources at 1555-2500 resources/min. Notably, the HIE's custom API outperformed, generating over 141 million resources at 12 000 resources/min. DISCUSSION: The HIE's custom API showcased superior performance, endorsing the effectiveness of SMART/HL7 Bulk FHIR in enabling large-scale data exchange while underlining the need for optimization in existing EHR platforms. Agility and scalability are essential for diverse health, research, and public health use cases. CONCLUSION: To fully realize the interoperability goals of the 21st Century Cures Act, addressing the performance limitations of Bulk FHIR API is critical. It would be beneficial to include performance metrics in both certification and reporting processes.

OBJECTIVE: To address challenges in large-scale electronic health record (EHR) data exchange, we sought to develop, deploy, and test an open source, cloud-hosted app "listener" that accesses standardized data across the SMART/HL7 Bulk FHIR Access application programming interface (API). METHODS: We advance a model for scalable, federated, data sharing and learning. Cumulus software is designed to address key technology and policy desiderata including local utility, control, and administrative simplicity as well as privacy preservation during robust data sharing, and artificial intelligence (AI) for processing unstructured text. RESULTS: Cumulus relies on containerized, cloud-hosted software, installed within a healthcare organization's security envelope. Cumulus accesses EHR data via the Bulk FHIR interface and streamlines automated processing and sharing. The modular design enables use of the latest AI and natural language processing tools and supports provider autonomy and administrative simplicity. In an initial test, Cumulus was deployed across 5 healthcare systems each partnered with public health. Cumulus output is patient counts which were aggregated into a table stratifying variables of interest to enable population health studies. All code is available open source. A policy stipulating that only aggregate data leave the institution greatly facilitated data sharing agreements. DISCUSSION AND CONCLUSION: Cumulus addresses barriers to data sharing based on (1) federally required support for standard APIs, (2) increasing use of cloud computing, and (3) advances in AI. There is potential for scalability to support learning across myriad network configurations and use cases.

OBJECTIVE: The annual American College of Medical Informatics (ACMI) symposium focused discussion on the national public health information systems (PHIS) infrastructure to support public health goals. The objective of this article is to present the strengths, weaknesses, threats, and opportunities (SWOT) identified by public health and informatics leaders in attendance. MATERIALS AND METHODS: The Symposium provided a venue for experts in biomedical informatics and public health to brainstorm, identify, and discuss top PHIS challenges. Two conceptual frameworks, SWOT and the Informatics Stack, guided discussion and were used to organize factors and themes identified through a qualitative approach. RESULTS: A total of 57 unique factors related to the current PHIS were identified, including 9 strengths, 22 weaknesses, 14 opportunities, and 14 threats, which were consolidated into 22 themes according to the Stack. Most themes (68%) clustered at the top of the Stack. Three overarching opportunities were especially prominent: (1) addressing the needs for sustainable funding, (2) leveraging existing infrastructure and processes for information exchange and system development that meets public health goals, and (3) preparing the public health workforce to benefit from available resources. DISCUSSION: The PHIS is unarguably overdue for a strategically designed, technology-enabled, information infrastructure for delivering day-to-day essential public health services and to respond effectively to public health emergencies. CONCLUSION: Most of the themes identified concerned context, people, and processes rather than technical elements. We recommend that public health leadership consider the possible actions and leverage informatics expertise as we collectively prepare for the future.

The COVID-19 pandemic exposed multiple weaknesses in the nation's public health system. Therefore, the American College of Medical Informatics selected "Rebuilding the Nation's Public Health Informatics Infrastructure" as the theme for its annual symposium. Experts in biomedical informatics and public health discussed strategies to strengthen the US public health information infrastructure through policy, education, research, and development. This article summarizes policy recommendations for the biomedical informatics community postpandemic. First, the nation must perceive the health data infrastructure to be a matter of national security. The nation must further invest significantly more in its health data infrastructure. Investments should include the education and training of the public health workforce as informaticians in this domain are currently limited. Finally, investments should strengthen and expand health data utilities that increasingly play a critical role in exchanging information across public health and healthcare organizations.

OBJECTIVE: To assess the practice- and market-level factors associated with the amount of provider health information exchange (HIE) use. MATERIALS AND METHODS: Provider and practice-level data was drawn from the Meaningful Use Stage 2 Public Use Files from the Centers for Medicare and Medicaid Services, the Physician Compare National Downloadable File, and the Compendium of US Health Systems, among other sources. We analyzed the relationship between provider HIE use and practice and market factors using multivariable linear regression and compared primary care providers (PCPs) to non-PCPs. Provider volume of HIE use is measured as the percentage of referrals sent with electronic summaries of care (eSCR) reported by eligible providers attesting to the Meaningful Use electronic health record (EHR) incentive program in 2016. RESULTS: Providers used HIE in 49% of referrals; PCPs used HIE in fewer referrals (43%) than non-PCPs (57%). Provider use of products from EHR vendors was negatively related to HIE use, while use of Athenahealth and Greenway Health products were positively related to HIE use. Providers treating, on average, older patients and greater proportions of patients with diabetes used HIE for more referrals. Health system membership, market concentration, and state HIE consent policy were unrelated to provider HIE use. DISCUSSION: HIE use during referrals is low among office-based providers with the capability for exchange, especially PCPs. Practice-level factors were more commonly associated with greater levels of HIE use than market-level factors. CONCLUSION: This furthers the understanding that market forces, like competition, may be related to HIE adoption decisions but are less important for use once adoption has occurred.

Journal of the American Medical Informatics Association, doi: 10.1093/jamia/ocab024 The author name “Julia Adler-Milstein” was incorrectly given as “Julia Adler-Milstien”. This has been corrected online.

OBJECTIVE: We sought to support public health surveillance and response to coronavirus disease 2019 (COVID-19) through rapid development and implementation of novel visualization applications for data amalgamated across sectors. MATERIALS AND METHODS: We developed and implemented population-level dashboards that collate information on individuals tested for and infected with COVID-19, in partnership with state and local public health agencies as well as health systems. The dashboards are deployed on top of a statewide health information exchange. One dashboard enables authorized users working in public health agencies to surveil populations in detail, and a public version provides higher-level situational awareness to inform ongoing pandemic response efforts in communities. RESULTS: Both dashboards have proved useful informatics resources. For example, the private dashboard enabled detection of a local community outbreak associated with a meat packing plant. The public dashboard provides recent trend analysis to track disease spread and community-level hospitalizations. Combined, the tools were utilized 133 637 times by 74 317 distinct users between June 21 and August 22, 2020. The tools are frequently cited by journalists and featured on social media. DISCUSSION: Capitalizing on a statewide health information exchange, in partnership with health system and public health leaders, Regenstrief biomedical informatics experts rapidly developed and deployed informatics tools to support surveillance and response to COVID-19. CONCLUSIONS: The application of public health informatics methods and tools in Indiana holds promise for other states and nations. Yet, development of infrastructure and partnerships will require effort and investment after the current pandemic in preparation for the next public health emergency.

OBJECTIVE: To examine the effectiveness of event notification service (ENS) alerts on health care delivery processes and outcomes for older adults. MATERIALS AND METHODS: We deployed ENS alerts in 2 Veterans Affairs (VA) medical centers using regional health information exchange (HIE) networks from March 2016 to December 2019. Alerts targeted VA-based primary care teams when older patients (aged 65+ years) were hospitalized or attended emergency departments (ED) outside the VA system. We employed a concurrent cohort study to compare postdischarge outcomes between patients whose providers received ENS alerts and those that did not (usual care). Outcome measures included: timely follow-up postdischarge (actual phone call within 7 days or an in-person primary care visit within 30 days) and all-cause inpatient or ED readmission within 30 days. Generalized linear mixed models, accounting for clustering by primary care team, were used to compare outcomes between groups. RESULTS: Compared to usual care, veterans whose primary care team received notification of non-VA acute care encounters were 4 times more likely to have phone contact within 7 days (AOR = 4.10, P < .001) and 2 times more likely to have an in-person visit within 30 days (AOR = 1.98, P = .007). There were no significant differences between groups in hospital or ED utilization within 30 days of index discharge (P = .057). DISCUSSION: ENS was associated with increased timely follow-up following non-VA acute care events, but there was no associated change in 30-day readmission rates. Optimization of ENS processes may be required to scale use and impact across health systems. CONCLUSION: Given the importance of ENS to the VA and other health systems, this study provides guidance for future research on ENS for improving care coordination and population outcomes. TRIAL REGISTRATION: ClinicalTrials.gov NCT02689076. "Regional Data Exchange to Improve Care for Veterans After Non-VA Hospitalization." Registered February 23, 2016.

OBJECTIVE: To assess primary care teams' perceptions of a health information exchange (HIE) event notification intervention for geriatric patients in 2 Veterans Health Administration (VHA) medical centers. MATERIALS AND METHODS: We conducted a qualitative evaluation of an event notification alerting primary care teams to non-VHA hospital admissions and emergency department visits. Data were collected through semistructured interviews (n = 23) of primary care team physicians, nurses and medical assistants. Study design and analysis were guided by the Consolidated Framework for Implementation Research (CFIR). RESULTS: Team members found the alerts necessary, helpful for filling information gaps, and effective in supporting timely follow-up care, although some expressed concern over scheduling capacity and distinguishing alerts from other VHA notices. Participants also suggested improvements including additional data on patients' diagnosis and discharge instructions, timing alerts to patients' discharge (including clear next steps), including additional team members to ensure alerts were acted upon, and implementing a single sign-on. DISCUSSION: Primary care team members perceived timely event notification of non-VHA emergency department visits and hospital admissions as potentially improving post-discharge follow-up and patient outcomes. However, they were sometimes unsure of next steps and suggested the alerts and platform could be streamlined for easier use. CONCLUSIONS: Event notifications may be a valuable tool in coordinating care for high-risk older patients. Future intervention research should explore the optimal amount and types of information and delivery method across sites and test the integration of alerts into broader care coordination efforts.

Prior research on health information exchange (HIE) typically measured provider usage through surveys or they summarized the availability of HIE services in a healthcare organization. Few studies utilized user log files. Using HIE access log files, we measured HIE use in real-world clinical settings over a 7-year period (2011-2017). Use of HIE increased in inpatient, outpatient, and emergency department (ED) settings. Further, while extant literature has generally viewed the ED as the most relevant setting for HIE, the greatest change in HIE use was observed in the inpatient setting, followed by the ED setting and then the outpatient setting. Our findings suggest that in addition to federal incentives, the implementation of features that address barriers to access (eg, Single Sign On), as well as value-added services (eg, interoperability with external data sources), may be related to the growth in user-initiated HIE.

OBJECTIVE: Biomedical informatics attracts few underrepresented racial minorities (URMs) into PhD programs. We examine graduation trends from 2002 to 2017 to determine how URM representation has changed over time. We also examine academic job placements by race and identify individual and institutional characteristics associated with URM graduates being successfully placed in academic jobs. MATERIALS AND METHODS: We analyze a near census of all research doctoral graduates from US-accredited institutions, surveyed at graduation by the National Science Foundation Survey of Earned Doctorates. Graduates of biomedical informatics-related programs were identified using self-reported primary and secondary disciplines. Data are analyzed using bivariate and multivariable logistic regressions. RESULTS: During the study period, 2426 individuals earned doctoral degrees in biomedical informatics-related disciplines. URM students comprised nearly 12% of graduates, and this proportion did not change over time (2002-2017). URMs included Hispanic (5.7%), Black (3.2%), and others, including multi-racial and indigenous American populations (2.8%). Overall, 82.3% of all graduates accepted academic positions at the time of graduation with significantly more Hispanic graduates electing to go into academia (89.2%; P < .001). URM graduates were more likely to be single (OR = 1.38; P < .05), have a dependent (1.95; P < .01), and not receive full tuition remission (OR = 1.37; P = .05) as a student. URM graduates accepting an academic position were less likely to be a graduate of a private institution (OR = 0.70; P < .05). DISCUSSION AND CONCLUSION: The proportion of URM candidates among biomedical informatics doctoral graduates has not increased over time and remains low. In order to improve URM recruitment and retention within academia, leaders in biomedical informatics should replicate strategies used to improve URM graduation rates in other fields.

Background: Information reconciliation is a common yet complex and often time-consuming task performed by healthcare providers. While electronic health record systems can receive "outside information" about a patient in electronic documents, rarely does the computer automate reconciling information about a patient across all documents. Materials and Methods: Using a mixed methods design, we evaluated an information system designed to reconcile information across multiple electronic documents containing health records for a patient received from a health information exchange (HIE) network. Nine healthcare providers participated in scenario-based sessions in which they manually consolidated information across multiple documents. Accuracy of consolidation was measured along with the time spent completing 3 different reconciliation scenarios with and without support from the information system. Participants also attended an interview about their experience. Perceived workload was evaluated quantitatively using the NASA-TLX tool. Qualitative analysis focused on providers' impression of the system and the challenges faced when reconciling information in practice. Results: While 5 providers made mistakes when trying to manually reconcile information across multiple documents, no participants made a mistake when the system supported their work. Overall perceived workload decreased significantly for scenarios supported by the system (37.2% in referrals, 18.4% in medications, and 31.5% in problems scenarios, P < 0.001). Information reconciliation time was reduced significantly when the system supported provider tasks (58.8% in referrals, 38.1% in medications, and 65.1% in problem scenarios). Conclusion: Automating retrieval and reconciliation of information across multiple electronic documents shows promise for reducing healthcare providers' task complexity and workload.

OBJECTIVE: To characterize patients who voluntarily enrolled in an electronic health information exchange (HIE) program designed to share data between Veterans Health Administration (VHA) and non-VHA institutions. MATERIALS AND METHODS: Patients who agreed to participate in the HIE program were compared to those who did not. Patient characteristics associated with HIE enrollment were examined using a multivariable logistic regression model. Variables selected for inclusion were guided by a health care utilization model adapted to explain HIE enrollment. Data about patients' sociodemographics (age, gender), comorbidity (Charlson index score), utilization (primary and specialty care visits), and access (distance to VHA medical center, insurance, VHA benefits) were obtained from VHA and HIE electronic health records. RESULTS: Among 57 072 patients, 6627 (12%) enrolled in the HIE program during its first year. The likelihood of HIE enrollment increased among patients ages 50-64, of female gender, with higher comorbidity, and with increasing utilization. Living in a rural area and being unmarried were associated with decreased likelihood of enrollment. DISCUSSION AND CONCLUSION: Enrollment in HIE is complex, with several factors involved in a patient's decision to enroll. To broaden HIE participation, populations less likely to enroll should be targeted with tailored recruitment and educational strategies. Moreover, inclusion of special populations, such as patients with higher comorbidity or high utilizers, may help refine the definition of success with respect to HIE implementation.

BACKGROUND: Healthcare providers sometimes receive multiple continuity of care documents (CCDs) for a single patient encompassing the patient's various encounters and medical history recorded in different information systems. It is cumbersome for providers to explore different pages of CCDs to find specific data which can be duplicated or even conflicted. This study describes initial steps toward a modular system that integrates and de-duplicates multiple CCDs into one consolidated document for viewing or processing patient-level data. MATERIALS AND METHODS: The authors developed a prototype system to consolidate and de-duplicate CCDs. The system is engineered to be scalable, extensible, and open source. Using a corpus of 150 de-identified CCDs synthetically generated from a single data source with a common vocabulary to represent 50 unique patients, the authors tested the system's performance and output. Performance was measured based on document throughput and reduction in file size and volume of data. The authors further compared the output of the system with manual consolidation and de-duplication. Testing across multiple vendor systems or implementations was not performed. RESULTS: All of the input CCDs was successfully consolidated, and no data were lost. De-duplication significantly reduced the number of entries in different sections (49% in Problems, 60.6% in Medications, and 79% in Allergies) and reduced the size of the documents (57.5%) as well as the number of lines in each document (58%). The system executed at a rate of approximately 0.009-0.03 s per rule depending on the complexity of the rule. DISCUSSION AND CONCLUSION: Given increasing adoption and use of health information exchange (HIE) to share data and information across the care continuum, duplication of information is inevitable. A novel system designed to support automated consolidation and de-duplication of information across clinical documents as they are exchanged shows promise. Future work is needed to expand the capabilities of the system and further test it using heterogeneous vocabularies across multiple HIE scenarios.

Recent investigations into appointment scheduling within facilities operated by the US Department of Veterans Affairs (VA) illuminate systemic challenges in meeting its goal of providing timely access to care for all Veterans. In the wake of these investigations, new policies have been enacted to expand access to care at VA facilities as well as non-VA facilities if the VA is unable to provide access within a reasonable timeframe or a Veteran lives more than 40 miles from a VA medical facility. These policies are similar to broader health reform efforts that seek to expand access to care for other vulnerable populations. In this perspective, we discuss the informatics implications of expanded access within the VA and its wider applicability across the US health system. Health systems will require robust health information exchange, to maintain coordination while access to care is expanded. Existing informatics research can guide short-term implementation; furthermore, new research is needed to generate evidence about how best to achieve the long-term aim of expanded access to care.

OBJECTIVE: To describe the perspectives of Regenstrief LOINC Mapping Assistant (RELMA) users before and after the deployment of Community Mapping features, characterize the usage of these new features, and analyze the quality of mappings submitted to the community mapping repository. METHODS: We evaluated Logical Observation Identifiers Names and Codes (LOINC) community members' perceptions about new "wisdom of the crowd" information and how they used the new RELMA features. We conducted a pre-launch survey to capture users' perceptions of the proposed functionality of these new features; monitored how the new features and data available via those features were accessed; conducted a follow-up survey about the use of RELMA with the Community Mapping features; and analyzed community mappings using automated methods to detect potential errors. RESULTS: Despite general satisfaction with RELMA, nearly 80% of 155 respondents to our pre-launch survey indicated that having information on how often other users had mapped to a particular LOINC term would be helpful. During the study period, 200 participants logged into the RELMA Community Mapping features an average of 610 times per month and viewed the mapping detail pages a total of 6686 times. Fifty respondents (25%) completed our post-launch survey, and those who accessed the Community Mapping features unanimously indicated that they were useful. Overall, 95.3% of the submitted mappings passed our automated validation checks. CONCLUSION: When information about other institutions' mappings was made available, study participants who accessed it agreed that it was useful and informed their mapping choices. Our findings suggest that a crowd-sourced repository of mappings is valuable to users who are mapping local terms to LOINC terms.

The aim of this study was to describe an integrated informatics approach to aggregating and displaying clinically relevant data that can identify problems with medication adherence and facilitate patient-provider communication about strategies to improve medication use. We developed a clinical dashboard within an electronic health record (EHR) system that uses data from three sources: the medical record, pharmacy claims, and a personal health record. The data are integrated to inform clinician-patient discussions about medication adherence. Whereas prior research on assessing patterns of medication adherence focused on a single approach using the EHR, pharmacy data, or patient-entered data, we present an approach that integrates multiple electronic data sources increasingly found in practice. Medication adherence is a complex challenge that requires patient and provider team input, necessitating an integrated approach using advanced EHR, clinical decision support, and patient-controlled technologies. Future research should focus on integrated strategies to provide patients and providers with the right combination of informatics tools to help them adequately address the challenge of adherence to complex medication therapies.

Proliferation of health information technologies creates opportunities to improve clinical and public health, including high quality, safer care and lower costs. To maximize such potential benefits, health information technologies must readily and reliably exchange information with other systems. However, evidence from public health surveillance programs in two states suggests that operational clinical information systems often fail to use available standards, a barrier to semantic interoperability. Furthermore, analysis of existing policies incentivizing semantic interoperability suggests they have limited impact and are fragmented. In this essay, we discuss three approaches for increasing semantic interoperability to support national goals for using health information technologies. A clear, comprehensive strategy requiring collaborative efforts by clinical and public health stakeholders is suggested as a guide for the long road towards better population health data and outcomes.

OBJECTIVE: To summarize the literature describing computer-based interventions aimed at improving bidirectional communication between clinical and public health. MATERIALS AND METHODS: A systematic review of English articles using MEDLINE and Google Scholar. Search terms included public health, epidemiology, electronic health records, decision support, expert systems, and decision-making. Only articles that described the communication of information regarding emerging health threats from public health agencies to clinicians or provider organizations were included. Each article was independently reviewed by two authors. RESULTS: Ten peer-reviewed articles highlight a nascent but promising area of research and practice related to alerting clinicians about emerging threats. Current literature suggests that additional research and development in bidirectional communication infrastructure should focus on defining a coherent architecture, improving interoperability, establishing clear governance, and creating usable systems that will effectively deliver targeted, specific information to clinicians in support of patient and population decision-making. CONCLUSIONS: Increasingly available clinical information systems make it possible to deliver timely, relevant knowledge to frontline clinicians in support of population health. Future work should focus on developing a flexible, interoperable infrastructure for bidirectional communications capable of integrating public health knowledge into clinical systems and workflows.

OBJECTIVE: The nationwide health information network (NHIN) has been proposed to securely link community and state health information exchange (HIE) entities to create a national, interoperable network for sharing healthcare data in the USA. This paper describes a framework for evaluating the costs, effort, and value of nationwide data exchange as the NHIN moves toward a production state. The paper further presents the results of an initial assessment of the framework by those engaged in HIE activities. DESIGN: Using a literature review and knowledge gained from active NHIN technology and policy development, the authors constructed a framework for evaluating the costs, effort, and value of data exchange between an HIE entity and the NHIN. MEASUREMENT: An online survey was used to assess the perceived usefulness of the metrics in the framework among HIE professionals and researchers. RESULTS: The framework is organized into five broad categories: implementation; technology; policy; data; and value. Each category enumerates a variety of measures and measure types. Survey respondents generally indicated the framework contained useful measures for current and future use in HIE and NHIN evaluation. Answers varied slightly based on a respondent's participation in active development of NHIN components. CONCLUSION: The proposed framework supports efforts to measure the costs, effort, and value associated with nationwide data exchange. Collecting longitudinal data along the NHIN's path to production should help with the development of an evidence base that will drive adoption, create value, and stimulate further investment in nationwide data exchange.