Causal discovery methods are powerful tools for uncovering the structure of relationships among variables, yet they face significant challenges in scalability and interpretability, especially in high-dimensional settings. In many domains, researchers are not only interested in causal links between individual variables, but also in relationships among sets or clusters of variables. Learning causal structure at the cluster level can both reveal higher-order relationships of interest and improve scalability. In this work, we introduce an approach for causal discovery over clusters in Markov causal systems. We propose a new graphical model that encodes knowledge of relationships between user-defined clusters while fully representing independencies and dependencies over clusters, faithful to a given distribution. We then define and characterize a graphical equivalence class of these models that share cluster-level independence information. Lastly, we present a sound and complete algorithm for causal discovery to represent learnable causal relationships between clusters of variables.

OBJECTIVE: Propose a framework to empirically evaluate and report validity of findings from observational studies using pre-specified objective diagnostics, increasing trust in real-world evidence (RWE). MATERIALS AND METHODS: The framework employs objective diagnostic measures to assess the appropriateness of study designs, analytic assumptions, and threats to validity in generating reliable evidence addressing causal questions. Diagnostic evaluations should be interpreted before the unblinding of study results or, alternatively, only unblind results from analyses that pass pre-specified thresholds. We provide a conceptual overview of objective diagnostic measures and demonstrate their impact on the validity of RWE from a large-scale comparative new-user study of various antihypertensive medications. We evaluated expected absolute systematic error (EASE) before and after applying diagnostic thresholds, using a large set of negative control outcomes. RESULTS: Applying objective diagnostics reduces bias and improves evidence reliability in observational studies. Among 11 716 analyses (EASE = 0.38), 13.9% met pre-specified diagnostic thresholds which reduced EASE to zero. Objective diagnostics provide a comprehensive and empirical set of tests that increase confidence when passed and raise doubts when failed. DISCUSSION: The increasing use of real-world data presents a scientific opportunity; however, the complexity of the evidence generation process poses challenges for understanding study validity and trusting RWE. Deploying objective diagnostics is crucial to reducing bias and improving reliability in RWE generation. Under ideal conditions, multiple study designs pass diagnostics and generate consistent results, deepening understanding of causal relationships. Open-source, standardized programs can facilitate implementation of diagnostic analyses. CONCLUSION: Objective diagnostics are a valuable addition to the RWE generation process.

The primary practice of healthcare artificial intelligence (AI) starts with model development, often using state-of-the-art AI, retrospectively evaluated using metrics lifted from the AI literature like AUROC and DICE score. However, good performance on these metrics may not translate to improved clinical outcomes. Instead, we argue for a better development pipeline constructed by working backward from the end goal of positively impacting clinically relevant outcomes using AI, leading to considerations of causality in model development and validation, and subsequently a better development pipeline. Healthcare AI should be "actionable," and the change in actions induced by AI should improve outcomes. Quantifying the effect of changes in actions on outcomes is causal inference. The development, evaluation, and validation of healthcare AI should therefore account for the causal effect of intervening with the AI on clinically relevant outcomes. Using a causal lens, we make recommendations for key stakeholders at various stages of the healthcare AI pipeline. Our recommendations aim to increase the positive impact of AI on clinical outcomes.

Human knowledge of diseases, treatments, and prevention techniques is constantly evolving. The generation of clinical evidence using randomized controlled trials on human subjects occurs notably slowly and inefficiently. The Learning Health System (LHS) has been proposed to facilitate the continuous improvement of individual and population health through a cycle of knowledge, practice, and data. However, the gap between the demand for high-quality evidence to support clinical decisions and the available evidence continues to enlarge. While the current LHS vision articulates the integration of Real-World Data (RWD), the rapid generation of RWD often outpaces the rate of effective evidence synthesis and implementation. Considering this, we propose a new framework that more effectively leverages RWD to support the entire clinical evidence lifecycle through a continuous learning mechanism. This framework, powered by modern data science and informatics, offers enhanced scalability and efficiency. In this vision, specifically, RWD is integrated into the clinical evidence lifecycle via four closed feedback loops: 1) guiding research prioritization and study design, 2) facilitating clinical guideline development, 3) assisting guideline evaluation, and 4) supporting shared decision-making. Our framework enables rapid responsiveness to emerging health data and evolving healthcare needs, timely development of clinical guidelines to optimize clinical recommendations, and sustained improvements in clinical practice and patient outcomes. This vision calls for informatics support for an efficient, scalable, and stakeholder-aware clinical evidence lifecycle.

High-dimensional healthcare data, such as electronic health records (EHR) data and claims data, present two primary challenges due to the large number of variables and the need to consolidate data from multiple clinical sites. The third key challenge is the potential existence of heterogeneity in terms of covariate shift. In this paper, we propose a distributed learning algorithm accounting for covariate shift to estimate the average treatment effect (ATE) for high-dimensional data, named DisC2o-HD. Leveraging the surrogate likelihood method, our method calibrates the estimates of the propensity score and outcome models to approximately attain the desired covariate balancing property, while accounting for the covariate shift across multiple clinical sites. We show that our distributed covariate balancing propensity score estimator can approximate the pooled estimator, which is obtained by pooling the data from multiple sites together. The proposed estimator remains consistent if either the propensity score model or the outcome regression model is correctly specified. The semiparametric efficiency bound is achieved when both the propensity score and the outcome models are correctly specified. We conduct simulation studies to demonstrate the performance of the proposed algorithm; additionally, we conduct an empirical study to present the readiness of implementation and validity.

OBJECTIVE: To evaluate the use of patient portal messaging to recruit individuals historically underrepresented in biomedical research (UBR) to the All of Us Research Program (AoURP) at a single recruitment site. MATERIALS AND METHODS: Patient portal-based recruitment was implemented at Columbia University Irving Medical Center. Patient engagement was assessed using patient's electronic health record (EHR) at four recruitment stages: Consenting to be contacted, opening messages, responding to messages, and showing interest in participating. Demographic and socioeconomic data were also collected from patient's EHR and univariate logistic regression analyses were conducted to assess patient engagement. RESULTS: Between October 2022 and November 2023, a total of 59 592 patients received patient portal messages inviting them to join the AoURP. Among them, 24 445 (41.0%) opened the message, 8983 (15.1%) responded, and 3765 (6.3%) showed interest in joining the program. Though we were unable to link enrollment data with EHR data, we estimate about 2% of patients contacted ultimately enrolled in the AoURP. Patients from underrepresented race and ethnicity communities had lower odds of consenting to be contacted and opening messages, but higher odds of showing interest after responding. DISCUSSION: Patient portal messaging provided both patients and recruitment staff with a more efficient approach to outreach, but patterns of engagement varied across UBR groups. CONCLUSION: Patient portal-based recruitment enables researchers to contact a substantial number of participants from diverse communities. However, more effort is needed to improve engagement from underrepresented racial and ethnic groups at the early stages of the recruitment process.

IMPORTANCE: The Observational Health Data Sciences and Informatics (OHDSI) is the largest distributed data network in the world encompassing more than 331 data sources with 2.1 billion patient records across 34 countries. It enables large-scale observational research through standardizing the data into a common data model (CDM) (Observational Medical Outcomes Partnership [OMOP] CDM) and requires a comprehensive, efficient, and reliable ontology system to support data harmonization. MATERIALS AND METHODS: We created the OHDSI Standardized Vocabularies-a common reference ontology mandatory to all data sites in the network. It comprises imported and de novo-generated ontologies containing concepts and relationships between them, and the praxis of converting the source data to the OMOP CDM based on these. It enables harmonization through assigned domains according to clinical categories, comprehensive coverage of entities within each domain, support for commonly used international coding schemes, and standardization of semantically equivalent concepts. RESULTS: The OHDSI Standardized Vocabularies comprise over 10 million concepts from 136 vocabularies. They are used by hundreds of groups and several large data networks. More than 8600 users have performed 50 000 downloads of the system. This open-source resource has proven to address an impediment of large-scale observational research-the dependence on the context of source data representation. With that, it has enabled efficient phenotyping, covariate construction, patient-level prediction, population-level estimation, and standard reporting. DISCUSSION AND CONCLUSION: OHDSI has made available a comprehensive, open vocabulary system that is unmatched in its ability to support global observational research. We encourage researchers to exploit it and contribute their use cases to this dynamic resource.

OBJECTIVE: Healthcare continues to grapple with the persistent issue of treatment disparities, sparking concerns regarding the equitable allocation of treatments in clinical practice. While various fairness metrics have emerged to assess fairness in decision-making processes, a growing focus has been on causality-based fairness concepts due to their capacity to mitigate confounding effects and reason about bias. However, the application of causal fairness notions in evaluating the fairness of clinical decision-making with electronic health record (EHR) data remains an understudied domain. This study aims to address the methodological gap in assessing causal fairness of treatment allocation with electronic health records data. In addition, we investigate the impact of social determinants of health on the assessment of causal fairness of treatment allocation. METHODS: We propose a causal fairness algorithm to assess fairness in clinical decision-making. Our algorithm accounts for the heterogeneity of patient populations and identifies potential unfairness in treatment allocation by conditioning on patients who have the same likelihood to benefit from the treatment. We apply this framework to a patient cohort with coronary artery disease derived from an EHR database to evaluate the fairness of treatment decisions. RESULTS: Our analysis reveals notable disparities in coronary artery bypass grafting (CABG) allocation among different patient groups. Women were found to be 4.4%-7.7% less likely to receive CABG than men in two out of four treatment response strata. Similarly, Black or African American patients were 5.4%-8.7% less likely to receive CABG than others in three out of four response strata. These results were similar when social determinants of health (insurance and area deprivation index) were dropped from the algorithm. These findings highlight the presence of disparities in treatment allocation among similar patients, suggesting potential unfairness in the clinical decision-making process. CONCLUSION: This study introduces a novel approach for assessing the fairness of treatment allocation in healthcare. By incorporating responses to treatment into fairness framework, our method explores the potential of quantifying fairness from a causal perspective using EHR data. Our research advances the methodological development of fairness assessment in healthcare and highlight the importance of causality in determining treatment fairness.

OBJECTIVE: The aim of this study was to analyze a publicly available sample of rule-based phenotype definitions to characterize and evaluate the variability of logical constructs used. MATERIALS AND METHODS: A sample of 33 preexisting phenotype definitions used in research that are represented using Fast Healthcare Interoperability Resources and Clinical Quality Language (CQL) was analyzed using automated analysis of the computable representation of the CQL libraries. RESULTS: Most of the phenotype definitions include narrative descriptions and flowcharts, while few provide pseudocode or executable artifacts. Most use 4 or fewer medical terminologies. The number of codes used ranges from 5 to 6865, and value sets from 1 to 19. We found that the most common expressions used were literal, data, and logical expressions. Aggregate and arithmetic expressions are the least common. Expression depth ranges from 4 to 27. DISCUSSION: Despite the range of conditions, we found that all of the phenotype definitions consisted of logical criteria, representing both clinical and operational logic, and tabular data, consisting of codes from standard terminologies and keywords for natural language processing. The total number and variety of expressions are low, which may be to simplify implementation, or authors may limit complexity due to data availability constraints. CONCLUSIONS: The phenotype definitions analyzed show significant variation in specific logical, arithmetic, and other operators but are all composed of the same high-level components, namely tabular data and logical expressions. A standard representation for phenotype definitions should support these formats and be modular to support localization and shared logic.

OBJECTIVE: Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations' variability on patient characteristics. MATERIALS AND METHODS: Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams' cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics. RESULTS: On average, the teams' interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts' size varied from one-third of the master cohort size to 10 times the cohort size (2159-63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3-16.2%). The teams' cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5. CONCLUSIONS: Independent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.

OBJECTIVES: Chart review as the current gold standard for phenotype evaluation cannot support observational research on electronic health records and claims data sources at scale. We aimed to evaluate the ability of structured data to support efficient and interpretable phenotype evaluation as an alternative to chart review. MATERIALS AND METHODS: We developed Knowledge-Enhanced Electronic Profile Review (KEEPER) as a phenotype evaluation tool that extracts patient's structured data elements relevant to a phenotype and presents them in a standardized fashion following clinical reasoning principles. We evaluated its performance (interrater agreement, intermethod agreement, accuracy, and review time) compared to manual chart review for 4 conditions using randomized 2-period, 2-sequence crossover design. RESULTS: Case ascertainment with KEEPER was twice as fast compared to manual chart review. 88.1% of the patients were classified concordantly using charts and KEEPER, but agreement varied depending on the condition. Missing data and differences in interpretation accounted for most of the discrepancies. Pairs of clinicians agreed in case ascertainment in 91.2% of the cases when using KEEPER compared to 76.3% when using charts. Patient classification aligned with the gold standard in 88.1% and 86.9% of the cases respectively. CONCLUSION: Structured data can be used for efficient and interpretable phenotype evaluation if they are limited to relevant subset and organized according to the clinical reasoning principles. A system that implements these principles can achieve noninferior performance compared to chart review at a fraction of time.

OBJECTIVE: Prediction of physiological mechanics are important in medical practice because interventions are guided by predicted impacts of interventions. But prediction is difficult in medicine because medicine is complex and difficult to understand from data alone, and the data are sparse relative to the complexity of the generating processes. Computational methods can increase prediction accuracy, but prediction with clinical data is difficult because the data are sparse, noisy and nonstationary. This paper focuses on predicting physiological processes given sparse, non-stationary, electronic health record data in the intensive care unit using data assimilation (DA), a broad collection of methods that pair mechanistic models with inference methods. METHODS: A methodological pipeline embedding a glucose-insulin model into a new DA framework, the constrained ensemble Kalman filter (CEnKF) to forecast blood glucose was developed. The data include tube-fed patients whose nutrition, blood glucose, administered insulins and medications were extracted by hand due to their complexity and to ensure accuracy. The model was estimated using an individual's data as if they arrived in real-time, and the estimated model was run forward producing a forecast. Both constrained and unconstrained ensemble Kalman filters were estimated to compare the impact of constraints. Constraint boundaries, model parameter sets estimated, and data used to estimate the models were varied to investigate their influence on forecasting accuracy. Forecasting accuracy was evaluated according to mean squared error between the model-forecasted glucose and the measurements and by comparing distributions of measured glucose and forecast ensemble means. RESULTS: The novel CEnKF produced substantial gains in robustness and accuracy while minimizing the data requirements compared to the unconstrained ensemble Kalman filters. Administered insulin and tube-nutrition were important for accurate forecasting, but including glucose in IV medication delivery did not increase forecast accuracy. Model flexibility, controlled by constraint boundaries and estimated parameters, did influence forecasting accuracy. CONCLUSION: Accurate and robust physiological forecasting with sparse clinical data is possible with DA. Introducing constrained inference, particularly on unmeasured states and parameters, reduced forecast error and data requirements. The results are not particularly sensitive to model flexibility such as constraint boundaries, but over or under constraining increased forecasting errors.

OBJECTIVE: Computing phenotypes that provide high-fidelity, time-dependent characterizations and yield personalized interpretations is challenging, especially given the complexity of physiological and healthcare systems and clinical data quality. This paper develops a methodological pipeline to estimate unmeasured physiological parameters and produce high-fidelity, personalized phenotypes anchored to physiological mechanics from electronic health record (EHR). METHODS: A methodological phenotyping pipeline is developed that computes new phenotypes defined with unmeasurable computational biomarkers quantifying specific physiological properties in real time. Working within the inverse problem framework, this pipeline is applied to the glucose-insulin system for ICU patients using data assimilation to estimate an established mathematical physiological model with stochastic optimization. This produces physiological model parameter vectors of clinically unmeasured endocrine properties, here insulin secretion, clearance, and resistance, estimated for individual patient. These physiological parameter vectors are used as inputs to unsupervised machine learning methods to produce phenotypic labels and discrete physiological phenotypes. These phenotypes are inherently interpretable because they are based on parametric physiological descriptors. To establish potential clinical utility, the computed phenotypes are evaluated with external EHR data for consistency and reliability and with clinician face validation. RESULTS: The phenotype computation was performed on a cohort of 109 ICU patients who received no or short-acting insulin therapy, rendering continuous and discrete physiological phenotypes as specific computational biomarkers of unmeasured insulin secretion, clearance, and resistance on time windows of three days. Six, six, and five discrete phenotypes were found in the first, middle, and last three-day periods of ICU stays, respectively. Computed phenotypic labels were predictive with an average accuracy of 89%. External validation of discrete phenotypes showed coherence and consistency in clinically observable differences based on laboratory measurements and ICD 9/10 codes and clinical concordance from face validity. A particularly clinically impactful parameter, insulin secretion, had a concordance accuracy of 83%±27%. CONCLUSION: The new physiological phenotypes computed with individual patient ICU data and defined by estimates of mechanistic model parameters have high physiological fidelity, are continuous, time-specific, personalized, interpretable, and predictive. This methodology is generalizable to other clinical and physiological settings and opens the door for discovering deeper physiological information to personalize medical care.

Confounding remains one of the major challenges to causal inference with observational data. This problem is paramount in medicine, where we would like to answer causal questions from large observational datasets like electronic health records (EHRs) and administrative claims. Modern medical data typically contain tens of thousands of covariates. Such a large set carries hope that many of the confounders are directly measured, and further hope that others are indirectly measured through their correlation with measured covariates. How can we exploit these large sets of covariates for causal inference? To help answer this question, this paper examines the performance of the large-scale propensity score (LSPS) approach on causal analysis of medical data. We demonstrate that LSPS may adjust for indirectly measured confounders by including tens of thousands of covariates that may be correlated with them. We present conditions under which LSPS removes bias due to indirectly measured confounders, and we show that LSPS may avoid bias when inadvertently adjusting for variables (like colliders) that otherwise can induce bias. We demonstrate the performance of LSPS with both simulated medical data and real medical data.

OBJECTIVE: Coronavirus disease 2019 (COVID-19) poses societal challenges that require expeditious data and knowledge sharing. Though organizational clinical data are abundant, these are largely inaccessible to outside researchers. Statistical, machine learning, and causal analyses are most successful with large-scale data beyond what is available in any given organization. Here, we introduce the National COVID Cohort Collaborative (N3C), an open science community focused on analyzing patient-level data from many centers. MATERIALS AND METHODS: The Clinical and Translational Science Award Program and scientific community created N3C to overcome technical, regulatory, policy, and governance barriers to sharing and harmonizing individual-level clinical data. We developed solutions to extract, aggregate, and harmonize data across organizations and data models, and created a secure data enclave to enable efficient, transparent, and reproducible collaborative analytics. RESULTS: Organized in inclusive workstreams, we created legal agreements and governance for organizations and researchers; data extraction scripts to identify and ingest positive, negative, and possible COVID-19 cases; a data quality assurance and harmonization pipeline to create a single harmonized dataset; population of the secure data enclave with data, machine learning, and statistical analytics tools; dissemination mechanisms; and a synthetic data pilot to democratize data access. CONCLUSIONS: The N3C has demonstrated that a multisite collaborative learning health network can overcome barriers to rapidly build a scalable infrastructure incorporating multiorganizational clinical data for COVID-19 analytics. We expect this effort to save lives by enabling rapid collaboration among clinicians, researchers, and data scientists to identify treatments and specialized care and thereby reduce the immediate and long-term impacts of COVID-19.

OBJECTIVE: A number of clinical decision support tools aim to use observational data to address immediate clinical needs, but few of them address challenges and biases inherent in such data. The goal of this article is to describe the experience of running a data consult service that generates clinical evidence in real time and characterize the challenges related to its use of observational data. MATERIALS AND METHODS: In 2019, we launched the Data Consult Service pilot with clinicians affiliated with Columbia University Irving Medical Center. We created and implemented a pipeline (question gathering, data exploration, iterative patient phenotyping, study execution, and assessing validity of results) for generating new evidence in real time. We collected user feedback and assessed issues related to producing reliable evidence. RESULTS: We collected 29 questions from 22 clinicians through clinical rounds, emails, and in-person communication. We used validated practices to ensure reliability of evidence and answered 24 of them. Questions differed depending on the collection method, with clinical rounds supporting proactive team involvement and gathering more patient characterization questions and questions related to a current patient. The main challenges we encountered included missing and incomplete data, underreported conditions, and nonspecific coding and accurate identification of drug regimens. CONCLUSIONS: While the Data Consult Service has the potential to generate evidence and facilitate decision making, only a portion of questions can be answered in real time. Recognizing challenges in patient phenotyping and designing studies along with using validated practices for observational research are mandatory to produce reliable evidence.

OBJECTIVE: Coronavirus disease 2019 (COVID-19) patients are at risk for resource-intensive outcomes including mechanical ventilation (MV), renal replacement therapy (RRT), and readmission. Accurate outcome prognostication could facilitate hospital resource allocation. We develop and validate predictive models for each outcome using retrospective electronic health record data for COVID-19 patients treated between March 2 and May 6, 2020. MATERIALS AND METHODS: For each outcome, we trained 3 classes of prediction models using clinical data for a cohort of SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2)-positive patients (n = 2256). Cross-validation was used to select the best-performing models per the areas under the receiver-operating characteristic and precision-recall curves. Models were validated using a held-out cohort (n = 855). We measured each model's calibration and evaluated feature importances to interpret model output. RESULTS: The predictive performance for our selected models on the held-out cohort was as follows: area under the receiver-operating characteristic curve-MV 0.743 (95% CI, 0.682-0.812), RRT 0.847 (95% CI, 0.772-0.936), readmission 0.871 (95% CI, 0.830-0.917); area under the precision-recall curve-MV 0.137 (95% CI, 0.047-0.175), RRT 0.325 (95% CI, 0.117-0.497), readmission 0.504 (95% CI, 0.388-0.604). Predictions were well calibrated, and the most important features within each model were consistent with clinical intuition. DISCUSSION: Our models produce performant, well-calibrated, and interpretable predictions for COVID-19 patients at risk for the target outcomes. They demonstrate the potential to accurately estimate outcome prognosis in resource-constrained care sites managing COVID-19 patients. CONCLUSIONS: We develop and validate prognostic models targeting MV, RRT, and readmission for hospitalized COVID-19 patients which produce accurate, interpretable predictions. Additional external validation studies are needed to further verify the generalizability of our results.

OBJECTIVE: Real-world data (RWD), defined as routinely collected healthcare data, can be a potential catalyst for addressing challenges faced in clinical trials. We performed a scoping review of database-specific RWD applications within clinical trial contexts, synthesizing prominent uses and themes. MATERIALS AND METHODS: Querying 3 biomedical literature databases, research articles using electronic health records, administrative claims databases, or clinical registries either within a clinical trial or in tandem with methodology related to clinical trials were included. Articles were required to use at least 1 US RWD source. All abstract screening, full-text screening, and data extraction was performed by 1 reviewer. Two reviewers independently verified all decisions. RESULTS: Of 2020 screened articles, 89 qualified: 59 articles used electronic health records, 29 used administrative claims, and 26 used registries. Our synthesis was driven by the general life cycle of a clinical trial, culminating into 3 major themes: trial process tasks (51 articles); dissemination strategies (6); and generalizability assessments (34). Despite a diverse set of diseases studied, <10% of trials using RWD for trial process tasks evaluated medications or procedures (5/51). All articles highlighted data-related challenges, such as missing values. DISCUSSION: Database-specific RWD have been occasionally leveraged for various clinical trial tasks. We observed underuse of RWD within conducted medication or procedure trials, though it is subject to the confounder of implicit report of RWD use. CONCLUSION: Enhanced incorporation of RWD should be further explored for medication or procedure trials, including better understanding of how to handle related data quality issues to facilitate RWD use.

Journal of the American Medical Informatics Association, 27(8), 2020, 1268–1277; doi: 10.1093/jamia/ocaa124 Upon the original publication of this article, several author corrections to the reference section were inadvertently left out, making literature referencing in the article inaccurate. This error has now been corrected online. The publisher apologises for the error.

Clinicians often attribute much of their burnout experience to use of the electronic health record, the adoption of which was greatly accelerated by the Health Information Technology for Economic and Clinical Health Act of 2009. That same year, AMIA's Policy Meeting focused on possible unintended consequences associated with rapid implementation of electronic health records, generating 17 potential consequences and 15 recommendations to address them. At the 2020 annual meeting of the American College of Medical Informatics (ACMI), ACMI fellows participated in a modified Delphi process to assess the accuracy of the 2009 predictions and the response to the recommendations. Among the findings, the fellows concluded that the degree of clinician burnout and its contributing factors, such as increased documentation requirements, were significantly underestimated. Conversely, problems related to identify theft and fraud were overestimated. Only 3 of the 15 recommendations were adjudged more than half-addressed.

[This corrects the article DOI: 10.1371/journal.pcbi.1005232.].

OBJECTIVE: Accurate electronic phenotyping is essential to support collaborative observational research. Supervised machine learning methods can be used to train phenotype classifiers in a high-throughput manner using imperfectly labeled data. We developed 10 phenotype classifiers using this approach and evaluated performance across multiple sites within the Observational Health Data Sciences and Informatics (OHDSI) network. MATERIALS AND METHODS: We constructed classifiers using the Automated PHenotype Routine for Observational Definition, Identification, Training and Evaluation (APHRODITE) R-package, an open-source framework for learning phenotype classifiers using datasets in the Observational Medical Outcomes Partnership Common Data Model. We labeled training data based on the presence of multiple mentions of disease-specific codes. Performance was evaluated on cohorts derived using rule-based definitions and real-world disease prevalence. Classifiers were developed and evaluated across 3 medical centers, including 1 international site. RESULTS: Compared to the multiple mentions labeling heuristic, classifiers showed a mean recall boost of 0.43 with a mean precision loss of 0.17. Performance decreased slightly when classifiers were shared across medical centers, with mean recall and precision decreasing by 0.08 and 0.01, respectively, at a site within the USA, and by 0.18 and 0.10, respectively, at an international site. DISCUSSION AND CONCLUSION: We demonstrate a high-throughput pipeline for constructing and sharing phenotype classifiers across sites within the OHDSI network using APHRODITE. Classifiers exhibit good portability between sites within the USA, however limited portability internationally, indicating that classifier generalizability may have geographic limitations, and, consequently, sharing the classifier-building recipe, rather than the pretrained classifiers, may be more useful for facilitating collaborative observational research.

OBJECTIVE: A growing body of observational data enabled its secondary use to facilitate clinical care for complex cases not covered by the existing evidence. We conducted a scoping review to characterize clinical decision support systems (CDSSs) that generate new knowledge to provide guidance for such cases in real time. MATERIALS AND METHODS: PubMed, Embase, ProQuest, and IEEE Xplore were searched up to May 2020. The abstracts were screened by 2 reviewers. Full texts of the relevant articles were reviewed by the first author and approved by the second reviewer, accompanied by the screening of articles' references. The details of design, implementation and evaluation of included CDSSs were extracted. RESULTS: Our search returned 3427 articles, 53 of which describing 25 CDSSs were selected. We identified 8 expert-based and 17 data-driven tools. Sixteen (64%) tools were developed in the United States, with the others mostly in Europe. Most of the tools (n = 16, 64%) were implemented in 1 site, with only 5 being actively used in clinical practice. Patient or quality outcomes were assessed for 3 (18%) CDSSs, 4 (16%) underwent user acceptance or usage testing and 7 (28%) functional testing. CONCLUSIONS: We found a number of CDSSs that generate new knowledge, although only 1 addressed confounding and bias. Overall, the tools lacked demonstration of their utility. Improvement in clinical and quality outcomes were shown only for a few CDSSs, while the benefits of the others remain unclear. This review suggests a need for a further testing of such CDSSs and, if appropriate, their dissemination.

OBJECTIVES: To demonstrate the application of the Large-scale Evidence Generation and Evaluation across a Network of Databases (LEGEND) principles described in our companion article to hypertension treatments and assess internal and external validity of the generated evidence. MATERIALS AND METHODS: LEGEND defines a process for high-quality observational research based on 10 guiding principles. We demonstrate how this process, here implemented through large-scale propensity score modeling, negative and positive control questions, empirical calibration, and full transparency, can be applied to compare antihypertensive drug therapies. We assess internal validity through covariate balance, confidence-interval coverage, between-database heterogeneity, and transitivity of results. We assess external validity through comparison to direct meta-analyses of randomized controlled trials (RCTs). RESULTS: From 21.6 million unique antihypertensive new users, we generate 6 076 775 effect size estimates for 699 872 research questions on 12 946 treatment comparisons. Through propensity score matching, we achieve balance on all baseline patient characteristics for 75% of estimates, observe 95.7% coverage in our effect-estimate 95% confidence intervals, find high between-database consistency, and achieve transitivity in 84.8% of triplet hypotheses. Compared with meta-analyses of RCTs, our results are consistent with 28 of 30 comparisons while providing narrower confidence intervals. CONCLUSION: We find that these LEGEND results show high internal validity and are congruent with meta-analyses of RCTs. For these reasons we believe that evidence generated by LEGEND is of high quality and can inform medical decision-making where evidence is currently lacking. Subsequent publications will explore the clinical interpretations of this evidence.

Evidence derived from existing health-care data, such as administrative claims and electronic health records, can fill evidence gaps in medicine. However, many claim such data cannot be used to estimate causal treatment effects because of the potential for observational study bias; for example, due to residual confounding. Other concerns include P hacking and publication bias. In response, the Observational Health Data Sciences and Informatics international collaborative launched the Large-scale Evidence Generation and Evaluation across a Network of Databases (LEGEND) research initiative. Its mission is to generate evidence on the effects of medical interventions using observational health-care databases while addressing the aforementioned concerns by following a recently proposed paradigm. We define 10 principles of LEGEND that enshrine this new paradigm, prescribing the generation and dissemination of evidence on many research questions at once; for example, comparing all treatments for a disease for many outcomes, thus preventing publication bias. These questions are answered using a prespecified and systematic approach, avoiding P hacking. Best-practice statistical methods address measured confounding, and control questions (research questions where the answer is known) quantify potential residual bias. Finally, the evidence is generated in a network of databases to assess consistency by sharing open-source analytics code to enhance transparency and reproducibility, but without sharing patient-level information. Here we detail the LEGEND principles and provide a generic overview of a LEGEND study. Our companion paper highlights an example study on the effects of hypertension treatments, and evaluates the internal and external validity of the evidence we generate.

Objective: To determine the effects of an inpatient portal intervention on patient activation, patient satisfaction, patient engagement with health information, and 30-day hospital readmissions. Methods and Materials: From March 2014 to May 2017, we enrolled 426 English- or Spanish-speaking patients from 2 cardiac medical-surgical units at an urban academic medical center. Patients were randomized to 1 of 3 groups: 1) usual care, 2) tablet with general Internet access (tablet-only), and 3) tablet with an inpatient portal. The primary study outcome was patient activation (Patient Activation Measure-13). Secondary outcomes included all-cause readmission within 30 days, patient satisfaction, and patient engagement with health information. Results: There was no evidence of a difference in patient activation among patients assigned to the inpatient portal intervention compared to usual care or the tablet-only group. Patients in the inpatient portal group had lower 30-day hospital readmissions (5.5% vs. 12.9% tablet-only and 13.5% usual care; P = 0.044). There was evidence of a difference in patient engagement with health information between the inpatient portal and tablet-only group, including looking up health information online (89.6% vs. 51.8%; P < 0.001). Healthcare providers reported that patients found the portal useful and that the portal did not negatively impact healthcare delivery. Conclusions: Access to an inpatient portal did not significantly improve patient activation, but it was associated with looking up health information online and with a lower 30-day hospital readmission rate. These results illustrate benefit of providing hospitalized patients with real-time access to their electronic health record data while in the hospital. Trial Registration: ClinicalTrials.gov Identifier: NCT01970852.

OBJECTIVE: We sought to assess the quality of race and ethnicity information in observational health databases, including electronic health records (EHRs), and to propose patient self-recording as an improvement strategy. MATERIALS AND METHODS: We assessed completeness of race and ethnicity information in large observational health databases in the United States (Healthcare Cost and Utilization Project and Optum Labs), and at a single healthcare system in New York City serving a racially and ethnically diverse population. We compared race and ethnicity data collected via administrative processes with data recorded directly by respondents via paper surveys (National Health and Nutrition Examination Survey and Hospital Consumer Assessment of Healthcare Providers and Systems). Respondent-recorded data were considered the gold standard for the collection of race and ethnicity information. RESULTS: Among the 160 million patients from the Healthcare Cost and Utilization Project and Optum Labs datasets, race or ethnicity was unknown for 25%. Among the 2.4 million patients in the single New York City healthcare system's EHR, race or ethnicity was unknown for 57%. However, when patients directly recorded their race and ethnicity, 86% provided clinically meaningful information, and 66% of patients reported information that was discrepant with the EHR. DISCUSSION: Race and ethnicity data are critical to support precision medicine initiatives and to determine healthcare disparities; however, the quality of this information in observational databases is concerning. Patient self-recording through the use of patient-facing tools can substantially increase the quality of the information while engaging patients in their health. CONCLUSIONS: Patient self-recording may improve the completeness of race and ethnicity information.

Drug-drug interactions (DDIs) constitute an important concern in drug development and postmarketing pharmacovigilance. They are considered the cause of many adverse drug effects exposing patients to higher risks and increasing public health system costs. Methods to follow-up and discover possible DDIs causing harm to the population are a primary aim of drug safety researchers. Here, we review different methodologies and recent advances using data mining to detect DDIs with impact on patients. We focus on data mining of different pharmacovigilance sources, such as the US Food and Drug Administration Adverse Event Reporting System and electronic health records from medical institutions, as well as on the diverse data mining studies that use narrative text available in the scientific biomedical literature and social media. We pay attention to the strengths but also further explain challenges related to these methods. Data mining has important applications in the analysis of DDIs showing the impact of the interactions as a cause of adverse effects, extracting interactions to create knowledge data sets and gold standards and in the discovery of novel and dangerous DDIs.

We introduce data assimilation as a computational method that uses machine learning to combine data with human knowledge in the form of mechanistic models in order to forecast future states, to impute missing data from the past by smoothing, and to infer measurable and unmeasurable quantities that represent clinically and scientifically important phenotypes. We demonstrate the advantages it affords in the context of type 2 diabetes by showing how data assimilation can be used to forecast future glucose values, to impute previously missing glucose values, and to infer type 2 diabetes phenotypes. At the heart of data assimilation is the mechanistic model, here an endocrine model. Such models can vary in complexity, contain testable hypotheses about important mechanics that govern the system (eg, nutrition's effect on glucose), and, as such, constrain the model space, allowing for accurate estimation using very little data.

OBJECTIVE: Birth month and climate impact lifetime disease risk, while the underlying exposures remain largely elusive. We seek to uncover distal risk factors underlying these relationships by probing the relationship between global exposure variance and disease risk variance by birth season. MATERIAL AND METHODS: This study utilizes electronic health record data from 6 sites representing 10.5 million individuals in 3 countries (United States, South Korea, and Taiwan). We obtained birth month-disease risk curves from each site in a case-control manner. Next, we correlated each birth month-disease risk curve with each exposure. A meta-analysis was then performed of correlations across sites. This allowed us to identify the most significant birth month-exposure relationships supported by all 6 sites while adjusting for multiplicity. We also successfully distinguish relative age effects (a cultural effect) from environmental exposures. RESULTS: Attention deficit hyperactivity disorder was the only identified relative age association. Our methods identified several culprit exposures that correspond well with the literature in the field. These include a link between first-trimester exposure to carbon monoxide and increased risk of depressive disorder (R = 0.725, confidence interval [95% CI], 0.529-0.847), first-trimester exposure to fine air particulates and increased risk of atrial fibrillation (R = 0.564, 95% CI, 0.363-0.715), and decreased exposure to sunlight during the third trimester and increased risk of type 2 diabetes mellitus (R = -0.816, 95% CI, -0.5767, -0.929). CONCLUSION: A global study of birth month-disease relationships reveals distal risk factors involved in causal biological pathways that underlie them.

Electronic health record phenotyping is the use of raw electronic health record data to assert characterizations about patients. Researchers have been doing it since the beginning of biomedical informatics, under different names. Phenotyping will benefit from an increasing focus on fidelity, both in the sense of increasing richness, such as measured levels, degree or severity, timing, probability, or conceptual relationships, and in the sense of reducing bias. Research agendas should shift from merely improving binary assignment to studying and improving richer representations. The field is actively researching new temporal directions and abstract representations, including deep learning. The field would benefit from research in nonlinear dynamics, in combining mechanistic models with empirical data, including data assimilation, and in topology. The health care process produces substantial bias, and studying that bias explicitly rather than treating it as merely another source of noise would facilitate addressing it.

Objective: To study the effect on patient cohorts of mapping condition (diagnosis) codes from source billing vocabularies to a clinical vocabulary. Materials and Methods: Nine International Classification of Diseases, Ninth Revision, Clinical Modification (ICD9-CM) concept sets were extracted from eMERGE network phenotypes, translated to Systematized Nomenclature of Medicine - Clinical Terms concept sets, and applied to patient data that were mapped from source ICD9-CM and ICD10-CM codes to Systematized Nomenclature of Medicine - Clinical Terms codes using Observational Health Data Sciences and Informatics (OHDSI) Observational Medical Outcomes Partnership (OMOP) vocabulary mappings. The original ICD9-CM concept set and a concept set extended to ICD10-CM were used to create patient cohorts that served as gold standards. Results: Four phenotype concept sets were able to be translated to Systematized Nomenclature of Medicine - Clinical Terms without ambiguities and were able to perform perfectly with respect to the gold standards. The other 5 lost performance when 2 or more ICD9-CM or ICD10-CM codes mapped to the same Systematized Nomenclature of Medicine - Clinical Terms code. The patient cohorts had a total error (false positive and false negative) of up to 0.15% compared to querying ICD9-CM source data and up to 0.26% compared to querying ICD9-CM and ICD10-CM data. Knowledge engineering was required to produce that performance; simple automated methods to generate concept sets had errors up to 10% (one outlier at 250%). Discussion: The translation of data from source vocabularies to Systematized Nomenclature of Medicine - Clinical Terms (SNOMED CT) resulted in very small error rates that were an order of magnitude smaller than other error sources. Conclusion: It appears possible to map diagnoses from disparate vocabularies to a single clinical vocabulary and carry out research using a single set of definitions, thus improving efficiency and transportability of research.

Objective: Unintentional medication discrepancies contribute to preventable adverse drug events in patients. Patient engagement in medication safety beyond verbal participation in medication reconciliation is limited. We conducted a pilot study to determine whether patients' use of an electronic home medication review tool could improve medication safety during hospitalization. Materials and Methods: Patients were randomized to use a tool before or after hospital admission medication reconciliation to review and modify their home medication list. We assessed the quantity, potential severity, and potential harm of patients' and clinicians' medication changes. We also surveyed clinicians to assess the tool's usefulness. Results: Of 76 patients approached, 65 (86%) participated. Forty-eight (74%) made changes to their home medication list [before: 29 (81%), after: 19 (66%), p = .170]. Before group participants identified 57 changes that clinicians subsequently missed on admission medication reconciliation. Thirty-nine (74%) had a significant or greater potential severity, and 19 (36%) had a greater than 50-50 chance of harm. After group patients identified 68 additional changes to their reconciled medication lists. Fifty-one (75%) had a significant or greater potential severity, and 33 (49%) had a greater than 50-50 chance of harm. Clinicians reported believing that the tool would save time, and patients would supply useful information. Discussion: The results demonstrate a high willingness of patients to engage in medication reconciliation, and show that patients were able to identify important medication discrepancies and often changes that clinicians missed. Conclusion: Engaging patients in admission medication reconciliation using an electronic home medication review tool may improve medication safety during hospitalization.

OBJECTIVE: To contribute a conceptual framework for evaluating data suitability to satisfy the research needs of observational studies. MATERIALS AND METHODS: Suitability considerations were derived from a systematic literature review on researchers' common data needs in observational studies and a scoping review on frequent clinical database design considerations, and were harmonized to construct a suitability conceptual framework using a bottom-up approach. The relationships among the suitability categories are explored from the perspective of 4 facets of data: intrinsic, contextual, representational, and accessible. A web-based national survey of domain experts was conducted to validate the framework. RESULTS: Data suitability for observational studies hinges on the following key categories: Explicitness of Policy and Data Governance, Relevance, Availability of Descriptive Metadata and Provenance Documentation, Usability, and Quality. We describe 16 measures and 33 sub-measures. The survey uncovered the relevance of all categories, with a 5-point Likert importance score of 3.9 ± 1.0 for Explicitness of Policy and Data Governance, 4.1 ± 1.0 for Relevance, 3.9 ± 0.9 for Availability of Descriptive Metadata and Provenance Documentation, 4.2 ± 1.0 for Usability, and 4.0 ± 0.9 for Quality. CONCLUSIONS: The suitability framework evaluates a clinical data source's fitness for research use. Its construction reflects both researchers' points of view and data custodians' design features. The feedback from domain experts rated Usability, Relevance, and Quality categories as the most important considerations.

We study the question of how to represent or summarize raw laboratory data taken from an electronic health record (EHR) using parametric model selection to reduce or cope with biases induced through clinical care. It has been previously demonstrated that the health care process (Hripcsak and Albers, 2012, 2013), as defined by measurement context (Hripcsak and Albers, 2013; Albers et al., 2012) and measurement patterns (Albers and Hripcsak, 2010, 2012), can influence how EHR data are distributed statistically (Kohane and Weber, 2013; Pivovarov et al., 2014). We construct an algorithm, PopKLD, which is based on information criterion model selection (Burnham and Anderson, 2002; Claeskens and Hjort, 2008), is intended to reduce and cope with health care process biases and to produce an intuitively understandable continuous summary. The PopKLD algorithm can be automated and is designed to be applicable in high-throughput settings; for example, the output of the PopKLD algorithm can be used as input for phenotyping algorithms. Moreover, we develop the PopKLD-CAT algorithm that transforms the continuous PopKLD summary into a categorical summary useful for applications that require categorical data such as topic modeling. We evaluate our methodology in two ways. First, we apply the method to laboratory data collected in two different health care contexts, primary versus intensive care. We show that the PopKLD preserves known physiologic features in the data that are lost when summarizing the data using more common laboratory data summaries such as mean and standard deviation. Second, for three disease-laboratory measurement pairs, we perform a phenotyping task: we use the PopKLD and PopKLD-CAT algorithms to define high and low values of the laboratory variable that are used for defining a disease state. We then compare the relationship between the PopKLD-CAT summary disease predictions and the same predictions using empirically estimated mean and standard deviation to a gold standard generated by clinical review of patient records. We find that the PopKLD laboratory data summary is substantially better at predicting disease state. The PopKLD or PopKLD-CAT algorithms are not meant to be used as phenotyping algorithms, but we use the phenotyping task to show what information can be gained when using a more informative laboratory data summary. In the process of evaluation our method we show that the different clinical contexts and laboratory measurements necessitate different statistical summaries. Similarly, leveraging the principle of maximum entropy we argue that while some laboratory data only have sufficient information to estimate a mean and standard deviation, other laboratory data captured in an EHR contain substantially more information than can be captured in higher-parameter models.

Explosion of the availability of big data sources along with the development in computational methods provides a useful framework to study drugs' actions, such as interactions with pharmacological targets and off-targets. Databases related to protein interactions, adverse effects and genomic profiles are available to be used for the construction of computational models. In this article, we focus on the description of biological profiles for drugs that can be used as a system to compare similarity and create methods to predict and analyze drugs' actions. We highlight profiles constructed with different biological data, such as target-protein interactions, gene expression measurements, adverse effects and disease profiles. We focus on the discovery of new targets or pathways for drugs already in the pharmaceutical market, also called drug repurposing, in the interaction with off-targets responsible for adverse reactions and in drug-drug interaction analysis. The current and future applications, strengths and challenges facing all these methods are also discussed. Biological profiles or signatures are an important source of data generation to deeply analyze biological actions with important implications in drug-related studies.

Type 2 diabetes leads to premature death and reduced quality of life for 8% of Americans. Nutrition management is critical to maintaining glycemic control, yet it is difficult to achieve due to the high individual differences in glycemic response to nutrition. Anticipating glycemic impact of different meals can be challenging not only for individuals with diabetes, but also for expert diabetes educators. Personalized computational models that can accurately forecast an impact of a given meal on an individual's blood glucose levels can serve as the engine for a new generation of decision support tools for individuals with diabetes. However, to be useful in practice, these computational engines need to generate accurate forecasts based on limited datasets consistent with typical self-monitoring practices of individuals with type 2 diabetes. This paper uses three forecasting machines: (i) data assimilation, a technique borrowed from atmospheric physics and engineering that uses Bayesian modeling to infuse data with human knowledge represented in a mechanistic model, to generate real-time, personalized, adaptable glucose forecasts; (ii) model averaging of data assimilation output; and (iii) dynamical Gaussian process model regression. The proposed data assimilation machine, the primary focus of the paper, uses a modified dual unscented Kalman filter to estimate states and parameters, personalizing the mechanistic models. Model selection is used to make a personalized model selection for the individual and their measurement characteristics. The data assimilation forecasts are empirically evaluated against actual postprandial glucose measurements captured by individuals with type 2 diabetes, and against predictions generated by experienced diabetes educators after reviewing a set of historical nutritional records and glucose measurements for the same individual. The evaluation suggests that the data assimilation forecasts compare well with specific glucose measurements and match or exceed in accuracy expert forecasts. We conclude by examining ways to present predictions as forecast-derived range quantities and evaluate the comparative advantages of these ranges.

Accurate prognosis of outcome events, such as clinical procedures or disease diagnosis, is central in medicine. The emergence of longitudinal clinical data, like the Electronic Health Records (EHR), represents an opportunity to develop automated methods for predicting patient outcomes. However, these data are highly dimensional and very sparse, complicating the application of predictive modeling techniques. Further, their temporal nature is not fully exploited by current methods, and temporal abstraction was recently used which results in symbolic time intervals representation. We present Maitreya, a framework for the prediction of outcome events that leverages these symbolic time intervals. Using Maitreya, learn predictive models based on the temporal patterns in the clinical records that are prognostic markers and use these markers to train predictive models for eight clinical procedures. In order to decrease the number of patterns that are used as features, we propose the use of three one class feature selection methods. We evaluate the performance of Maitreya under several parameter settings, including the one-class feature selection, and compare our results to that of atemporal approaches. In general, we found that the use of temporal patterns outperformed the atemporal methods, when representing the number of pattern occurrences.

OBJECTIVE: Maintaining patient privacy is a challenge in large-scale observational research. To assist in reducing the risk of identifying study subjects through publicly available data, we introduce a method for obscuring date information for clinical events and patient characteristics. METHODS: The method, which we call Shift and Truncate (SANT), obscures date information to any desired granularity. Shift and Truncate first assigns each patient a random shift value, such that all dates in that patient's record are shifted by that amount. Data are then truncated from the beginning and end of the data set. RESULTS: The data set can be proven to not disclose temporal information finer than the chosen granularity. Unlike previous strategies such as a simple shift, it remains robust to frequent - even daily - updates and robust to inferring dates at the beginning and end of date-shifted data sets. Time-of-day may be retained or obscured, depending on the goal and anticipated knowledge of the data recipient. CONCLUSIONS: The method can be useful as a scientific approach for reducing re-identification risk under the Privacy Rule of the Health Insurance Portability and Accountability Act and may contribute to qualification for the Safe Harbor implementation.

OBJECTIVE: To investigate subjective experiences and patterns of engagement with a novel electronic tool for facilitating reflection and problem solving for individuals with type 2 diabetes, Mobile Diabetes Detective (MoDD). METHODS: In this qualitative study, researchers conducted semi-structured interviews with individuals from economically disadvantaged communities and ethnic minorities who are participating in a randomized controlled trial of MoDD. The transcripts of the interviews were analyzed using inductive thematic analysis; usage logs were analyzed to determine how actively the study participants used MoDD. RESULTS: Fifteen participants in the MoDD randomized controlled trial were recruited for the qualitative interviews. Usage log analysis showed that, on average, during the 4 weeks of the study, the study participants logged into MoDD twice per week, reported 120 blood glucose readings, and set two behavioral goals. The qualitative interviews suggested that individuals used MoDD to follow the steps of the problem-solving process, from identifying problematic blood glucose patterns, to exploring behavioral triggers contributing to these patterns, to selecting alternative behaviors, to implementing these behaviors while monitoring for improvements in glycemic control. DISCUSSION: This qualitative study suggested that informatics interventions for reflection and problem solving can provide structured scaffolding for facilitating these processes by guiding users through the different steps of the problem-solving process and by providing them with context-sensitive evidence and practice-based knowledge related to diabetes self-management on each of those steps. CONCLUSION: This qualitative study suggested that MoDD was perceived as a useful tool in engaging individuals in self-monitoring, reflection, and problem solving.

The increasing availability of multivariate temporal data in many domains, such as biomedical, security and more, provides exceptional opportunities for temporal knowledge discovery, classification and prediction, but also challenges. Temporal variables are often sparse and in many domains, such as in biomedical data, they have huge number of variables. In recent decades in the biomedical domain events, such as conditions, drugs and procedures, are stored as time intervals, which enables to discover Time Intervals Related Patterns (TIRPs) and use for classification or prediction. In this study we present a framework for outcome events prediction, called Maitreya, which includes an algorithm for TIRPs discovery called KarmaLegoD, designed to handle huge number of symbols. Three indexing strategies for pairs of symbolic time intervals are proposed and compared, showing that the use of FullyHashed indexing is only slightly slower but consumes minimal memory. We evaluated Maitreya on eight real datasets for the prediction of clinical procedures as outcome events. The use of TIRPs outperform the use of symbols, especially with horizontal support (number of instances) as TIRPs feature representation.

OBJECTIVE: An individual's birth month has a significant impact on the diseases they develop during their lifetime. Previous studies reveal relationships between birth month and several diseases including atherothrombosis, asthma, attention deficit hyperactivity disorder, and myopia, leaving most diseases completely unexplored. This retrospective population study systematically explores the relationship between seasonal affects at birth and lifetime disease risk for 1688 conditions. METHODS: We developed a hypothesis-free method that minimizes publication and disease selection biases by systematically investigating disease-birth month patterns across all conditions. Our dataset includes 1 749 400 individuals with records at New York-Presbyterian/Columbia University Medical Center born between 1900 and 2000 inclusive. We modeled associations between birth month and 1688 diseases using logistic regression. Significance was tested using a chi-squared test with multiplicity correction. RESULTS: We found 55 diseases that were significantly dependent on birth month. Of these 19 were previously reported in the literature (P < .001), 20 were for conditions with close relationships to those reported, and 16 were previously unreported. We found distinct incidence patterns across disease categories. CONCLUSIONS: Lifetime disease risk is affected by birth month. Seasonally dependent early developmental mechanisms may play a role in increasing lifetime risk of disease.

BACKGROUND: Fields like nonlinear physics offer methods for analyzing time series, but many methods require that the time series be stationary-no change in properties over time.Objective Medicine is far from stationary, but the challenge may be able to be ameliorated by reparameterizing time because clinicians tend to measure patients more frequently when they are ill and are more likely to vary. METHODS: We compared time parameterizations, measuring variability of rate of change and magnitude of change, and looking for homogeneity of bins of temporal separation between pairs of time points. We studied four common laboratory tests drawn from 25 years of electronic health records on 4 million patients. RESULTS: We found that sequence time-that is, simply counting the number of measurements from some start-produced more stationary time series, better explained the variation in values, and had more homogeneous bins than either traditional clock time or a recently proposed intermediate parameterization. Sequence time produced more accurate predictions in a single Gaussian process model experiment. CONCLUSIONS: Of the three parameterizations, sequence time appeared to produce the most stationary series, possibly because clinicians adjust their sampling to the acuity of the patient. Parameterizing by sequence time may be applicable to association and clustering experiments on electronic health record data. A limitation of this study is that laboratory data were derived from only one institution. Sequence time appears to be an important potential parameterization.

Consistent collection and use of social and behavioral determinants of health can improve clinical care, prevention and general health, patient satisfaction, research, and public health. A recent Institute of Medicine committee defined a panel of 11 domains and 12 measures to be included in electronic health records. Incorporating the panel into practice creates a number of informatics research opportunities as well as challenges. The informatics issues revolve around standardization, efficient collection and review, decision support, and support for research. The informatics community can aid the effort by simultaneously optimizing the collection of the selected measures while also partnering with social science researchers to develop and validate new sources of information about social and behavioral determinants of health.

OBJECTIVE: Large and complex terminologies, such as Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT), are prone to errors and inconsistencies. Abstraction networks are compact summarizations of the content and structure of a terminology. Abstraction networks have been shown to support terminology quality assurance. In this paper, we introduce an abstraction network derivation methodology which can be applied to SNOMED CT target hierarchies whose classes are defined using only hierarchical relationships (ie, without attribute relationships) and similar description-logic-based terminologies. METHODS: We introduce the tribal abstraction network (TAN), based on the notion of a tribe-a subhierarchy rooted at a child of a hierarchy root, assuming only the existence of concepts with multiple parents. The TAN summarizes a hierarchy that does not have attribute relationships using sets of concepts, called tribal units that belong to exactly the same multiple tribes. Tribal units are further divided into refined tribal units which contain closely related concepts. A quality assurance methodology that utilizes TAN summarizations is introduced. RESULTS: A TAN is derived for the Observable entity hierarchy of SNOMED CT, summarizing its content. A TAN-based quality assurance review of the concepts of the hierarchy is performed, and erroneous concepts are shown to appear more frequently in large refined tribal units than in small refined tribal units. Furthermore, more erroneous concepts appear in large refined tribal units of more tribes than of fewer tribes. CONCLUSIONS: In this paper we introduce the TAN for summarizing SNOMED CT target hierarchies. A TAN was derived for the Observable entity hierarchy of SNOMED CT. A quality assurance methodology utilizing the TAN was introduced and demonstrated.

Large amounts of personal health data are being collected and made available through existing and emerging technological media and tools. While use of these data has significant potential to facilitate research, improve quality of care for individuals and populations, and reduce healthcare costs, many policy-related issues must be addressed before their full value can be realized. These include the need for widely agreed-on data stewardship principles and effective approaches to reduce or eliminate data silos and protect patient privacy. AMIA's 2012 Health Policy Meeting brought together healthcare academics, policy makers, and system stakeholders (including representatives of patient groups) to consider these topics and formulate recommendations. A review of a set of Proposed Principles of Health Data Use led to a set of findings and recommendations, including the assertions that the use of health data should be viewed as a public good and that achieving the broad benefits of this use will require understanding and support from patients.

The New York City Clinical Data Research Network (NYC-CDRN), funded by the Patient-Centered Outcomes Research Institute (PCORI), brings together 22 organizations including seven independent health systems to enable patient-centered clinical research, support a national network, and facilitate learning healthcare systems. The NYC-CDRN includes a robust, collaborative governance and organizational infrastructure, which takes advantage of its participants' experience, expertise, and history of collaboration. The technical design will employ an information model to document and manage the collection and transformation of clinical data, local institutional staging areas to transform and validate data, a centralized data processing facility to aggregate and share data, and use of common standards and tools. We strive to ensure that our project is patient-centered; nurtures collaboration among all stakeholders; develops scalable solutions facilitating growth and connections; chooses simple, elegant solutions wherever possible; and explores ways to streamline the administrative and regulatory approval process across sites.

We describe the architecture of the Patient Centered Outcomes Research Institute (PCORI) funded Scalable Collaborative Infrastructure for a Learning Healthcare System (SCILHS, http://www.SCILHS.org) clinical data research network, which leverages the $48 billion dollar federal investment in health information technology (IT) to enable a queryable semantic data model across 10 health systems covering more than 8 million patients, plugging universally into the point of care, generating evidence and discovery, and thereby enabling clinician and patient participation in research during the patient encounter. Central to the success of SCILHS is development of innovative 'apps' to improve PCOR research methods and capacitate point of care functions such as consent, enrollment, randomization, and outreach for patient-reported outcomes. SCILHS adapts and extends an existing national research network formed on an advanced IT infrastructure built with open source, free, modular components.

OBJECTIVE: The study of utilization patterns can quantify potential overuse of laboratory tests and find new ways to reduce healthcare costs. We demonstrate the use of distributional analytics for comparing electronic health record (EHR) laboratory test orders across time to diagnose and quantify overutilization. MATERIALS AND METHODS: We looked at hemoglobin A1c (HbA1c) testing across 119,000 patients and 15 years of hospital records. We examined the patterns of HbA1c ordering before and after the publication of the 2002 American Diabetes Association guidelines for HbA1c testing. We conducted analyses to answer three questions. What are the patterns of HbA1c ordering? Do HbA1c orders follow the guidelines with respect to frequency of measurement? If not, how and why do they depart from the guidelines? RESULTS: The raw number of HbA1c orderings has steadily increased over time, with a specific increase in low-measurement orderings (<6.5%). There is a change in ordering pattern following the 2002 guideline (p<0.001). However, by comparing ordering distributions, we found that the changes do not reflect the guidelines and rather exhibit a new practice of rapid-repeat testing. The rapid-retesting phenomenon does not follow the 2009 guidelines for diabetes diagnosis either, illustrated by a stratified HbA1c value analysis. DISCUSSION: Results suggest HbA1c test overutilization, and contributing factors include lack of care coordination, unexpected values prompting retesting, and point-of-care tests followed by confirmatory laboratory tests. CONCLUSIONS: We present a method of comparing ordering distributions in an EHR across time as a useful diagnostic approach for identifying and assessing the trend of inappropriate use over time.

OBJECTIVE: To systematically review existing literature regarding patient engagement technologies used in the inpatient setting. METHODS: PubMed, Association for Computing Machinery (ACM) Digital Library, Institute of Electrical and Electronics Engineers (IEEE) Xplore, and Cochrane databases were searched for studies that discussed patient engagement ('self-efficacy', 'patient empowerment', 'patient activation', or 'patient engagement'), (2) involved health information technology ('technology', 'games', 'electronic health record', 'electronic medical record', or 'personal health record'), and (3) took place in the inpatient setting ('inpatient' or 'hospital'). Only English language studies were reviewed. RESULTS: 17 articles were identified describing the topic of inpatient patient engagement. A few articles identified design requirements for inpatient engagement technology. The remainder described interventions, which we grouped into five categories: entertainment, generic health information delivery, patient-specific information delivery, advanced communication tools, and personalized decision support. CONCLUSIONS: Examination of the current literature shows there are considerable gaps in knowledge regarding patient engagement in the hospital setting and inconsistent use of terminology regarding patient engagement overall. Research on inpatient engagement technologies has been limited, especially concerning the impact on health outcomes and cost-effectiveness.

BACKGROUND: Hospital readmission risk prediction remains a motivated area of investigation and operations in light of the hospital readmissions reduction program through CMS. Multiple models of risk have been reported with variable discriminatory performances, and it remains unclear how design factors affect performance. OBJECTIVES: To study the effects of varying three factors of model development in the prediction of risk based on health record data: (1) reason for readmission (primary readmission diagnosis); (2) available data and data types (e.g. visit history, laboratory results, etc); (3) cohort selection. METHODS: Regularized regression (LASSO) to generate predictions of readmissions risk using prevalence sampling. Support Vector Machine (SVM) used for comparison in cohort selection testing. Calibration by model refitting to outcome prevalence. RESULTS: Predicting readmission risk across multiple reasons for readmission resulted in ROC areas ranging from 0.92 for readmission for congestive heart failure to 0.71 for syncope and 0.68 for all-cause readmission. Visit history and laboratory tests contributed the most predictive value; contributions varied by readmission diagnosis. Cohort definition affected performance for both parametric and nonparametric algorithms. Compared to all patients, limiting the cohort to patients whose index admission and readmission diagnoses matched resulted in a decrease in average ROC from 0.78 to 0.55 (difference in ROC 0.23, p value 0.01). Calibration plots demonstrate good calibration with low mean squared error. CONCLUSION: Targeting reason for readmission in risk prediction impacted discriminatory performance. In general, laboratory data and visit history data contributed the most to prediction; data source contributions varied by reason for readmission. Cohort selection had a large impact on model performance, and these results demonstrate the difficulty of comparing results across different studies of predictive risk modeling.

Much of what is currently documented in the electronic health record is in response toincreasingly complex and prescriptive medicolegal, reimbursement, and regulatory requirements. These requirements often result in redundant data capture and cumbersome documentation processes. AMIA's 2011 Health Policy Meeting examined key issues in this arena and envisioned changes to help move toward an ideal future state of clinical data capture and documentation. The consensus of the meeting was that, in the move to a technology-enabled healthcare environment, the main purpose of documentation should be to support patient care and improved outcomes for individuals and populations and that documentation for other purposes should be generated as a byproduct of care delivery. This paper summarizes meeting deliberations, and highlights policy recommendations and research priorities. The authors recommend development of a national strategy to review and amend public policies to better support technology-enabled data capture and documentation practices.

The national adoption of electronic health records (EHR) promises to make an unprecedented amount of data available for clinical research, but the data are complex, inaccurate, and frequently missing, and the record reflects complex processes aside from the patient's physiological state. We believe that the path forward requires studying the EHR as an object of interest in itself, and that new models, learning from data, and collaboration will lead to efficient use of the valuable information currently locked in health records.

We demonstrate the importance of explicit definitions of electronic health record (EHR) data completeness and how different conceptualizations of completeness may impact findings from EHR-derived datasets. This study has important repercussions for researchers and clinicians engaged in the secondary use of EHR data. We describe four prototypical definitions of EHR completeness: documentation, breadth, density, and predictive completeness. Each definition dictates a different approach to the measurement of completeness. These measures were applied to representative data from NewYork-Presbyterian Hospital's clinical data warehouse. We found that according to any definition, the number of complete records in our clinical database is far lower than the nominal total. The proportion that meets criteria for completeness is heavily dependent on the definition of completeness used, and the different definitions generate different subsets of records. We conclude that the concept of completeness in EHR is contextual. We urge data consumers to be explicit in how they define a complete record and transparent about the limitations of their data.

We describe a clinical research visit scheduling system that can potentially coordinate clinical research visits with patient care visits and increase efficiency at clinical sites where clinical and research activities occur simultaneously. Participatory Design methods were applied to support requirements engineering and to create this software called Integrated Model for Patient Care and Clinical Trials (IMPACT). Using a multi-user constraint satisfaction and resource optimization algorithm, IMPACT automatically synthesizes temporal availability of various research resources and recommends the optimal dates and times for pending research visits. We conducted scenario-based evaluations with 10 clinical research coordinators (CRCs) from diverse clinical research settings to assess the usefulness, feasibility, and user acceptance of IMPACT. We obtained qualitative feedback using semi-structured interviews with the CRCs. Most CRCs acknowledged the usefulness of IMPACT features. Support for collaboration within research teams and interoperability with electronic health records and clinical trial management systems were highly requested features. Overall, IMPACT received satisfactory user acceptance and proves to be potentially useful for a variety of clinical research settings. Our future work includes comparing the effectiveness of IMPACT with that of existing scheduling solutions on the market and conducting field tests to formally assess user adoption.

Time series are essential to health data research and data mining. We aim to study the properties of one of the more commonly available but historically unreliable types of data: administrative diagnoses in the form of the International Classification of Diseases, Ninth Revision (ICD9) codes. We use differential entropy of ICD9 code time series as a surrogate measure for disease time course and also explore Gaussian kernel smoothing to characterize the time course of diseases in a more fine-grained way. Compared to a gold standard created by a panel of clinicians, the first model classified diseases into acute and chronic groups with a receiver operating characteristic area under curve of 0.83. In the second model, several characteristic temporal profiles were observed including permanent, chronic, and acute. In addition, condition dynamics such as the refractory period for giving birth following childbirth were observed. These models demonstrate that ICD9 codes, despite well-documented concerns, contain valid and potentially valuable temporal information.

The performance of a classification system depends on the context in which it will be used, including the prevalence of the classes and the relative costs of different types of errors. Metrics such as accuracy are limited to the context in which the experiment was originally carried out, and metrics such as sensitivity, specificity, and receiver operating characteristic area--while independent of prevalence--do not provide a clear picture of the performance characteristics of the system over different contexts. Graphing a prevalence-specific metric such as F-measure or the relative cost of errors over a wide range of prevalence allows a visualization of the performance of the system and a comparison of systems in different contexts.

OBJECTIVE: To understand the nature of emerging electronic documentation practices, disconnects between documentation workflows and computing systems designed to support them, and ways to improve the design of electronic documentation systems. MATERIALS AND METHODS: Time-and-motion study of resident physicians' note-writing practices using a commercial electronic health record system that includes an electronic documentation module. The study was conducted in the general medicine unit of a large academic hospital. RESULTS: During the study, 96 note-writing sessions by 11 resident physicians, resulting in close to 100 h of observations were seen. Seven of the 10 most common transitions between activities during note composition were between documenting, and gathering and reviewing patient data, and updating the plan of care. DISCUSSION: The high frequency of transitions seen in the study suggested that clinical documentation is fundamentally a synthesis activity, in which clinicians review available patient data and summarize their impressions and judgments. At the same time, most electronic health record systems are optimized to support documentation as uninterrupted composition. This mismatch leads to fragmentation in clinical work, and results in inefficiencies and workarounds. In contrast, we propose that documentation can be best supported with tools that facilitate data exploration and search for relevant information, selective reading and annotation, and composition of a note as a temporal structure. CONCLUSIONS: Time-and-motion study of clinicians' electronic documentation practices revealed a high level of fragmentation of documentation activities and frequent task transitions. Treating documentation as synthesis rather than composition suggests new possibilities for supporting it more effectively with electronic systems.

Clinical research is the foundation for advancing the practice of medicine. However, the lack of seamless integration between clinical research and patient care workflow impedes recruitment efficiency, escalates research costs, and hence threatens the entire clinical research enterprise. Increased use of electronic health records (EHRs) holds promise for facilitating this integration but must surmount regulatory obstacles. Among the unintended consequences of current research oversight are barriers to accessing patient information for prescreening and recruitment, coordinating scheduling of clinical and research visits, and reconciling information about clinical and research drugs. We conclude that the EHR alone cannot overcome barriers in conducting clinical trials and comparative effectiveness research. Patient privacy and human subject protection policies should be clarified at the local level to exploit optimally the full potential of EHRs, while continuing to ensure participant safety. Increased alignment of policies that regulate the clinical and research use of EHRs could help fulfill the vision of more efficiently obtaining clinical research evidence to improve human health.

This paper analyzes a Question & Answer site for programmers, Stack Overflow, that dramatically improves on the utility and performance of Q&A systems for technical domains. Over 92% of Stack Overflow questions about expert topics are answered - in a median time of 11 minutes. Using a mixed methods approach that combines statistical data analysis with user interviews, we seek to understand this success. We argue that it is not primarily due to an a priori superior technical design, but also to the high visibility and daily involvement of the design team within the community they serve. This model of continued community leadership presents challenges to both CSCW systems research as well as to attempts to apply the Stack Overflow model to other specialized knowledge domains.

OBJECTIVE: To demonstrate that a large, heterogeneous clinical database can reveal fine temporal patterns in clinical associations; to illustrate several types of associations; and to ascertain the value of exploiting time. MATERIALS AND METHODS: Lagged linear correlation was calculated between seven clinical laboratory values and 30 clinical concepts extracted from resident signout notes from a 22-year, 3-million-patient database of electronic health records. Time points were interpolated, and patients were normalized to reduce inter-patient effects. RESULTS: The method revealed several types of associations with detailed temporal patterns. Definitional associations included low blood potassium preceding 'hypokalemia.' Low potassium preceding the drug spironolactone with high potassium following spironolactone exemplified intentional and physiologic associations, respectively. Counterintuitive results such as the fact that diseases appeared to follow their effects may be due to the workflow of healthcare, in which clinical findings precede the clinician's diagnosis of a disease even though the disease actually preceded the findings. Fully exploiting time by interpolating time points produced less noisy results. DISCUSSION: Electronic health records are not direct reflections of the patient state, but rather reflections of the healthcare process and the recording process. With proper techniques and understanding, and with proper incorporation of time, interpretable associations can be derived from a large clinical database. CONCLUSION: A large, heterogeneous clinical database can reveal clinical associations, time is an important feature, and care must be taken to interpret the results.

OBJECTIVE: To measure the time spent authoring and viewing documentation and to study patterns of usage in healthcare practice. DESIGN: Audit logs for an electronic health record were used to calculate rates, and social network analysis was applied to ascertain usage patterns. Subjects comprised all care providers at an urban academic medical center who authored or viewed electronic documentation. MEASUREMENT: Rate and time of authoring and viewing clinical documentation, and associations among users were measured. RESULTS: Users spent 20-103 min per day authoring notes and 7-56 min per day viewing notes, with physicians spending less than 90 min per day total. About 16% of attendings' notes, 8% of residents' notes, and 38% of nurses' notes went unread by other users, and, overall, 16% of notes were never read by anyone. Viewing of notes dropped quickly with the age of the note, but notes were read at a low but measurable rate, even after 2 years. Most healthcare teams (77%) included a nurse, an attending, and a resident, and those three users' groups were the first to write notes during an admission. Limitations The limitations were restriction to a single academic medical center and use of log files without direct observation. CONCLUSIONS: Care providers spend a significant amount of time viewing and authoring notes. Many notes are never read, and rates of usage vary significantly by author and viewer. While the rate of viewing a note drops quickly with its age, even after 2 years inpatient notes are still viewed.

We measured the prevalence (or rate) of patient-note mismatches (clinical notes judged to pertain to another patient) in the electronic medical record. The rate ranged from 0.5% (95% CI 0.2% to 1.7%) before a pop-up window intervention to 0.3% (95% CI 0.1% to 1.1%) after the intervention. Clinicians discovered patient-note mismatches in 0.05-0.03% of notes, or about 10% of actual mismatches. The reduction in rates after the intervention was statistically significant. Therefore, while the patient-note mismatch rate is low compared to published rates of other documentation errors, it can be further reduced by the design of the user interface.

BACKGROUND: The availability of up-to-date, executable, evidence-based medical knowledge is essential for many clinical applications, such as pharmacovigilance, but executable knowledge is costly to obtain and update. Automated acquisition of environmental and phenotypic associations in biomedical and clinical documents using text mining has showed some success. The usefulness of the association knowledge is limited, however, due to the fact that the specific relationships between clinical entities remain unknown. In particular, some associations are indirect relations due to interdependencies among the data. RESULTS: In this work, we develop methods using mutual information (MI) and its property, the data processing inequality (DPI), to help characterize associations that were generated based on use of natural language processing to encode clinical information in narrative patient records followed by statistical methods. Evaluation based on a random sample consisting of two drugs and two diseases indicates an overall precision of 81%. CONCLUSION: This preliminary study demonstrates that the proposed method is effective for helping to characterize phenotypic and environmental associations obtained from clinical reports.

OBJECTIVE: To measure the uncertainty of temporal assertions like "3 weeks ago" in clinical texts. DESIGN: Temporal assertions extracted from narrative clinical reports were compared to facts extracted from a structured clinical database for the same patients. MEASUREMENTS: The authors correlated the assertions and the facts to determine the dependence of the uncertainty of the assertions on the semantic and lexical properties of the assertions. RESULTS: The observed deviation between the stated duration and actual duration averaged about 20% of the stated deviation. Linear regression revealed that assertions about events further in the past tend to be more uncertain, smaller numeric values tend to be more uncertain (1 mo v. 30 d), and round numbers tend to be more uncertain (10 versus 11 yrs). CONCLUSIONS: The authors empirically derived semantics behind statements of duration using "ago," and verified intuitions about how numbers are used.

OBJECTIVE: To assess the performance of electronic health record data for syndromic surveillance and to assess the feasibility of broadly distributed surveillance. DESIGN: Two systems were developed to identify influenza-like illness and gastrointestinal infectious disease in ambulatory electronic health record data from a network of community health centers. The first system used queries on structured data and was designed for this specific electronic health record. The second used natural language processing of narrative data, but its queries were developed independently from this health record. Both were compared to influenza isolates and to a verified emergency department chief complaint surveillance system. MEASUREMENTS: Lagged cross-correlation and graphs of the three time series. RESULTS: For influenza-like illness, both the structured and narrative data correlated well with the influenza isolates and with the emergency department data, achieving cross-correlations of 0.89 (structured) and 0.84 (narrative) for isolates and 0.93 and 0.89 for emergency department data, and having similar peaks during influenza season. For gastrointestinal infectious disease, the structured data correlated fairly well with the emergency department data (0.81) with a similar peak, but the narrative data correlated less well (0.47). CONCLUSIONS: It is feasible to use electronic health records for syndromic surveillance. The structured data performed best but required knowledge engineering to match the health record data to the queries. The narrative data illustrated the potential performance of a broadly disseminated system and achieved mixed results.

Electronic clinical documentation can be useful for activities such as public health surveillance, quality improvement, and research, but existing methods of de-identification may not provide sufficient protection of patient data. The general-purpose natural language processor MedLEE retains medical concepts while excluding the remaining text so, in addition to processing text into structured data, it may be able provide a secondary benefit of de-identification. Without modifying the system, the authors tested the ability of MedLEE to remove protected health information (PHI) by comparing 100 outpatient clinical notes with the corresponding XML-tagged output. Of 809 instances of PHI, 26 (3.2%) were detected in output as a result of processing and identification errors. However, PHI in the output was highly transformed, much appearing as normalized terms for medical concepts, potentially making re-identification more difficult. The MedLEE processor may be a good enhancement to other de-identification systems, both removing PHI and providing coded data from clinical text.

Advances in high-throughput and mass-storage technologies have led to an information explosion in both biology and medicine, presenting novel challenges for analysis and modeling. With regards to multivariate analysis techniques such as clustering, classification, and regression, large datasets present unique and often misunderstood challenges. The authors' goal is to provide a discussion of the salient problems encountered in the analysis of large datasets as they relate to modeling and inference to inform a principled and generalizable analysis and highlight the interdisciplinary nature of these challenges. The authors present a detailed study of germane issues including high dimensionality, multiple testing, scientific significance, dependence, information measurement, and information management with a focus on appropriate methodologies available to address these concerns. A firm understanding of the challenges and statistical technology involved ultimately contributes to better science. The authors further suggest that the community consider facilitating discussion through interdisciplinary panels, invited papers and curriculum enhancement to establish guidelines for analysis and reporting.

OBJECTIVE It is vital to detect the full safety profile of a drug throughout its market life. Current pharmacovigilance systems still have substantial limitations, however. The objective of our work is to demonstrate the feasibility of using natural language processing (NLP), the comprehensive Electronic Health Record (EHR), and association statistics for pharmacovigilance purposes. DESIGN Narrative discharge summaries were collected from the Clinical Information System at New York Presbyterian Hospital (NYPH). MedLEE, an NLP system, was applied to the collection to identify medication events and entities which could be potential adverse drug events (ADEs). Co-occurrence statistics with adjusted volume tests were used to detect associations between the two types of entities, to calculate the strengths of the associations, and to determine their cutoff thresholds. Seven drugs/drug classes (ibuprofen, morphine, warfarin, bupropion, paroxetine, rosiglitazone, ACE inhibitors) with known ADEs were selected to evaluate the system. RESULTS One hundred thirty-two potential ADEs were found to be associated with the 7 drugs. Overall recall and precision were 0.75 and 0.31 for known ADEs respectively. Importantly, qualitative evaluation using historic roll back design suggested that novel ADEs could be detected using our system. CONCLUSIONS This study provides a framework for the development of active, high-throughput and prospective systems which could potentially unveil drug safety profiles throughout their entire market life. Our results demonstrate that the framework is feasible although there are some challenging issues. To the best of our knowledge, this is the first study using comprehensive unstructured data from the EHR for pharmacovigilance.

OBJECTIVE: Explore the automated acquisition of knowledge in biomedical and clinical documents using text mining and statistical techniques to identify disease-drug associations. DESIGN: Biomedical literature and clinical narratives from the patient record were mined to gather knowledge about disease-drug associations. Two NLP systems, BioMedLEE and MedLEE, were applied to Medline articles and discharge summaries, respectively. Disease and drug entities were identified using the NLP systems in addition to MeSH annotations for the Medline articles. Focusing on eight diseases, co-occurrence statistics were applied to compute and evaluate the strength of association between each disease and relevant drugs. RESULTS: Ranked lists of disease-drug pairs were generated and cutoffs calculated for identifying stronger associations among these pairs for further analysis. Differences and similarities between the text sources (i.e., biomedical literature and patient record) and annotations (i.e., MeSH and NLP-extracted UMLS concepts) with regards to disease-drug knowledge were observed. CONCLUSION: This paper presents a method for acquiring disease-specific knowledge and a feasibility study of the method. The method is based on applying a combination of NLP and statistical techniques to both biomedical and clinical documents. The approach enabled extraction of knowledge about the drugs clinicians are using for patients with specific diseases based on the patient record, while it is also acquired knowledge of drugs frequently involved in controlled trials for those same diseases. In comparing the disease-drug associations, we found the results to be appropriate: the two text sources contained consistent as well as complementary knowledge, and manual review of the top five disease-drug associations by a medical expert supported their correctness across the diseases.

In order to assess the mission and strategic direction in an academic department of biomedical informatics, we used social network analysis to identify patterns of common interest among the department's multidisciplinary faculty. Data representing faculty and their self-identified research methods and expertise were analyzed by applying a network modularity algorithm to detect community structure. Three distinct communities of practice emerged: empirical discovery and prediction; human and organizational factors; and information management. This analysis made intuitive sense and served the goal of stimulating discussion from new perspectives. The findings will guide future direction and faculty recruitment efforts. Communities of practice present a novel view of interdisciplinarity in biomedical informatics.

The advent of electronic medical records and health information exchange raise the possibility of expanding public health reporting to detect a broad range of clinical conditions and of monitoring the health of the public on a broad scale. Expanding public health reporting may require patient anonymity, matching records, re-identifying cases, and recording patient characteristics for localization. The privacy regulations under the Health Insurance Portability and Accountability Act of 1996 (HIPAA) provide several mechanisms for public health surveillance, including using laws and regulations, public health activities, de-identification, research waivers, and limited data sets, and in addition, surveillance may be distributed with aggregate reporting. The appropriateness of these approaches varies with the definition of what data may be included, the requirements of the minimum necessary standard, the accounting of disclosures, and the feasibility of the approach.

CONTEXT: TimeText is a temporal reasoning system designed to represent, extract, and reason about temporal information in clinical text. OBJECTIVE: To measure the accuracy of the TimeText for processing clinical discharge summaries. DESIGN: Six physicians with biomedical informatics training served as domain experts. Twenty discharge summaries were randomly selected for the evaluation. For each of the first 14 reports, 5 to 8 clinically important medical events were chosen. The temporal reasoning system generated temporal relations about the endpoints (start or finish) of pairs of medical events. Two experts (subjects) manually generated temporal relations for these medical events. The system and expert-generated results were assessed by four other experts (raters). All of the twenty discharge summaries were used to assess the system's accuracy in answering time-oriented clinical questions. For each report, five to ten clinically plausible temporal questions about events were generated. Two experts generated answers to the questions to serve as the gold standard. We wrote queries to retrieve answers from system's output. MEASUREMENTS: Correctness of generated temporal relations, recall of clinically important relations, and accuracy in answering temporal questions. RESULTS: The raters determined that 97% of subjects' 295 generated temporal relations were correct and that 96.5% of the system's 995 generated temporal relations were correct. The system captured 79% of 307 temporal relations determined to be clinically important by the subjects and raters. The system answered 84% of the temporal questions correctly. CONCLUSION: The system encoded the majority of information identified by experts, and was able to answer simple temporal questions.

MOTIVATION: The ambiguity of biomedical entities, particularly of gene symbols, is a big challenge for text-mining systems in the biomedical domain. Existing knowledge sources, such as Entrez Gene and the MEDLINE database, contain information concerning the characteristics of a particular gene that could be used to disambiguate gene symbols. RESULTS: For each gene, we create a profile with different types of information automatically extracted from related MEDLINE abstracts and readily available annotated knowledge sources. We apply the gene profiles to the disambiguation task via an information retrieval method, which ranks the similarity scores between the context where the ambiguous gene is mentioned, and candidate gene profiles. The gene profile with the highest similarity score is then chosen as the correct sense. We evaluated the method on three automatically generated testing sets of mouse, fly and yeast organisms, respectively. The method achieved the highest precision of 93.9% for the mouse, 77.8% for the fly and 89.5% for the yeast. AVAILABILITY: The testing data sets and disambiguation programs are available at http://www.dbmi.columbia.edu/~hux7002/gsd2006

Our goal is to assess how clinical information from previous visits is used in the emergency department. We used detailed user audit logs to measure access to different data types. We found that clinician-authored notes and laboratory and radiology data were used most often (common data types were used up to 5% to 20% of the time). Data were accessed less than half the time (up to 20% to 50%) even when the user was alerted to the presence of data. Our access rate indicates that health information exchange projects should be conservative in estimating how often shared data will be used and the wide breadth of data accessed indicates that although a clinical summary is likely to be useful, an ideal solution will supply a broad variety of data.

Information retrieval studies that involve searching the Internet or marking phrases usually lack a well-defined number of negative cases. This prevents the use of traditional interrater reliability metrics like the kappa statistic to assess the quality of expert-generated gold standards. Such studies often quantify system performance as precision, recall, and F-measure, or as agreement. It can be shown that the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts and that kappa approaches these measures as the number of negative cases grows large. Positive specific agreement-or the equivalent F-measure-may be an appropriate way to quantify interrater reliability and therefore to assess the reliability of a gold standard in these studies.

OBJECTIVE: To model the temporal information contained in medical narrative reports as a simple temporal constraint satisfaction problem. DESIGN: A constraint satisfaction problem is defined by time points and constraints (inequalities between points). A time interval comprises a pair of points and a constraint. Five complete electronic discharge summaries and paragraphs from 226 other discharge summaries were studied. Medical events were represented as intervals, and assertions about events were represented as constraints. Through a consensus process, a set of encoding procedures and a list of issues related to encoding were generated. MEASUREMENTS: Instances of temporal disjunction and contradiction and distribution of temporal constraints were used. RESULTS: An average of 95 medical events (range, 46-151) and 234 temporal assertions (range, 118-388) were identified per complete discharge summary. Nondefinitional assertions were explicit (36%) or implicit (64%) and absolute (17%), qualitative (72%), or metric (11%). Implicit assertions were based on domain knowledge and assumptions, e.g., the section of the report determined the ordering of events. Issues included linking events, intermittence, periodicity, granularity, vagueness, ambiguity, uncertainty, and plans. ions such as intermittence were not represented explicitly. The temporal network was sparse: Only 0.80% (range, 0.42%-1.38%) of possible constraints were instantiated. No instances of discontinuous temporal disjunction were found in the complete summaries or the 226 paragraphs. One instance of temporal contradiction was found (intrareport rate of 0.2 with a 95% confidence interval of 0.005-1.114). CONCLUSION: A simple temporal constraint satisfaction problem appears sufficient to represent most temporal assertions in discharge summaries and may be useful for encoding electronic medical records.

OBJECTIVE: To determine whether natural language processing (NLP) can effectively detect adverse events defined in the New York Patient Occurrence Reporting and Tracking System (NYPORTS) using discharge summaries. DESIGN: An adverse event detection system for discharge summaries using the NLP system MedLEE was constructed to identify 45 NYPORTS event types. The system was first applied to a random sample of 1,000 manually reviewed charts. The system then processed all inpatient cases with electronic discharge summaries for two years. All system-identified events were reviewed, and performance was compared with traditional reporting. MEASUREMENTS: System sensitivity, specificity, and predictive value, with manual review serving as the gold standard. RESULTS: The system correctly identified 16 of 65 events in 1,000 charts. Of 57,452 total electronic discharge summaries, the system identified 1,590 events in 1,461 cases, and manual review verified 704 events in 652 cases, resulting in an overall sensitivity of 0.28 (95% confidence interval [CI]: 0.17-0.42), specificity of 0.985 (CI: 0.984-0.986), and positive predictive value of 0.45 (CI: 0.42-0.47) for detecting cases with events and an average specificity of 0.9996 (CI: 0.9996-0.9997) per event type. Traditional event reporting detected 322 events during the period (sensitivity 0.09), of which the system identified 110 as well as 594 additional events missed by traditional methods. CONCLUSION: NLP is an effective technique for detecting a broad range of adverse events in text documents and outperformed traditional and previous automated adverse event detection methods.

Electronic prescribing (e-prescribing) systems can provide computer-based support for the creation, transmission, dispensing, and monitoring of pharmacological therapies. In the United States and other countries, such systems have been documented, under certain conditions, to increase the safety and quality of patient care.1–5 The authors applaud the initial efforts of Teich and colleagues in the Joint Clinical Decision Support Workgroup (Joint CDS WG) to outline e-prescribing desiderata, as reported in this issue of JAMIA by Teich et al.6 Their article is published as an endorsed policy of the American Medical Informatics Association (AMIA). Previously, Bell et al. published an excellent list of desiderata for outpatient e-prescribing and sorted the desiderata into functional categories.7 Subsequently, Wang et al. surveyed e-prescribing vendor systems to determine that existing systems on average met only half the desiderata, with none exceeding 64% fulfillment.8 The recommendations outlined in the tables of the Joint CDS WG provide a useful point of departure for future discussions. Of note, the Joint CDS WG guidelines were developed as a “commissioned work” with externally determined foci, time limitations, and priorities, so that those guidelines do not fully cover all relevant areas. The Joint CDS WG document therefore represents an important first step in an evolving approach to a complex set of problems. The Joint CDS WG recommendations present a scenario of how e-prescribing features might be rolled out. The authors of this commentary would like to supplement, from what we believe is a broader perspective, the focused set of Joint CDS WG recommendations. The Joint CDS WG proposal has several strengths, including the recommendations that the United States should develop and promote shareable standards for e-prescribing and related decision support systems, a consensus should be developed on how to implement and evaluate decision support systems, and certain organizations, (such as the Office of the National Coordinator for Health Information Technology, the Agency for Healthcare Research and Quality, the U.S. Food and Drug Administration (FDA), the National Library of Medicine, AMIA, the e-Health Initiative, and the Health Information and Management Systems Society) should take leadership roles in the e-prescribing efforts. The authors note that developers and implementers should consider the variability that currently exists among users, clinical settings, information systems, and environments when determining how and when to install and support an e-prescribing system. For example, even when clinical systems provide net benefits to an institution, the implementation of electronic systems to improve the quality of care can introduce unwanted, potentially harmful side effects that must be detected, monitored, and addressed.9,10 It is therefore important to consider the potential adverse effects of e-prescribing implementation. Electronic prescribing systems represent only one genre of electronic health record system activity (others include departmental pharmacy, radiology, and laboratory systems and systems for record keeping, ordering, results display, monitoring, and decision support). Because the current state of the art for complex, comprehensive electronic health record systems is immature,11 there is not yet a scientific basis for selecting among the many potential courses of action related to implementation and use of e-prescribing systems. It is the authors' opinion that human (end-user) factors and electronic information interchanges among e-prescribing and other clinical systems play critically important roles in determining the success or failure of e-prescribing systems. These considerations should be combined with the Joint CDS WG suggestions when making implementation decisions. Several articles in the current issue of JAMIA illustrate how intricate and difficult it is to implement and evaluate such systems. There are few operational systems in place that have documented the success or failure of e-prescribing guidelines outlined. The authors note that e-prescribing systems alone may not suffice; more comprehensive electronic health systems may be required to address the needs of both healthcare facilities and individual practitioners. Clinicians should be wary of developing a false sense of security and unrealistic expectations based on use of e-prescribing applications alone, when more complex systems may be required. To reap the benefits of several decades of dedicated work by biomedical informaticians, commercial vendors, and health care providers (institutions and individuals), a responsible approach to e-prescribing must be advocated. All parties with a stake in e-prescribing must develop a common, overarching framework for its development and dissemination. The remainder of this commentary examines the environmental factors, technical factors, and strategic factors relevant to e-prescribing before concluding with a recommended framework that builds on and supplements the Joint CDS WG e-prescribing recommendations. Environmental factors relevant to e-prescribing include individual practitioners' specialties and roles; the variety of practice settings in which care is delivered in the United States; the standard of care in the clinical community; and end-user constraints imposed by human limitations in knowledge, habits, and work flows. Pragmatically, clinicians' individual practice circumstances should determine their access to, choice of, and use of e-prescribing technology. Currently in the United States, the status of e-prescribing systems varies by geographic regions and by federal, state, or local governmental jurisdictions; practice setting/type; and commercial vendor application. As a result, there are widely varying e-prescribing adoption rates. It might be argued that in today's health care environment, a rural general practitioner who manages inpatients in the morning, outpatients in a small office in the afternoon, and who makes house calls as needed cannot and should not use the same e-prescribing system in each setting, even though in the future one system should suffice. Today's systems with one underlying knowledge base cannot easily switch between inpatient and outpatient formularies (which differ significantly). The rich information environment of hospital-based electronic medical record and computerized provider order entry (CPOE) systems is not easily replicated in outpatients' homes, even with remote wireless connectivity. Large academic medical centers have the resources (adequate teams of talented informaticians, available expert clinicians, and an annual budget of millions of dollars to support their work) to develop or purchase, customize, roll out, and evolve state-of-the-art e-prescribing systems. By contrast, both solo providers and small rural hospitals have limited access to informatics expertise, and little time or money to develop or install complex systems.12,13 In the inpatient setting, when a clinician orders a medication, there is typically only one inpatient pharmacy system through which the order will be processed, and it is the same system for all providers and all orders. In the outpatient setting, patients typically receive multiple prescriptions from multiple care providers and may fill them at different pharmacies. Each retail pharmacy store (or pharmacy chain) may have its own software system that provides various levels of alerts regarding doses and drug interactions to pharmacists as they fill prescriptions, but the same prescription taken to different pharmacies will generate different alerts. Electronic connectivity is rare between free-standing outpatient pharmacies and the hospital or clinic-based, patient information–rich practice settings where providers generate prescriptions. Most connectivity that exists takes the form of fax machines, which, it is hoped, produce legible prescriptions but which still do not preclude transcription (and other) errors. For example, it would not be uncommon for a physician to write, “warfarin 5 mg one tablet by mouth daily” and the pharmacist to dispense (due to a temporary shortage) 2.5 mg tablets, with the instruction “take two tablets daily.” When in the following week, the physician receives a subtherapeutic international normalized ratio (INR) blood test result for the patient (indicating that the dose should be increased), a phone call to the patient's home to “take one and one-half pills daily” may have disastrous consequences. The patient would take a decreased dose (3.75 mg) instead of the physician's intended dose (7.5 mg) due to failed communication among systems, providers, and the patient. “Closed loop” e-prescribing feedback that matches clinicians' orders with pharmacy dispensing annotations requires a bidirectional interface from prescribing site to the dispensing pharmacy. What is required is that the systems “match up” the dispensing record with the prescribing order in a manner in which any clinician reviewing the patient's chart could easily determine what was ordered and how it was dispensed, so as to avoid the previous scenario. The few bidirectional interfaces now in existence often generate a request for the attending physician's countersignature whenever the pharmacist changes the dispensing record. The latter model is unworkable in terms of introducing an unnecessary and often confusing extra burden on the physician, who may not understand which medication order is being changed for what dispensing reason if multiple changes are made. Clinicians who are told to use an e-prescribing system cannot be expected to do so if using the system compromises the clinicians' existing standard of care for patients. Weight-based dosing represents the standard of care for many medications and ages in the pediatric population. A 1998 article in Pediatrics titled “Prevention of Medication Errors in the Pediatric Inpatient Setting” listed “confirm that the patient's weight is correct for weight-based dosages” as its first recommendation under “medication ordering to reduce errors.”14 In a related article in 2004, Kaushal et al. cited error-prone behaviors in clinicians' manual calculations of weight-based pediatric medication dosages.15 The authors believe that weight-based dose calculations for pediatric patients is an established standard of practice in the community that should be adopted within e-prescribing systems immediately (i.e., by 2006), even though many current vendor pharmacy and CPOE products poorly support pediatric dosing (and especially dosing in premature neonates). A number of academic centers have demonstrated that pediatric dosing can be done as part of electronic prescribing. Those centers have developed, deployed, and evaluated reliable pediatric and neonatal dosing systems over the In the current issue of et present a of a that potentially in part by use of an inpatient e-prescribing system. the of complex interactions among system and environmental factors that can to whenever and feedback Systems that clinician by not all relevant information for decision making into one place the of the clinician must still the record (or a clinical results as as with an e-prescribing system the result can be more work and for the clinician as as more to by important a or so with can a of errors. The in this issue by and provides a of how systems that might be can effects when in the clinical environment due to a of failure of to the system clinical of the environment, and of various among different electronic systems, among clinicians, or between and systems can to errors. must be to the environment as as to the systems being In the current issue of the by et al. the factors that clinician with and that environmental factors can play roles in or to use of between and using the not with the implementation of recommended of clinical recommendation and interface the number of at a strategic of the of into and the to document system and receive In previous of et al. the human factors required for implementation of clinical and and argued that the important implementation are factors relevant to e-prescribing systems include the information of a the for system the development for the system the to which clinician in feedback at the implementation and use the electronic interface of the system to other electronic the interface of the the of a combined who and system the of local to have into system and the to changes within to not or and and for quality both and developers of such systems. an base exists in the under e-prescribing systems can improve patient safety and quality of the under which such cannot be easily Most the of CPOE and electronic medical record systems have from academic medical centers that the resources and both informatics and to and clinical decision support the information knowledge underlying commercial CPOE and pharmacy system vendor products are typically by commercial (such as and and to CPOE The quality and of pharmacy information have been and their has at been into The efforts reported by et are of this the success reported by that such when can be The and safety of e-prescribing systems on their underlying pharmacy information the of the and the circumstances of software and system implementation. In the systems available to the of hospitals and in the United States were developed by commercial vendors, systems typically the state of the art in the academic centers by several For example, that vendor systems are in the of pediatric prescribing. The authors are of a number of of hospitals pharmacy systems and or the pediatric prescribing due to of by the systems. of one commercial drug information system in an e-prescribing (such as dose of in and for and that use of the system local was The of and the of with many commercial vendor systems (and are not to the Most commercial pharmacy information systems generate a (and number of drug et al. a for all based on a failure of the to how often interactions in patient and the of those when they As et al. note, often at the physician to in the et al. the of resources that must be dedicated to and the commercial drug information that must be at medical that such systems. The of at each local medical the are both and at local is required at system desiderata, including use of were in the of Bell et the Joint CDS WG et and in the for system developers outlined by and In to desiderata, it is to consider and when e-prescribing systems. For example, information should be from only when the information will be for important decisions. the should be only from the individual to the correct a clinician in patient or laboratory results so that the information can be as or on a prescription may be the e-prescribing system should decision support that laboratory results or if to provide dosing recommendations and The to e-prescribing system reviewing alerts and taken is for quality in As by and in this issue of such as may the of for use by decision support systems and may be to of e-prescribing of factors relevant to e-prescribing include of or how to e-prescribing systems through published related to the of e-prescribing system (such as governmental and Joint on of Healthcare related to and and such as development of standards monitoring of an e-prescribing system over time on the quality and of the knowledge base underlying the e-prescribing system at any point in the quality and of the software system the knowledge base to a patient's clinical for prescribing the and for of both the knowledge base and the The Joint CDS WG that drug information should be if a number of such are of one and if their knowledge and terms of quality of and over they might not to the quality of drug information systems. The of for e-prescribing systems is an but we cannot how each of the to by the Joint CDS WG will to it is important to of the underlying drug information knowledge base from of the related software cannot for quality drug and an quality knowledge base may be by can of one so that a system as may from over time if the knowledge base is not or if software include The authors believe that it is to if will work or if the for Healthcare Information is the for such can be demonstrated to be and The authors believe that a framework should be developed and to promote adoption of e-prescribing systems in outline the that such a framework might both now and in the system must improve over time in a and are limited to e-prescribing systems and their and do not address more comprehensive medication safety such as of both patients and medications and of medication orders and doses at the point of care to A approach to what has been and will be through e-prescribing is to making Electronic prescribing cannot an activity is developed for its in settings academic cannot be in alone should not patients e-prescribing system that is the patient information from an system can produce clinical should in The governmental and should the development of standards for transmission, and monitoring of information and play a in the development and of quality drug information knowledge The United States has taken a through in developing standards for the prescribing and dispensing of The and related and Drug products represent excellent A step would be to develop consensus standards for e-prescribing information For example, a is needed for and drug including the and of standards for the and of interactions within drug information should be each vendor its own systems for such information and only this to an of e-prescribing as reported by et in this but it potentially in which, for example, a CPOE system cannot to pharmacy system (or what is to be potentially for a patient the CPOE system might as to but the pharmacy system might not be to the (or A standard for drug information (or for adverse effects of individual should have an it might for example, the following of the that (or of the drug with an adverse a but standard set of for the clinical of the interactions A the of drug A and drug or A of drug an a of the of the base for the based on in or to based on knowledge of but reported in reported as or reported as a of done replicated by multiple a for the of but or potential for or or potentially or and to in a on a of how often each has been reported to for example, such a might 5 more of to of one in to in to such an standard set of for drug adverse effects and drug e-prescribing systems could more determine how to the of alerts to the for and potentially more alerts in a more manner alerts. There are a number of academic and commercial developers who now have from developing e-prescribing systems. such as up” information (such as current patient or with a or when the is to a if the system laboratory results medication ordering (such as and levels when an is it is important to the and time that the laboratory results were a physician might and as being current when ordering today's when in the laboratory results are (and might if the clinician In the current issue of et al. that the same delivered through different interface can to different by their interface to a for drug information The should develop a that on the of et and the general recommendations of et to useful information for future developers and for to install e-prescribing systems. and academic centers in should development of e-prescribing standards as for both information and consider their drug information knowledge to be and and as a result, they are not to an of by In the current environment, e-prescribing systems may not even have access to the drug information knowledge underlying the systems, making quality a In an both their own systems with such and the same systems with the same would produce the same or can and of for drug information knowledge for e-prescribing system software might drug information knowledge to a should that the clinical standard of care in the such as weight-based dosing for is in e-prescribing systems. When it is should be and system under such circumstances might be if weight-based dosing is not the system could not generate prescriptions for should be developed that provide to which vendor and systems have been evaluated at what in time the to the results or academic medical centers should develop based on of vendor (or e-prescribing systems that are available for which of systems or system features (such as the desiderata of the Joint CDS WG) are recommended for in various A approach might be might the current of practice environment on a that with of the would the of the e-prescribing a solo practitioner in an a small a a practice or with a to medical a small a a hospital and a academic medical would various of systems (or system as or and the resources in and information required to use recommendations for e-prescribing systems would with each so that a solo practitioner with e-prescribing system now might be to one of a number of computer-based e-prescribing systems and to only use it for prescribing when needed as to not practice but to provide information and in any By contrast, for a academic medical with an electronic medical record the might of a e-prescribing approach and relevant to use in selecting such a system. organizations, and academic medical centers should promote and for adverse with a system for is an important first the authors that only the has the and resources to the of to a common, of drug information to support e-prescribing over efforts to as as the National Library of to and the Medical can as for how a drug information knowledge base could be The has a among governmental academic medical and commercial to develop and a useful the such an the drug information of medications might include recommended doses for each medication dose form by and and interactions with and of quality of number of of the and other and The drug information knowledge base could be to and commercial In the of a drug information knowledge developers and should both the drug information underlying their e-prescribing products and the related software to for and quality (which will should through and academic centers and to develop the to such and develop standard for the should the results of of their products they are must evolve the and at local to drug information making them more to and to from the should in any governmental to a drug information knowledge base recommended now developing and drug information knowledge and e-prescribing systems not of if the at future provides a drug information knowledge base for all to The could the knowledge base at a and by both quality information resources and useful e-prescribing software with decision support through such as the Health Information and Management Systems should be to develop a system that from and the to provide to their systems on a or or basis to improve patient A set of standards to should be developed for the e-prescribing that e-prescribing systems have the to products and to feedback for quality must determine that the burden of and using e-prescribing systems not the quality of even if the quality of prescribing is For example, if the of using e-prescribing is for each practitioner to patients the net of an excellent system may be individual care providers must first to use e-prescribing systems, must and must provide feedback for quality By patients that the provider is using an e-prescribing the provider the and or use of and the provider alerts the patient in the patient is by a which to be an of a The time for developing a approach to e-prescribing is at The desiderata by the Joint CDS WG present an important step this list of desiderata is it is only a first should be developed over time to for among users, settings, and information systems. In the authors note that individual e-prescribing recommendations may not be for all settings pediatric on of the recommendations are and will to improve the guidelines over is including the of effects of such systems, with development of reliable to in a The authors believe that of such systems will be complex and we the and of that patient safety cannot be done in an manner or to and correct as they work will be required through of governmental academic and The authors applaud the Joint CDS WG recommendations as a step that will evolve based on feedback and in

This paper brings together methods from two different disciplines: statistics and machine learning. We address the problem of estimating the variance of cross-validation (CV) estimators of the generalization error. In particular, we approach the problem of variance estimation of the CV estimators of generalization error as a problem in approximating the moments of a statistic. The approximation illustrates the role of training and test sets in the performance of the algorithm. It provides a unifying approach to evaluation of various methods used in obtaining training and test sets and it takes into account the variability due to different training and test sets. For the simple problem of predicting the sample mean and in the case of smooth loss functions, we show that the variance of the CV estimator of the generalization error is a function of the moments of the random variables Y=Card(Sj ∩ Sj') and Y*=Card(Sjc ∩ Sj'c), where Sj, Sj' are two training sets, and Sjc, Sj'c are the corresponding test sets. We prove that the distribution of Y and Y* is hypergeometric and we compare our estimator with the one proposed by Nadeau and Bengio (2003). We extend these results in the regression case and the case of absolute error loss, and indicate how the methods can be extended to the classification case. We illustrate the results through simulation.

OBJECTIVE: The aim of this study was to develop a method based on natural language processing (NLP) that automatically maps an entire clinical document to codes with modifiers and to quantitatively evaluate the method. METHODS: An existing NLP system, MedLEE, was adapted to automatically generate codes. The method involves matching of structured output generated by MedLEE consisting of findings and modifiers to obtain the most specific code. Recall and precision applied to Unified Medical Language System (UMLS) coding were evaluated in two separate studies. Recall was measured using a test set of 150 randomly selected sentences, which were processed using MedLEE. Results were compared with a reference standard determined manually by seven experts. Precision was measured using a second test set of 150 randomly selected sentences from which UMLS codes were automatically generated by the method and then validated by experts. RESULTS: Recall of the system for UMLS coding of all terms was .77 (95% CI.72-.81), and for coding terms that had corresponding UMLS codes recall was .83 (.79-.87). Recall of the system for extracting all terms was .84 (.81-.88). Recall of the experts ranged from .69 to .91 for extracting terms. The precision of the system was .89 (.87-.91), and precision of the experts ranged from .61 to .91. CONCLUSION: Extraction of relevant clinical information and UMLS coding were accomplished using a method based on NLP. The method appeared to be comparable to or better than six experts. The advantage of the method is that it maps text to codes along with other related information, rendering the coded output suitable for effective retrieval.

CONTEXT: Although patient safety is a major problem, most health care organizations rely on spontaneous reporting, which detects only a small minority of adverse events. As a result, problems with safety have remained hidden. Chart review can detect adverse events in research settings, but it is too expensive for routine use. Information technology techniques can detect some adverse events in a timely and cost-effective way, in some cases early enough to prevent patient harm. OBJECTIVE: To review methodologies of detecting adverse events using information technology, reports of studies that used these techniques to detect adverse events, and study results for specific types of adverse events. DESIGN: Structured review. METHODOLOGY: English-language studies that reported using information technology to detect adverse events were identified using standard techniques. Only studies that contained original data were included. MAIN OUTCOME MEASURES: Adverse events, with specific focus on nosocomial infections, adverse drug events, and injurious falls. RESULTS: Tools such as event monitoring and natural language processing can inexpensively detect certain types of adverse events in clinical databases. These approaches already work well for some types of adverse events, including adverse drug events and nosocomial infections, and are in routine use in a few hospitals. In addition, it appears likely that these techniques will be adaptable in ways that allow detection of a broad array of adverse events, especially as more medical information becomes computerized. CONCLUSION: Computerized detection of adverse events will soon be practical on a widespread basis.

In health care today, most adverse events are detected using spontaneous reporting, which identifies only a small number of adverse events.1 This is probably the major reason that problems with patient safety have been overlooked until recently. However, information technology can be used in a variety of ways to detect adverse events continuously and relatively inexpensively. In an accompanying paper,2 we review the methodologies for detecting adverse events using information technology and the evidence regarding their efficacy. This editorial presents some of what we believe are future possibilities in this domain and discusses policy issues regarding the development of strategies that may result in wider use of such tools. Spontaneous reporting is attractive because it is inexpensive compared with other approaches for detecting adverse events. Events detected via this route can be useful for quality improvement. However, because reported events represent only a tiny fraction of all adverse events that occur, absolute rates of spontaneous reporting or changes in them are not particularly useful, except to assess safety culture or whether strategies to improve reporting have worked.3 In contrast, a variety of information tchnology approaches can be used to identify a large proportion of all adverse events that occur,4 and this proportion can be expected to increase as more electronic data become available and tools are refined. Although the data regarding automated detection of some types of adverse events are now substantial,5–9 many research and development issues remain to be addressed. For nosocomial infections, adverse drug events and falls, there is a major need for studies that compare different approaches to detection and identify methods that will improve the positive predictive value (which is generally low) for individual signals. This is important because the major cost of such detection strategies is the time of the personnel who respond to the signals. Another key research area involves the definition of approaches that will allow exportation of such detection modules to hospitals in general and to small rural and community hospitals in particular. Finally, tools that allow detection of a wide array of adverse events are needed. Although claims data can provide limited information, especially for inpatients, they do not include sufficient detail to identify a large proportion of adverse events.10 A key benefit of electronic medical records may be that it will be possible to search them using computerized detection tools. Such approaches appear promising based on early data,11 but they need much more evaluation with respect to performance and generalizability. Finally, standards regarding definitions and representation of adverse events would be useful, as would better tools for classifying what went wrong in preventable events. Patient safety is extraordinarily important to the public, but the policy issues around adverse event detection and malpractice are nettlesome. Unfortunately, given the current structures of health care in the U.S., there are strong incentives for organizations to turn a blind eye to adverse events. In particular, serious, preventable adverse events typically must be reported to the state, and such events often lead to multiple visits from the department of public health or end up in the press, with adverse consequences for the institution. Thus, adverse events have negative connotations to many, and our current system offers few incentives to organizations to look for them aggressively. In particular, those who ultimately must approve resources for monitoring systems (chief executive officers and chief operating officers) can avoid investing in them, especially since there are so many competing demands for funds. As a result, financial incentives or regulation may be needed to achieve widespread adoption of routine automated monitoring for adverse events. Several years ago the Centers for Medicare and Medicaid Services (the former Health Care Financing Admin-istration) published draft regulations in the Federal Register that would have mandated computerized monitoring of adverse drug events in inpatients.12 These regulations had a number of unrelated problems, and are undergoing revision. We believe that setting up incentives for hospitals to monitor both adverse drug events and nosocomial infections using computerized detection would be desirable now. Eventually, health care systems should look routinely for adverse events using computerized detection approaches both inside and outside of hospitals, but they will not take on this burden without incentives. Incentives may come in the form of carrots (e.g., higher reimbursement for compliant organizations) or as sticks (e.g., making such monitoring a condition of participation). Another enabler is for the government to make monitoring tools available; the Center for Medicare and Medicaid Services is currently considering this possibility. A related issue that must be addressed is how individuals and organizations that identify and improve their systems using error and adverse event detection should be treated. Although this issue is complicated, the current “blame-and-shame” approach is highly counterproductive. In aviation, nonpunitive ap-proaches have been highly effective in determining the causes of adverse events and developing strategies and interventions for prevention.13 If good techniques for identifying adverse events are developed and used broadly, it will be possible to use such information to improve safety in an ongoing way and, in particular, to use it to assess the impact of systemic changes. We believe that achieving widespread adoption of these techniques may require regulation, because such screening requires resources and organizations are justifiably fearful that uncovering problems may increase litigation risk. However, if legislation or regulations were enacted to provide better protection for health care organizations, substantial improvement in patient safety may result.

OBJECTIVE: To analyze the effect of expert knowledge on the inductive learning process in creating classifiers for medical text reports. DESIGN: The authors converted medical text reports to a structured form through natural language processing. They then inductively created classifiers for medical text reports using varying degrees and types of expert knowledge and different inductive learning algorithms. The authors measured performance of the different classifiers as well as the costs to induce classifiers and acquire expert knowledge. MEASUREMENTS: The measurements used were classifier performance, training-set size efficiency, and classifier creation cost. RESULTS: Expert knowledge was shown to be the most significant factor affecting inductive learning performance, outweighing differences in learning algorithms. The use of expert knowledge can affect comparisons between learning algorithms. This expert knowledge may be obtained and represented separately as knowledge about the clinical task or about the data representation used. The benefit of the expert knowledge is more than that of inductive learning itself, with less cost to obtain. CONCLUSION: For medical text report classification, expert knowledge acquisition is more significant to performance and more cost-effective to obtain than knowledge discovery. Building classifiers should therefore focus more on acquiring knowledge from experts than trying to learn this knowledge inductively.

Knowledge on interactions between molecules in living cells is indispensable for theoretical analysis and practical applications in modern genomics and molecular biology. Building such networks relies on the assumption that the correct molecular interactions are known or can be identified by reading a few research articles. However, this assumption does not necessarily hold, as truth is rather an emerging property based on many potentially conflicting facts. This paper explores the processes of knowledge generation and publishing in the molecular biology literature using modelling and analysis of real molecular interaction data. The data analysed in this article were automatically extracted from 50000 research articles in molecular biology using a computer system called GeneWays containing a natural language processing module. The paper indicates that truthfulness of statements is associated in the minds of scientists with the relative importance (connectedness) of substances under study, revealing a potential selection bias in the reporting of research results. Aiming at understanding the statistical properties of the life cycle of biological facts reported in research articles, we formulate a stochastic model describing generation and propagation of knowledge about molecular interactions through scientific publications. We hope that in the future such a model can be useful for automatically producing consensus views of molecular interaction data.

In medical informatics research, study questions frequently involve individuals who are grouped into clusters. For example, an intervention may be aimed at a clinician (who treats a cluster of patients) with the intention of improving the health of individual patients. Correlation among individuals within a cluster can lead to incorrect estimates of the sample size required to detect an effect and inappropriate estimates of the confidence intervals and the statistical significance of the intervention effects. Contamination, which is the spread of the effect of an intervention or control treatment to the opposite group, often occurs between individuals within clusters. It leads to an attenuation of the effect of the intervention and reduced power to detect a difference. If individuals are randomized in a clinical trial (individual-randomized trial), then correlation must be taken into account in the analysis, and the sample size may need to be increased to compensate for contamination. Randomizing clusters rather than individuals (cluster-randomized trials) can eliminate contamination and may be preferred for logistical reasons. Cluster-randomized trials are generally less efficient than individual-randomized trials, so the tradeoffs must be assessed. Correlation must be taken into account in the analysis and in the sample-size calculations for cluster-randomized trials.

Medical informatics systems are often designed to perform at the level of human experts. Evaluation of the performance of these systems is often constrained by lack of reference standards, either because the appropriate response is not known or because no simple appropriate response exists. Even when performance can be assessed, it is not always clear whether the performance is sufficient or reasonable. These challenges can be addressed if an evaluator enlists the help of clinical domain experts. 1) The experts can carry out the same tasks as the system, and then their responses can be combined to generate a reference standard. 2)The experts can judge the appropriateness of system output directly. 3) The experts can serve as comparison subjects with which the system can be compared. These are separate roles that have different implications for study design, metrics, and issues of reliability and validity. Diagrams help delineate the roles of experts in complex study designs.

The Columbia University Informatics for Diabetes Education and Telemedicine (IDEATel) Project is a four-year demonstration project funded by the Centers for Medicare and Medicaid Services with the overall goals of evaluating the feasibility, acceptability, effectiveness, and cost-effectiveness of telemedicine in the management of older patients with diabetes. The study is designed as a randomized controlled trial and is being conducted by a state-wide consortium in New York. Eligibility requires that participants have diabetes, are Medicare beneficiaries, and reside in federally designated medically underserved areas. A total of 1,500 participants will be randomized, half in New York City and half in other areas of the state. Intervention participants receive a home telemedicine unit that provides synchronous videoconferencing with a project-based nurse, electronic transmission of home fingerstick glucose and blood pressure data, and Web access to a project Web site. End points include glycosylated hemoglobin, blood pressure, and lipid levels; patient satisfaction; health care service utilization; and costs. The project is intended to provide data to help inform regulatory and reimbursement policies for electronically delivered health care services.

The Columbia University Informatics for Diabetes Education and Telemedicine IDEATel) project is a four-year demonstration project funded by the Centers for Medicare and Medicaid Services with the overall goal of evaluating the feasibility, acceptability, effectiveness, and cost-effectiveness of telemedicine. The focal point of the intervention is the home telemedicine unit (HTU), which provides four functions: synchronous videoconferencing over standard telephone lines, electronic transmission for fingerstick glucose and blood pressure readings, secure Web-based messaging and clinical data review, and access to Web-based educational materials. The HTU must be usable by elderly patients with no prior computer experience. Providing these functions through the HTU requires tight integration of six components: the HTU itself, case management software, a clinical information system, Web-based educational material, data security, and networking and telecommunications. These six components were integrated through a variety of interfaces, providing a system that works well for patients and providers. With more than 400 HTUs installed, IDEATel has demonstrated the feasibility of large-scale home telemedicine.

OBJECTIVE: To develop methods that automatically map abbreviations to their full forms in biomedical articles. METHODS: The authors developed two methods of mapping defined and undefined abbreviations (defined abbreviations are paired with their full forms in the articles, whereas undefined ones are not). For defined abbreviations, they developed a set of pattern-matching rules to map an abbreviation to its full form and implemented the rules into a software program, AbbRE (for "abbreviation recognition and extraction"). Using the opinions of domain experts as a reference standard, they evaluated the recall and precision of AbbRE for defined abbreviations in ten biomedical articles randomly selected from the ten most frequently cited medical and biological journals. They also measured the percentage of undefined abbreviations in the same set of articles, and they investigated whether they could map undefined abbreviations to any of four public abbreviation databases (GenBank LocusLink, SWISSPROT, LRABR of the UMLS Specialist Lexicon, and BioABACUS). RESULTS: AbbRE had an average 0.70 recall and 0.95 precision for the defined abbreviations. The authors found that an average of 25 percent of abbreviations were defined in biomedical articles and that of a randomly selected subset of undefined abbreviations, 68 percent could be mapped to any of four abbreviation databases. They also found that many abbreviations are ambiguous (i.e., they map to more than one full form in abbreviation databases). CONCLUSION: AbbRE is efficient for mapping defined abbreviations. To couple AbbRE with abbreviation databases for the mapping of undefined abbreviations, not only exhaustive abbreviation databases but also a method to resolve the ambiguity of abbreviations in the databases are needed.

OBJECTIVE: To design a document model that provides reliable and efficient access to clinical information in patient reports for a broad range of clinical applications, and to implement an automated method using natural language processing that maps textual reports to a form consistent with the model. METHODS: A document model that encodes structured clinical information in patient reports while retaining the original contents was designed using the extensible markup language (XML), and a document type definition (DTD) was created. An existing natural language processor (NLP) was modified to generate output consistent with the model. Two hundred reports were processed using the modified NLP system, and the XML output that was generated was validated using an XML validating parser. RESULTS: The modified NLP system successfully processed all 200 reports. The output of one report was invalid, and 199 reports were valid XML forms consistent with the DTD. CONCLUSIONS: Natural language processing can be used to automatically create an enriched document that contains a structured component whose elements are linked to portions of the original textual report. This integrated document model provides a representation where documents containing specific information can be accurately and efficiently retrieved by querying the structured components. If manual review of the documents is desired, the salient information in the original reports can also be identified and highlighted. Using an XML model of tagging provides an additional benefit in that software tools that manipulate XML documents are readily available.

GOAL: To assess the reliability of a reference standard for an information extraction task. SETTING: Twenty-four physician raters from two sites and two specialties judged whether clinical conditions were present based on reading chest radiograph reports. METHODS: Variance components, generalizability (reliability) coefficients, and the number of expert raters needed to generate a reliable reference standard were estimated. RESULTS: Per-rater reliability averaged across conditions was 0.80 (95% CI, 0.79-0.81). Reliability for the nine individual conditions varied from 0.67 to 0.97, with central line presence and pneumothorax the most reliable, and pleural effusion (excluding CHF) and pneumonia the least reliable. One to two raters were needed to achieve a reliability of 0.70, and six raters, on average, were required to achieve a reliability of 0.95. This was far more reliable than a previously published per-rater reliability of 0.19 for a more complex task. Differences between sites were attributable to changes to the condition definitions. CONCLUSION: In these evaluations, physician raters were able to judge very reliably the presence of clinical conditions based on text reports. Once the reliability of a specific rater is confirmed, it would be possible for that rater to create a reference standard reliable enough to assess aggregate measures on a system. Six raters would be needed to create a reference standard sufficient to assess a system on a case-by-case basis. These results should help evaluators design future information extraction studies for natural language processors and other knowledge-based systems.

OBJECTIVE: To measure the accuracy of automated tuberculosis case detection. SETTING: An inner-city medical center. INTERVENTION: An electronic medical record and a clinical event monitor with a natural language processor were used to detect tuberculosis cases according to Centers for Disease Control criteria. MEASUREMENT: Cases identified by the automated system were compared to the local health department's tuberculosis registry, and positive predictive value and sensitivity were calculated. RESULTS: The best automated rule was based on tuberculosis cultures; it had a sensitivity of .89 (95% CI.75-.96) and a positive predictive value of .96 (.89-.99). All other rules had a positive predictive value less than .20. A rule based on chest radiographs had a sensitivity of .41 (.26-.57) and a positive predictive value of .03 (.02-.05), and rule the represented the overall Centers for Disease Control criteria had a sensitivity of .91 (.78-.97) and a positive predictive value of .15 (.12-.18). The culture-based rule was the most useful rule for automated case reporting to the health department, and the chest radiograph-based rule was the most useful rule for improving tuberculosis respiratory isolation compliance. CONCLUSIONS: Automated tuberculosis case detection is feasible and useful, although the predictive value of most of the clinical rules was low. The usefulness of an individual rule depends on the context in which it is used. The major challenge facing automated detection is the availability and accuracy of electronic clinical data.

OBJECTIVE: To evaluate the performance of tools for authoring patient database queries. DESIGN: Query by Review, a tool that exploits the training that users have undergone to master a result review system, was compared with AccessMed, a vocabulary browser that supports lexical matching and the traversal of hierarchical and semantic links. Seven subjects (Medical Logic Module authors) were asked to use both tools to gather the vocabulary terms necessary to perform each of eight laboratory queries. MEASUREMENTS: The proportion of queries that were correct; intersubject agreement. RESULTS: Query by Review had better performance than AccessMed (38% correct queries versus 18%, p = 0.002), but both figures were low. Poor intersubject agreement (28% for Query by Review and 21% for AccessMed) corroborated the relatively low performance. Subjects appeared to have trouble distinguishing laboratory tests from laboratory batteries, picking terms relevant to the particular data type required, and using classes in the vocabulary's hierarchy. CONCLUSION: Query by Review, with its more constrained user interface, performed somewhat better than AccessMed, a more general tool. Neither tool achieved adequate performance, however, which points to the difficulty of formulating a query for a clinical database and the need for further work.

Abstract This paper describes a natural language text extraction system, called MEDLEE, that has been applied to the medical domain. The system extracts, structures, and encodes clinical information from textual patient reports. It was integrated with the Clinical Information System (CIS), which was developed at Columbia-Presbyterian Medical Center (CPMC) to help improve patient care. MEDLEE is currently used on a daily basis to routinely process radiological reports of patients at CPMC. In order to describe how the natural language system was made compatible with the existing CIS, this paper will also discuss engineering issues which involve performance, robustness, and accessibility of the data from the end users' viewpoint. Also described are the three evaluations that have been performed on the system. The first evaluation was useful primarily for further refinement of the system. The two other evaluations involved an actual clinical application which consisted of retrieving reports that were associated with specified diseases. Automated queries were written by a medical expert based on the structured output forms generated as a result of text processing. The retrievals obtained by the automated system were compared to the retrievals obtained by independent medical experts who read the reports manually to determine whether they were associated with the specified diseases. MEDLEE was shown to perform comparably to the experts. The technique used to perform the last two evaluations was found to be a realistic evaluation technique for a natural language processor.

OBJECTIVE: Develop a knowledge-based representation for a controlled terminology of clinical information to facilitate creation, maintenance, and use of the terminology. DESIGN: The Medical Entities Dictionary (MED) is a semantic network, based on the Unified Medical Language System (UMLS), with a directed acyclic graph to represent multiple hierarchies. Terms from four hospital systems (laboratory, electrocardiography, medical records coding, and pharmacy) were added as nodes in the network. Additional knowledge about terms, added as semantic links, was used to assist in integration, harmonization, and automated classification of disparate terminologies. RESULTS: The MED contains 32,767 terms and is in active clinical use. Automated classification was successfully applied to terms for laboratory specimens, laboratory tests, and medications. One benefit of the approach has been the automated inclusion of medications into multiple pharmacologic and allergenic classes that were not present in the pharmacy system. Another benefit has been the reduction of maintenance efforts by 90%. CONCLUSION: The MED is a hybrid of terminology and knowledge. It provides domain coverage, synonymy, consistency of views, explicit relationships, and multiple classification while preventing redundancy, ambiguity (homonymy) and misclassification.