The timely and accurate identification of adverse drug reactions (ADRs) following drug approval is a persistent and serious public health challenge. Aggregated data drawn from anonymized logs of Web searchers has been shown to be a useful source of evidence for detecting ADRs. However, prior studies have been based on the analysis of established ADRs, the existence of which may already be known publically. Awareness of these ADRs can inject existing knowledge about the known ADRs into online content and online behavior, and thus raise questions about the ability of the behavioral log-based methods to detect new ADRs. In contrast to previous studies, we investigate the use of search logs for the early detection of known ADRs. We use a large set of recently labeled ADRs and negative controls to evaluate the ability of search logs to accurately detect ADRs in advance of their publication. We leverage the Internet Archive to estimate when evidence of an ADR first appeared in the public domain and adjust the index date in a backdated analysis. Our results demonstrate how search logs can be used to detect new ADRs, the central challenge in pharmacovigilance.

Data mining is a crucial tool for identifying risk signals of potential adverse drug reactions (ADRs). However, mining of ADR signals is currently limited to leveraging a single data source at a time. It is widely believed that combining ADR evidence from multiple data sources will result in a more accurate risk identification system. We present a methodology based on empirical Bayes modeling to combine ADR signals mined from ~5 million adverse event reports collected by the FDA, and healthcare data corresponding to 46 million patients' the main two types of information sources currently employed for signal detection. Based on four sets of test cases (gold standard), we demonstrate that our method leads to a statistically significant and substantial improvement in signal detection accuracy, averaging 40% over the use of each source independently, and an area under the ROC curve of 0.87. We also compare the method with alternative supervised learning approaches, and argue that our approach is preferable as it does not require labeled (training) samples whose availability is currently limited. To our knowledge, this is the first effort to combine signals from these two complementary data sources, and to demonstrate the benefits of a computationally integrative strategy for drug safety surveillance.

OBJECTIVE: Data-mining algorithms that can produce accurate signals of potentially novel adverse drug reactions (ADRs) are a central component of pharmacovigilance. We propose a signal-detection strategy that combines the adverse event reporting system (AERS) of the Food and Drug Administration and electronic health records (EHRs) by requiring signaling in both sources. We claim that this approach leads to improved accuracy of signal detection when the goal is to produce a highly selective ranked set of candidate ADRs. MATERIALS AND METHODS: Our investigation was based on over 4 million AERS reports and information extracted from 1.2 million EHR narratives. Well-established methodologies were used to generate signals from each source. The study focused on ADRs related to three high-profile serious adverse reactions. A reference standard of over 600 established and plausible ADRs was created and used to evaluate the proposed approach against a comparator. RESULTS: The combined signaling system achieved a statistically significant large improvement over AERS (baseline) in the precision of top ranked signals. The average improvement ranged from 31% to almost threefold for different evaluation categories. Using this system, we identified a new association between the agent, rasburicase, and the adverse event, acute pancreatitis, which was supported by clinical review. CONCLUSIONS: The results provide promising initial evidence that combining AERS with EHRs via the framework of replicated signaling can improve the accuracy of signal detection for certain operating scenarios. The use of additional EHR data is required to further evaluate the capacity and limits of this system and to extend the generalizability of these results.

The concept of "data squashing" was introduced by DuMouchel et al [4] as a method of summarizing massive data sets that preserves statistical relationships among variables. The idea is to create a smaller data set that allows statistical modeling to take place using in-memory algorithms, and to preserve the modeling results more accurately than would a same-size random sample from the massive data set. This research attempts to avoid several limitations of previous approaches to data squashing. Our method avoids the curse of dimensionality by a double use of principal components transformations that makes computing time linear in the number of cases and quadratic in the number of variables. Categorical and continuous variables are smoothly integrated. Because the binning is based on principal components, which are uncorrelated, we can use fractional factorial designs that sample less than one point per bin. We also investigate various weighting schemes for the squashed sample to see whether matching moments or matching subregion data counts is more effective. Finally, previous work required the specification of a statistical model, either to perform the squashing algorithm or to compare the worth of different squashing methods. Our approach to evaluation is model free and does not even require the specification of variables as responses or predictors. Instead, we develop a chi-squared like measure of accuracy to compare the closeness of various discrete densities (the squashed data sets) to the discrete massive data set.

Because of practical limits in characterizing the safety profiles of therapeutic products prior to marketing, manufacturers and regulatory agencies perform post-marketing surveillance based on the collection of adverse reaction reports ("pharmacovigilance").The resulting databases, while rich in real-world information, are notoriously difficult to analyze using traditional techniques. Each report may involve multiple medicines, symptoms, and demographic factors, and there is no easily linked information on drug exposure in the reporting population. KDD techniques, such as association finding, are well-matched to the problem, but are difficult for medical staff to apply and interpret.To deploy KDD effectively for pharmacovigilance, Lincoln Technologies and GlaxoSmithKline collaborated to create a webbased safety data mining web environment. The analytical core is a high-performance implementation of the MGPS (Multi-Item Gamma Poisson Shrinker) algorithm described previously by DuMouchel and Pregibon, with several significant extensions and enhancements. The environment offers an interface for specifying data mining runs, a batch execution facility, tabular and graphical methods for exploring associations, and drilldown to case details. Substantial work was involved in preparing the raw adverse event data for mining, including harmonization of drug names and removal of duplicate reports.The environment can be used to explore both drug-event and multi-way associations (interactions, syndromes). It has been used to study age/gender effects, to predict the safety profiles of proposed combination drugs, and to separate contributions of individual drugs to safety problems in polytherapy situations.

This paper considers the framework of the so-called "market basket problem", in which a database of transactions is mined for the occurrence of unusually frequent item sets. In our case, "unusually frequent" involves estimates of the frequency of each item set divided by a baseline frequency computed as if items occurred independently. The focus is on obtaining reliable estimates of this measure of interestingness for all item sets, even item sets with relatively low frequencies. For example, in a medical database of patient histories, unusual item sets including the item "patient death" (or other serious adverse event) might hopefully be flagged with as few as 5 or 10 occurrences of the item set, it being unacceptable to require that item sets occur in as many as 0.1% of millions of patient reports before the data mining algorithm detects a signal. Similar considerations apply in fraud detection applications. Thus we abandon the requirement that interesting item sets must contain a re...

A feature of data mining that distinguishes it from "classical" machine learning (ML) and statistical modeling (SM) is scale. The community seems to agree on this yet progress to this point has been limited. We present a methodology that addresses scale in a novel fashion that has the potential for revolutionizing the field. While the methodology applies most directly to flat (row by column) data sets we believe that it can be adapted to other representations. Our approach to the problem is not to scale up individual ML and SM methods. Rather we prefer to leverage the entire collection of existing methods by scaling down the data set. We call the method squashing. Our method demonstrably outperforms random sampling and a theoretical argument suggests how and why it works well. Squashing consists of three modular steps: grouping, momentizing, and generating (GMG). These three steps describe the squashing pipeline whereby the original (very large data set) is sectioned off into mutual...

OBJECTIVE: Computer-based reminder systems have the potential to change physician and patient behaviors and to improve patient outcomes. We performed a meta-analysis of published randomized controlled trials to assess the overall effectiveness of computer-based reminder systems in ambulatory settings directed at preventive care. DESIGN: Meta-analysis. SEARCH STRATEGY: Searches of the Medline (1966-1994), Nursing and Allied Health (1982-1994), and Health Planning and Administration (1975-1994) databases identified 16 randomized, controlled trials of computer-based reminder systems in ambulatory settings. STATISTICAL METHODS: A weighted mixed effects model regression analysis was used to estimate intervention effects for computer and manual reminder systems for six classes of preventive practices. MAIN OUTCOME MEASURE: Adjusted odds ratio for preventive practices. RESULTS: Computer reminders improved preventive practices compared with the control condition for vaccinations (adjusted odds ratio [OR] 3.09; 95% confidence interval [CI] 2.39-4.00), breast cancer screening (OR 1.88; 95% CI 1.44-2.45), colorectal cancer screening (OR 2.25; 95% CI 1.74-2.91), and cardiovascular risk reduction (OR 2.01; 95% CI 1.55-2.61) but not cervical cancer screening (OR 1.15; 95% CI 0.89-1.49) or other preventive care (OR 1.02; 95% CI 0.79-1.32). For all six classes of preventive practices combined the adjusted OR was 1.77 (95% CI 1.38-2.27). CONCLUSION: Evidence from randomized controlled studies supports the effectiveness of data-driven computer-based reminder systems to improve prevention services in the ambulatory care setting.

OBJECTIVE: With the advent of hospital payment by diagnosis-related group (DRG), length of stay (LOS) has become a major issue in hospital efforts to control costs. Because the Columbia-Presbyterian Medical Center (CPMC) has had above-average LOSs for many DRGs, the authors tested the hypothesis that a computer-generated informational message directed to physicians would shorten LOS. DESIGN: Randomized clinical trial with the patient as the unit of randomization. SETTING AND STUDY POPULATION: From June 1991 to April 1993, at CPMC in New York, 7,109 patient admissions were randomly assigned to an intervention (informational message) group and 6,990 to a control (no message) group. INTERVENTION: A message giving the average LOS for the patient's admission or provisional DRG, as assigned by hospital utilization review, and the current LOS, in days, was included in the main menu for review of test results in the hospital's clinical information system, available at all nursing stations in the hospital. MAIN OUTCOME MEASURE: Hospital LOS. RESULTS: The median LOS for study patients was 7 days. After adjustment for covariates including age, sex, payor, patient care unit, and time trends, the mean LOS in the intervention group was 3.2% shorter than that in the control group (p = 0.022). CONCLUSION: Computer-generated patient-specific LOS information directed to physicians was associated with a reduction in hospital LOS.

Abstract This paper describes a natural language text extraction system, called MEDLEE, that has been applied to the medical domain. The system extracts, structures, and encodes clinical information from textual patient reports. It was integrated with the Clinical Information System (CIS), which was developed at Columbia-Presbyterian Medical Center (CPMC) to help improve patient care. MEDLEE is currently used on a daily basis to routinely process radiological reports of patients at CPMC. In order to describe how the natural language system was made compatible with the existing CIS, this paper will also discuss engineering issues which involve performance, robustness, and accessibility of the data from the end users' viewpoint. Also described are the three evaluations that have been performed on the system. The first evaluation was useful primarily for further refinement of the system. The two other evaluations involved an actual clinical application which consisted of retrieving reports that were associated with specified diseases. Automated queries were written by a medical expert based on the structured output forms generated as a result of text processing. The retrievals obtained by the automated system were compared to the retrievals obtained by independent medical experts who read the reports manually to determine whether they were associated with the specified diseases. MEDLEE was shown to perform comparably to the experts. The technique used to perform the last two evaluations was found to be a realistic evaluation technique for a natural language processor.