In many critical applications, sensitive data is inherently distributed and cannot be centralized due to privacy concerns. A wide range of federated learning approaches have been proposed to train models locally at each client without sharing their sensitive data, typically by exchanging model parameters, or probabilistic predictions (soft labels) on a public dataset or a combination of both. However, these methods still disclose private information and restrict local models to those that can be trained using gradient-based methods. We propose a federated co-training (FEDCT) approach that improves privacy by sharing only definitive (hard) labels on a public unlabeled dataset. Clients use a consensus of these shared labels as pseudo-labels for local training. This federated co-training approach empirically enhances privacy without compromising model quality. In addition, it allows the use of local models that are not suitable for parameter aggregation in traditional federated learning, such as gradient-boosted decision trees, rule ensembles, and random forests. Furthermore, we observe that FEDCT performs effectively in federated fine-tuning of large language models, where its pseudo-labeling mechanism is particularly beneficial. Empirical evaluations and theoretical analyses suggest its applicability across a range of federated learning scenarios.

Consider the following healthcare trends: (1) There is a tremendous increase in the amount of patient, life sciences and process data in electronic form, fueled by advances in healthcare IT technology, and health reform legislation. (2) The amount of medical information (e.g., evidence-based knowledge) and published knowledge is said to be doubling every few years. (3) There is an explosion in the number of available therapies and diagnostic options for patient care, often enabling precise targeting of therapy to disease conditions. In this talk we will discuss these trends and some of the reasons why, despite these advances, healthcare is facing a crisis: namely, there is a steady unsustainable increase in medical costs without a corresponding improvement of patient outcomes. We believe that analysis of clinical, life sciences and medical process data can play a key role in tackling these fundamental challenges. Two technology advances, in particular, can play a key role: cloud computing and mobility will make it possible to analyze vast amounts of data and quickly deliver useful information to clinicians, consumers and researchers at the point where it can have the most impact. Some of this is already happening today, with medical records being analyzed to reduce fraud, waste and abuse, improve patient outcomes, and to improve compliance with standards of care and policy guidelines. We conclude the talk with a glimpse of a future where medical systems could be continually analyzed for optimizing healthcare costs and outcomes.

We propose a novel Bayesian multiple instance learning (MIL) algorithm. This algorithm automatically identifies the relevant feature subset, and utilizes inductive transfer when learning multiple (conceptually related) classifiers. Experimental results indicate that the proposed MIL method is more accurate than previous MIL algorithms and selects a much smaller set of useful features. Inductive transfer further improves the accuracy of the classifier as compared to learning each task individually.

Privacy-preserving data mining (PPDM) is an emergent research area that addresses the incorporation of privacy preserving concerns to data mining techniques. In this paper we propose a privacy-preserving (PP) Cox model for survival analysis, and consider a real clinical setting where the data is horizontally distributed among different institutions. The proposed model is based on linearly projecting the data to a lower dimensional space through an optimal mapping obtained by solving a linear programming problem. Our approach differs from the commonly used random projection approach since it instead finds a projection that is optimal at preserving the properties of the data that are important for the specific problem at hand. Since our proposed approach produces an sparse mapping, it also generates a PP mapping that not only projects the data to a lower dimensional space but it also depends on a smaller subset of the original features (it provides explicit feature selection). Real data from several European healthcare institutions are used to test our model for survival prediction of non-small-cell lung cancer patients. These results are also confirmed using publicly available benchmark datasets. Our experimental results show that we are able to achieve a near-optimal performance without directly sharing the data across different data sources. This model makes it possible to conduct large-scale multi-centric survival analysis without violating privacy-preserving requirements.

Cross validation allows models to be tested using the full training set by means of repeated resampling; thus, maximizing the total number of points used for testing and potentially, helping to protect against overfitting. Improvements in computational power, recent reductions in the (computational) cost of classification algorithms, and the development of closed-form solutions (for performing cross validation in certain classes of learning algorithms) makes it possible to test thousand or millions of variants of learning models on the data. Thus, it is now possible to calculate cross validation performance on a much larger number of tuned models than would have been possible otherwise. However, we empirically show how under such large number of models the risk for overfitting increases and the performance estimated by cross validation is no longer an effective estimate of generalization; hence, this paper provides an empirical reminder of the dangers of cross validation. We use a closed-form solution that makes this evaluation possible for the cross validation problem of interest. In addition, through extensive experiments we expose and discuss the effects of the overuse/misuse of cross validation in various aspects, including model selection, feature selection, and data dimensionality. This is illustrated on synthetic, benchmark, and real-world data sets.

In biology and medical sciences, highly parallel biological assays spurred a revolution leading to the emergence of the '-omics' era. Dimensionality reduction techniques are necessary to be able to analyze, interpret, validate and take advantage of the tremendous wealth of highly dimensional data they provide. This paper is based on a DNA microarray study providing gene signatures for hypoxia. These gene signatures were tested on a large breast cancer data set for assessing their prognostic power by means of Kaplan-Meier survival, univariate, and multivariate analyses. We explore the use of several mathematical programming-based techniques that aim to reduce the gene signature sizes as much as possible while maintaining the key characteristics of the original signature, more precisely: the signature prognostic and diagnostic significance. The proposed signature reduction techniques have very interesting potential uses. Indeed, by downsizing the relevant data to a manageable size, one can then patent the core set of biomarkers and also create a dedicated assay (e.g.: on a customized array) for routine applications (e.g.: in the clinical set up) leading to individualized medicine capabilities. Our experiments show that the reduced hypoxia signatures reproduced qualitatively and quantitatively in a similar way that of the original ones.

We apply a new active learning formulation to the problem of learning medical concepts from unstructured text. The new formulation is based on maximizing the mutual information that a sample labeling provides about the retrieval/classification model. This methodology is related to and extends the Query-by-Committee approach (QBC) (Seung et al., 1992) by exploiting unlabeled data in novel ways, beyond their common use only as potential query points. Unlike QBC, this method allows us to employ unlabeled data in addition to labeled data in order to select more appropriate samples for labeling. The samples thus chosen are both informative and also relevant according to a distribution of interest. This flexibility allows us to also tailor the model to arbitrary distributions relevant to the task at hand, in particular to the distribution of the test data. This formulation has implications in scenarios where the training and test distributions are different, or when a general model is adapted to a more specific model. Experiments were conducted to evaluate retrieval performance of natural-language text associated to various concepts of interest in the medical domain. We demonstrate the advantages of our formulation compared with QBC, the state-of-the art active learning approach, and against random sample selection.

In this paper, we apply weighted ridge regression to tackle the highly unbalanced data issue in automatic large-scale ICD-9 coding of medical patient records. Since most of the ICD-9 codes are unevenly represented in the medical records, a weighted scheme is employed to balance positive and negative examples. The weights turn out to be associated with the instance priors from a probabilistic interpretation, and an efficient EM algorithm is developed to automatically update both the weights and the regularization parameter. Experiments on a large-scale real patient database suggest that the weighted ridge regression outperforms the conventional ridge regression and linear support vector machines (SVM).

We present LungCAD, a computer aided diagnosis (CAD) system that employs a classification algorithm for detecting solid pulmonary nodules from CT thorax studies. We briefly describe some of the machine learning techniques developed to overcome the real world challenges in this medical domain. The most significant hurdle in transitioning from a machine learning research prototype that performs well on an in-house dataset into a clinically deployable system, is the requirement that the CAD system be tested in a clinical trial. We describe the clinical trial in which LungCAD was tested: a large scale multi-reader, multi-case (MRMC) retrospective observational study to evaluate the effect of CAD in clinical practice for detecting solid pulmonary nodules from CT thorax studies. The clinical trial demonstrates that every radiologist that participated in the trial had a significantly greater accuracy with LungCAD, both for detecting nodules and identifying potentially actionable nodules; this, along with other findings from the trial, has resulted in FDA approval for LungCAD in late 2006.

We propose a Bayesian undirected graphical model for co-training, or more generally for semi-supervised multi-view learning. This makes explicit the previously unstated assumptions of a large class of co-training type algorithms, and also clarifies the circumstances under which these assumptions fail. Building upon new insights from this model, we propose an improved method for co-training, which is a novel co-training kernel for Gaussian process classifiers. The resulting approach is convex and avoids local-maxima problems, unlike some previous multi-view learning methods. Furthermore, it can automatically estimate how much each view should be trusted, and thus accommodate noisy or unreliable views. Experiments on toy data and real world data sets illustrate the benefits of this approach.

We study the joint feature selection problem when learning multiple related classification or regression tasks. By imposing an automatic relevance determination prior on the hypothesis classes associated with each of the tasks and regularizing the variance of the hypothesis parameters, similar feature patterns across different tasks are encouraged and features that are relevant to all (or most) of the tasks are identified. Our analysis shows that the proposed probabilistic framework can be seen as a generalization of previous result from adaptive ridge regression to the multi-task learning setting. We provide a detailed description of the proposed algorithms for simultaneous model construction and justify the proposed algorithms in several aspects. Our experimental results show that this approach outperforms a regularized multi-task learning approach and the traditional methods where individual tasks are solved independently on synthetic data and the real-world data sets for lung cancer prognosis.

This paper describes a novel classification method for computer aided detection (CAD) that identifies structures of interest from medical images. CAD problems are challenging largely due to the following three characteristics. Typical CAD training data sets are large and extremely unbalanced between positive and negative classes. When searching for descriptive features, researchers often deploy a large set of experimental features, which consequently introduces irrelevant and redundant features. Finally, a CAD system has to satisfy stringent real-time requirements.This work is distinguished by three key contributions. The first is a cascade classification approach which is able to tackle all the above difficulties in a unified framework by employing an asymmetric cascade of sparse classifiers each trained to achieve high detection sensitivity and satisfactory false positive rates. The second is the incorporation of feature computational costs in a linear program formulation that allows the feature selection process to take into account different evaluation costs of various features. The third is a boosting algorithm derived from column generation optimization to effectively solve the proposed cascade linear programs.We apply the proposed approach to the problem of detecting lung nodules from helical multi-slice CT images. Our approach demonstrates superior performance in comparison against support vector machines, linear discriminant analysis and cascade AdaBoost. Especially, the resulting detection system is significantly sped up with our approach.

Many computer aided diagnosis (CAD) problems can be best modelled as a multiple-instance learning (MIL) problem with unbalanced data: i.e. , the training data typically consists of a few positive bags, and a very large number of negative instances. Existing MIL algorithms are much too computationally expensive for these datasets. We describe CH, a framework for learning a Convex Hull representation of multiple instances that is significantly faster than existing MIL algorithms. Our CH framework applies to any standard hyperplane-based learning algorithm, and for some algorithms, is guaranteed to find the global optimal solution. Experimental studies on two different CAD applications further demonstrate that the proposed algorithm significantly improves diagnostic accuracy when compared to both MIL and traditional classifiers. Although not designed for standard MIL problems (which have both positive and negative bags and relatively balanced datasets), comparisons against other MIL methods on benchmark problems also indicate that the proposed method is competitive with the state-of-the-art.

The task of learning models for many real-world problems requires incorporating domain knowledge into learning algorithms, to enable accurate learning from a realistic volume of training data. This paper considers a variety of types of domain knowledge for constraining parameter estimates when learning Bayesian networks. In particular, we consider domain knowledge that constrains the values or relationships among subsets of parameters in a Bayesian network with known structure. We incorporate a wide variety of parameter constraints into learning procedures for Bayesian networks, by formulating this task as a constrained optimization problem. The assumptions made in module networks, dynamic Bayes nets and context specific independence models can be viewed as particular cases of such parameter constraints. We present closed form solutions or fast iterative algorithms for estimating parameters subject to several specific classes of parameter constraints, including equalities and inequalities among parameters, constraints on individual parameters, and constraints on sums and ratios of parameters, for discrete and continuous variables. Our methods cover learning from both frequentist and Bayesian points of view, from both complete and incomplete data. We present formal guarantees for our estimators, as well as methods for automatically learning useful parameter constraints from data. To validate our approach, we apply it to the domain of fMRI brain image analysis. Here we demonstrate the ability of our system to first learn useful relationships among parameters, and then to use them to constrain the training of the Bayesian network, resulting in improved cross-validated accuracy of the learned model. Experiments on synthetic data are also presented.

We propose an algorithm to construct classification models with a mixture of kernels from labeled and unlabeled data. The derived classifier is a mixture of models, each based on one kernel choice from a library of kernels. The sparse-favoring 1-norm regularization method is employed to restrict the complexity of mixture models and to achieve the sparsity of solutions. By modifying the column generation boosting algorithm LPBoost to a more general linear programming formulation, we are able to efficiently solve mixture-of-kernel problems and automatically select kernel basis functions centered at labeled data as well as unlabeled data. The effectiveness of the proposed approach is proved by experimental results on benchmark datasets.

Coronary Heart Disease is the single leading cause of death world-wide, with lack of early diagnosis being a key contributory factor. This disease can be diagnosed by measuring and scoring regional motion of the heart wall in echocardiography images of the left ventricle (LV) of the heart. We describe a completely automated and robust technique that detects diseased hearts based on automatic detection and tracking of the endocardium and epicardium of the LV. We describe a novel feature selection technique based on mathematical programming that results in a robust hyperplane-based classifier. The classifier depends only on a small subset of numerical feature extracted from dualcontours tracked through time. We verify the robustness of our system on echocardiograms collected in routine clinical practice at one hospital, both with the standard crossvalidation analysis, and then on a held-out set of completely unseen echocardiography images.

We describe an algorithm for converting linear support vector machines and any other arbitrary hyperplane-based linear classifiers into a set of non-overlapping rules that, unlike the original classifier, can be easily interpreted by humans. Each iteration of the rule extraction algorithm is formulated as a constrained optimization problem that is computationally inexpensive to solve. We discuss various properties of the algorithm and provide proof of convergence for two different optimization criteria We demonstrate the performance and the speed of the algorithm on linear classifiers learned from real-world datasets, including a medical dataset on detection of lung cancer from medical images. The ability to convert SVM's and other "black-box" classifiers into a set of human-understandable rules, is critical not only for physician acceptance, but also to reducing the regulatory barrier for medical-decision support systems based on such classifiers.

We describe a method for sparse feature selection for a class of problems motivated by our work in Computer-Aided Detection (CAD) systems for identifying structures of interest in medical images. We propose a sparse formulation for Fisher Linear Discriminant (FLD) that scales well to large datasets; our method inherits all the desirable properties of FLD, while improving on handling large numbers of irrelevant and redundant features. We demonstrate that our sparse FLD formulation outperforms conventional FLD and two other methods for feature selection from the literature on both an artificial dataset and a real-world Colon CAD dataset.

Building accurate models from a small amount of available training data can sometimes prove to be a great challenge. Expert domain knowledge can often be used to alleviate this burden. Parameter Sharing is one such important form of domain knowledge. Graphical models like HMMs, DBNs and Module Networks use different forms of Parameter Sharing to reduce the variance in the parameter estimates. The goal of this paper is to present a theoretical approach for learning in presence of several other types of Parameter Related Domain Knowledge that go beyond the ones in the above models. First, we introduce a General Parameter Sharing Framework that describes the models just mentioned, but allows for much finer grained parameter sharing assumptions. In this framework, we present sound procedures for parameter learning from both a Frequentist and a Bayesian point of view, from both complete and incomplete data, in the case where a domain expert specifies in advance the structure of the graphical model, and the subsets of parameters to be shared. Second, we describe a hierarchical extension of this framework based on Parameter Sharing Trees. Finally we present algorithms for using domain knowledge that specifies that certain groups of parameters share certain properties. In particular, we consider two kinds of constraints: first kind states certain groups of parameters share the same aggregate probability mass and second kind states the ratio of the parameters is preserved (shared) in several groups. As an example, we derive a novel form of parameter sharing for Bayesian Multinetworks.

We propose a fast iterative classification algorithm for Kernel Fisher Discriminant (KFD) using heterogeneous kernel models. In contrast with the standard KFD that requires the user to predefine a kernel function, we incorporate the task of choosing an appropriate kernel into the optimization problem to be solved. The choice of kernel is defined as a linear combination of kernels belonging to a potentially large family of different positive semidefinite kernels. The complexity of our algorithm does not increase significantly with respect to the number of kernels on the kernel family. Experiments on several benchmark datasets demonstrate that generalization performance of the proposed algorithm is not significantly different from that achieved by the standard KFD in which the kernel parameters have been tuned using cross validation. We also present results on a real-life colon cancer dataset that demonstrate the efficiency of the proposed method.

Inference from hospital patient records is difficult because data collection is done at arbitrary (not evenly-spaced) time intervals, and key clinical information is recorded only in unstructured form (as free text in doctors’ notes). We present REMIND, a framework for performing inference from patient records based upon continuous-time Markov models and Bayesian networks. We empirically justify the need for such a complex model. REMIND only uses easily-available domain knowledge which we obtain from physicians and medical literature, which may be inaccurate. We also demonstrate the robustness of our approach to parameter assignment.

Existing patient records are a valuable resource for automated outcomes analysis and knowledge discovery. However, key clinical data in these records is typically recorded in unstructured form as free text and images, and most structured clinical information is poorly organized. Time-consuming interpretation and analysis is required to convert these records into structured clinical data. Thus, only a tiny fraction of this resource is utilized. We present REMIND, a Bayesian Framework for Reliable Extraction and Meaningful Inference from Nonstructured Data. REMIND integrates and blends the structured and unstructured clinical data in patient records to automatically created high-quality structured clinical data. This structuring allows existing patient records to be mined for quality assurance, regulatory compliance, and to relate financial and clinical factors. We demonstrate REMIND on two medical applications: (a) Extract "recurrence", the key outcome for measuring treatment effectiveness, for colon cancer patients (ii) Extract key diagnoses and complications for acute myocardial infarction (heart attack) patients, and demonstrate the impact of these clinical factors on financial outcomes.

CARDWATCH, a database mining system used for credit card fraud detection, is presented. The system is based on a neural network learning module, provides an interface to a variety of commercial databases and has a comfortable graphical user interface. Test results obtained for synthetically generated credit card data and an autoassociative neural network model show very successful fraud detection rates.