Gradient boosting method has been successfully used for single target prediction problems. In real world applications, however, problems involving the prediction of multiple target attributes are often of interest. In this paper, a multi-target boosting method for regression problems, named as MTBR, is proposed. Although MTBR builds one model for each target attribute separately, all the target attributes are utilized when building each model. In each boosting iteration, the base learner, the regression tree in particular, is learned by selecting the best models from all the target attributes. We also introduce a novel knowledge transfer approach. That is, the tree structure learned from one target attribute, representing a way to partition the feature space, is used to predict another target attribute. Experiments with six data sets compare MTBR to other ensemble regression methods, and prove the effectiveness of MTBR in leveraging the knowledge of multiple target attributes and improving the model accuracy.

MOTIVATION: Matched case-control analysis is widely used in biomedical studies to identify exposure variables associated with health conditions. The matching is used to improve the efficiency. Existing variable selection methods for matched case-control studies are challenged in high-dimensional settings where interactions among variables are also important. We describe a quite different method for high-dimensional matched case-control data, based on the potential outcome model, which is not only flexible regarding the number of matching and exposure variables but also able to detect interaction effects. RESULTS: We present Matched Forest (MF), an algorithm for variable selection in matched case-control data. The method preserves the case and control values in each instance but transforms the matched case-control data with added counterfactuals. A modified variable importance score from a supervised learner is used to detect important variables. The method is conceptually simple and can be applied with widely available software tools. Simulation studies show the effectiveness of MF in identifying important variables. MF is also applied to data from the biomedical domain and its performance is compared with alternative approaches. AVAILABILITY AND IMPLEMENTATION: R code for implementing MF is available at https://github.com/NooshinSh/Matched_Forest. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

BACKGROUND: In biomarker discovery, applying domain knowledge is an effective approach to eliminating false positive features, prioritizing functionally impactful markers and facilitating the interpretation of predictive signatures. Several computational methods have been developed that formulate the knowledge-based biomarker discovery as a feature selection problem guided by prior information. These methods often require that prior information is encoded as a single score and the algorithms are optimized for biological knowledge of a specific type. However, in practice, domain knowledge from diverse resources can provide complementary information. But no current methods can integrate heterogeneous prior information for biomarker discovery. To address this problem, we developed the Know-GRRF (know-guided regularized random forest) method that enables dynamic incorporation of domain knowledge from multiple disciplines to guide feature selection. RESULTS: Know-GRRF embeds domain knowledge in a regularized random forest framework. It combines prior information from multiple domains in a linear model to derive a composite score, which, together with other tuning parameters, controls the regularization of the random forests model. Know-GRRF concurrently optimizes the weight given to each type of domain knowledge and other tuning parameters to minimize the AIC of out-of-bag predictions. The objective is to select a compact feature subset that has a high discriminative power and strong functional relevance to the biological phenotype. Via rigorous simulations, we show that Know-GRRF guided by multiple-domain prior information outperforms feature selection methods guided by single-domain prior information or no prior information. We then applied Known-GRRF to a real-world study to identify prognostic biomarkers of prostate cancers. We evaluated the combination of cancer-related gene annotations, evolutionary conservation and pre-computed statistical scores as the prior knowledge to assemble a panel of biomarkers. We discovered a compact set of biomarkers with significant improvements on prediction accuracies. CONCLUSIONS: Know-GRRF is a powerful novel method to incorporate knowledge from multiple domains for feature selection. It has a broad range of applications in biomarker discoveries. We implemented this method and released a KnowGRRF package in the R/CRAN archive.

Aggressive driving behavior is one of the leading causes of road accidents worldwide. One way to ameliorate this situation is to collect and analyze driving patterns with the intention of promoting driving awareness, a task that has been called driving analytics (DA). DA employing the driver's smartphone has attracted attention from the community given its good capabilities to capture, via its integrated sensors, data that could be exploited to infer driving style. Most works in the related literature have represented this sensor information either as statistical scores or in raw format that are later fed into threshold-based heuristics or machine learning (ML) approaches. Based on the hypothesis that better data representations do exist, in this paper, we propose a second-order representation, based on the bag-of-words' strategy, to model accelerometer timestamps associated with aggressive driving maneuvers. We evaluate this representation in two scenarios and three data sets against the best reported work in each of them. In the first scenario, we classify accelerometer samples as either belonging to aggressive or safe driving style. In the second scenario, we approach a multi-class problem, where we are now interested in identifying the exact aggressive maneuver that the accelerometer sample represents. The results show that this novel representation outperforms both state-of-the-art works with 6% and 15% in F-measure for each scenario, respectively. To further investigate the strength of our representation, we make a comparison against similar second-order strategies that have also proved to be successful. Overall, this analysis suggests that this representation constitutes an attractive method for driving behavior classification, boosting discriminative performance of ML approaches.

Clustering is an important aspect of data mining, while clustering high-dimensional mixed-attribute data in a scalable fashion still remains a challenging problem. In this paper, we propose a tree-ensemble clustering algorithm for static datasets, CRAFTER, to tackle this problem. CRAFTER is able to handle categorical and numeric attributes simultaneously, and scales well with the dimensionality and the size of datasets. CRAFTER leverages the advantages of a tree-ensemble to handle mixed attributes and high dimensionality. The concept of the class probability estimates is utilized to identify the representative data points for clustering. Through a series of experiments on both synthetic and real datasets, we have demonstrated that CRAFTER is superior than Random Forest Clustering (RFC), an existing tree-based clustering method, in terms of both the clustering quality and the computational cost.

Autoassociative neural networks (ANNs) have been proposed as a nonlinear extension of principal component analysis (PCA), which is commonly used to identify linear variation patterns in high-dimensional data. While principal component scores represent uncorrelated features, standard backpropagation methods for training ANNs provide no guarantee of producing distinct features, which is important for interpretability and for discovering the nature of the variation patterns in the data. Here, we present an alternating nonlinear PCA method, which encourages learning of distinct features in ANNs. A new measure motivated by the condition of orthogonal loadings in PCA is proposed for measuring the extent to which the nonlinear principal components represent distinct variation patterns. We demonstrate the effectiveness of our method using a simulated point cloud data set as well as a subset of the MNIST handwritten digits data. The results show that standard ANNs consistently mix the true variation sources in the low-dimensional representation learned by the model, whereas our alternating method produces solutions where the patterns are better separated in the low-dimensional space.

In this paper, we propose a new end-to-end deep neural network model for time-series classification (TSC) with emphasis on both the accuracy and the interpretation. The proposed model contains a convolutional network component to extract high-level features and a recurrent network component to enhance the modeling of the temporal characteristics of TS data. In addition, a feedforward fully connected network with the sparse group lasso (SGL) regularization is used to generate the final classification. The proposed architecture not only achieves satisfying classification accuracy, but also obtains good interpretability through the SGL regularization. All these networks are connected and jointly trained in an end-to-end framework, and it can be generally applied to TSC tasks across different domains without the efforts of feature engineering. Our experiments in various TS data sets show that the proposed model outperforms the traditional convolutional neural network model for the classification accuracy, and also demonstrate how the SGL contributes to a better model interpretation.

Time series classification is an important task with many challenging applications. A nearest neighbor (NN) classifier with dynamic time warping (DTW) distance is a strong solution in this context. On the other hand, feature-based approaches have been proposed as both classifiers and to provide insight into the series, but these approaches have problems handling translations and dilations in local patterns. Considering these shortcomings, we present a framework to classify time series based on a bag-of-features representation (TSBF). Multiple subsequences selected from random locations and of random lengths are partitioned into shorter intervals to capture the local information. Consequently, features computed from these subsequences measure properties at different locations and dilations when viewed from the original series. This provides a feature-based approach that can handle warping (although differently from DTW). Moreover, a supervised learner (that handles mixed data types, different units, etc.) integrates location information into a compact codebook through class probability estimates. Additionally, relevant global features can easily supplement the codebook. TSBF is compared to NN classifiers and other alternatives (bag-of-words strategies, sparse spatial sample kernels, shapelets). Our experimental results show that TSBF provides better results than competitive methods on benchmark datasets from the UCR time series database.

We propose a tree regularization framework, which enables many tree models to perform feature selection efficiently. The key idea of the regularization framework is to penalize selecting a new feature for splitting when its gain (e.g. information gain) is similar to the features used in previous splits. The regularization framework is applied on random forest and boosted trees here, and can be easily applied to other tree models. Experimental studies show that the regularized trees can select high-quality feature subsets with regard to both strong and weak classifiers. Because tree models can naturally deal with categorical and numerical variables, missing values, different scales between variables, interactions and nonlinearities etc., the tree regularization framework provides an effective and efficient feature selection solution for many practical problems.

Kernel principal component analysis (KPCA) is a method widely used for denoising multivariate data. Using geometric arguments, we investigate why a projection operation inherent to all existing KPCA denoising algorithms can sometimes cause very poor denoising. Based on this, we propose a modification to the projection operation that remedies this problem and can be incorporated into any of the existing KPCA algorithms. Using toy examples and real datasets, we show that the proposed algorithm can substantially improve denoising performance and is more robust to misspecification of an important tuning parameter.

Discretizing continuous attributes is necessary before association rules mining or using several inductive learning algorithms with a heterogeneous data space. This data preprocessing step should be carried out with a minimum information loss; that is the mutual information between attributes on the one hand and between attributes and the class labels on the other should not be destroyed. This paper introduces a novel supervised, global and dynamic discretization algorithm, called RFDisc (Random Forests Discretizer). It derives its ability in conserving the data properties from the Random Forests learning algorithm. RFDisc is simple, relatively fast and learns automatically the number of bins into which each continuous attribute is to be discretized. Empirical results indicate that the accuracies of classification algorithms such as CART when used with several data sets are comparable before and after discretization using RFDisc. Furthermore, C5.0 achieves the highest classification accuracy with data discretized with RFDisc when compared with other well known discretization algorithms.

MOTIVATION: Gene expression profiling technologies can generally produce mRNA abundance data for all genes in a genome. A dearth of proteomic data persists because identification range and sensitivity of proteomic measurements lag behind those of transcriptomic measurements. Using partial proteomic data, it is likely that integrative transcriptomic and proteomic analysis may introduce significant bias. Developing methodologies to accurately estimate missing proteomic data will allow better integration of transcriptomic and proteomic datasets and provide deeper insight into metabolic mechanisms underlying complex biological systems. RESULTS: In this study, we present a non-linear data-driven model to predict abundance for undetected proteins using two independent datasets of cognate transcriptomic and proteomic data collected from Desulfovibrio vulgaris. We use stochastic gradient boosted trees (GBT) to uncover possible non-linear relationships between transcriptomic and proteomic data, and to predict protein abundance for the proteins not experimentally detected based on relevant predictors such as mRNA abundance, cellular role, molecular weight, sequence length, protein length, guanine-cytosine (GC) content and triple codon counts. Initially, we constructed a GBT model using all possible variables to assess their relative importance and characterize the behavior of the predictive model. A strong plateau effect in the regions of high mRNA values and sparse data occurred in this model. Hence, we removed genes in those areas based on thresholds estimated from the partial dependency plots where this behavior was captured. At this stage, only the strongest predictors of protein abundance were retained to reduce the complexity of the GBT model. After removing genes in the plateau region, mRNA abundance, main cellular functional categories and few triple codon counts emerged as the top-ranked predictors of protein abundance. We then created a new tuned GBT model using the five most significant predictors. The construction of our non-linear model consists of a set of serial regression trees models with implicit strength in variable selection. The model provides variable relative importance measures using as a criterion mean square error. The results showed that coefficients of determination for our nonlinear models ranged from 0.393 to 0.582 in both datasets, providing better results than linear regression used in the past. We evaluated the validity of this non-linear model using biological information of operons, regulons and pathways, and the results demonstrated that the coefficients of variation of estimated protein abundance values within operons, regulons or pathways are indeed smaller than those for random groups of proteins. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

This paper proposes a new feature selection methodology. The methodology is based on the stepwise variable selection procedure, but, instead of using the traditional discriminant metrics such as Wilks' Lambda, it uses an estimation of the misclassification error as the figure of merit to evaluate the introduction of new features. The expected misclassification error rate (MER) is obtained by using the densities of a constructed function of random variables, which is the stochastic representation of the conditional distribution of the quadratic discriminant function estimate. The application of the proposed methodology results in significant savings of computational time in the estimation of classification error over the traditional simulation and cross-validation methods. One of the main advantages of the proposed method is that it provides a direct estimation of the expected misclassification error at the time of feature selection, which provides an immediate assessment of the benefits of introducing an additional feature into an inspection/classification algorithm.

Automated visual inspection (AVI) systems these days are considered essential in the assembly of surface-mounted devices (SMDs) in the electronics industry. This industry has faced the problem of rapid introduction and retirement of SMD-based products with the consequent obsolescence of the inspection systems already in the assembly lines. The constant introduction of new products has caused AVI systems to become rapidly obsolete. The general goal of this research centers on developing self-training AVI systems for the inspection of SMD components. The premise is that these systems would be less prone to obsolescence. In this paper, the authors describe the methodology being used for automatically selecting the features to inspect new components. In particular, this paper explores the use of multivariate stepwise discriminant analysis techniques, such as Wilks' Lambda, in order to automate the feature selection process. All of these techniques are applied to a case study of the inspection of SMD components. Note to Practitioners-In this paper, we present a methodology that would allow the automation of the tedious task of exploring and selecting features to develop algorithms for automated visual inspection systems. In particular, the proposed method selects a subset of features among all of the known features. The chosen subset seeks to minimize inspection errors while keeping the algorithmic development time to a minimum. This is particularly useful for adapting pre-existing systems to inspect new components, especially when the characteristics of the new components are similar to those of components already inspected by the inspection system. We applied this methodology to a case of study of the inspection of surface-mounted devices

A military air operations plan contains a number of sequential and/or parallel tasks. Given information about the battlefield situation (including a task status estimate) during the plan's execution, it is necessary to assess the situation impact on the plan, especially whether the entire plan would be delayed. Plan performance assessment fuses the task status estimate into the status estimate of the entire plan. The plan performance assessment assists the air operations commander in deciding the need for plan modification to assure the air operations goal. The performance assessment of the plan must deal with uncertainty in the task status estimate. This paper presents a comparative study of two information fusion techniques that infer the entire plan's status estimate from the task status estimate under uncertainty: a decomposition technique with low computational cost and a composition technique with high computational cost. The test results of these two techniques indicate their similar estimation accuracies. Due to its low computational cost, the decomposition technique is recommended. Guidelines for applying the decomposition technique to a large-scale complex task plan are also provided.

In this paper, we introduce the concept of fine-tuned learning which relies on the notion of data approximation followed by sequential data refinement. We seek to determine whether fine-tuned learning is a viable approach to use when trying to solve combinatorial optimization problems. In particular, we conduct an extensive computational experiment to study the performance of fine-tuned-learning-based heuristics for the traveling salesman problem (TSP). We provide important insight that reveals how fine-tuned learning works and why it works well, and conclude that it is a meritorious concept that deserves serious consideration by researchers solving difficult problems.