The prediction of molecular properties is a critical step in drug discovery campaigns. Computational methods such as graph neural networks (GNNs) and Transformers have effectively leveraged the small-range and long-range dependencies in molecules to preserve the local and global patterns for multiple molecular property prediction tasks. However, the dependence of these models on large amounts of experimental data poses a challenge, particularly on smaller biological datasets prevalent across the drug discovery pipeline. This paper introduces FS-GCvTR, a few-shot graph-based convolutional Transformer architecture designed to predict chemical properties with a small amount of labeled compounds. The convolutional Transformer is presented as a crucial component, effectively integrating both local and global dependencies of molecular graph embeddings by propagating a set of convolutional tokens across Transformer attention layers for molecular property prediction. Furthermore, a few-shot meta-learning approach is introduced to iteratively adapt model parameters across multiple few-shot tasks while generalizing to new chemical properties with limited available data. Experiments including few-shot evaluations on multi-property datasets show that the FS-GCvTR model outperformed other few-shot graph-based baselines in specific molecular property prediction tasks. • A few-shot GNN-Transformer, FS-GCvTR is proposed for molecular property discovery. • A Convolutional Transformer learns local and global information in graph embeddings. • A meta-learning approach adapts FS-GCvTR across tasks to predict molecular properties. • Experiments show that FS-GCvTR outperforms standard graph-based methods.

Due to the constant increase in cancer rates, the disease has become a leading cause of death worldwide, enhancing the need for its detection and treatment. In the era of personalized medicine, the main goal is to incorporate individual variability in order to choose more precisely which therapy and prevention strategies suit each person. However, predicting the sensitivity of tumors to anticancer treatments remains a challenge. In this work, we propose two deep neural network models to predict the impact of anticancer drugs in tumors through the half-maximal inhibitory concentration (IC50). These models join biological and chemical data to apprehend relevant features of the genetic profile and the drug compounds, respectively. In order to predict the drug response in cancer cell lines, this study employed different DL methods, resorting to Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs). In the first stage, two autoencoders were pre-trained with high-dimensional gene expression and mutation data of tumors. Afterward, this genetic background is transferred to the prediction models that return the IC50 value that portrays the potency of a substance in inhibiting a cancer cell line. When comparing RSEM Expected counts and TPM as methods for displaying gene expression data, RSEM has been shown to perform better in deep models and CNNs model can obtain better insight in these types of data. Moreover, the obtained results reflect the effectiveness of the extracted deep representations in the prediction of the IC50 value that portrays the potency of a substance in inhibiting a tumor, achieving a performance of a mean squared error of 1.06 and surpassing previous state-of-the-art models.

With the ongoing digitization of the manufacturing industry and the ability to bring together data from specific manufacturing processes, there is enormous potential to use machine learning (ML) techniques to improve such processes. In this context, the competitive automotive industry can take advantage of the ML power by predicting defects before they occur, aiming to reduce the scrap rate and increase the robustness and reliability of the production processes. In a real world scenario, small and medium size companies don’t have the amount of data the big companies have, which can prevent the usage of ML models in this vital niche for the industry. A collaboration in terms of data usage to develop powerful and general industry solutions is hindered by data privacy concerns despite similar problems. This paper addresses these concerns by providing a framework based on the Federated Learning (FL) method combined with Digital Envelopes (DE) to allow the ML models training while keeping the data of the partners and the models parameters private and protected against external cyber-attacks, which is one of the weaknesses of FL as of now. A case study was carried out to demonstrate the effectiveness of the proposed framework on handling data poisoning attacks to the training data and also the models’ weights.

The prediction of chemical toxicity and adverse side effects is a crucial task in drug discovery.Graph neural networks (GNNs) have accelerated the discovery of compounds with improved molecular profiles for effective drug development.Recently, Transformer networks have also managed to capture the long-range dependence in molecules to preserve the global aspects of molecular embeddings for molecular property prediction.In this paper, we propose a few-shot GNN-Transformer, FS-GNNCvTR to face the challenge of low-data toxicity and side effect prediction.Specifically, we introduce a convolutional Transformer to model the local spatial context of molecular graph embeddings while preserving the global information of deep representations.Furthermore, a two-module meta-learning framework is proposed to iteratively update model parameters across fewshot tasks with limited available data.Experiments on small-sized biological datasets for toxicity and side effect prediction, Tox21 and SIDER, demonstrate a superior performance of FS-GNNCvTR compared to standard graph-based methods.The code and data underlying this article are available in the repository, https://github.com/larngroup/FS-GNNCvTR.

This paper focuses on the effects of applying quantization during training to Recurrent Neural Networks (RNNs) used in Simplified Molecular-Input Line-Entry System (SMILES) generation, a form of line notation for molecular information used in the development of pharmaceutical drugs, from the PubChem database. It offers the flexibility to choose the precision used by the model, by defining the number of bits at each layer. The RNNs are the focus of the current study, by comparing the performance of three of the most used algorithms, Simple RNN, Long Short-Term Memory (LSTM), and Gated Recurrent Unit (GRU). The models were trained on a selection of SMILES. By exploiting the QK-eras library, quantization performance was compared to their floating-point equivalent for several combinations of parameters. The goal of the testing program developed is to generate a large number of novel SMILES, facilitating the process of Drug Discovery which is traditionally long, and thus very expensive and difficult. By understanding how the behavior of quantized networks deviates from the regular model, in relation to the parameters used, we are able to control the process of choosing whether to quantize a model and to which degree it becomes more or less efficient. In this study, we observed good performance even for 4-bit models making use of LSTM and GRU layers, the same way we concluded that Simple RNN quantization does not compensate the effort.

Molecular property prediction is an essential task in drug discovery. Recently, deep neural networks have accelerated the discovery of compounds with improved molecular profiles for effective drug development. In particular, graph neural networks (GNNs) have played a pivotal role in identifying promising drug candidates with desirable molecular properties. However, it is common for only a few molecules to share the same set of properties, which presents a low-data problem unanswered by regular machine learning (ML) approaches. Transformer networks have also emerged as a promising solution to model the long-range dependence in molecular embeddings and achieve encouraging results across a wide range of molecular property prediction tasks. Nonetheless, these methods still require a large number of data points per task to achieve acceptable performance. In this study, we propose a few-shot GNN-Transformer architecture, FS-GNNTR to face the challenge of low-data in molecular property prediction. The proposed model accepts molecules in the form of molecular graphs to model the local spatial context of molecular graph embeddings while preserving the global information of deep representations. Furthermore, we introduce a two-module meta-learning framework to iteratively update model parameters across few-shot tasks and predict new molecular properties with limited available data. Finally, we conduct multiple experiments on small-sized biological datasets for molecular property prediction, Tox21 and SIDER, and our results demonstrate the superior performance of FS-GNNTR compared to simpler graph-based baselines. The code and data underlying this article are available in the repository, https://github.com/ltorres97/FS-GNNTR.

Abstract Graph neural networks and convolutional architectures have proven to be pivotal in improving the prediction of molecular properties in drug discovery. However, this is fundamentally a low data problem that is incompatible with regular deep learning approaches. Contemporary deep networks require large amounts of training data, which severely limits the prediction of new molecular entities from limited available data. In this paper, we address the challenge of low data in molecular property prediction by: (1) defining a set of deep learning architectures that accept compound chemical structures in the form of molecular graphs, (2) creating a few-shot learning strategy across graph neural networks and convolutional neural networks to leverage the rich information of graph embeddings, and (3) proposing a two-module meta-learning framework to learn from task-transferable knowledge and predict molecular properties on few-shot data. Furthermore, we conduct multiple experiments on two benchmark multiproperty datasets to demonstrate a superior performance over conventional graph-based baselines. ROC-AUC results for 10-shot experiments show an average improvement of $$+11.37\%$$ + 11.37 % on Tox21 and $$+0.53\%$$ + 0.53 % on SIDER, which are representative small-sized biological datasets for molecular property prediction.

Cancer is among the deadliest diseases, enhancing the need for its detection and treatment. In the era of precision medicine, the main goal is to take into account individual vari-ability in order to choose more accurately which treatment and prevention strategies suit each person. However, drug response prediction for cancer therapy remains a challenge. In this work, we propose a deep neural network model to predict the effect of anticancer drugs in tumors through the half-maximal inhibitory concentration (IC50). The model can be seen as two-fold: first, we pre-trained two autoencoders with high-dimensional gene expression and mutation data to capture the crucial features from tumors; then, this genetic background is translated to cancer cell lines to predict the impact of the genetic variants on a given drug. Moreover, SMILES structures were introduced so that the model can apprehend relevant features regarding the drug compound. Finally, we use drug sensitivity data correlated to the genomic and drugs data to identify features that predict the IC50 value for each pair of drug-cell line. The obtained results demonstrate the effectiveness of the extracted deep representations in the prediction of drug-target interactions, achieving a performance of a mean squared error of 1.07 and surpassing previous state-of-the-art models.

In drug discovery, Deep Learning algorithms are emerging as a potential method to generate novel chemical structures since they can speed up the traditional process and decrease expenditure. Recurrent architectures are amongst the most promising methods for computational de novo drug design. One current challenge consists in finding the optimal architecture and parameters for the recurrent network that assures the generation of valid molecules that span the chemical space. In this work we perform an evaluation on Recurrent Neural Networks which can learn the syntax of molecular representation in terms of SMILES notation. We optimize the computational framework based on the recurrent architecture and its hyper-parameters. Moreover, we evaluate the performance of two types of encoding and spatial arrangement of molecules: Embedding and One-hot Encoding, and datasets with and without stereo-chemical information, respectively. The proposed model showed improved performance when compared to the current literature, both in terms of percentage of valid generated SMILES and diversity with 98.7% and 0.88, for the ChEMBL dataset, respectively. Even when considering the ZINC biogenic library, with stereochemical information, the values were 94.5% and 0.90. The obtained results reveal the potential of the recurrent architectures in learning the SMILES syntax and adding novelty to generate promising compounds.

Machine learning has been increasingly applied with success in generating synthetically reasonable molecules.However, a complete system capable of both producing valid molecules and optimizing multiple traits has remained elusive.This paper employs multiobjective reinforcement learning to draw a framework to design compounds.Different multiobjective techniques have been evaluated, such as weighted sum and Chebyshev.The results show that the implemented model can be effectively optimized towards different and competing molecular properties.Nonetheless, the model implemented with the weighted sum scalarization technique with a weight of 0.55 for biological affinity is the one with the most appropriate trade-off for the different evaluated properties.

We propose two novel Federated Learning (FL) algorithms based on decaying momentum (Demon): Federated Demon (FedDemon) and Federated Demon Adam (FedDemonAdam).In particular, we apply Demon to Momentum Stochastic Gradient Descent (SGD) and Adam in a Federated setting, which has shown to improve results in a centralized environment.We empirically show that FedDemon and FedDemonAdam have a faster convergence rate and performance improvements compared to state-of-the-art algorithms including FedAvg, FedAvgM and FedAdam.17

This paper provides a generative network framework that can replicate the molecular space distribution to satisfy a set of desirable features.The approach incorporates two effective machine learning techniques: an Encoder-Decoder architecture that converts the string notations of molecules into latent space and a generative adversarial network to learn the data distribution and generate new compounds.We train this joint model on a dataset that includes stereo-chemical information.The results show an improvement in the Encoder-Decoder performance, reaching 89% of correctly reconstructed molecules.The framework can generate a wide variety of compounds biased towards specific molecular properties using Transfer Learning.

MOTIVATION: The process of placing new drugs into the market is time-consuming, expensive and complex. The application of computational methods for designing molecules with bespoke properties can contribute to saving resources throughout this process. However, the fundamental properties to be optimized are often not considered or conflicting with each other. In this work, we propose a novel approach to consider both the biological property and the bioavailability of compounds through a deep reinforcement learning framework for the targeted generation of compounds. We aim to obtain a promising set of selective compounds for the adenosine A2A receptor and, simultaneously, that have the necessary properties in terms of solubility and permeability across the blood-brain barrier to reach the site of action. The cornerstone of the framework is based on a recurrent neural network architecture, the Generator. It seeks to learn the building rules of valid molecules to sample new compounds further. Also, two Predictors are trained to estimate the properties of interest of the new molecules. Finally, the fine-tuning of the Generator was performed with reinforcement learning, integrated with multi-objective optimization and exploratory techniques to ensure that the Generator is adequately biased. RESULTS: The biased Generator can generate an interesting set of molecules, with approximately 85% having the two fundamental properties biased as desired. Thus, this approach has transformed a general molecule generator into a model focused on optimizing specific objectives. Furthermore, the molecules' synthesizability and drug-likeness demonstrate the potential applicability of the de novo drug design in medicinal chemistry. AVAILABILITY AND IMPLEMENTATION: All code is publicly available in the https://github.com/larngroup/De-Novo-Drug-Design. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

The discovery of potential Drug-Target Interactions (DTIs) is a determining step in the drug discovery and repositioning process, as the effectiveness of the currently available antibiotic treatment is declining. Although putting efforts on the traditional in vivo or in vitro methods, pharmaceutical financial investment has been reduced over the years. Therefore, establishing effective computational methods is decisive to find new leads in a reasonable amount of time. Successful approaches have been presented to solve this problem but seldom protein sequences and structured data are used together. In this paper, we present a deep learning architecture model, which exploits the particular ability of Convolutional Neural Networks (CNNs) to obtain 1D representations from protein sequences (amino acid sequence) and compounds SMILES (Simplified Molecular Input Line Entry System) strings. These representations can be interpreted as features that express local dependencies or patterns that can then be used in a Fully Connected Neural Network (FCNN), acting as a binary classifier. The results achieved demonstrate that using CNNs to obtain representations of the data, instead of the traditional descriptors, lead to improved performance. The proposed end-to-end deep learning method outperformed traditional machine learning approaches in the correct classification of both positive and negative interactions.

The application of deep neural networks in drug discovery is mainly due to their enormous potential to significantly increase the predictive power when inferring the properties and activities of small-molecules. However, in the traditional drug discovery process, where supervised data is scarce, the lead-optimization step is a low-data problem, making it difficult to find molecules with the desired therapeutic activity and obtain accurate predictions for candidate compounds. One major requirement to ensure the validity of the obtained neural network models is the need for a large number of training examples per class, which is not always feasible in drug discovery applications. This invalidates the use of instances whose classes were not considered in the training phase or in data where the number of classes is high and oscillates dynamically. The main objective of the study is to optimize the discovery of novel compounds based on a reduced set of candidate drugs. We propose a Siamese neural network architecture for one-shot classification, based on Convolutional Neural Networks (CNNs), that learns from a similarity score between two input molecules according to a given similarity function. Using a one-shot learning strategy, few instances per class are needed for training, and a small amount of data and computational resources are required to build an accurate model. The results achieved demonstrate that using a Siamese Deep Neural Network for one-shot classification leads to overall improved performance when compared to other state-of the-art models. The proposed architecture provides an accurate and reliable prediction of novel compounds considering the lack of biological data available for drug discovery tasks.

Concept Blending is one of the most prominent computational approaches to study and understand the underlying processes related to creativity. In this article, we show how to use the Regulated Activation Network (RAN) cognitive model to reconstruct abstract concepts and their blends. The MNIST dataset is used in this work to build a representation of abstract concepts. For the demonstration, three experiments were designed: first, shows how a high dimensional input image is encoded into a low dimension vector and further reconstructed back into an image; second, reconstruction of blends of abstract concepts that represent same digits; third, reconstructing blends of abstract concepts which represent different digits. The reconstructed images in all three experiments were visually analyzed. The best reconstructions were observed with the encoded image experiment obtaining Mean Squared Error of 0.00562 and an Rsquare score of 0.9193. The blends of similar abstract concepts also reconstructed the expected blend of a digit. The blends of dissimilar abstract concepts reconstructed the images by creating interesting symbols such as character x.

In the context of expensive and time-consuming acquisition of reliably labeled data, how to utilize the unlabeled instances that can potentially improve the classification accuracy becomes an attractive problem with significant importance in practice. Semi-supervised classification that fills the g ap between supervised learning and unsupervised learning is designed to take advantage of the unlabeled data in regular supervised learning procedure for classification tasks. In this paper we proposed a self-learning framework, that firstly pre-learns a classification model using the labeled data, then makes the prediction of unlabeled instances in the form of soft class labels, and re-learned a model based on the enlarged training data. Two multi-label Learning Vector Quantization Neural Networks (LVQ-NNs) are proposed, namely multi-label online LVQ-NN (mLVQo) and multi-label batch LVQ-NN (mLVQb), to work with the soft labels of training instances. The experiments demonstrate that the semi-supervised models using multi-label LVQ-NN as the base classifier can produce better generalization accuracy than the supervised counterpart.

Developing learning systems is challenging in many ways: often there is the need to optimise the learning algorithm structure and parameters, and it is necessary to decide which is the best data representation to use, i.e., we usually have to design features and select the most representative and useful ones. In this work we focus on the later and investigate whether or not it is possible to obtain good performances with compressed versions of the original data, possibly reducing the learning time. The process of compressing the data, i.e., reducing its dimensionality, is typically conducted by someone who has domain knowledge and expertise, and engineers features in a trial-and-error endless cycle. Our goal is to achieve such compressed versions automatically; for that, we use an Evolutionary Algorithm to generate the structure of AutoEncoders. Instead of targeting the reconstruction of the images, we focus on the reconstruction of the mean signal of each class, and therefore the goal is to acquire the most representative characteristics of each class. Results on the MNIST dataset show that the proposed approach can not only reduce the original dataset dimensionality, but the performance of the classifiers over the compressed representation is superior to the performance on the original uncompressed images.

Urban environments are systems of overwhelming complexity and dynamism, involving numerous variables and idiosyncrasies which not usually easy to model from a functional perspective. Simulation modeling is a common and well-accepted approach to study such systems, specially those that prove to be too complex to be analyzed by standard analytic methods. However, such urban simulation models can become computationally very expensive to run. To address this drawback, simulation metamodels can be employed to approximate the underlying simulation function. In this paper, we propose a batch-mode active learning strategy based on Gaussian Processes metamodeling that searches for the most informative data points in batches with respect to their corresponding predictive variances. These points are selected in such a way that they originate from different high variance neighborhoods. Eventually, this allows us to analyze the simulation output behavior with fewer simulation requests. Using an illustrative traffic simulation example, the results show that the proposed restricted batch-mode strategy is able to increase the simulation input space exploration efficiency in comparison with standard batch-mode strategies.

Current challenges in machine learning include dealing with temporal data streams, drift and non-stationary scenarios, often with text data, whether in social networks or in business systems. This dynamic nature tends to limit the performance of traditional static learning models and dynamic learning strategies must be put forward. However, acquiring the performance of those strategies is not a straightforward issue, as sample's dependency undermines the use of validation techniques, like crossvalidation. In this paper we propose to use the McNemar's test to compare two distinct approaches that tackle adaptive learning in dynamic environments, namely DARK (Drift Adaptive Retain Knowledge) and Learn++. NSE (Learn++ for Non-Stationary Environments). The validation is based on a Twitter case study benchmark constructed using the DOTS (Drift Oriented Tool System) dataset generator. The results obtained demonstrate the usefulness and adequacy of using McNemar's statistical test in dynamic environments where time is crucial for the learning algorithm.

Any cognizable thing (object, or idea) can be recognized as a concept, and they are usually described by modality-specific, experience-dependent, or localist-distributed representations. There has been a significant effort in studying concrete concepts, along with their underlying cognitive and psychological processes through conceptual representations produced by computational approaches (like Artificial Neural Networks). However, the notion of abstract concepts, though debated, is seldom explored. In this article, we propose an approach to learn abstract representation of concepts with Regulated Activation Networks (RANs) evolving computational modeling. RANs methodology is illustrated using a toydata problem, and evaluated for metrics Precision, Recall, Fl-score, and Accuracy, along with ROC curve analysis. RANs comparison is presented using a benchmark data from human activity recognition domain, yielding Precision= 98.20% (ca.), Recall= 97.62% (ca.), Fl-score= 97.8 (ca.), and Accuracy= 97.62% (ca.). Sleep Detection data from SOCIALITE project is used to model active and inactive students, showing a convincing performance. Further, the model is used in studying 3 students in order to deduce biomarkers for understanding their psychological conditions.

Vertical Ground Reaction Forces (VGRF) reference profiles of human gait are important in medicine, for the recognition of gait disorders and in the assessment of rehabilitation treatments. A walking human is a dynamic varying system and in spite of the VGRF reference dependence on several patient's variables, doctors traditionally use the same static reference for all patients. The purpose of this study is to find out if an Extreme Learning Machine (ELM) is adequate to generate the dynamic VGRF reference profiles of healthy people depending on the subject's age, weight, height and stride duration. The ELM is compared with two other baseline Computational Intelligence (CI) methods, the Backpropagation Neural Network (BNN) and Multioutput Support Vector Regression (MSVR). Data from 28 healthy males walking at five different stride durations, collected using instrumented shoes, were used to train and test the CI models. The results showed that ELM is a well suited method to generate the dynamic VGRF reference profile for both dominant and non-dominant limbs, showing the lowest root mean square errors for the test set, 0.0201 and 0.0243 (fraction of body weight) for the dominant and non-dominant limbs respectively. This study reveals a promising methodology that can be implemented in real time gait analysis, allowing doctors to find a specific reference gait pattern for the gait analysis of an unhealthy person by specifying the age, weight, height and stride duration.

Simulation modeling is a well-known and recurrent approach to study the performance of urban systems. Taking into account the recent and continuous transformations within increasingly complex and multidimensional cities, the use of simulation tools is, in many cases, the only feasible and reliable approach to analyze such dynamic systems. However, simulation models can become very time consuming when detailed input-space exploration is needed. To tackle this problem, simulation metamodels are often used to approximate the simulators' results. In this paper, we propose an active learning algorithm based on the Gaussian process (GP) framework that gathers the most informative simulation data points in batches, according to both their predictive variances and to the relative distance between them. This allows us to explore the simulators' input space with fewer data points and in parallel, and thus in a more efficient way, while avoiding computationally expensive simulation runs in the process. We take advantage of the closeness notion encoded into the GP to select batches of points in such a way that they do not belong to the same high-variance neighborhoods. In addition, we also suggest two simple and practical user-defined stopping criteria so that the iterative learning procedure can be fully automated. We illustrate this methodology using three experimental settings. The results show that the proposed methodology is able to improve the exploration efficiency of the simulation input space in comparison with non-restricted batch-mode active learning procedures.

The effectiveness of Artificial Neural Networks (ANNs) depends on a non-trivial manual crafting of their topology and parameters. Typically, practitioners resort to a time consuming methodology of trial-and-error to find and/or adjust the models to solve specific tasks. To minimise this burden one might resort to algorithms for the automatic selection of the most appropriate properties of a given ANN. A remarkable example of such methodologies is Grammar-based Genetic Programming. This work analyses and compares the use of two grammar-based methods, Grammatical Evolution (GE) and Structured Grammatical Evolution (SGE), to automatically design and configure ANNs. The evolved networks are used to tackle several classification datasets. Experimental results show that SGE is able to automatically build better models than GE, and that are competitive with the state of the art, outperforming hand-designed ANNs in all the used benchmarks.

Big Data has been a catalyst force for the Machine Learning (ML) area, forcing us to rethink existing strategies in order to create innovative solutions that will push forward the field. This paper presents an overview of the strategies for using machine learning in Big Data with emphasis on the high-performance parallel implementations on many-core hardware. The rationale is to increase the practical applicability of ML implementations to large-scale data problems. The common underlying thread has been the recent progress in usability, cost effectiveness and diversity of parallel computing platforms, specifically, the Graphics Processing Units (GPUs), tailored for a broad set of data analysis and Machine Learning tasks. In this context, we provide the main outcomes of a GPU Machine Learning Library (GPUMLib) framework, which empowers researchers with the capacity to tackle larger and more complex problems, by using high-performance implementations of wellknown ML algorithms. Moreover, we attempt to give insights on the future trends of Big Data Analytics and the challenges lying ahead.

Public special events, like sports games, concerts and festivals are well known to create disruptions in transportation systems, often catching the operators by surprise. Although these are usually planned well in advance, their impact is difficult to predict, even when organisers and transportation operators coordinate. The problem highly increases when several events happen concurrently. To solve these problems, costly processes, heavily reliant on manual search and personal experience, are usual practice in large cities like Singapore, London or Tokyo. This paper presents a Bayesian additive model with Gaussian process components that combines smart card records from public transport with context information about events that is continuously mined from the Web. We develop an efficient approximate inference algorithm using expectation propagation, which allows us to predict the total number of public transportation trips to the special event areas, thereby contributing to a more adaptive transportation system. Furthermore, for multiple concurrent event scenarios, the proposed algorithm is able to disaggregate gross trip counts into their most likely components related to specific events and routine behavior. Using real data from Singapore, we show that the presented model outperforms the best baseline model by up to 26 percent in R2 and also has explanatory power for its individual components.

The growing need to analyze large collections of documents has led to great developments in topic modeling. Since documents are frequently associated with other related variables, such as labels or ratings, much interest has been placed on supervised topic models. However, the nature of most annotation tasks, prone to ambiguity and noise, often with high volumes of documents, deem learning under a single-annotator assumption unrealistic or unpractical for most real-world applications. In this article, we propose two supervised topic models, one for classification and another for regression problems, which account for the heterogeneity and biases among different annotators that are encountered in practice when learning from crowds. We develop an efficient stochastic variational inference algorithm that is able to scale to very large datasets, and we empirically demonstrate the advantages of the proposed model over state-of-the-art approaches.

In real-world applications, the assumption of independent and identical distribution is no longer consistent. To alleviate the significant mismatch between source and target domains, importance weighting import vector machine, which is an adaptive classifier, is proposed. This adaptive probabilistic classification method, which is sparse and computationally efficient, can be used for unsupervised domain adaptation (DA). The effectiveness of the proposed approach is demonstrated via a toy problem, and a real-world cross-domain object recognition task. Even though the sparseness, the proposed method outperforms the state-of-the-art in both unsupervised and semisupervised DA scenarios. We also introduce a reliable importance weighted cross validation (RIWCV), which is an improvement of importance weighted cross validation, for parameter and model selection. The RIWCV avoid falling down in local minimum, by selecting a more reliable combination of the parameters instead of the best parameters.

Brand recognition is a very challenging topic with many useful applications in localization recognition, advertisement and marketing. In this paper we present an automatic graphic logo detection system that robustly handles unconstrained imaging conditions. Our approach is based on Fast Region-based Convolutional Networks (FRCN) proposed by Ross Girshick, which have shown state-of-the-art performance in several generic object recognition tasks (PASCAL Visual Object Classes challenges). In particular, we use two CNN models pretrained with the ILSVRC ImageNet dataset and we look at the selective search of windows `proposals' in the pre-processing stage and data augmentation to enhance the logo recognition rate. The novelty lies in the use of transfer learning to leverage powerful Convolutional Neural Network models trained with large-scale datasets and repurpose them in the context of graphic logo detection. Another benefit of this framework is that it allows for multiple detections of graphic logos using regions that are likely to have an object. Experimental results with the FlickrLogos-32 dataset show not only the promising performance of our developed models with respect to noise and other transformations a graphic logo can be subject to, but also its superiority over state-of-the-art systems with hand-crafted models and features.

With the massive data challenges nowadays and the rapid growing of technology, stream mining has recently received considerable attention. To address the large number of scenarios in which this phenomenon manifests itself suitable tools are required in various research fields. Instance-based data stream algorithms generally employ the Euclidean distance for the classification task underlying this problem. A novel way to look into this issue is to take advantage of a more flexible metric due to the increased requirements imposed by the data stream scenario. In this paper we present a new algorithm that learns a Mahalanobis metric using similarity and dissimilarity constraints in an online manner. This approach hybridizes a Mahalanobis distance metric learning algorithm and a k-NN data stream classification algorithm with concept drift detection. First, some basic aspects of Mahalanobis distance metric learning are described taking into account key properties as well as online distance metric learning algorithms. Second, we implement specific evaluation methodologies and comparative metrics such as Q statistic for data stream classification algorithms. Finally, our algorithm is evaluated on different datasets by comparing its results with one of the best instance-based data stream classification algorithm of the state of the art. The results demonstrate that our proposal is better in some scenarios and has shown to be competitive in others.

The growing need to analyze large collections of documents has led to great developments in topic modeling. Since documents are frequently associated with other related variables, such as labels or ratings, much interest has been placed on supervised topic models. However, the nature of most annotation tasks, prone to ambiguity and noise, often with high volumes of documents, deem learning under a single-annotator assumption unrealistic or unpractical for most real-world applications. In this paper, we propose a supervised topic model that accounts for the heterogeneity and biases among different annotators that are encountered in practice when learning from crowds. We develop an efficient stochastic variational inference algorithm that is able to scale to very large datasets, and we empirically demonstrate the advantages of the proposed model over state of the art approaches.

Social networks are making part of the daily routine of millions of users. Twitter is among Facebook and Instagram one of the most used, and can be seen as a relevant source of information as users share not only daily status, but rapidly propagate news and events that occur worldwide. Considering the dynamic nature of social networks, and their potential in information spread, it is imperative to find learning strategies able to learn in these environments and cope with their dynamic nature. Time plays an important role by easily out-dating information, being crucial to understand how informative can past events be to current learning models and for how long it is relevant to store previously seen information, to avoid the computation burden associated with the amount of data produced. In this paper we study the impact of longstanding messages in micro-blogging classification by using different training time-window sizes in the learning process. Since there are few studies dealing with drift in Twitter and thus little is known about the types of drift that may occur, we simulate different types of drift in an artificial dataset to evaluate and validate our strategy. Results shed light on the relevance of previously seen examples according to different types of drift.

Handwritten signature recognition is one important component of biometric authentication. This is a central process in a broad range of areas requiring personal identification, such as security, legal contracts and bank transactions. Extensive efforts have been put into the research towards the verification of handwritten signatures, which contain biometric information. Although many successful methods have been used, they often disregard the size of databases, which can be very large, posing scalability problems to their application in real-world scenarios. To overcome this problem, in this paper, we use binary embeddings of high-dimensional data which is an efficient tool for indexing big datasets of biometric images. The rationale is to find a good hash function such that similar data points in Euclidean space preserve their similarities in the resulting Hamming space for fast data retrieval and state-of-the-art classification performance. In the settings of an handwritten signature retrieval system, an indexing hashing-based scheme is presented. We propose to learn k-bits hash code with a generalised regression neural network (GRNN), which yielded competitive results in the GPDS database.

Ensemble is a recently emerged computing technique to provide promising decisions by a consensus of multiple classifiers. The benefit of classifier ensembles has been demonstrated in a vast number of studies in the scope of credit risk management. Ye

Learning from multiple annotators took a valuable step towards modelling data that does not fit the usual single annotator setting. However, multiple annotators sometimes offer varying degrees of expertise. When disagreements arise, the establishment of the correct label through trivial solutions such as majority voting may not be adequate, since without considering heterogeneity in the annotators, we risk generating a flawed model. In this paper, we extend GP classification in order to account for multiple annotators with different levels expertise. By explicitly handling uncertainty, Gaussian processes (GPs) provide a natural framework to build proper multiple-annotator models. We empirically show that our model significantly outperforms other commonly used approaches, such as majority voting, without a significant increase in the computational cost of approximate Bayesian inference. Furthermore, an active learning methodology is proposed, which is able to reduce annotation cost even further.

Learning in non-stationary environments is not an easy task and requires a distinctive approach. The learning model must not only have the ability to continuously learn, but also the ability to acquired new concepts and forget the old ones. Additionally, given the significant importance that social networks gained as information networks, there is an ever-growing interest in the extraction of complex information used for trend detection, promoting services or market sensing. This dynamic nature tends to limit the performance of traditional static learning models and dynamic learning strategies must be put forward. In this paper we present a learning strategy to learn with drift in the occurrence of concepts in Twitter. We propose three different models: a time-window model, an ensemble-based model and an incremental model. Since little is known about the types of drift that can occur in Twitter, we simulate different types of drift by artificially time stamping real Twitter messages in order to evaluate and validate our strategy. Results are so far encouraging regarding learning in the presence of drift, along with classifying messages in Twitter streams.

The performance of household electrical seasonal consumption disaggregation is explored in this paper. Firstly, given a tensor composed by the data for the several devices in the house, non-negative tensor factorization is performed in order to extract the most relevant components. Secondly, the outcome is embedded in the test step, where only the whole-home measured consumption is available. Lastly, the disaggregated data by device is obtained by factorizing the associated matrix regarding the learned model. This source separation approach thus requires prior data, needed to learn the source models. Nevertheless, the consumer behaviors vary along time particularly from season to season, and hence also the electrical consumption. Consequently, the assessment of performance at long-term and across different times of the year is essential. We evaluate the performance of load disaggregation by this supervised method along several years and across seasons. Towards this end, computational experiments were yielded using real-world data from a household electrical consumption measurements along several years. The analysis of the computational results illustrates the adequacy of the method for handling the shifts between seasons.

Automatic keyword extraction (AKE) from textual sources took a valuable step towards harnessing the problem of efficient scanning of large document collections. Particularly in the context of urban mobility, where the most relevant events in the city are advertised on-line, it becomes difficult to know exactly what is happening in a place. In this paper we tackle this problem by extracting a set of keywords from different kinds of textual sources, focusing on the urban events context. We propose an ensemble of automatic keyword extraction systems KEA (Keyphrase Extraction Algorithm) and KUSCO (Knowledge Unsupervised Search for instantiating Concepts on lightweight Ontologies) and Conditional Random Fields (CRF). Unlike KEA and KUSCO which are well-known tools for automatic keyword extraction, CRF needs further preprocessing. Therefore, a tool for handling AKE from the documents using CRF is developed. The architecture for the AKE ensemble system is designed and efficient integration of component applications is achieved. Finally, we empirically show that our AKE ensemble system significantly succeeds on baseline sources and urban events collections.

The problem of handwritten signature recognition is considered significant in biometrics, in particular for determining the validity of official documents. The rationale consists of creating an off-line classifier to discriminate between fake (forged) and genuine digitalized signatures. In such applications containing thousands of samples machine learning techniques such as Support Vector Machines (SVM) play a preponderant role in overcoming the challenges inherent to this problematic. However, to deal with the computational burden of calculating the large Gram matrix, approaches such as Graphics Processing Units (GPU) computing are required for efficiently processing big image biometric data. In this paper, first, we present an empirical study for efficient feature selection concerning the signature identification problem. Second, an GPU-based SVM classifier that integrates a component of the open source Machine Learning Library (GPUMLib) supporting several kernels is developed. Third, we ran several experiments with improved performance over baseline approaches. From our study, we gain insights in both performance and computational cost under a number of experimental conditions, and conclude that the most appropriate model is usually a trade-off between performance and computational cost for a given experimental setup and dataset.

Skewed class distribution and non-uniform misclassification cost are pervasive in many real-world domains such as bankruptcy prediction, medical diagnosis, and intrusion detection. Although class imbalance learning and cost-sensitive learning can be

The cascade classifier is a usual approach in object detection based on vision, since it successively rejects negative occurrences, e.g., background images, in a cascade structure, keeping the processing time suitable for on-the-fly applications. On the other hand, similar to other classifier ensembles, cascade classifiers are likely to have high Vapnik-Chervonenkis (VC) dimension, which may lead to overfitting the training data. Therefore, this work aims at improving the generalization capacity of the cascade classifier by controlling its complexity, which depends on the model of their classifier stages, the number of stages, and the feature space dimension of each stage, which can be controlled by integrating the parameter setting of the feature extractor (in our case an image descriptor) into the maximum-margin framework of support vector machine training, as will be shown in this paper. Moreover, to set the number of cascade stages, bounds on the false positive rate (FP) and on the true positive rate (TP) of cascade classifiers are derived based on a VC-style analysis. These bounds are applied to compose an enveloping receiver operating curve (EROC), i.e., a new curve in the TP–FP space in which each point is an ordered pair of upper bound on the FP and lower bound on the TP. The optimal number of cascade stages is forecasted by comparing EROCs of cascades with different numbers of stages.

Deep learning architecture models by contrast with shallow models draw on the insights of biological inspiration which has been a challenge since the inception of the idea of simulating the brain. In particular their (many) hierarchical levels of composition track the development of parallel implementation in an attempt to become accessibly fast. When it comes to performance enhancement Graphics Processing Units (GPU) have carved their own strength in machine learning. In this paper, we present an approach that relies mainly on three kernels for implementing both the Restricted Boltzmann Machines (RBM) and Deep Belief Networks (DBN) algorithms. Instead of considering the neuron as the smallest unit of computation each thread represents the connection between two (one visible and one hidden) neurons. Although conceptually it may seem weird, the rationale behind is to think of a connection as performing a simple function that multiplies the clamped input by its weight. Thus, we maximize the GPU workload avoiding idle cores. Moreover, we placed great emphasis on the kernels to avoid uncoalesced memory accesses as well as to take advantage of the shared memory to reduce global memory accesses. Additionally, our approach uses a step adaptive learning rate procedure which accelerates convergence. The approach yields very good speedups (up to 46×) as compared with a straightforward implementation when both GPU and CPU implementations are tested on the MINST database.

Non-negative Matrix Factorization (NMF) is an unsupervised technique that projects data into lower dimensional spaces, effectively reducing the number of features of a dataset while retaining the basis information necessary to reconstruct the original data. In this paper we present a semi-supervised NMF approach that reduces the computational cost while improving the accuracy of NMF-based models. The advantages inherent to the proposed method are supported by the results obtained in two well-known face recognition benchmarks.

Many dimensionality reduction algorithms have been proposed easing both tasks of visualization and classification in high dimension problems. Despite the different motivations they can be cast in a graph embedding framework. In this paper we address weighted graph subspace learning methods for bankruptcy analysis. The rationale behind re-embedding the data in a lower dimensional space that would be better filled is twofold: to get the most compact representation (visualization) and to make subsequent processing of data more easy (classification). The approaches used, Graph regularized Non-Negative Matrix Factorization (GNMF) and Spatially Smooth Subspace Learning (SSSL), construct an affinity weight graph matrix to encode geometrical information and to learn in the training set the subspace models that enhance visualization and are able to ease the task of bankruptcy prediction. The experimental results on a real problem of French companies show that from the perspective of financial problem analysis the methodology is quite effective.

Crowdsourcing is an emergent trend for general-purpose classification problem solving. Over the past decade, this notion has been embodied by enlisting a crowd of humans to help solve problems. There are a growing number of real-world problems that take advantage of this technique, such as Wikipedia, Linux or Amazon Mechanical Turk. In this paper, we evaluate its suitability for classification, namely if it can outperform state-of-the-art models by combining it with active learning techniques. We propose two approaches based on crowdsourcing and active learning and empirically evaluate the performance of a baseline Support Vector Machine when active learning examples are chosen and made available for classification to a crowd in a web-based scenario. The proposed crowdsourcing active learning approach was tested with Jester data set, a text humour classification benchmark, resulting in promising improvements over baseline results.

Emerging trends for energy monitoring as in Smart Energy Systems require intelligent solutions for appliances identification. Non-intrusive load monitoring (NILM) systems are able to extract particular features from the aggregate consumption of the electrical network. However, the whole-consumption signal is contaminated with noise, which hinders successful load disambiguation of individual appliances. In this work, we propose a novel approach to denoise a signal based on the techniques of Embedding, Wavelet Shrinkage and Diagonal Averaging. The embedding stage transforms the one-dimensional signal into a sequence of lagged vectors. These vectors are denoised using wavelet decomposition. Finally, the denoised signal is obtained by taking the diagonal averages of the resultant matrix. Our approach is compared to Wavelet Decomposition and Singular Spectrum Analysis methods for electrical signal denoising. The results are very favorable since they yield better performance as highlighted by the statistical tests performed.

The aggregated electrical load of a household network contains relevant information. Specifically, which loads are related to electrical appliances switched on. However, when real-world data is at stake, not only this specific data must be individually recognized, but also there is other non-relevant information that can be thought as noise in the electrical signal. Therefore, to extract the important information we need to use signal denoising algorithms. This work presents a comparison for the application of an algorithm based on Wavelet Decomposition versus the Singular Spectrum Analysis to the denoise of aggregated electrical signal. These techniques were applied both in an artificially generated signal as well as to the analysis of a signal obtained from an ordinary household. For the latter, the experiments highlighted a small set of wavelet functions that are more suitable for the problem being tackled. Finally, the comparison of the performance of each of the approaches applied to the same sampled signal data indicate the effectiveness of both denoising techniques for use over real data sets.

Credit rating is involved in many financial applications to estimate the creditworthiness of corporations or individuals. In addition to building accurate credit rating models, the stability of models is of significant importance to economic performa

The Graphics Processing Unit (GPU) originally designed for rendering graphics and which is difficult to program for other tasks, has since evolved into a device suitable for general-purpose computations. As a result graphics hardware has become progressively more attractive yielding unprecedented performance at a relatively low cost. Thus, it is the ideal candidate to accelerate a wide variety of data parallel tasks in many fields such as in Machine Learning (ML). As problems become more and more demanding, parallel implementations of learning algorithms are crucial for a useful application. In particular, the implementation of Neural Networks (NNs) in GPUs can significantly reduce the long training times during the learning process. In this paper we present a GPU parallel implementation of the Back-Propagation (BP) and Multiple Back-Propagation (MBP) algorithms, and describe the GPU kernels needed for this task. The results obtained on well-known benchmarks show faster training times and improved performances as compared to the implementation in traditional hardware, due to maximized floating-point throughput and memory bandwidth. Moreover, a preliminary GPU based Autonomous Training System (ATS) is developed which aims at automatically finding high-quality NNs-based solutions for a given problem.

The Graphics Processing Unit (GPU) is a highly parallel, many-core device with enormous computational power, especially well-suited to address Machine Learning (ML) problems that can be expressed as data-parallel computations. As problems become increasingly demanding, parallel implementations of ML algorithms become critical for developing hybrid intelligent real-world applications. The relative low cost of GPUs combined with the unprecedent computational power they offer, make them particularly well-positioned to automatically analyze and capture relevant information from large amounts of data. In this paper, we propose the creation of an open source GPU Machine Learning Library (GPUMLib) that aims to provide the building blocks for the scientific community to develop GPU ML algorithms. Experimental results on benchmark datasets demonstrate that the GPUMLib components already implemented achieve significant savings over the counterpart CPU implementations.

Neural Networks (NN) have proven to be able to successfully solve problems in many areas. However, for large scale real problems, data is often incomplete. This is a serious problem, because NN cannot handle directly missing values. The usual approach to solve this problem consists of removing attributes and/or samples containing unknown values. This strategy is very attractive since it is simple to implement and reduces the dimensionality of data, therefore potentially reducing the complexity of the problem. However removing features or instances containing vital information, which can not be compensated by the remaining data, may result in the unattainability of accurate NN models. Another strategy consists of estimating missing values. However, wrong estimations of crucial data can lead to unpredicted results. Moreover these techniques do not account for real situations (e.g. where sensors may fail) and the output of such NN models may cause instabilities in the whole process. In this paper we propose a technique for handling missing values that accounts for the creation of different transparent NN models with respect to the missing features instead of relying on tedious data pre-processing techniques. The resulting models are bounded to share information among them. Contrary to the imputation of data (estimate) our models take into account the uncertainty caused by unknown values. Moreover the presented technique is prepared to deal with faulty sensors. The preliminary results obtained in several datasets show the efficacy of the proposed approach.

In recent years the the potential and programmability of Graphics Processing Units (GPU) has raised a note-worthy interest in the research community for applications that demand high-computational power. In particular, in financial applications containing thousands of high-dimensional samples, machine learning techniques such as neural networks are often used. One of their main limitations is that the learning phase can be extremely consuming due to the long training times required which constitute a hard bottleneck for their use in practice. Thus their implementation in graphics hardware is highly desirable as a way to speed up the training process. In this paper we present a bankruptcy prediction model based on the parallel implementation of the Multiple BackPropagation (MBP) algorithm which is tested on a real data set of French companies (healthy and bankrupt). Results by running the MBP algorithm in a sequential processing CPU version and in a parallel GPU implementation show reduced computational costs with respect to the latter while yielding very competitive performance.

Financial distress prediction is of great importance to all stakeholders in order to enable better decision-making in evaluating firms. In recent years, the rate of bankruptcy has risen and it is becoming harder to estimate as companies become more complex and the asymmetric information between banks and firms increases. Although a great variety of techniques have been applied along the years, no comprehensive method incorporating an holistic perspective had hitherto been considered. Recently, SVM+ a technique proposed by Vapnik [17] provides a formal way to incorporate privileged information onto the learning models improving generalization. By exploiting additional information to improve traditional inductive learning we propose a prediction model where data is naturally separated into several groups according to the size of the firm. Experimental results in the setting of a heterogeneous data set of French companies demonstrated that the proposed model showed superior performance in terms of prediction accuracy in bankruptcy prediction and misclassification cost.

Cancer diagnosis is generally the process of using some form of physical or genetic tests or exams, usually referred as patient data, to detect the disease. One of the main problems with cancer diagnosis systems is the lack of labeled data, as well as the difficulties of labeling pre-existing unlabeled data. Thus, there is a growing interest in exploring the use of unlabeled data as a way to improve classification performance in cancer diagnosis. The possible availability of this kind of data for some applications makes it an appealing source of information. In this work we explore an Incremental Background Knowledge (IBK) technique to introduce unlabeled data into the training set by expanding it using initial classifiers to better aid decisions, namely by improving recall values. The defined incremental SVM margin-based method was tested in the Wisconsin-Madison breast cancer diagnosis problem to examine the effectiveness of such techniques in supporting diagnosis.

Constructing a single text classifier that excels in any given application is a rather inviable goal. As a result, ensemble systems are becoming an important resource, since they permit the use of simpler classifiers and the integration of different knowledge in the learning process. However, many text-classification ensemble approaches have an extremely high computational burden, which poses limitations in applications in real environments. Moreover, state-of-the-art kernel-based classifiers, such as support vector machines and relevance vector machines, demand large resources when applied to large databases. Therefore, we propose the use of a new systematic distributed ensemble framework to tackle these challenges, based on a generic deployment strategy in a cluster distributed environment. We employ a combination of both task and data decomposition of the text-classification system, based on partitioning, communication, agglomeration, and mapping to define and optimize a graph of dependent tasks. Additionally, the framework includes an ensemble system where we exploit diverse patterns of errors and gain from the synergies between the ensemble classifiers. The ensemble data partitioning strategy used is shown to improve the performance of baseline state-of-the-art kernel-based machines. The experimental results show that the performance of the proposed framework outperforms standard methods both in speed and classification.

Nowadays machine learning are faced with an overload of data, both in terms of examples and features. Although recent algorithms, like support vector machines, can handle high dimensionality, it remains valuable to find smaller and more fitted spaces to perform learning tasks. We propose a twofold approach to tackle these high dimensionality issues in a text classification setting. First we use manifold learning as a pre-processing step to nonlinearly reduce the feature space. Second we use support vector machines to implement an active learning strategy, where the kernel trick is used to define the active examples. This approach deals with the high dimensionality both reducing the features and the number of examples needed to reach a desired performance.Results on a real-world benchmark corpus from Reuters and also on a reduced realistic version of the corpus show first the visualization capabilities of manifold learning and the performance improvement achieved with the active learning strategy.

One of the Internet’s hallmark is the rapid spread of the use of information and communication technology. This has boosted methods for hiding stego information inside digital cover content images which is a concerning issue in information security. On the other hand, attack of steganographic schemes has leveraged methods for steganalysis which is a challenging problem. In this paper, first we look at the design of classifiers, such as, Support Vector Machines (SVM) and neural networks (RBF and MLP) which are able to detect the presence of Least Significant Bit (LSB) matching steganography of gray scale images. Second, by combining with feature ranking methods (SVM-Recursive Feature Elimination, Kruskal Wallis) and reduction techniques (PCA) pattern classification of stego is successfully achieved. It is of utmost importance to look at the large set of features extracted from images and find ranking methods able, namely, to exclude correlated and redundant features, avoid the curse of dimensionality or circumvent the need of the steganalyzer to be re-designed. Results show that desirable properties of robustness and resilience are attained by designing classifiers able to deal with redundancy and noise. Moreover, comparison of classifiers performance emphasizes the chosen model for the steganalyser.

In this article, based on the Markov approach proposed by shietal., we expand it to the inter-blocks of the DCT domain, calculate the difference of the expanded Markov features between the testing image and the calibrated version, and combine these difference features and the polynomial fitting features on the histogram of the DCT coefficients as detectors. We reasonably improve the detection performance in multi-class JPEG images. We also compare the steganalysis performance among the feature reduction/selection methods based on principal component analysis, singular value decomposition, and Fisherpsilas linear discriminant.

Learning in online open networking environments turned out possible by today's widespread use of Internet technologies, has led to the development of a broad range of products for web-based courses at University level. However, although elearning in education is well established, there are a few attempts to extract information during the course final evaluation phase. In other words, while evaluation of e-learning applications has boosted the need to design effective methodologies for better tools, little attention was devoted to extract information for discovery of student's behavior.

In this paper we develop and analyze methods for expanding automated learning of Relevance Vector Machines (RVM) to large scale text sets. RVM rely on Bayesian inference learning and while maintaining state-of-the-art performance, offer sparse and probabilistic solutions. However, efforts towards applying RVM to large scale sets have met with limited success in the past, due to computational constraints. We propose a diversified set of divide-and-conquer approaches where decomposition techniques promote the definition of smaller working sets that permit the use of all training examples. The rationale is that by exploring incremental, ensemble and boosting strategies, it is possible to improve classification performance, taking advantage of the large training set available. Results on Reuters-21578 and RCV1 are presented, showing performance gains and maintaining sparse solutions that can be deployed in distributed environments.

Relevance vector machines (RVM) have proven successful in many learning tasks. However, in large applications, they scale poorly. In many settings there is a large amount of unlabeled data which could be actively chosen by a learner and integrated in the learning procedure. The idea is to improve performance meanwhile reducing costs from data categorization. In this paper we propose an active learning RVM method based on the kernel trick. The underpinning idea is to define a working space between the relevance vectors (RV) initially obtained in a small labeled data set and the new unlabeled examples, where the most informative instances are chosen. By using kernel distance metrics, such a space can be defined and more informative examples can be added to the training set, increasing performance even though the problem dimension is not significantly affected. We detail the proposed method giving illustrative examples in the Reuters-21578 benchmark. Results show performance improvement and scalability.

The risk of developing life-threatening ventricular arrhythmias in patients with structural heart disease is higher with increased occurrence of premature ventricular complex (PVC). Therefore, reliable detection of these arrhythmias is a challenge for a cardiovascular diagnosis system. While early diagnosis is critical, the task of its automatic detection and classification becomes crucial. Therefore, the underlying models should be efficient, albeit ensuring robustness. Although neural networks (NN) have proven successful in this setting, we show that kernel-based learning algorithms achieve superior performance. In particular, recently developed sparse Bayesian methods, such as, Relevance Vector Machines (RVM), present a parsimonious solution when compared with Support Vector Machines (SVM), yet revealing competitive accuracy. This can lead to significant reduction in the computational complexity of the decision function, thereby making RVM more suitable for real-time applications.

Support vector machines (SVM) and relevance vector machines (RVM) constitute two state-of-the-art learning machines that are currently focus of cutting-edge research. SVM present accuracy and complexity preponderance, but are surpassed by RVM when probabilistic outputs or kernel selection come to discussion. We propose a two-level hierarchical hybrid SVM-RVM model to combine the best of both learning machines. The proposed model first level uses an RVM to determine the less confident classified examples and the second level then makes use of an SVM to learn and classify the tougher examples. We show the benefits of the hierarchical approach on a text classification task, where the two-levels outperform both learning machines

Making inferences and choosing appropriate responses based on incomplete, uncertainty and noisy data is challenging in financial settings particularly in bankruptcy detection. In an increasingly globalized economy, bankruptcy results both in huge economic losses and tremendous social impact. While early prediction for a bankruptcy, if done appropriately, is of great importance to banks, insurance firms, creditors, and investors, the need of substantially more accurately predicting models becomes crucial. This problem has been approached by various methods ranging from statistics to machine learning, however they find a class decision estimate rather than a probabilistic confidence of the class distribution. In this paper we show that sparse Bayesian models also known as Relevance Vector Machine (RVMs) are superior to the state-of-the-art machine learning algorithms such as Support Vector Machines (SVMs) therefore leading to predictors of choice. The advantage of RVM approach is that the classifier can yield a decision function that is much sparser than the SVM while maintaining its detection accuracy. This can lead to significant reduction in the computational complexity of the decision function, thereby making it more suitable for real-time applications. Preliminary experiments on Coface Data set (French credit risk provider) show that RVM classifiers outperform SVM, lead to more sparse and accurate prediction models.

Text classification (TC) is a complex ubiquitous task that handles a huge amount of data. Current research has recently proved that kernel learning based methods are quite effective in this problem. As opposed to support vector machines (SVM), the relevance vector machine (RVM) in particular yields a probabilistic output while preserving its accuracy. However, few research efforts have addressed the issue of scalability that arises when applying RVM to large scale problems like TC. We propose a new model which consists of a two-step RVM classifier able to (i) be competitive regarding processing time, (ii) use all available training elements and (iii) improve RVM classification performance. The paper also shows that a convenient similitude measure among documents can be defined on all the collection data, which does not only make the process swifter but also parallelizable. Using REUTERS-21578, we show that deployment of successful real-time applications is possible through reduction of the computational complexity and improvement of overall performance, obtained by the proposed model.

The amount of texts available in digital form has dramatically increased, giving rise to the need of fast text classifiers. The tasks involved can be parallelized and distributed in a grid environment. This paper reports a study conducted on Reuters-21578 corpus, using a SVM learning machine. The task of text categorization is distributed in several platforms. The results achieved are very promising for speeding-up text categorization tasks and are valid independently of the learning machine.

Support vector machines (SVMs) are receiving increased attention in different application domains for which neural networks (NNs) have had a prominent role. However, in quality monitoring little attention has been given to this more recent development encompassing a technique with foundations in statistic learning theory. In this paper, we compare C-SVM and /spl nu/-SVM classifiers with radial basis function (RBF) NNs in data sets corresponding to product faults in an industrial environment concerning a plastics injection molding machine. The goal is to monitor in-process data as a means of indicating product quality and to be able to respond quickly to unexpected process disturbances. Our approach based on SVMs exploits the first part of this goal. Model selection which amounts to search in hyperparameter space is performed for study of suitable condition monitoring. In the multiclass problem formulation presented, classification accuracy is reported for both strategies. Experimental results obtained thus far indicate improved generalization with the large margin classifier as well as better performance enhancing the strength and efficacy of the chosen model for the practical case study.

Text mining, also known as intelligent text analysis, text data mining or knowledge-discovery in text, refers generally to the process of extracting interesting and nontrivial information and knowledge from text. One of the main problems with text mining and classification systems is the lack of labeled data, as well as the cost of labeling unlabeled data (Kiritchenko and Matwin 2001). Thus, there is a growing interest in exploring the use of unlabeled data as a way to improve classification performance in text classification. The ready availability of this kind of data in most applications makes it an appealing source of information. In this work we evaluate the benefits of introducing unlabeled data in a support vector machine automatic text classifier. We further evaluate the possibility of learning actively and propose a method for choosing the samples to be learned.

Light sensor spectral calibration is an ill-defined problem. For the identification problem one needs a priori knowledge of the characteristics of the sensor which is difficult to get in most situations. A new methodology is presented in this paper that does not rely on any a priori knowledge of the sensor's characteristics. The method uses an extended generalized cross-validation function to measure predictability of the identified sensor's spectral behavior. The prediction error is minimized with a hybrid genetic algorithm. Further an extended image formation model is introduced to model changes in additive and multiplicative errors. The calibration problem is formulated to be independent of these changes by previously identifying and removing them from the images.

Spectral data estimation from image data is an ill-posed problem since (i) due to the integral nature of solid-state light sensors, the same output can be obtained from an infinity of input signals and (ii) color signals are spectrally smooth in nature and therefore limit the number of linear independent equations that can be formulated for the identification problem. To enable the solution of these problems most methods rely on exact a priori knowledge, such as smoothness and modality, to formulate hard constraints. A new method based on an extended generalized cross-validation measure is introduced for this type of problems. The solution is obtained with a genetic algorithm that maximizes its prediction ability. The method does not require exact a priori knowledge on the solution, since it is able to extract this information from the input data.

This paper presents a new neural network to solve the shortest path problem for inter-network routing. The proposed solution extends the traditional single-layer recurrent Hopfield architecture introducing a two-layer architecture that automatically guarantees an entire set of constraints held by any valid solution to the shortest path problem. This new method addresses some of the limitations of previous solutions, in particular the lack of reliability in what concerns successful and valid convergence. Experimental results show that an improvement in successful convergence can be achieved in certain classes of graphs. Additionally, computation performance is also improved at the expense of slightly worse results.

A technique is proposed to detect and avoid obstacles for a mobile robot in an unknown environment. The usual problem of having too much sensorial information is dealt with by using several neural networks that cooperate in the guidance of the robot. Several unknown obstacle configurations were presented to the modular networks, proving that the MONODA architecture is very effective for obstacle avoidance when there is neither a priori nor a posteriori maps of the environment.

Injection molding technology should assure a high level of quality control of the molded parts via automation. Inherent complexities of the process make mathematical modeling difficult, hindering the control quality demands of the conventional methods. Neural networks adaptive data based technology has been successfully applied in industrial applications as they rely on highly nonlinear models and are able to provide enough rich data for modelling the required process relationships. Neural networks are used herein for fault detection in the injection molding process. The next step is to develop a system for automatic tuning of machine setups.