Domain adaptation arises as an important problem in statistical learning theory, arising when the data-generating processes differ between the training and test samples, respectively called source and target domains. Recent theoretical advances have demonstrated that the success of domain adaptation algorithms heavily relies on their ability to minimize the divergence between the probability distributions of the source and target domains. However, minimizing this divergence cannot be achieved independently of other key ingredients, such as the source risk or the combined error of the ideal joint hypothesis. The trade-off between these terms is often ensured through algorithmic solutions that remain implicit and are not directly reflected by the theoretical guarantees. To get to the bottom of this issue, we propose in this paper a new theoretical framework for domain adaptation through hierarchical optimal transport. This framework provides more explicit generalization bounds and enables us to consider the natural hierarchical organization of samples in both domains into structures, i.e. classes or clusters. Additionally, we provide a new divergence measure between the source and target domains called Hierarchical Wasserstein distance that indicates under mild assumptions, which structures need to be aligned to achieve successful adaptation.

Domain adaptation is a subfield of statistical learning theory that takes into account the shift between the distribution of training and test data, typically known as source and target domains, respectively. In this context, this paper presents an incremental approach to tackle the intricate challenge of unsupervised domain adaptation, where labeled data within the target domain is unavailable. The proposed approach, OTP-DA, endeavors to learn a sequence of joint subspaces from both the source and target domains using Linear Discriminant Analysis (LDA), such that the projected data into these subspaces are domain-invariant and well-separated. Nonetheless, the necessity of labeled data for LDA to derive the projection matrix presents a substantial impediment, given the absence of labels within the target domain in the setting of unsupervised domain adaptation. To circumvent this limitation, we introduce a selective label propagation technique grounded on optimal transport (OTP), to generate pseudo-labels for target data, which serve as surrogates for the unknown labels. We anticipate that the process of inferring labels for target data will be substantially streamlined within the acquired latent subspaces, thereby facilitating a self-training mechanism. Furthermore, our paper provides a rigorous theoretical analysis of OTP-DA, underpinned by the concept of weak domain adaptation learners, thereby elucidating the requisite conditions for the proposed approach to solve the problem of unsupervised domain adaptation efficiently. Experimentation across a spectrum of visual domain adaptation problems suggests that OTP-DA exhibits promising efficacy and robustness, positioning it favorably compared to several state-of-the-art methods.

Hand is a formidable and very complex organ. Each anatomical structure, even the smallest, can play a crucial role: arteries, tendons, nerves, work together to generate a large range of movements. Therefore, hand surgery remains a complex operation, and requires a lot of skills and experience. In this paper, we explore two different object detection problems linked to hand surgery: The detection of anatomical structures and lesions, using modular machine learning, and the Endoscopic Carpal Tunnel Release, with the help of Self-Supervised Learning. We use object detection to detect anatomical structures, and lesions. We show that modular machine learning and Self-Supervised Learning can be used to improve the results, even with only a small amount of labeled data.

Deep learning-based applications have seen a lot of success in recent years. Text, audio, image, and video have all been explored with great success using deep learning approaches. The use of convolutional neural networks (CNN) in computer vision, in particular, has yielded reliable results. In order to achieve these results, a large amount of data is required. However, the dataset cannot always be accessible. Moreover, annotating data can be difficult and time-consuming. Self-training is a semi-supervised approach that managed to alleviate this problem and achieve state-of-the-art performances. Theoretical analysis even proved that it may result in a better generalization than a normal classifier. Another problem neural networks can face is the increasing complexity of modern problems, requiring a high computational and storage cost. One way to mitigate this issue, a strategy that has been inspired by human cognition known as modular learning, can be employed. The principle of the approach is to decompose a complex problem into simpler sub-tasks. This approach has several advantages, including faster learning, better generalization, and enables interpretability. In the first part of this paper, we introduce and evaluate different architectures of modular learning for Dorsal Capsulo-Scapholunate Septum (DCSS) instability classification. Our experiments have shown that modular learning improves performances compared to non-modular systems. Moreover, we found that weighted modular, that is to weight the output using the probabilities from the gating module, achieved an almost perfect classification. In the second part, we present our approach for data labeling and segmentation with self-training applied on shoulder arthroscopy images.

In this paper, we address the problem of unsupervised domain adaptation where we ask to infer a low target risk classifier, while labeled data are only available from the source domain. Our proposed approach, called DA-OTP, aims to learn a gradual subspace alignment of the source and target domains through Supervised Locality Preserving Projection, so that projected data in the joint low-dimensional latent subspace can be domain-invariant and easily separable. However, this objective can be rather challenging to achieve because of the absence of labeled data in the target domain. To overcome this conundrum, we use an incremental label propagation technique based on optimal transport, which performs selective pseudo-labeling in the target domain. The selected pseudo-labeled target samples are then combined with labeled source samples to learn in a self-training fashion a robust classifier after the incremental subspace alignment. Experiments show the competitiveness of the proposed approach across contemporary state-of-the-art methods over a range of domain adaptation problems. We make our code publicly available.11Code is available at: https://github.com/DA-OTP/DA-OTP

Non-negative matrix factorization (NMF) is an unsupervised algorithm for clustering where a non-negative data matrix is factorized into (usually) two matrices with the property that all the matrices have no negative elements. This factorization raises the problem of instability, which means whenever we run NMF for the same dataset, we get different factorization. In order to solve the problem of non-uniqueness and to have a more stable solution, we propose a new approach that consists on collaborating different NMF models followed by a consensus. The proposed approach was validated on several datasets and the experimental results showed the effectiveness of our approach which is based on the reducing of standard reconstruction error in NMF model.

In this paper, we tackle the transductive semi-supervised learning problem that aims to obtain label predictions for the given unlabeled data points according to Vapnik's principle. Our proposed approach is based on optimal transport, a mathematical theory that has been successfully used to address various machine learning problems, and is starting to attract renewed interest in semi-supervised learning community. The proposed approach, Optimal Transport Propagation (OTP), performs in an incremental process, label propagation through the edges of a complete bipartite edge-weighted graph, whose affinity matrix is constructed from the optimal transport plan between empirical measures defined on labeled and unlabeled data. OTP ensures a high degree of predictions certitude by controlling the propagation process using a certainty score based on Shannon's entropy. We also provide a convergence analysis of our algorithm. Experiments task show the superiority of the proposed approach over the state-of-the-art. We make our code publicly available.11Code is available at: https://github.com/MouradElHamri/OTP

Recently, more researchers are interested in the domain of quantum machine learning as it can manipulate and classify large numbers of vectors in high dimensional space in reasonable time. In this paper, we propose a new approach called Quantum Collaborative K-means which is based on combining several clustering models based on quantum K-means. This collaboration consists of exchanging the information of each algorithm locally in order to find a common underlying structure for clustering. Comparing the classical version of collaborative clustering to our approach, we notice that we have an exponential speed up: while the classical version takes O(K×L×M×N), the quantum version takes only O(K×L×log(M×N)). And comparing to the quantum version of K-means, we get a better solution in terms of the criteria of validation which means in terms of clustering. The empirical evaluations validate the benefits of the proposed approach.

The task of clustering is at the same time challenging and very important in Artificial Intelligence. One of the most popular family of clustering algorithms is the prototype-based approach. Prototype-based algorithms compute a representation of the clusters in the form of a set of prototypes, usually vectors approximating each cluster's barycenter. However, the objects in a data set are not necessarily vectors, especially in real-world applications. These non-vectorial data sets are often represented by the dissimilarities, distances, or relations between all pairs of objects. They are usually referred as relational data sets. For this kind of data, the algorithms must be adapted to different measures of distance. There are a few state-of-the-art algorithms adapted to relational data sets through the use of barycentric coordinates formalism, in which the objects of a relational data sets are embedded in a space defined by the distances between a subset of the objects, called support points. In this paper, we propose an approach that is able to automatically select the optimal set of support points. We also extend the method to relational data streams, in order to detect variations in the intrinsic dimensionality of the representation space over time. We have compared experimentally the quality of the proposed algorithms on real and artificial data sets. We show that the automatic selection of support points allows an optimal quality in a minimal computation time.

In multi-label learning, each learning instance is associated with multiple class labels simultaneously and the task is to learn a transition from the features space to the labels space. Generally, it is costly and time consuming to get labels for learning samples, especially for the task of multi-label annotation where many class labels must be assigned to the same instance. To overcome this difficulty, semi-supervised multi-label learning aims to exploit unlabeled data readily available to help build a multi-label predictive model. Nevertheless, most semi-supervised solutions for the multi-label learning tasks use a batch method for labeling (e.g., label propagation), and are not suitable for online classification tasks. In this paper, a new online semi-supervised multi-label classifier based on the Growing Neural Gas (GNG) algorithm is presented. Online GNG-based algorithms are already used in several applications, but the existing algorithms are unable to tackle the problem of multi-label classification. There is a real need of online semi-supervised multi-label algorithms. Our main contribution is to propose an online multi-label algorithm based on GNG able to provide on the fly annotation of data without explicit storage of the training instances, i.e, with limited memory usage. The proposed algorithm is experimentally tested on five different data-sets from three different application domains. The performance results of the approach is compared with several state-of-the-art methods. The experiments show that the proposed Semi-Supervised algorithm outperforms the existing multi-label classifiers.

In this work we explore an original strategy for building deep networks, based on stacking layers of Self-Organizing Maps (SOM) with finite weights. We aim to show that our approach, with enough hidden variables, is capable to represents any “soft” distribution over the visible variables, where “soft” means that the distribution does not contain any probabilities of 1 or 0. The algorithm compute the model one layer at a time. The first layer receives the input observations and compute a probability of membership for each observation and each neuron. These probabilities become the input of the second layer, which compute a new set of probabilities and so on. The number of neurons decrease in each layer after the first. The proposed algorithm is experimentally tested on artificial and real data-sets. The effect of the added hidden layers for the representation of data structure is experimentally demonstrated.

Graph clustering techniques are very useful for detecting densely connected groups in large graphs. Many existing graph clustering methods mainly focus on the topological structure, but ignore the vertex properties. Existing graph clustering methods have been recently extended to deal with nodes attribute. In this paper we propose a new method which uses the nodes attributes information along with the topological structure of the network in the clustering process. In order to use the information about the attributes nodes, the collaborative clustering can be employed in the model. The aim of collaborative clustering is to reveal the common underlying structure of data spread across multiple sites by applying different clustering algorithms and therefore improve the final clustering result. The purpose of this article is to introduce a new attributed collaborative multi-view networks based on community detection in networks and topological collaborative learning. The idea consists in modifying databases by adding virtual points which convey clustering information, to change the position of centers of the clustering solution. Experimental results demonstrate the effectiveness of the proposed method through comparisons with the state-of-the-art graph clustering methods on synthetic and real datasets.

Collaborative clustering is a recent learning paradigm concerned with the unsupervised analysis of complex multi-view data using several algorithms working together. Well known applications of collaborative clustering include multi-view clustering and distributed data clustering, where several algorithms exchange information in order to mutually improve each others based on the diversity of their models or their view of the data. However, many of the proposed algorithms and statistical models in these fields lack the capability to properly detect noisy views and sub-optimal collaborators. As a result, these weak collaborators and noisy views often go undetected during the collaborative process, and end up deteriorating the results of all other algorithms. In this article, we propose a weighting optimization method for the collaborative version of the SOM algorithm that will help detect whether local self-organizing maps should or should not exchange their information based on the diversity between their topologies. This method can further be used to detect noisy views and discard them in unsupervised collaborative and multi-view processes.

In this paper, we present a novel method for co-clustering, an unsupervised learning approach that aims at discovering homogeneous groups of data instances and features by grouping them simultaneously. The proposed method uses the entropy regularized optimal transport between empirical measures defined on data instances and features in order to obtain an estimated joint probability density function represented by the optimal coupling matrix. This matrix is further factorized to obtain the induced row and columns partitions using multiscale representations approach. To justify our method theoretically, we show how the solution of the regularized optimal transport can be seen from the variational inference perspective thus motivating its use for co-clustering. The algorithm derived for the proposed method and its kernelized version based on the notion of Gromov-Wasserstein distance are fast, accurate and can determine automatically the number of both row and column clusters. These features are vividly demonstrated through extensive experimental evaluations.

The aim of collaborative clustering is to reveal the common underlying structure of data spread across multiple sites by applying different clustering algorithms and therefore improve the final clustering result. The purpose of this article is to introduce a new collaborative topological clustering approach based on Self-Organizing Maps (SOM) with more flexible structure. The main drawback of the previously proposed collaborative methods is they require a strong condition to carry out the collaboration, i.e. all the map related to each site must be learned with the same number of neurones. An alternative approach to overcome the problem of this dimensionality has been proposed in the case of k-means. The idea consists in modifying databases by adding virtual points which convey clustering information, to change the position of centers of the clustering solution. This approach seems promising because it is very flexible. In this paper we propose a method called VP2SOM (Virtual Points to SOM) which uses virtual points to realize the collaboration between SOM maps with different sizes. We have tested the proposed approach on several datasets and the early results seem promising.

Unsupervised machine learning approaches involving several clustering algorithms working together to tackle difficult data sets are a recent area of research with a large number of applications such as clustering of distributed data, multi-expert clustering, multi-scale clustering analysis or multi-view clustering. Most of these frameworks can be regrouped under the umbrella of collaborative clustering, the aim of which is to reveal the common underlying structures found by the different algorithms while analyzing the data. Within this context, the purpose of this article is to propose a collaborative framework lifting the limitations of many of the previously proposed methods: Our proposed collaborative learning method makes possible for a wide range of clustering algorithms from different families to work together based solely on their clustering solutions, thus lifting previous limitation requiring identical prototypes between the different collaborators. Our proposed framework uses a variational EM as its theoretical basis for the collaboration process and can be applied to any of the previously mentioned collaborative contexts. In this article, we give the main ideas and theoretical foundations of our method, and we demonstrate its effectiveness in a series of experiments on real data sets as well as data sets from the literature.

Recent studies have shown that the use of a priori knowledge can significantly improve the results of unsupervised classification. However, capturing and formatting such knowledge as constraints is not only very expensive requiring the sustained involvement of an expert but it is also very difficult because some valuable information can be lost when it cannot be encoded as constraints. In this paper, we propose a new constraint-based clustering approach based on ontology reasoning for automatically generating constraints and bridging the semantic gap in satellite image labeling. The use of ontology as a priori knowledge has many advantages that we leverage in the context of satellite image interpretation. The experiments we conduct have shown that our proposed approach can deal with incomplete knowledge while completely exploiting the available one.

The ability of a human being to extrapolate previously gained knowledge to other domains inspired a new family of methods in machine learning called transfer learning. Transfer learning is often based on the assumption that objects in both target and source domains share some common feature and/or data space. In this paper, we propose a simple and intuitive approach that minimizes iteratively the distance between source and target task distributions by optimizing the kernel target alignment (KTA). We show that this procedure is suitable for transfer learning by relating it to Hilbert-Schmidt Independence Criterion (HSIC) and Quadratic Mutual Information (QMI) maximization. We run our method on benchmark computer vision data sets and show that it can outperform some state-of-art methods.

Generative Topographic Mapping (GTM) is a popular probabilistic framework for modeling non-linear relationships in high-dimensional data as well as for unsupervised learning and visualization of such data. It is also known as to provide a principled probabilistic alternative to the well-known Self-Organizing Map (SOM) in the neural networks community, thanks to its flexible mixture model formulation and the desirable properties of the expectation-maximization (EM) algorithm. However, much attention has been focused on the use of GTM for multivariate data, in general assumed to be independent and identically distributed (i.i.d) and the problem of modeling sequences using GTM is less investigated. In this paper, we focus on GTM for unsupervised modeling and visualization of sequential data. We consider modeling sequences of continuous multidimensional observations and we propose a GTM through time (GTM-TT) approach based on hidden Markov models (HMM) where the observations are a sent of independent sequences, rather than a signle sequence. We further extend the model to the clustering of multiple sequences by proposing a GTM-TT mixture model. The model parameters are estimated by maximum likelihood via the EM algorithm. The proposed approach is evaluated using simulated data and real-world data.

The concept of nonnegative matrix factorization is a recent machine learning technique that is used to decompose large data matrices imposing the non-negativity constraints on the factors. This technique is now used in many data mining applications and thus remains a topic of ongoing interest. In this paper we are particularly interested in the Multilayer NMF - a model that can be seen as a pretraining step of Deep NMF model for learning hidden representations. We analyze the factors obtained using Multilayer NMF and show that the process of building layers can be seen as a repeated application of the Hoyer's projection operator applied sequentially to the factor of the second layer. We also provide the sparsity analysis for matrices obtained during the optimization procedure at each layer. We conclude that the overall sparsity decreases with the increasing number of layers despite the general assumption that Multilayer NMF is efficient due to the fact that it increases the sparsity of learned factors.

The aim of collaborative clustering is to reveal the common underlying structures found by different algorithms while analyzing data. The fundamental concept of collaboration is that the clustering algorithms operate locally but collaborate by exchanging information about the local structures found by each algorithm. In this framework, the one purpose of this article is to introduce a new method which allows to reinforce the clustering process by exchanging information between several results acquired by different clustering algorithms. The originality of our proposed approach is that the collaboration step can use clustering results obtained from any type of algorithm during the local phase. This article gives the theoretical foundations of our approach as well as some experimental results. The proposed approach has been validated on several data sets and the results have shown to be very competitive.

In this paper, we propose a Collaborative Clustering method based on Variational Bayesian Generative Topographic Mapping (VBGTM). To do so, we first propose a method that combines VBGTM and Fuzzy c-means (FCM). Collaborative clustering is useful to achieve interaction between different sources of information for the purpose of revealing underlying structures and regularities within data sets. It can be treated as a process of consensus building where we attempt to reveal a structure that is common across all sets of data. VBGTM was introduced as a variational approximation of Generative Topographic Mapping (GTM) to control data overfitting. It provides an analytical approximation to the posterior probability of the latent variables and the distribution of the input data in the latent space. It can be effectively applied to visualize and explore properties of the data. But when the number of latent points is large, similar units need to be grouped (i.e., clustered) to facilitate quantitative analysis of the map and the data. We use FCM to determine the prototypes as well as the resultant clusters and the corresponding membership functions of the input data, based on the latent variables obtained from VBGTM. So, by combining the two algorithms, we develop a method that can do visualization and clustering at the same time. We observe that the hybrid method (F-VBGTM) performs very well in terms of many cluster-validity indexes.

We present a generative approach to train a new probabilistic self-organizing map (PrSOMS) for dependent and nonidentically distributed data sets. Our model defines a low-dimensional manifold allowing friendly visualizations. To yield the topology preserving maps, our model has the SOM like learning behavior with the advantages of probabilistic models. This new paradigm uses hidden Markov models (HMM) formalism and introduces relationships between the states. This allows us to take advantage of all the known classical views associated to topographic map. The objective function optimization has a clear interpretation, which allows us to propose expectation-maximization (EM) algorithm, based on the forward–backward algorithm, to train the model. We demonstrate our approach on two data sets: The real-world data issued from the "French National Audiovisual Institute" and handwriting data captured using a WACOM tablet.

The aim of collaborative clustering is to reveal the common structure of data which are distributed on different sites. The topological collaborative clustering, based on Self-Organizing Maps (SOM) is an unsupervised learning method which is able to use the output of other SOMs from other sites during the learning. This paper investigates the impact of the diversity between collaborators on the collaboration's quality and presents a study of different diversity indexes for collaborative clustering. Based on experiments on artificial and real datasets, we demonstrated that the quality and the diversity of the collaboration can have an important impact on the quality of the collaboration and that not all diversity indexes are relevant for this task.

In this paper we study a variation of a Non-negative Matrix Factorization (NMF) called the Orthogonal NMF(ONMF). This special type of NMF was proposed in order to increase the quality of clustering results of standard NMF by imposing orthogonality on clustering indicator matrix and/or the matrix of basis vectors. We develop an extension of ONMF which we call Weighted ONMF and propose a novel approach for imposing orthogonality on the matrix of basis vectors obtained via NMF using Gram-Schmidt process.

In this paper we propose a new unsupervised transfer learning approach which aims at finding a partition of unlabeled data in target domain using the knowledge obtained from clustering a source domain unlabeled data. The key idea behind our method is that finding partitions in different feature's subspaces of a source task can help to obtain a more accurate partition in a target one. From the set of source partitions we select only k nearest neighbors using some measure of similarity. Finally, multi-layer non-negative matrix factorization is performed to obtain a partition of objects in target domain. Experimental results show high potential and effectiveness of the proposed technique.

The purpose of this article is to introduce a new collaborative multi-view clustering approach based on a probabilistic model. The aim of collaborative clustering is to reveal the common underlying structure of data spread across multiple data sites by applying clustering techniques. The strength of the collaboration between each pair of data repositories is determined by a fixed parameter. Previous works considered deterministic techniques such as Fuzzy C-Means (FCM) and Self-Organizing Maps (SOM). In this paper, we present a new approach for the collaborative clustering using a generative model, which is the Generative Topographic Mappings (GTM). Maps representing different sites could collaborate without recourse to the original data, preserving their privacy. We present the approach for multi-view collaboration using GTM, where data sets have the same observations but presented in different feature space; i.e. different dimensions. The proposed approach has been validated on several data sets, and experimental results have shown very promising performance.

In many cases, databases are in constant evolution, new data is arriving continuously. Data streams pose several unique problems that make obsolete the applications of standard data analysis methods. Indeed, these databases are constantly on-line, growing with the arrival of new data. In addition, the probability distribution associated with the data may change over time. We propose in this paper a method of synthetic representation of the data structure for efficient storage of information, and a measure of dissimilarity between these representations for the detection of change in the stream structure.

In this paper, we propose a study on the use of weighted topological learning and matrix factorization methods to transform the representation space of a sparse dataset in order to increase the quality of learning, and adapt it to the case of transfer learning. The matrix factorization allows us to find latent variables, weighted topological learning is used to detect the most relevant among them. New data representation is based on their projections on the weighted topological model. Each object in the dataset is described by a new representation consisting of the distances of this object to all components of the topological model (prototypes). For transfer learning, we propose a new method where the representation of data is done in the same way as in the first phase, but using a pruned topological model. This pruning is performed after labeling the units of the topological model using the labels available for transfer. The experiments are presented as a part of an International Challenge [1] where we have obtained promising results (5th rank).

The aim of collaborative clustering is to reveal the common structure of data distributed on different sites. In this paper, we present a formalism of topological collaborative clustering using prototype-based clustering techniques; in particular we formulate our approach using Kohonen's Self-Organizing Maps. Maps representing different sites could collaborate without recourse to the original data, preserving their privacy. We present two different approaches of collaborative clustering: horizontal and vertical. The strength of collaboration (confidence exchange) between each pair of datasets is determined by a parameter, we call coefficient of collaboration, to be estimated iteratively during the collaboration phase using a gradient-based optimization, for both the approaches. The proposed approaches have been validated on several datasets and experimental results have shown very promising performance.

The exponential growth of data generates terabytes of very large databases. The growing number of data dimensions and data objects presents tremendous challenges for effective data analysis and data exploration methods and tools. One solution commonly proposed is the use of a condensed description of the properties and structure of data. Thus, it becomes crucial to have visualization tools capable of representing the data structure, not from the data themselves, but from these condensed descriptions. The purpose of our work described in this paper is to develop and put a synergistic visualization of data and knowledge into the knowledge discovery process. We propose here a method of describing data from enriched and segmented prototypes using a clustering algorithm. We then introduce a visualization tool that can enhance the structure within and between groups in data. We show, using some artificial and real databases, the relevance of the proposed method.

The aim of collaborative clustering is to reveal the common structure of data which are distributed on different sites. The topological collaborative clustering (based on Kohonen Self-Organizing Maps) allows to take into account other maps without recourse to the data in an unsupervised learning. In this paper, the approach is presented in the case of SOM and it is valid for all prototypes based classifications methods. The strength of the collaboration between each pair of datasets is determined by a fixed parameter for the both, vertical and horizontal topological collaborative clustering. In this study, learning the confidence exchange is presented for the both topological collaborative clustering approaches by using the topological knowledge. The gradient based optimization is used to set the value of the confidence parameter for each collaboration. The paper presents the formalism of the approach and its validation. The proposed approach has been validated on several datasets and experimental results have shown very promising performance.

In this paper we propose a novelty detection framework based on the orthogonal projection operators and the bootstrap idea. Our approach called Random Subspace Novelty Detection Filter (RS - NDF) combines the sampling technique and the ensemble idea. RS - NDF is an ensemble of NDF, induced from bootstrap samples of the training data, using random feature selection in the NDF induction process. Prediction is made by aggregating the predictions of the ensemble. RS - NDF generally exhibits a substantial performance improvement over the single NDF. Thanks to an online learning algorithm, the RS-NDF approach is also able to track changes in data over time. The RS - NDF method is compared to single NDF and other novelty detection methods with tenfold cross-validation experiments on publicly available datasets, where the methods superiority is demonstrated. Performance metrics such as precision and recall, false positive rate and false negative rate, F-measure, AUC and G-mean are computed. The proposed approach is shown to improve the prediction accuracy of the novelty detection, and have favorable performance compared to the existing algorithms.

This paper describes a new algorithm to learn a new probabilistic Self-Organizing Map for not independent and not identically distributed data set. This new paradigm probabilistic self-organizing map uses HMM (Hidden Markov Models) formalism and introduces relationships between the states of the map. The map structure is integrated in the parameter estimation of Markov model using a neighborhood function to learn a topographic clustering. We have applied this novel model to cluster and to reconstruct the data captured using a WACOM tablet.

This paper addresses the problem of cluster characterization by selecting a subset of the most relevant features for each cluster from a categorical dataset in an autonomous way. The proposed autonomous model is based on the Relational Topological Clustering (RTC) associated with a statistical test which allows to detect the most important variables in an automatic way without setting any parameters. The RTC approach is used to build a prototypes matrix which contains continuous variables, where each prototype vector represents correlated categorical data. Thereafter, the statistical ScreeTest is used to detect relevant and correlated features (or modalities) for each prototype. The proposed method requires simple computational techniques and the RTC topology technique is based on the principle of the self-organizing map (SOM) model. This method allows the dimensionality reduction, visualization and cluster characterization simultaneously. Empirical results based on real datasets from the UCI repository, are given and discussed.

Several aspects could affect the existing machine learning algorithms. One of these aspects is related to unbalanced classes in which the number of observations belonging to a class, greatly exceeds the observations in other classes. We propose in this paper an under-sampling method which uses self-organizing map to cluster the majority class guided with minority class. The proposed approach has been validated on multiple data sets using decision trees as a classifier with cross validation. The experimental results showed that elimination from majority class by integrating Neighborhood Cleaning Rule in SOM algorithm, produce high and very promising performance.

This paper introduces a new topological clustering formalism, dedicated to categorical data arising in the form of a binary matrix or a sum of binary matrices. The proposed approach is based on the principle of the Kohonen's model (conservation of topological order) and uses the Relational Analysis formalism by optimizing a cost function defined as a Condorcet criterion. We propose an hybrid algorithm, which deals linearly with large datasets, provides a natural clusters identification and allows a visualization of the clustering result on a two dimensional grid while preserving the a priori topological order of the data. The proposed approach called RTC was validated on several datasets and the experimental results showed very promising performances.

This paper introduces a relational topological map model, dedicated to multidimensional categorial data (or qualitative data) arising in the form of a binary matrix or a sum of binary matrices. This approach is based on the principle of Kohonen's model (conservation of topological order) and uses the Relational Analysis formalism by maximizing a modified Condorcet criterion. This proposed method is developed from the classical Relational Analysis approach by adding a neighborhood constraint to the Condorcet criterion. We propose a hybrid algorithm, which deals linearly with large data sets, provides a natural clusters identification and allows a visualization of the clustering result on a two-dimensional grid while preserving the a priori topological order of this data. The proposed approach called Relational Topological Map (RTM) was validated on several databases and the experimental results showed very promising performances.

Radio Frequency IDentification (RFID) is an advanced tracking technology that can be used to study the spatio-temporal behavior of customers in a supermarket. The aim of this work is to build a new RFID-based autonomous system to follow individuals' spatio-temporal activity, a tool not currently available, and to develop new methods for automatic data mining. Here, we study how to transform these data to investigate the customers' behaviors. We propose a new unsupervised data mining method to deal with this complex and very noisy data. This method is fast, efficient and allows some useful analysis to understand how the customers behave during shopping.

This paper presents a new learning strategy for the clustering algorithms based on Self-Organizing Map. Our contribution relies on the competitive phase of this unsupervised learning algorithm and proposes a new strategy for choosing the most active cell/neuron. This new strategy is to choose the most active neuron taking into account its historical activations, learned in a voting matrix from the dataset. Indeed, the use of this historic neighbourhood, allows introducing of topological constraints in the final geometry of the map. This new unsupervised learning approach allows discovering of data structure with a better quality (lower topographic error and better clustering purity). The proposed approach was validated on multiple datasets of different sizes and complexities, and the experimental validation shows promising results.

We introduce a new learning approach, which provides simultaneously self-organizing map (SOM) and local weight vector for each cluster. The proposed approach is computationally simple, and learns a different features vector weights for each cell (relevance vector). Based on the self-organizing map approach, we present two new simultaneously clustering and weighting algorithms: local weighting observation lwo-SOM and local weighting distance lwd-SOM. Both algorithms achieve the same goal by minimizing different cost functions. After learning phase, a selection method with weight vectors is used to prune the irrelevant variables and thus we can characterize the clusters. We illustrate the performance of the proposed approach using different data sets. A number of synthetic and real data are experimented on to show the benefits of the proposed local weighting using self-organizing models.

This paper deals with the problem of combining multiple clustering algorithms using the same data set to get a single consensus clustering. Our contribution is to formally define the cluster consensus problem as an optimization problem. to reach this goal, we propose an original existing algorithm but still relatively unknown method named relational analysis (RA). This method has several advantages among which we can quote: its low computational complexity, it does not require a number of clusters and does not neglect the weak clustering result. The unsupervised clustering consensus method implemented in this work is quite general. We evaluate the effectiveness of cluster consensus in three qualitatively different data sets. Promising results are provided in all three situations for synthetic as well as real data sets.

This paper introduces a new probabilistic topological map as generative model that includes mixture of Gaussian and Bernoulli distribution. This model is dedicated to cluster mixed data with continuous and categorical variables. This model is fitted by maximum likelihood using the EM algorithm. Examples using real data set allow to validate our model. The proposed approach has the advantage comparing to existing topological map of providing a set of prototype with the same coding as the learning data. More information is produced with this model that could be used in practical applications.

Determining the optimum number of clusters is an ill posed problem for which there is no simple way of knowing that number without a priori knowledge. The purpose of this paper is to provide a simultaneous two-level clustering algorithm based on self organizing map, called DS2L-SOM, which learn at the same time the structure of the data and its segmentation. The algorithm is based both on distance and density measures in order to accomplish a topographic clustering. An important feature of the algorithm is that the cluster number is discovered automatically. A great advantage of the proposed algorithm, compared to the common partitional clustering methods, is that it is not restricted to convex clusters but can recognize arbitrarily shaped clusters and touching clusters. The validity and the stability of this algorithm are superior to standard two-level clustering methods such as SOM+K-means and SOM+hierarchical agglomerative clustering. This is demonstrated on a set of critical clustering problems.

This paper introduces a probabilistic self-organizing map for topographic clustering, analysis and visualization of multivariate binary data or categorical data using binary coding. We propose a probabilistic formalism dedicated to binary data in which cells are represented by a Bernoulli distribution. Each cell is characterized by a prototype with the same binary coding as used in the data space and the probability of being different from this prototype. The learning algorithm, Bernoulli on self-organizing map, that we propose is an application of the EM standard algorithm. We illustrate the power of this method with six data sets taken from a public data set repository. The results show a good quality of the topological ordering and homogenous clustering.

Natural Language Processing has emerged as an active field of research in the machine learning community. Several methods based on statistical information have been proposed. However, with the linguistic complexity of the texts, semantic-based approaches have been investigated. In this paper, we propose a Semantic Kernel for semi- structured biomedical documents. The semantic meanings of words are extracted using the UMLS framework. The kernel, with a SVM classifier, has been applied to a text categorization task on a medical corpus of free text documents. The results have shown that the Semantic Kernel outperforms the Linear Kernel and the Naive Bayes classifier. Moreover, this kernel was ranked in the top ten of the best algorithms among 44 classification methods at the 2007 CMC Medical NLP International Challenge.

One of the most crucial questions in many real-world cluster applications is determining a suitable number of clusters, also known as the model selection problem. Determining the optimum number of clusters is an ill posed problem for which there is no simple way of knowing that number without a priori knowledge. In this paper we propose a new two-level clustering algorithm based on self organizing map, called S2L-SOM, which allows an automatic determination of the number of clusters during learning. Estimating true numbers of clusters is related to the cluster stability which involved the validity of clusters generated by the learning algorithm. To measure this stability we use the sub-sampling method. The great advantage of our proposed algorithm, compared to the common partitional clustering methods, is that it is not restricted to convex clusters but can recognize arbitrarily shaped clusters. The validity of this algorithm is superior to standard two-level clustering methods such as SOM+k-means and SOM+Hierarchical agglomerative clustering. This is demonstrated on a set of critical clustering problems.

This paper introduces a probabilistic self-organizing map for clustering, analysis and visualization of multivariate binary data. We propose a probabilistic formalism dedicated to binary data in which cells are represented by a Bernoulli distribution. Each cell is characterized by a prototype with the same binary coding as used in the data space and the probability of being different from this prototype. The learning algorithm, BeSOM, that we propose is an application of the EM standard algorithm. We illustrate the power of this method with two data sets taken from a public data set repository: a handwritten digit data set and a zoo data set. The results show a good quality of the topological ordering and homogenous clustering.

Spatio-temporal connectionist networks comprise an important class of neural models that can deal with patterns distributed in both time and space. In this article, we present new models of self-organizing maps for sequence clustering and classification. We have introduced the temporal dynamics in these maps and we have proposed several new models based on covariance matrices computation. In the first models, the inputs are modeled using its associated covariance matrix. These models, used in speaker recognition, do not take into account the order of the vectors in the sequence. To overcome this drawback, we have proposed new models, which introduce the temporal dynamics in the covariance matrix associated to the input sequences. In order to obtain a network that can learn new knowledge without forgetting the previous learned ones, we have introduced the plasticity and stability properties into one proposed temporal model using the adaptive resonance theory paradigm.

This work presents approaches for sequence clustering and classification. These approaches use the self organizing map "SOM". The inputs of the map are modelled in order to take into account the information and the correlation of the patterns contained in the sequences. The first approaches represent the input of the map by a representative vector or by a covariance matrix in order to take into account the correlations between the sequence components. These approaches do not take into account the temporal order in the sequences (the dynamics). The other approaches introduce the dynamics in the covariance matrix. When covariance matrices represent sequences, the SOM is modified in order to take into account the fact that the inputs are matrices. The experimentations show that our approaches are better than some other temporal self organizing maps for user Web navigation classification.

Proposes a new version of the "topological maps" algorithm, which has been used to cluster Web site visitors. These are characterized by partially redundant variables over time. In this version, we only consider those input vectors' neurons that participate in the selection of the winning neuron in the map. In order to identify these neurons, we use a binary function. Subsequently, we apply a partial modification on the weights that relates them to the winning neuron. Using this new version, we obtained a clustering of Web site visitors' behaviors, which has been difficult to analyse before. This clustering allows a recommendation system to satisfy the Web site visitors' needs based on their cluster membership at each step in time.

In Neural Networks (NN) applications, it is common practice to train several different networks and select the best one on the basis of performance on a validation set. There is a major disadvantage of such an approach: the network with the best performance on the validation set might not be the best one on new test data. In fact, the generalization performance on the validation set has a random component that is due to the data noise. An alternative approach to this problem is to use a combination of multiple NN classifiers. It is well known that such combinations can produce better performance than the best single network used in isolation. Several studies in different fields of the pattern recognition have experimentally shown that an appropriate combination of NN classifiers allows to capture complex phenomena, and to make decisions even with a great deal of information.3,11,12 The combination of multiple classifiers is a general problem in the pattern recognition area. Several methods for combining the outputs of multiple classifiers have been proposed: ensemble methods,14 boosting approaches,9 stacking techniques,27 multi-expert systems,2,16 and multi-modular architectures.3,13,18 A very rich synthesis of different methods of combining classifiers can be found in Ref. 28. An analytical framework to quantify the improvements in classification results that is due to combining classifiers is provided in Ref. 26. In this paper some contributions in combining multiple NN classifiers are presented. We give a combinational approach based on multi-modular systems for local diagnosis problem in the telephone network. In addition, we propose a combinational method for discrimination task using a fusion. The system validation will be performed on a disturbance identification problem.

A classifier's ability to respond to novel patterns is not unique, and different classifiers provide different generalization. We investigate the co-operation of two neural network (NN) MLP-based classifiers (with two different feature sets as entries), through a committee classifier implementing a modified generalized committee principle for the combined decision. The training and test phase are performed on the data extracted from the NIST database. A rejection criteria is implemented and the final decision of the committee classifier integrates the additional information derived from the output of the trained NN member classifiers. The final classification system is a multistage system integrating the rule-based reasoning with improved recognition and reliability rates.

In this paper, we propose a features selection measure and an architecture optimization procedure for Multi-Layer Perceptrons (MLP). The algorithm presented in this contribution employs a heuristic measure named HVS (Heuristic for Variable Selection). This new measure allows us to identify and select important variables in the features space. This can be achieved by eliminating redundant features and those which do not contain enough relevant information. The proposed measure is used in a new procedure aimed at selecting the "best" MLP architecture given an initial structure. Application results for two generic problems: regression and discrimination, demonstrates the proposed selection algorithm's effectiveness in identifying optimized connectionist models with higher accuracy. Finally, an extension of HVS, named epsilonHVS, is proposed for discriminative features detection and architecture optimization for Time Delay Neural Networks models (TDNN).

Irrelevant or redundant features may have negative effects on classification algorithms. The designer of a classification algorithm typically require a few very significant features characterising the class membership of the patterns. The discriminatory information is encoded in a very complex manner and features which are the most important for pattern classification may not be apparent. One way to address this problem is the use of feature weighting procedure as a data preprocessing step. In this paper we propose a two-step algorithm as an extension of the learning vector quantization algorithm (LVQ). This approach is based on weighting features depending on their contribution to discrimination. Adapting weighting coefficients and codewords is done simultaneously by using a new global learning algorithm named /spl omega/LVQ2. Experiments are undertaken on a synthetic problem and on real problems in speech and speaker recognition domain, to show significant improvement over the standard learning algorithm.

We propose an integrated approach to feature and architecture optimization for convolutional connectionist models. The goal is to select single features which are likely to have good discriminatory power and extract nonlinear combinations of features with the same aim. In particular, the focus is on the interaction of the feature extraction and selection modules with the recognizer design. We propose a pruning-based method called /spl epsi/HVS (extended HVS), where the use of a priori knowledge is adaptively optimized during a discrimination training criterion aiming at minimum classification error. Results demonstrate the selection approach's effectiveness in identifying reduced architectures with the same recognition accuracy.

This paper presents and evaluates a modular/hybrid connectionist system for speaker identification. Modularity has emerged as a powerful technique for reducing the complexity of connectionist systems, and allowing a priori knowledge to be incorporated into their design. Text-independent speaker identification is an inherently complex task where the amount of training data is often limited. It thus provides an ideal domain to test the validity of the modular/hybrid connectionist approach. To achieve such identification, we develop, in this paper, an architecture based upon the cooperation of several connectionist modules, and a Hidden Markov Model module. When tested on a population of 102 speakers extracted from the DARPA-TIMIT database, perfect identification was obtained.

This paper presents and evaluates a modular/hybrid connectionist system for speaker identification. Modularity has emerged as a powerful technique for reducing the complexity of connectionist systems, allowing a priori knowledge to be incorporated into their design. In problems where training data are scarce, such modular systems are likely to generalize significantly better than a monolithic connectionist system. In addition, modules are not restricted to be connectionist: hybrid systems, with e.g. Hidden Markov Models (HMMs), can be designed, combining the advantages of connectionist and non-connectionist approaches. Text independent speaker identification is an inherently complex task where the amount of training data is often limited. It thus provides an ideal domain to test the validity of the modular/hybrid connectionist approach. An architecture is developed in this paper which achieves this identification, based upon the cooperation of several connectionist modules, together with an HMM module. When tested on a population of 102 speakers extracted from the DARPA-TIMIT database, perfect identification was obtained. Overall, our recognition results are among the best for any text-independent speaker identification system handling this population size. In a specific comparison with a system based on multivariate auto-regressive models, the modular/hybrid connectionist approach was found to be significantly better in terms of both accuracy and speed. Our design also allows for easy incorporation of new speakers.

Using two speech recognition tasks, the authors compared the performance and behavior of time delay neural networks (NN), learning vector quantization, and a modular architecture. This set of experiments makes it possible to investigate the capabilities of the models and demonstrate some of their weaknesses. Good performance was obtained through the use of sophisticated architectures which encompass the limitations of more basic NN models. This is particularly clear for a phoneme experiment where it was possible to increase the performances until they were far better than those of traditional classifiers. This improvement was obtained in successive steps by using modified cost functions or algorithms and building a combined architecture. These results illustrate that current NN algorithms can be greatly improved. Modular architectures like the one used are a promising way to do this.>

The authors propose a novel model for text-independent talker identification which uses TDNN (time-delay neural network) extracted feature information. This model has been tested on 20 speakers (10 male and 10 female) from the TIMIT database using an LPC (linear predictive coding) parameterization. An average identification of 98% was observed.>

A connectionist approach to automatic speaker identification based on the learning vector quantization (VQ) algorithm is presented. For each adherent to the identification system, a number of references is fixed. The algorithm is based on a nearest-neighbor principle, with adaptation through learning. The identification is realized by comparing to a given threshold the distance of the unknown utterance to the nearest reference. Preliminary tests run on a ten-speaker set show an identification rate of 97% for MFC coefficients. The identification system and database used and the results obtained for different combinations of parameters are given. The system is evaluated by comparing its performances with a Bayesian system.>