• Propose SCS-SupCon, a supervised contrastive learning framework for image tasks. • Introduce sigmoid-based contrastive loss with learnable adaptive boundaries. • Emphasize hard negatives to reduce negative-sample dilution effectively. • Add style-distance constraint for better disentangling of style and content. • Achieve state-of-the-art accuracy on fine-grained benchmark datasets. Image classification can be inherently challenging due to subtle inter-class differences and substantial intra-class variations, limiting the effectiveness of existing contrastive learning approaches. In particular, supervised contrastive methods based on the InfoNCE loss often suffer from the negative-sample dilution issue and lack explicit mechanisms for adaptive decision-boundary control, significantly weakening their discriminative capability on fine-grained image classification tasks. To address these challenges, we propose a novel supervised contrastive learning framework, termed Sigmoid-based Common and Style Supervised Contrastive Learning (SCS-SupCon). In this framework, we introduce a sigmoid-based pairwise contrastive loss with adaptive decision boundaries, explicitly parameterized by learnable temperature and bias terms. This design places greater emphasis on critical discriminative information from hard negatives, thereby alleviating the problem of negative-sample dilution while fully leveraging supervision signals in contrastive learning. Furthermore, we incorporate an explicit style-distance constraint to disentangle style and content representations, leading to more robust and discriminative feature learning. Comprehensive experiments on six benchmark datasets, including prominent fine-grained datasets such as CUB200-2011 and Stanford Dogs, consistently demonstrate that our proposed SCS-SupCon achieves superior performance over the most closely related InfoNCE-based supervised contrastive baselines (SupCon, SelfCon, CS-SupCon and its overlapping variant) across diverse CNN and Transformer backbones. In particular, on CIFAR-100 with a ResNet-50 encoder, SCS-SupCon improves the mean top-1 accuracy over SupCon by about 3.9 percentage points and over CS-SupCon by about 1.7 percentage points under a five-fold cross-validation protocol. On challenging fine-grained datasets such as CUB200-2011 and Stanford Dogs with both CNN and Transformer architectures, our method achieves absolute improvements of approximately 0.4–3.0 percentage points over CS-SupCon. Extensive ablation studies and paired statistical tests further confirm the robustness and effectiveness of our framework, and a Friedman test with Nemenyi post-hoc analysis shows that SCS-SupCon attains the best average rank among the evaluated methods, even though pairwise differences with other strong competitors are not always statistically significant at the 0.05 level.

Multi-view semi-supervised classification (Mv-SSC) aims to leverage complementary information from multiple views and unlabeled data to enhance classification performance. However, existing methods often struggle with constructing robust graphs, effectively fusing heterogeneous features. Therefore, we propose a novel framework, Multi-view Semi-supervised Classification with Graph Construction Innovation and Smoothness-aware Graph Convolution (GCSGC), which introduces three key innovations. First, we employ sparse autoencoders (SAEs) coupled with a learnable feature fusion module based on a one-layer fully connected network. The SAEs extract compact and discriminative representations from each view by enforcing sparsity to remove redundancies, while the fully connected layer adaptively aligns the multi-view latent features with the learnable shared features to preserve complementary semantic information. Second, GCSGC introduces two types of view-based graphs: (1) a Cosine-KNN-based Collaborative Graph (CKG) that integrates K-Nearest Neighbor relationships with cosine similarity of the learned view-specific latent features, and (2) a robust semi-supervised graph that enhances inter-view consistency and structural stability. Third, we design a hybrid loss function that combines cross-entropy and smoothness regularization to guide the optimization of the graph convolutional network. Extensive experiments on multiple benchmark datasets demonstrate that GCSGC consistently outperforms state-of-the-art Mv-SSC methods, validating the effectiveness of its innovative graph construction strategy, sparse feature fusion mechanism, and dual-loss optimization. This work provides a new perspective on integrating structural modeling, feature transformation, and loss design in multi-view semi-supervised learning.

The advent of graph convolutional network (GCN)-based multi-view learning provides a powerful framework for integrating structural information from heterogeneous views, enabling effective modeling of complex multi-view data. However, existing methods often fail to fully exploit the complementary information across views, leading to suboptimal feature representations and limited performance. To address this, we propose MV-SupGCN, a semi-supervised GCN model that integrates several complementary components with clear motivations and mutual reinforcement. First, to better capture discriminative features and improve model generalization, we design a joint loss function that combines Cross-Entropy loss with Supervised Contrastive loss, encouraging the model to simultaneously minimize intra-class variance and maximize inter-class separability in the latent space. Second, recognizing the instability and incompleteness of single graph construction methods, we combine both KNN-based and semi-supervised graph construction approaches on each view, thereby enhancing the robustness of the data structure representation and reducing generalization error. Third, to effectively utilize abundant unlabeled data and enhance semantic alignment across multiple views, we propose a unified framework that integrates contrastive learning in order to enforce consistency among multi-view embeddings and capture meaningful inter-view relationships, together with pseudo-labeling, which provides additional supervision applied to both the cross-entropy and contrastive loss functions to enhance model generalization. Extensive experiments demonstrate that MV-SupGCN consistently surpasses state-of-the-art methods across multiple benchmarks, validating the effectiveness of our integrated approach. The source code is available at https://github.com/HuaiyuanXiao/MVSupGCN

Age estimation from facial images is a complex and multifaceted challenge in computer vision. In this study, we present a novel hybrid architecture that combines ConvNeXt, a state-of-the-art advancement of convolutional neural networks (CNNs), with Vision Transformers (ViT). While each model independently delivers excellent performance on a variety of tasks, their integration leverages the complementary strengths of the CNNs’ localized feature extraction capabilities and the Transformers’ global attention mechanisms. Our proposed ConvNeXt-ViT hybrid solution was thoroughly evaluated on benchmark age estimation datasets, including MORPH II, CACD, and AFAD, and achieved superior performance in terms of mean absolute error (MAE). To address computational constraints, we leverage pre-trained models and systematically explore different configurations, using linear layers and advanced regularization techniques to optimize the architecture. Comprehensive ablation studies highlight the critical role of individual components and training strategies, and in particular emphasize the importance of adapted attention mechanisms within the CNN framework to improve the model’s focus on age-relevant facial features. The results show that the ConvNeXt-ViT hybrid not only outperforms traditional methods, but also provides a robust foundation for future advances in age estimation and related visual tasks. This work underscores the transformative potential of hybrid architectures and represents a promising direction for the seamless integration of CNNs and transformers to address complex computer vision challenges. • We developed a hybrid deep learning model for age estimation from facial images . • It integrates ViT with ConvNeXt , retaining both global and local features. • This fusion captures both fine-grained facial details and holistic structural cues . • It was evaluated on four public datasets : MORPH II, CACD, AFAD, and IMDB-Clean . • It achieved state-of-the-art performance across all datasets, demonstrating its robustness.

Facial beauty prediction (FBP) is an emerging area of artificial intelligence (AI) that focuses on the development of models that analyze facial features to assess beauty, based on human perception. This task is particularly challenging due to the subjective nature of beauty and limited resources. Deep learning methods have proven their exceptional ability to capture complex features and are therefore well suited for FBP tasks. This paper reviews recent advances in FBP, focusing on deep learning techniques and benchmark datasets. A proposed taxonomy organizes the main methods according to their design and applications, and comparative analyzes highlight trends and the performance of different models. Finally, the study outlines future research directions to drive progress in this evolving field.

Multi-view data significantly improves the accuracy of machine learning algorithms by providing a holistic representation of object features. However, previous research on the use of Graph Convolutional Networks (GCNs) for processing node connectivity and data features is still limited. Current methods mainly focus on weighted summation of graph matrices, while only a few integrate the features into graphs. To address these challenges, this paper presents an integrated deep learning architecture: the Feature Fusion and Multi-Graph Fusion Learning Framework (CGCN-FMF). The framework consists of two key modules: (1) a Feature Fusion Network designed to extract essential features from multiple views, and (2) a Multi-Graph Fusion Network that constructs multiple graphs for each view and optimizes both the graph weights and the GCN model. Experimental results from multiple multi-view datasets show that CGCN-FMF outperforms state-of-the-art methods on semi-supervised multi-view classification tasks. • We address deep graph-based semi-supervised classification for multi-view data. • Data structures are set using both KNN graphs and semi-supervised graph schemes. • Autoencoders and 1D convolution layers are leveraged to learn unified features. • A learnable attention mechanism and staged training approach are employed. • Extensive experiments are conducted on six publicly available multi-view datasets.

In the realm of graph-based semi-supervised learning (GSSL), traditional methodologies often struggle to effectively handle labeled samples and scale to accommodate large datasets. To increase supervision information in semi-supervised learning, the self-training paradigm is often used, mainly in datasets with moderate sizes. On the other hand, the use of anchors was adopted with large datasets. In this research endeavor, we propose a novel framework for GSSL that leverages a novel self-training principle tailored for very large datasets, and introduces an advanced method for automatic graph construction using anchors. Our approach focuses on utilizing generated labels of random batches of unlabeled samples, subsequently incorporating these predictions into the training set to enhance the model’s accuracy. Pseudo-labeling, a specific instance of self-training, assigns pseudo-labels to the most confidently predicted unlabeled examples, treating them as ground truth during the training phase. By constructing anchor-to-anchor affinity graphs that incorporate both feature and label information, our method facilitates robust learning on large-scale datasets. Through comprehensive experimentation across diverse large datasets, our approach demonstrates its efficacy in achieving scalable and reliable semi-supervised learning outcomes. These findings represent a significant advancement in the field of GSSL, with wide-ranging implications for various applications across different domains. Our method not only addresses the scalability issue but also ensures the effective integration of both labeled and pseudo labeled data, thereby enhancing the overall learning process.

Semi-supervised learning with a graph-based approach has become increasingly popular in machine learning, particularly when dealing with situations where labeling data is a costly process. Graph Convolution Networks (GCNs) have been widely employed in semi-supervised learning, primarily on graph-structured data like citations and social networks. However, there exists a significant gap in applying these methods to non-graph multi-view data, such as collections of images. To bridge this gap, we introduce a novel deep semi-supervised multi-view classification model tailored specifically for non-graph data. This model independently reconstructs individual graphs using a powerful semi-supervised approach and subsequently merges them adaptively into a unified consensus graph. The consensus graph feeds into a unified GCN framework incorporating a label smoothing constraint. To assess the efficacy of the proposed model, experiments were conducted across seven multi-view image datasets. Results demonstrate that this model excels in both the graph generation and semi-supervised classification phases, consistently outperforming classical GCNs and other existing semi-supervised multi-view classification approaches. 1 1 Source code: https://github.com/BiJingjun/SCFG .

Abstract The goal of multi-view clustering is to partition unlabeled objects into disjoint clusters or groups using consistent and complementary information provided by the different features of the same object. Most existing methods perform this clustering task sequentially: Computation of the individual or consistent graph matrices, spectral embedding, and clustering. In this work, we present an approach that can overcome some of the limitations of previous multiview clustering methods. We introduce a single objective function whose minimization can jointly determine the consistent graph matrix for all views, the unified spectral data representation, the soft cluster indices, and the view weights. We present a constraint term that relates the cluster indices to the convolution of the consistent spectral data representations over the consistent graph. The method we present has two interesting features that are not simultaneously present in recent work. First, the cluster indices can be estimated directly without the need for an additional clustering step, which depends heavily on initialization. Second, the soft cluster indices are directly linked to the kernel representation of the features of the views. Moreover, our proposed method automatically determines the weights of each view, thus requiring fewer hyperparameters. A series of experiments have been conducted on real datasets. These demonstrate the efficiency of the proposed method, which compares favorably to many multi-view clustering methods.

The development of a robust and adaptive deep learning technique for the diagnosis of pneumonia and the assessment of its severity was a major challenge. Indeed, both chest X-rays (CXR) and CT scans have been widely studied for the diagnosis, detection and quantification of pneumonia. In this paper, a novel approach (PViTGAtt-IP) based on a parallel array of vision transformers is presented, in which the input image is divided into regions of interest. Each region is fed into an individual model and the collective output gives the severity score. Three parallel architectures were also derived and tested. The proposed models were subjected to rigorous tests on two different datasets: RALO CXRs and Per COVID-19 CT scans. The experimental results showed that the proposed models exhibited high performance in accurately predicting scores for both datasets. In particular, the parallel transformers with multi-gate attention proved to be the best performing model. Furthermore, a comparative analysis using state-of-the-art methods showed that our proposed approach consistently achieved competitive or even better performance in terms of the Mean Absolute Error (MAE) and the Pearson Correlation Coefficient (PC). This emphasizes the effectiveness and superiority of our models in the context of diagnosing and assessing the severity of pneumonia.

Data augmentation can mitigate overfitting problems in data exploration without increasing the size of the model. Existing cutmix-based data augmentation has been proven to significantly enhance deep learning performance. However, many existing methods overlook the discriminative local context of the image and rely on ad hoc regions consisting of square or rectangular local regions, resulting in the loss of complete semantic object parts. In this work, we propose a superpixelwise local-context-aware efficient image mixing approach for data augmentation, aiming to overcome the limitations previously mentioned. Our approach only requires one forward propagation using a superpixel attention-based label mixing with lower computational complexity. The model is trained using a combination of a global classification of the mixed (augmented) image loss, a superpixel-wise weighted local classification loss, and a superpixel-based weighted contrastive learning loss. The last two losses are based on the superpixel-aware attentive embeddings. Thus, the resulting deep encoder can learn both local and global features of the images, capturing object-part local context information. Experiments on diverse benchmarks, such as ImageNet-1K and CUB-200-2011, indicate that the proposed method out-performs many augmentation methods for visual recognition. We have not only demonstrated its effectiveness on CNN models, but also on transformer models.

Automatic classification methods widely used for diagnosing and analyzing COVID-19 cases. These methods assume known labels and rely on a single view of the dataset. Given the prevalence of COVID-19 cases and the extensive volume of patient records lacking labels, this communication underscores our unique approach—conducting the first study on COVID-19 case diagnosis in an unsupervised manner. Our work operates under the assumption of prior knowledge regarding the number of classes, such as COVID-19, pneumonia, and normal, in a case study. By adopting an unsupervised learning paradigm, we leverage the wealth of unlabeled data, reducing dependence on human experts for annotating numerous images. This paper introduces an enhanced version of a recent direct method where non-negative cluster indices and spectral embeddings are jointly estimated. Beyond the inherent advantages of this method, our proposed model introduces improvements through two additional types of constraints: (i) ensuring consistent smoothing of cluster labels across all views and (ii) imposing an orthogonality constraint on the matrix of cluster assignments. The efficacy of the proposed method is demonstrated using the public COVIDx dataset with three classes, showcasing promising results in categorizing radiographs. The proposed approach is tested on other public image datasets to assess its effectiveness.

Recently, clustering data collected from various sources has become a hot topic in real-world applications. The most common methods for multi-view clustering can be divided into several categories: Spectral clustering algorithms, subspace multi-view clustering algorithms, matrix factorization approaches, and kernel methods. Despite the high performance of these methods, they directly fuse all similarity matrices of all views and separate the affinity learning process from the multiview clustering process. The performance of these algorithms can be affected by noisy affinity matrices. To overcome this drawback, this paper presents a novel method called One Step Multi-view Clustering via Consensus Graph Learning and Nonnegative Embedding (OSMGNE). Instead of directly merging the similarity matrices of different views, which may contain noise, a step of learning a consensus similarity matrix is performed. This step forces the similarity matrices of different views to be too similar, which eliminates the problem of noisy data. Moreover, the use of the nonnegative embedding matrix (soft cluster assignment matrix makes it possible to directly obtain the final clustering result without any extra step. The proposed method can solve five subtasks simultaneously. It jointly estimates the similarity matrix of all views, the similarity matrix of each view, the corresponding spectral projection matrix, the unified clustering indicator matrix, and automatically gives the weight of each view without the use of hyper-parameters. In addition, another version of our method is also studied in this paper. This method differs from the first one by using a consensus spectral projection matrix and a consensus Laplacian matrix over all views. An iterative algorithm is proposed to solve the optimization problem of these two methods. The two proposed methods are tested on several real datasets, which prove their superiority.

Clustering data from different sources or views is a key challenge in real-world applications. While traditional graph-based methods are effective at capturing data structures, they often require separate steps to estimate graphs of views or a consensus graph from the raw data. This reliance on intermediate steps can make these clustering methods susceptible to noisy graphs, which affects the overall performance of clustering. In response to this limitation, and with an emphasis on advocating end-to-end solutions for multi-view clustering, two comprehensive approaches are presented in this paper. Each approach starts from either the raw data or its kernelized features. The first proposal introduces a unified objective function that enables the simultaneous recovery of the graph for each view, the unified graph, the spectral projection matrices for all views, the soft cluster assignments, and the scores assigned to each view. The second proposal uses a global criterion that integrates regularization and constraints for the soft cluster assignment matrix based on the consensus graph matrix and the consensus data representation. Both proposed methods enable direct and straightforward clustering of the data without the need for additional steps. Extensive tests with various real-world image and text datasets confirm the superior performance of the two proposed methods.

Cutmix-based data augmentation, which uses a cut-and-paste strategy, has shown remarkable generalization capabilities in deep learning. However, existing methods primarily consider global semantics with image-level constraints, which excessively reduces attention to the discriminative local context of the class and leads to a performance improvement bottleneck. Moreover, existing methods for generating augmented samples usually involve cutting and pasting rectangular or square regions, resulting in a loss of object part information. To mitigate the problem of inconsistency between the augmented image and the generated mixed label, existing methods usually require double forward propagation or rely on an external pre-trained network for object centering, which is inefficient. To overcome the above limitations, we propose LGCOAMix, an efficient context-aware and object-part-aware superpixel-based grid blending method for data augmentation. To the best of our knowledge, this is the first time that a label mixing strategy using a superpixel attention approach has been proposed for cutmix-based data augmentation. It is the first instance of learning local features from discriminative superpixel-wise regions and cross-image superpixel contrasts. Extensive experiments on various benchmark datasets show that LGCOAMix outperforms state-of-the-art cutmix-based data augmentation methods on classification tasks, and weakly supervised object location on CUB200-2011. We have demonstrated the effectiveness of LGCOAMix not only for CNN networks, but also for Transformer networks. Source codes are available at https://github.com/DanielaPlusPlus/LGCOAMix.

Multi-view clustering aims to partition unlabeled patterns into disjoint clusters using consistent and complementary information derived from features of patterns in multiple views. Downstream methods perform this clustering sequentially: estimation of individual or consistent similarity matrices, spectral embedding, and clustering. In this article, we present an approach that can address some of the shortcomings of previous multiview clustering methods. We propose a single objective function whose optimization can jointly provide the consistent graph matrix for all views, the unified spectral data representation, the cluster assignments, and the view weights. We propose a new constraint term that sets the cluster index matrix to the convolution of the consistent spectral projection matrix over the consistent graph. Our proposed scheme has two interesting properties that the recent works do not have simultaneously. First, the cluster assignments can be estimated directly without the need for an additional clustering phase, which depends heavily on initialization. Second, the soft cluster assignments are directly linked to the kernel representation of the features of the views. Moreover, our method automatically computes the weights of each view, requiring fewer hyperparameters. We have conducted a series of experiments on real datasets. These demonstrate the effectiveness of the proposed approach, which compares favorably to many competing multi-view clustering methods.

Recently, graph-based semi-supervised learning (GSSL) has garnered significant interest in the realms of machine learning and pattern recognition. Although some of the proposed methods have made some progress, there are still some shortcomings that need to be overcome. There are three main limitations. First, the graphs used in these approaches are usually predefined regardless of the task at hand. Second, due to the use of graphs, almost all approaches are unable to process and consider data with a very large number of unlabeled samples. Thirdly, the imbalance of the topology of the samples is very often not taken into account. In particular, processing large datasets with GSSL might pose challenges in terms of computational resource feasibility. In this article, we present a scalable and inductive GSSL method. We broaden the scope of the graph topology imbalance paradigm to extensive databases. Second, we employ the calculated weights of the labeled sample for the label-matching term in the global objective function. This leads to a unified, scalable, semi-supervised learning model that allows simultaneous labeling of unlabeled data, projection of the feature space onto the labeling space, along with the graph matrix of anchors. In the proposed scheme, the integration of labels and features from anchors is applied for the adaptive construction of the anchor graph. Experimental results were performed on four large databases: NORB, RCV1, Covtype, and MNIST. These experiments demonstrate that the proposed method exhibits superior performance when compared to existing scalable semi-supervised learning models.

Abstract In recent years, estimating beauty of faces has attracted growing interest in the fields of computer vision and machine learning. This is due to the emergence of face beauty datasets (such as SCUT-FBP, SCUT-FBP5500 and KDEF-PT) and the prevalence of deep learning methods in many tasks. The goal of this work is to leverage the advances in Deep Learning architectures to provide stable and accurate face beauty estimation from static face images. To this end, our proposed approach has three main contributions. To deal with the complicated high-level features associated with the FBP problem by using more than one pre-trained Convolutional Neural Network (CNN) model, we propose an architecture with two backbones (2B-IncRex). In addition to 2B-IncRex, we introduce a parabolic dynamic law to control the behavior of the robust loss parameters during training. These robust losses are ParamSmoothL1, Huber, and Tukey. As a third contribution, we propose an ensemble regression based on five regressors, namely Resnext-50, Inception-v3 and three regressors based on our proposed 2B-IncRex architecture. These models are trained with the following dynamic loss functions: Dynamic ParamSmoothL1, Dynamic Tukey, Dynamic ParamSmoothL1, Dynamic Huber, and Dynamic Tukey, respectively. To evaluate the performance of our approach, we used two datasets: SCUT-FBP5500 and KDEF-PT. The dataset SCUT-FBP5500 contains two evaluation scenarios provided by the database developers: 60-40% split and five-fold cross-validation. Our approach outperforms state-of-the-art methods on several metrics in both evaluation scenarios of SCUT-FBP5500. Moreover, experiments on the KDEF-PT dataset demonstrate the efficiency of our approach for estimating facial beauty using transfer learning, despite the presence of facial expressions and limited data. These comparisons highlight the effectiveness of the proposed solutions for FBP. They also show that the proposed Dynamic robust losses lead to more flexible and accurate estimators.

Since the emergence of the Covid-19 pandemic in late 2019, medical imaging has been widely used to analyze this disease. Indeed, CT-scans of the lungs can help diagnose, detect, and quantify Covid-19 infection. In this paper, we address the segmentation of Covid-19 infection from CT-scans. To improve the performance of the Att-Unet architecture and maximize the use of the Attention Gate, we propose the PAtt-Unet and DAtt-Unet architectures. PAtt-Unet aims to exploit the input pyramids to preserve the spatial awareness in all of the encoder layers. On the other hand, DAtt-Unet is designed to guide the segmentation of Covid-19 infection inside the lung lobes. We also propose to combine these two architectures into a single one, which we refer to as PDAtt-Unet. To overcome the blurry boundary pixels segmentation of Covid-19 infection, we propose a hybrid loss function. The proposed architectures were tested on four datasets with two evaluation scenarios (intra and cross datasets). Experimental results showed that both PAtt-Unet and DAtt-Unet improve the performance of Att-Unet in segmenting Covid-19 infections. Moreover, the combination architecture PDAtt-Unet led to further improvement. To Compare with other methods, three baseline segmentation architectures (Unet, Unet++, and Att-Unet) and three state-of-the-art architectures (InfNet, SCOATNet, and nCoVSegNet) were tested. The comparison showed the superiority of the proposed PDAtt-Unet trained with the proposed hybrid loss (PDEAtt-Unet) over all other methods. Moreover, PDEAtt-Unet is able to overcome various challenges in segmenting Covid-19 infections in four datasets and two evaluation scenarios.

In recent years, semi-supervised learning on graphs has gained importance in many fields and applications. The goal is to use both partially labeled data (labeled examples) and a large amount of unlabeled data to build more effective predictive models. Deep Graph Neural Networks (GNNs) are very useful in both unsupervised and semi-supervised learning problems. As a special class of GNNs, Graph Convolutional Networks (GCNs) aim to obtain data representation through graph-based node smoothing and layer-wise neural network transformations. However, GCNs have some weaknesses when applied to semi-supervised graph learning: (1) it ignores the manifold structure implicitly encoded by the graph; (2) it uses a fixed neighborhood graph and focuses only on the convolution of a graph, but pays little attention to graph construction; (3) it rarely considers the problem of topological imbalance. To overcome the above shortcomings, in this paper, we propose a novel semi-supervised learning method called Re-weight Nodes and Graph Learning Convolutional Network with Manifold Regularization (ReNode-GLCNMR). Our proposed method simultaneously integrates graph learning and graph convolution into a unified network architecture, which also enforces label smoothing through an unsupervised loss term. At the same time, it addresses the problem of imbalance in graph topology by adaptively reweighting the influence of labeled nodes based on their distances to the class boundaries. Experiments on 8 benchmark datasets show that ReNode-GLCNMR significantly outperforms the state-of-the-art semi-supervised GNN methods.1

Regional dropout strategies have demonstrated to be very effective in improving both the performance and the generalization capability of deep learning models. However, when such strategies are performed in a totally random manner, the background noise and label mismatch problems arise. To tackle such problems, existing approaches typically focus on regions with the highest distinctiveness. Yet, there are two main drawbacks of existing approaches: (I) Many existing region-based augmentation methods can only use rectangular regions, resulting in the loss of object contour information; (II) Deterministic selection of the most discriminative regions leads to poor diversification in data augmentation. In fact, a trade-off is needed between diversification and concentration, which can decrease the undesirable noise. In this paper, we propose a novel object-centric contour-aware CutMix data augmentation strategy with arbitrary- shape and size superpixel supports, which is hereafter referred to as OcCaMix for short. It not only captures the most discriminative regions, but also effectively preserves the contour details of the objects. Moreover, it enables the search of natural object parts of different sizes. Extensive experiments on a large number of benchmark datasets show that OcCaMix significantly outperforms state-of-the-art CutMix based data augmentation methods in classification tasks. The source codes and trained models are available at https://github.com/DanielaPlusPlus/OcCaMix.

Clustering is inherently a process of exploratory data analysis. It has attracted more attention recently because much real-world data consists of multiple representations or views. However, it becomes increasingly problematic when dealing with large and heterogeneous data. It is worth noting that several approaches have been developed to increase computational efficiency, although most of them have some drawbacks: (1) Most existing techniques consider equal or static weights to quantify importance across different views and samples, so common and complementary features cannot be used. (2) The clustering task is performed by arbitrary initialization without caring about the rich structure of the joint discrete representation, and thus poorly executed. In this paper, we propose a novel approach called “Auto-Weighted Binary Multi-View Clustering Via Deep Initialization” for large-scale multi-view clustering based on two main scenarios. First, we consider the distinction between different views based on the importance of samples, and therefore apply a dynamic learning strategy for the automatic weighting of views and samples. Second, in the context of initializing binary clustering, we develop a new CNN feature and use a low-dimensional binary embedding by exploiting the efficient capabilities of Fourier mapping. Moreover, our approach simultaneously learns a joint discrete representation and performs direct clustering using a constrained binary matrix factorization; the optimization problem is perfectly solved in a unified learning model. Experimental results conducted on several challenging datasets demonstrate the effectiveness and superiority of the proposed approach over state-of-the-art methods in terms of accuracy, normalized mutual information, and purity.

Recently, structured computing has become an interesting topic in the world of artificial intelligence, especially in the field of machine learning, as most researchers focus on the development of graph-based semi-supervised learning models. In this article, we present a new framework for graph-based semi-supervised learning. We present a powerful method for simultaneous label inference and linear transform estimation. The targeted linear transformation is used to obtain a discriminant subspace. To improve semi-supervised learning, our framework focuses on exploiting the data structure and soft labels of the available unlabeled samples. In the iterative optimization scheme used, the prior estimation of the label increases the supervision information indirectly through an introduced informative matrix called the label graph, thus avoiding the use of hard confidence-based decisions as used in self-supervised methods. In addition, the estimation of labels and projected data is made more robust by using smoothing concepts based on hybrid graphs. For each type of smoothing, the hybrid graph is an adaptive fusion of the two graphs encoding the similarity of the data and the similarity of the labels. The proposed method leads to an improved discriminative linear transformation. Several experimental results on real image datasets confirm the effectiveness of the proposed method. They also show superior performance compared to semi-supervised methods that use integration and label inference simultaneously.

Graph Convolutional Networks (GCNs) have received a lot of attention in pattern recognition and machine learning. In this paper, we present a revisited scheme for the new method called ”GCNs with Manifold Regularization” (GCNMR). While manifold regularization can add additional information, the GCN-based semi-supervised classification process cannot consider the full layer-wise structured information. Inspired by graph-based label propagation approaches, we will integrate high-order feature propagation into each GCN layer. High-order feature propagation over the graph can fully exploit the structured information provided by the latter at all the GCN’s layers. It fully exploits the clustering assumption, which is valid for structured data but not well exploited in GCNs. Our proposed scheme would lead to more informative GCNs. Using the revisited model, we will conduct several semi-supervised classification experiments on public image datasets containing objects, faces and digits: Extended Yale, PF01, Caltech101 and MNIST. We will also consider three citation networks. The proposed scheme performs well compared to several semi-supervised methods. With respect to the recent GCNMR approach, the average improvements were 2.2%, 4.5%, 1.0% and 10.6% on Extended Yale, PF01, Caltech101 and MNIST, respectively.

Recently, the estimation of facial age has attracted much attention. This letter extends and improves a recently developed method (Guehairia et al., 2020) for fusing multiple deep facial features for age estimation. This method was based on deep random forests. We propose a new pipeline that integrates tensor-based subspace learning before applying DRFs. Deep face features of a training set are represented as a 3D tensor. Multi-linear Whitened Principal Component (MWPCA) and Tensor Exponential Discriminant (TEDA) are used to extract the most discriminative information. The tensor subspace features are then fed into DRFs to predict age. Experiments conducted on five public face databases show that our method can compete with many state-of-the-art methods.

Recently, one-step clustering methods have shown good performance. However, very few one-step approaches have addressed the multi-view case, where an instance may have multiple representations. Data can be represented with multiple heterogeneous views. Clustering with multiple views faces the challenge of how to combine all the different views. A general scheme is to represent the views by view-based graphs and/or a consensus graph. Graphs can be well suited for clustering problems since they can capture the local and global structure of the data. In this paper, we present a novel approach to one-step graph-based multi-view clustering. In contrast to existing graph-based one-step clustering methods, our proposed method introduces two key innovations. First, we build an additional graph by using the cluster label correlation to the graphs associated with the data space. Second, a smoothing constraint is exploited to constrain the cluster-label matrix and make it more consistent with the original data graphs as well as with and label graphs. Experimental results on several public datasets show the efficiency of the proposed approach. All cluster evaluation metrics show significant improvement by applying our method to different types and sizes of datasets. The average improvement (across all datasets) is the difference between the indicator obtained by our approach and the indicator obtained by the most competitive method. The average improvement is approximately 4%, 2%, 3%, and 2% for the Accuracy indicator, the Normalized Mutual Information indicator, the Purity indicator, and the Adjusted Rand index, respectively.

In the last decade, several studies have shown that facial attractiveness can be learned by machines. In this paper, we address Facial Beauty Prediction from static images. The paper contains three main contributions. First, we propose a two-branch architecture (REX-INCEP) based on merging the architecture of two already trained networks to deal with the complicated high-level features associated with the FBP problem. Second, we introduce the use of a dynamic law to control the behaviour of the following robust loss functions during training: ParamSmoothL1, Huber and Tukey. Third, we propose an ensemble regression based on Convolutional Neural Networks (CNNs). In this ensemble, we use both the basic networks and our proposed network (REX-INCEP). The proposed individual CNN regressors are trained with different loss functions, namely MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey. Our approach is evaluated on the SCUT-FBP5500 database using the two evaluation scenarios provided by the database creators: 60%–40% split and five-fold cross-validation. In both evaluation scenarios, our approach outperforms the state of the art on several metrics. These comparisons highlight the effectiveness of the proposed solutions for FBP. They also show that the proposed dynamic robust losses lead to more flexible and accurate estimators.

Recently, multi-view clustering has received much attention in the fields of machine learning and pattern recognition. Spectral clustering for single and multiple views has been the common solution. Despite its good clustering performance, it has a major limitation: it requires an extra step of clustering. This extra step, which could be the famous k-means clustering, depends heavily on initialization, which may affect the quality of the clustering result. To overcome this problem, a new method called Multi-view Clustering via Consensus Graph Learning and Nonnegative Embedding (MVCGE) is presented in this paper. In the proposed approach, the consensus affinity matrix (graph matrix), consensus representation and cluster index matrix (nonnegative embedding) are learned simultaneously in a unified framework. Our proposed method takes as input the different kernel matrices corresponding to the different views. The proposed learning model integrates two interesting constraints: (i) the cluster indices should be as smooth as possible over the consensus graph and (ii) the cluster indices are set to be as close as possible to the graph convolution of the consensus representation. In this approach, no post-processing such as k-means or spectral rotation is required. Our approach is tested with real and synthetic datasets. The experiments performed show that the proposed method performs well compared to many state-of-the-art approaches.

Abstract The performance of machine learning and pattern recognition algorithms generally depends on data representation. That is why, much of the current effort in performing machine learning algorithms goes into the design of preprocessing frameworks and data transformations able to support effective machine learning. The method proposed in this work consists of a hybrid linear feature extraction scheme to be used in supervised multi-class classification problems. Inspired by two recent linear discriminant methods: robust sparse linear discriminant analysis (RSLDA) and inter-class sparsity-based discriminative least square regression (ICS_DLSR), we propose a unifying criterion that is able to retain the advantages of these two powerful methods. The resulting transformation relies on sparsity-promoting techniques both to select the features that most accurately represent the data and to preserve the row-sparsity consistency property of samples from the same class. The linear transformation and the orthogonal matrix are estimated using an iterative alternating minimization scheme based on steepest descent gradient method and different initialization schemes. The proposed framework is generic in the sense that it allows the combination and tuning of other linear discriminant embedding methods. According to the experiments conducted on several datasets including faces, objects, and digits, the proposed method was able to outperform competing methods in most cases.

Abstract Over the past two decades, several studies have paid great attention to biometric palmprint recognition. Recently, most methods in literature adopted deep learning due to their high recognition accuracy and the capability to adapt with different acquisition palmprint images. However, high-dimensional data with a large number of uncorrelated and redundant features remain a challenge due to computational complexity issues. Feature selection is a process of selecting a subset of relevant features, which aims to decrease the dimensionality, reduce the running time, and improve the accuracy. In this paper, we propose efficient unimodal and multimodal biometric systems based on deep learning and feature selection. Our approach called simplified PalmNet–Gabor concentrates on the improvement of the PalmNet for fast recognition of multispectral and contactless palmprint images. Therefore, we used Log-Gabor filters in the preprocessing to increase the contrast of palmprint features. Then, we reduced the number of features using feature selection and dimensionality reduction procedures. For the multimodal system, we fused modalities at the matching score level to improve system performance. The proposed method effectively improves the accuracy of the PalmNet and reduces the number of features as well the computational time. We validated the proposed method on four public palmprint databases, two multispectral databases, CASIA and PolyU, and two contactless databases, Tongji and PolyU 2D/3D. Experiments show that our approach achieves a high recognition rate while using a substantially lower number of features.

Eye blink detection is a challenging problem that many researchers are working on because it has the potential to solve many facial analysis tasks, such as face anti-spoofing, driver drowsiness detection, and some health disorders. There have been few attempts to detect blinking in the wild scenario, while most of the work has been done under controlled conditions. Moreover, current learning approaches are designed to process sequences that contain only a single blink ignoring the case of the presence of multiple eye blinks. In this work, we propose a fast framework for eye blink detection and eye blink verification that can effectively extract multiple blinks from image sequences considering several challenges such as lighting changes, variety of poses, and change in appearance. The proposed framework employs fast landmarks detector to extract multiple facial key points including the ones that identify the eye regions. Then, an SVD-based method is proposed to extract the potential eye blinks in a moving time window that is updated with new images every second. Finally, the detected blink candidates are verified using a 2D Pyramidal Bottleneck Block Network (PBBN). We also propose an alternative approach that uses a sequence of frames instead of an image as input and employs a continuous 3D PBBN that follows most of the state-of-the-art approaches schemes. Experimental results show the better performance of the proposed approach compared to the state-of-the-art approaches.

Graph construction plays an essential role in graph-based label propagation since graphs give some information on the structure of the data manifold. While most graph construction methods rely on predefined distance calculation, recent algorithms merge the task of label propagation and graph construction in a single process. Moreover, the use of several descriptors is proved to outperform a single descriptor in representing the relation between the nodes. In this article, we propose a Multiple-View Consistent Graph construction and Label propagation algorithm (MVCGL) that simultaneously constructs a consistent graph based on several descriptors and performs label propagation over unlabeled samples. Furthermore, it provides a mapping function from the feature space to the label space with which we estimate the label of unseen samples via a linear projection. The constructed graph does not rely on a predefined similarity function and exploits data and label smoothness. Experiments conducted on three face and one handwritten digit databases show that the proposed method can gain better performance compared to other graph construction and label propagation methods.

Graph-based learning in semisupervised models provides an effective tool for modeling big data sets in high-dimensional spaces. It has been useful for propagating a small set of initial labels to a large set of unlabeled data. Thus, it meets the requirements of many emerging applications. However, in real-world applications, the scarcity of labeled data can negatively affect the performance of the semisupervised method. In this article, we present a new framework for semisupervised learning called joint label inference and discriminant embedding for soft label inference and linear feature extraction. The proposed criterion and its associated optimization algorithm take advantage of both labeled and unlabeled data samples in order to estimate the discriminant transformation. This type of criterion should allow learning more discriminant semisupervised models. Nine public image data sets are used in the experiments and method comparisons. These experimental results show that the performance of the proposed method is superior to that of many advanced semisupervised graph-based algorithms.

This article introduces a scheme for semi-supervised learning by estimating a flexible non-linear data representation that exploits Spectral Graph Convolutions structure. Structured data are exploited in order to determine non-linear and linear models. The introduced scheme takes advantage of data-driven graphs at two levels. First, it incorporates manifold smoothness that is naturally encoded by the graph itself. Second, the regression model is built on the convolved data samples that are derived from the data and their associated graph. The proposed semi-supervised embedding can tackle the problem of over-fitting on neighborhood structures for image data. The proposed Graph Convolution-based Semi-supervised Embedding paves the way to new theoretical and application perspectives related to the non-linear embedding. Indeed, building flexible models that adopt convolved data samples can enhance both the data representation and the final performance of the learning system. Several experiments are conducted on six image datasets for comparing the introduced scheme with some state-of-the-art semi-supervised approaches. This empirical evaluation shows the effectiveness of the proposed embedding scheme.

In recent times, graph-based semi-supervised learning proved to be a powerful paradigm for processing and mining large datasets. The main advantage relies on the fact that these methods can be useful in propagating a small set of known labels to a large set of unlabeled data. The scarcity of labeled data may affect the performance of the semi-learning. This paper introduces a new semi-supervised framework for simultaneous linear feature extraction and label propagation. The proposed method simultaneously estimates a discriminant transformation and the unknown label by exploiting both labeled and unlabeled data. In addition, the unknowns of the learning model are estimated by integrating two types of graph-based smoothness constraints. The resulting semi-supervised model is expected to learn more discriminative information. Experiments are conducted on six public image datasets. These experimental results show that the performance of the proposed method can be better than that of many state-of-the-art graph-based semi-supervised algorithms.

Feature extraction methods have been successfully applied to many real-world applications. The classical Linear Discriminant Analysis (LDA) and its variants are widely used as feature extraction methods. Although they have been used for different classification tasks, these methods have some shortcomings. The main one is that the projection axes obtained are not informative about the relevance of original features. In this paper, we propose a linear embedding method that merges two interesting properties: Robust LDA and inter-class sparsity. Furthermore, the targeted projection transformation focuses on the most discriminant original features, The proposed method is called Robust Discriminant Analysis with Feature Selection and Inter-class Sparsity (RDA _FSIS). Two kinds of sparsity are explicitly included in the proposed model. The first kind is obtained by imposing the l2,1 constraint on the projection matrix in order to perform feature ranking. The second kind is obtained by imposing the inter-class sparsity constraint used for getting a common sparsity structure in each class. Comprehensive experiments on five real-world image datasets demonstrate the effectiveness and advantages of our framework over existing linear methods.

The main goal of discriminant embedding is to extract features that can be compact and informative representations of the original set of features. This paper introduces a hybrid scheme for linear feature extraction for supervised multiclass classification. We introduce a unifying criterion that is able to retain the advantages of robust sparse LDA and Interclass sparsity. Thus, the estimated transformation includes two types of discrimination which are the inter-class sparsity and robust Linear Discriminant Analysis with feature selection. In order to optimize the proposed objective function, we deploy an iterative alternating minimization scheme for estimating the linear transformation and the orthogonal matrix. The introduced scheme is generic in the sense that it can be used for combining and tuning many other linear embedding methods. In the lights of the experiments conducted on six image datasets including faces, objects, and digits, the proposed scheme was able to outperform competing methods in most of the cases.

Person re-identification is one of the indispensable elements for visual surveillance. It assigns consistent labeling for the same person within the field of view of the same camera or even across multiple cameras. While handcrafted feature extraction is certainly one way of approaching this problem, in many cases, these features are becoming more and more complex. Besides, training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. This paper explores the following three main strategies for solving the person re-identification problem: (i) using handcrafted features, (ii) using transfer learning based on a pre-trained deep CNN (trained for object categorization) and (iii) training a deep CNN from scratch. Our experiments consistently demonstrated that: (1) The handcrafted features may still have favorable characteristics and benefits especially in cases where the learning database is not sufficient to train a deep network. (2) A fully trained Siamese CNN outperforms handcrafted approaches and the combination of pre-trained CNN with different re-identification processes. (3) Moreover, our experiments demonstrated that pre-trained features and handcrafted features perform equally well. These experiments have also revealed the most discriminative parts in the human body.

The graph-based embedding is an effective and useful method in reducing the dimension and extracting relevant data. This paper introduces a framework for classifying high dimensional data via a joint graph-based embedding and weighting method which could be used in semi-supervised or supervised learning. We design on effective optimization algorithm to solve the objective function. Experiments on image classification show that our proposed method can have a performance that is better than that of many state-of-the-art methods including linear and nonlinear methods.

Facial expression conveys important signs about the human affective state, cognitive activity, intention and personality. In fact, the automatic facial expression recognition systems are getting more interest year after year due to its wide range of applications in several interesting fields such as human computer/robot interaction, medical applications, animation and video gaming. In this study, the authors propose to combine between different descriptors features (histogram of oriented gradients, local phase quantisation and binarised statistical image features) after applying principal component analysis on each of them to recognise the six basic expressions and the neutral face from the static images. Their proposed fusion method has been tested on four popular databases which are: JAFFE, MMI, CASIA and CK+, using two different cross‐validation schemes: subject independent and leave‐one–subject‐out. The obtained results show that their method outperforms both the raw features concatenation and state‐of‐the‐art methods.

The Sparse Representation Classifier, the Collaborative Representation Classifier (CRC), and the Two Phase Test Sample Sparse Representation (TPTSSR) classifier were introduced in recent times. All these frameworks are supervised and passive in the sense that they cannot benefit from unlabeled data samples. In this paper, inspired by active learning paradigms, we introduce an active CRC that can be used by these frameworks. More precisely, we are interested in the TPTSSR framework due to its good performance and its reasonable computational cost. Our proposed Active Two Phase Collaborative Representation Classifier (ATPCRC) starts by predicting the label of the available unlabeled samples. At testing stage, two coding processes are carried out separately on the set of originally labeled samples and the whole set (original and predicted label). The two types of class-wise reconstruction errors are blended in order to decide the class of any test image. Experiments conducted on four public image datasets show that the proposed ATPCRC can outperform the classic TPTSSR as well as many state-of-the-art methods that exploit label and unlabeled data samples.

This paper introduces a robust and flexible graph-based semi-supervised embedding method for generic classification and recognition tasks. It combines the merits of sparsity preserving projections, margin maximization, and robust loss function. The latter reduces the effect of outliers on the regression model needed for mapping unseen examples. Furthermore, unlike label propagation semi-supervised schemes, our proposed method is a data embedding into a space whose dimension is not limited to the number of classes. The used robust norm combines the merits of matrix l1,2 and l2 norms. It is suited for the Laplacian distribution of outliers and the Gaussian distribution of samples with small losses. We provide experiments on four benchmark image datasets in order to study the performance of the proposed method. These experiments show that the proposed methods can be more discriminative than other state-of-the-art methods.

In the past years, various graph-based data embedding algorithms were proposed and used in machine learning and pattern recognition fields. This paper introduces a graph-based non-linear embedding learning algorithm for image classification and recognition. The proposed embedding method can be used for supervised and semi-supervised learning settings. The proposed criterion allows the simultaneous estimation of a linear and a non-linear embedding. It integrates manifold smoothness, Sparse Regression and Margin Discriminant Embedding. The deployed sparse regression implicitly performs feature selection on the original features of the data matrix and of the linear transform. The proposed method is applied to four image datasets: 8 Sports Event Categories dataset, Scene 15 dataset, ORL Face dataset and COIL-20 Object dataset. The experiments demonstrate the effectiveness of the proposed embedding method.

Sparse modeling representative selection (SMRS) has been recently introduced for selecting the most relevant examples in data sets. SMRS exploits data self-representativeness coding in order to infer a coding matrix with block sparsity constraint. The relevance scores of samples are then derived from the estimated matrix of coefficients. Since SMRS is based on a linear model for data self-representation, it cannot always provide good relevant samples. Besides, most of its selected samples can be found in dense areas in input space. In this paper, we propose to overcome the SMRS method's shortcomings that are related to the coding matrix estimation. We introduce two nonlinear data self-representativeness coding schemes that are based on Hilbert space and column generation. Experimental evaluation is carried out on summarizing a video movie and on summarizing training image data sets used for classification tasks. These experiments demonstrated that the proposed nonlinear methods can outperform state-of-the art selection methods.

In recent years, software-based face presentation attack detection (PAD) methods have seen a great progress. However, most existing schemes are not able to generalize well in more realistic conditions. The objective of this competition is to evaluate and compare the generalization performances of mobile face PAD techniques under some real-world variations, including unseen input sensors, presentation attack instruments (PAI) and illumination conditions, on a larger scale OULU-NPU dataset using its standard evaluation protocols and metrics. Thirteen teams from academic and industrial institutions across the world participated in this competition. This time typical liveness detection based on physiological signs of life was totally discarded. Instead, every submitted system relies practically on some sort of feature representation extracted from the face and/or background regions using hand-crafted, learned or hybrid descriptors. Interesting results and findings are presented and discussed in this paper.

Recently, Convolutional Network (ConvNet) features permit to achieve state-of-the-art performance in robotic fields such as visual navigation and SLAM. In this paper, a visual localization technique was proposed based on ConvNet networks by combining the powerful ConvNet features and image sequence matching. The pre-trained networks provided by MatConvNet are used to extract the features and then a sequence search technique is applied for visual recognition. Compared with the traditional approaches based on handcraft features and single image matching, the proposed method shows good performances even in presence of appearance and illumination changes. We present extensive experiments on five real world datasets to evaluate each of the specific challenges in visual recognition. We also conduct a comprehensive performance comparison of different ConvNet layers (each defining a level of features) considering both appearance and illumination changes.

This paper introduces a graph-based semi-supervised embedding method as well as its kernelized version for generic classification and recognition tasks. The aim is to combine the merits of flexible manifold embedding and nonlinear graph-based embedding for semi-supervised learning. The proposed linear method will be flexible since it estimates a nonlinear manifold that is the closest one to a linear embedding. The proposed kernelized method will also be flexible since it estimates a kernel-based embedding that is the closest to a nonlinear manifold. In both proposed methods, the nonlinear manifold and the mapping (linear transform for the linear method and the kernel multipliers for the kernelized method) are simultaneously estimated, which overcomes the shortcomings of a cascaded estimation. The dimension of the final embedding obtained by the two proposed methods is not limited to the number of classes. They can be used by any kind of classifiers once the data are embedded into the new subspaces. Unlike nonlinear dimensionality reduction approaches, which suffer from out-of-sample problem, our proposed methods have an obvious advantage that the learnt subspace has a direct out-of-sample extension to novel samples, and are thus easily generalized to the entire high-dimensional input space. We provide extensive experiments on seven public databases in order to study the performance of the proposed methods. These experiments demonstrate much improvement over the state-of-the-art algorithms that are based on label propagation or graph-based semi-supervised embedding.

In this paper, we develop a new efficient graph construction algorithm that is useful for many learning tasks. Unlike the main stream for graph construction, our proposed data self-representativeness approach simultaneously estimates the graph structure and its edge weights through sample coding. Compared with the recent l1 graph that is based on sparse coding, our proposed objective function has an analytical solution (based on self-representativeness of data) and thus is more efficient. This paper has two main contributions. Firstly, we introduce the Two Phase Weighted Regularized Least Square (TPWRLS) graph construction. Secondly, the obtained data graph is used, in a semi-supervised context, in order to categorize detected objects in driving/urban scenes using Local Binary Patterns as image descriptors. The experiments show that the proposed method can outperform competing methods.

Local discriminant embedding (LDE) has been recently proposed to overcome some limitations of the global linear discriminant analysis method. In the case of a small training data set, however, LDE cannot directly be applied to high-dimensional data. This case is the so-called small-sample-size (SSS) problem. The classical solution to this problem was applying dimensionality reduction on the raw data (e.g., using principal component analysis). In this paper, we introduce a novel discriminant technique called "exponential LDE" (ELDE). The proposed ELDE can be seen as an extension of LDE framework in two directions. First, the proposed framework overcomes the SSS problem without discarding the discriminant information that was contained in the null space of the locality preserving scatter matrices associated with LDE. Second, the proposed ELDE is equivalent to transforming original data into a new space by distance diffusion mapping (similar to kernel-based nonlinear mapping), and then, LDE is applied in such a new space. As a result of diffusion mapping, the margin between samples belonging to different classes is enlarged, which is helpful in improving classification accuracy. The experiments are conducted on five public face databases: Yale, Extended Yale, PF01, Pose, Illumination, and Expression (PIE), and Facial Recognition Technology (FERET). The results show that the performances of the proposed ELDE are better than those of LDE and many state-of-the-art discriminant analysis techniques.

In this paper we present relevant results of a work in progress1that deals with the texturing of 3D urban facade models by fusing terrestrial multi-source data acquired by a Mobile Mapping System (MMS). Some of current 3D urban facade models often are textured by using images that contain parts of urban objects that belong to the street. These urban objects represent in this case occlusions since they are located between the acquisition system and the facades. We show the potential use of georeferenced images and 3D point cloud that are acquired at street level by the MMS in generating occlusion-free facade textures. We describe a methodology for reconstructing texture parts of facades that are highly occluded by wide frontal objects.

Many natural image sets, depicting objects whose appearance is changing due to motion, pose or light variations, can be considered samples of a low-dimension nonlinear manifold embedded in the high-dimensional observation space (the space of all possible images). The main contribution of our work is represented by a Supervised Laplacian Eigemaps (S-LE) algorithm, which exploits the class label information for mapping the original data in the embedded space. Our proposed approach benefits from two important properties: i) it is discriminative, and ii) it adaptively selects the neighbors of a sample without using any predefined neighborhood size. Experiments were conducted on four face databases and the results demonstrate that the proposed algorithm significantly outperforms many linear and non-linear embedding techniques. Although we've focused on the face recognition problem, the proposed approach could also be extended to other category of objects characterized by large variance in their appearance.

This article addresses the problem of registering high-resolution, small field-of-view images with low-resolution panoramic images provided by a panoramic catadioptric video sensor. Such systems may find application in surveillance and telepresence systems that require a large field of view and high resolution at selected locations. Although image registration has been studied in more conventional applications, the problem of registering panoramic and conventional video has not previously been addressed, and this problem presents unique challenges due to (i) the extreme differences in resolution between the sensors (more than a 16:1 linear resolution ratio in our application), and (ii) the resolution inhomogeneity of panoramic images. The main contributions of this article are as follows. First, we introduce our foveated panoramic sensor design. Second, we show how a coarse registration can be computed from the raw images using parametric template matching techniques. Third, we propose two refinement methods allowing automatic and near real-time registration between the two image streams. The first registration method is based on matching extracted interest points using a closed form method. The second registration method is featureless and based on minimizing the intensity discrepancy allowing the direct recovery of both the geometric and the photometric transforms. Fourth, a comparison between the two registration methods is carried out, which shows that the featureless method is superior in accuracy. Registration examples using the developed methods are presented.

This paper presents a pipeline in progress for generating virtual 3D model of urban street facades by fusing cadastral map with terrestrial images and laser data. A Mobile Mapping System (MMS) has been employed to massively gather georeferenced images and laser raw data at street level. Firstly, we describe the context as well as our motivation that aims at enriching the quality of the virtual urban environments. Secondly, we briefly present advanced works on 3D street facades mapping by merging the cadastral map and laser data. These works addressed the street facade plane estimation, their delimitation, and their texturing. Thirdly, we present a novel extension we are working on for extracting 3D detailed facade limits by fusing the cadastral map with laser and image data. This extension is essentially focused to the detection of fine 2D skylines from acquired high resolution facade images and to the estimation of the detailed 3D facade top boundaries. The presented work deals with datasets acquired in the city of Paris under real conditions. Advances and results show the potential of the envisaged pipeline for applications such as the visualization and the large scale navigation in virtual 3D street model.

This paper proposes a new framework for real-time estimation of the onboard stereo head's position and orientation relative to the road surface, which is required for any advanced driver-assistance application. This framework can be used with all road types: highways, urban, etc. Unlike existing works that rely on feature extraction in either the image domain or 3-D space, we propose a framework that directly estimates the unknown parameters from the stream of stereo pairs' brightness. The proposed approach consists of two stages that are invoked for every stereo frame. The first stage segments the road region in one monocular view. The second stage estimates the camera pose using a featureless registration between the segmented monocular road region and the other view in the stereo pair. This paper has two main contributions. The first contribution combines a road segmentation algorithm with a registration technique to estimate the online stereo camera pose. The second contribution solves the registration using a featureless method, which is carried out using two different optimization techniques: 1) the differential evolution algorithm and 2) the Levenberg-Marquardt (LM) algorithm. We provide experiments and evaluations of performance. The results presented show the validity of our proposed framework.

This paper presents a new approach for efficient object detection and matching in images and videos. We propose a stage based on a classification scheme that classifies the extracted features in new images into object features and non-object features. This binary classification scheme has turned out to be an efficient tool that can be used for object detection and matching. By means of this classification not only the matching process becomes more robust and faster but also the robust object registration becomes fast. We provide quantitative evaluations showing the advantages of using the classification stage for object matching and registration. Our approach could lend itself nicely to real-time object tracking and detection.

This paper presents a new method for person-specific face shape estimation under varying head pose of a previously unseen person from a single image. We describe a featureless approach based on a deformable 3D model and a learned face subspace. The proposed approach is based on maximizing a likelihood measure associated with a learned face subspace, which is carried out by a stochastic and genetic optimizer. We conducted the experiments on a subset of Honda Video Database showing the feasibility and robustness of the proposed approach. For this reason, our approach could lend itself nicely to complex frameworks involving 3D face tracking and face gesture recognition in monocular videos.

In this paper, we propose an integrated framework for tracking, modelling and recognition of facial expressions. The main contributions are: (i) a view- and texture independent scheme that exploits facial action parameters estimated by an appearance-based 3D face tracker; (ii) the complexity of the non-linear facial expression space is modelled through a manifold, whose structure is learned using Laplacian Eigenmaps. The projected facial expressions are afterwards recognized based on Nearest Neighbor classifier; (iii) with the proposed approach, we developed an application for an AIBO robot, in which it mirrors the perceived facial expression.

This paper presents a new approach for the simultaneous estimation of the 3D pose and specific shape of a previously unseen face from a single image. The face pose is not limited to a frontal view. We describe a holistic approach based on a deformable 3D model and a learned statistical facial texture model. Rather than obtaining a person-specific facial surface, the goal of this work is to compute person-specific 3D face shape in terms of a few control parameters that are used by many applications. The proposed holistic approach estimates the 3D pose parameters as well as the face shape control parameters by registering the warped texture to a statistical face texture, which is carried out by a stochastic and genetic optimizer. The proposed approach has several features that make it very attractive: (i) it uses a single grey-scale image, (ii) it is person-independent, (iii) it is featureless (no facial feature extraction is required), and (iv) its learning stage is easy. The proposed approach lends itself nicely to 3D face tracking and face gesture recognition in monocular videos. We describe extensive experiments that show the feasibility and robustness of the proposed approach.

Recently, we have proposed a real-time tracker that simultaneously tracks the 3-D head pose and facial actions in monocular video sequences that can be provided by low quality cameras. This paper has two main contributions. First, we propose an automatic 3-D face pose initialization scheme for the real-time tracker by adopting a 2-D face detector and an eigenface system. Second, we use the proposed methods-the initialization and tracking-for enhancing the human-machine interaction functionality of an AIBO robot. More precisely, we show how the orientation of the robot's camera (or any active vision system) can be controlled through the estimation of the user's head pose. Applications based on head-pose imitation such as telepresence, virtual reality, and video games can directly exploit the proposed techniques. Experiments on real videos confirm the robustness and usefulness of the proposed methods.

The dynamical recognition of facial gestures and expressions in image sequences is an important and challenging problem. Most of the existing methods adopt the following paradigm. First, facial feature movements are retrieved from the images, then the facial expression is recognized based on these retrieved movements. In contrast to this mainstream approach, this paper introduces a new approach allowing the simultaneous retrieval of facial feature movements and expression using a particle filter adopting multi-class and multi-person dynamics. The proposed fast scheme is either as robust as, or more robust than existing ones in a number of respects. We provide evaluations of performance to show the feasibility and robustness of the proposed approach.

This paper describes a human machine interaction application for building panoramic views easily and efficiently. The panoramas are not limited to the 1D problem (one axis of rotation). The viewing direction of the camera acquiring snapshots is directly controlled by the userpsilas tracked gaze direction through a 3D face tracker. Natural face motions can be used to control local or remote camera in order to build panoramic views. The resulting system may find applications in online environment mapping as well as in video surveillance. The developed system was applied to map some indoor and outdoor scenes.

Faces play a major role in many HCI systems, because they represent a rich source of information. Being able to estimate the 3D face pose in real-time, we can get a clue about user's intentions or to assess which object become his/her focus of attention. This paper has two main contributions. First, we propose an automatic 3D face pose initialization scheme for our real-time tracker by adopting a 2D face detector and an eigenface system. Second, we use the proposed methods - the initialization and tracking - for controlling the orientation of an AIBO camera. We show how the changes in user's face movement can be imitated by the robot's camera and how it can be applied to map an indoor scene.

This paper presents an efficient technique for estimating the pose of an onboard stereo vision system relative to the environment's dominant surface area, which is supposed to be the road surface. Unlike previous approaches, it can be used either for urban or highway scenarios since it is not based on a specific visual traffic feature extraction but on 3D raw data points. The whole process is performed in the Euclidean space and consists of two stages. Initially, a compact 2D representation of the original 3D data points is computed. Then, a RANdom SAmple Consensus (RANSAC) based least-squares approach is used to fit a plane to the road. Fast RANSAC fitting is obtained by selecting points according to a probability function that takes into account the density of points at a given depth. Finally, stereo camera height and pitch angle are computed related to the fitted road plane. The proposed technique is intended to be used in driver-assistance systems for applications such as vehicle or pedestrian detection. Experimental results on urban environments, which are the most challenging scenarios (i.e., flat/uphill/downhill driving, speed bumps, and car's accelerations), are presented. These results are validated with manually annotated ground truth. Additionally, comparisons with previous works are presented to show the improvements in the central processing unit processing time, as well as in the accuracy of the obtained results.

In this paper, we propose a novel approach for facial expression analysis and recognition. The proposed approach relies on tracked facial actions provided by an appearance-based 3D face tracker. For each universal expression, a dynamical model for facial actions given by an auto-regressive process is learned from training data. We classify a given image in an unseen video into one of the universal facial expression categories using an analysis-synthesis scheme. This scheme uses all models and select the one that provides the most consistent synthesized spatio-temporal facial actions. The dynamical models can be utilized in the tasks of synthesis and prediction. Experiments using unseen videos demonstrated the effectiveness of the developed method.

In this work, we address the problem of tracking faces and facial actions in a single video sequence. The main contributions of the paper are as follows. First, we develop a particle filter based framework for tracking the global 3-D motion of a face using a statistical facial appearance model. Second, we propose a framework for tracking the 3-D face pose as well as the local motion of inner features of the face due for instance to spontaneous facial actions, using an adaptive appearance model. We allow the statistics of the facial appearance as well as the dynamics to be adaptively updated during tracking. Third, we propose a variant of the second framework based on a heuristic search. Tracking real video sequences demonstrated the effectiveness of the developed methods. Accurate tracking was obtained even in the presence of perturbing factors including significant head pose and facial expression variations, occlusions, and illumination changes

The recognition of facial gestures and expressions in image sequences is an important and challenging problem. Most of the existing methods adopt the following paradigm. First, facial actions/features are retrieved from the images, and then facial expressions are recognized based on the retrieved temporal parameters. Unlike this main strewn, this paper introduces a new approach allowing the simultaneous recovery of facial actions and expression using a particle filter adopting, multiclass dynamics that are conditioned on the expression. For each frame in the video sequence, our approach is split in two consecutive stages. In the first stage, the 3D head pose is recovered using a deterministic registration technique based on online appearance models. In the second stage, the facial actions as well as the facial expression are simultaneously recovered using the stochastic framework with, mixed states. The proposed fast scheme is either as robust as existing ones or more robust with respect to many regards. Experimental results show the feasibility and robustness of the proposed approach

In this paper, we address the recognition of facial expressions in continuous videos. We introduce a fast and texture-independent approach that exploits facial action parameters estimated by an appearance-based 3D tracker. We represent the learned facial actions associated with different facial expressions by time series. We adopt a dynamic time warping technique for recognizing the facial expressions using similarity measures between the learned time series and the tracked ones. The developed approach can be used online and can deal with non-frontal face images since it is view-independent. Preliminary experiments demonstrated the effectiveness of the developed method.

Traditionally, the structure from motion (SFM) problem has been solved using feature correspondences. This approach requires reliably detected and tracked features between images taken from widespread locations. In this paper, we present a new paradigm to the SFM problem for planar scenes. The novelty of the paradigm lies in the fact that instead of image feature correspondences, only image derivatives are used. We use robust statistics in order to get robust solutions in presence of outliers. Experiments on both synthetic and real image sequences demonstrated the effectiveness of the developed methods.

This paper addresses the three-dimensional (3-D) tracking of pose and animation of the human face in monocular image sequences using active appearance models. The major problem of the classical appearance-based adaptation is the high computational time resulting from the inclusion of a synthesis step in the iterative optimization. Whenever the dimension of the face space is large, a real-time performance cannot be achieved. In this paper, we aim at designing a fast and stable active appearance model search for 3-D face tracking. The main contribution is a search algorithm whose CPU-time is not dependent on the dimension of the face space. Using this algorithm, we show that both the CPU-time and the likelihood of a nonaccurate tracking are reduced. Experiments evaluating the effectiveness of the proposed algorithm are reported, as well as method comparison and tracking synthetic and real image sequences.

In this paper, we consider the potentialities of adapting a 3D deformable face model to video sequences. Two adaptation methods are proposed. The first method computes the adaptation using a locally exhaustive and directed search in the parameter space. The second method decouples the estimation of head and facial feature motion. It computes the 3D head pose by combining: (i) a robust feature-based pose estimator, and (ii) a global featureless criterion. The facial animation parameters are then estimated with a combined exhaustive and directed search. Tracking experiments and performance evaluation demonstrate the feasibility and usefulness of the developed methods. These experiments also show that the proposed methods can outperform the adaptation based on a directed continuous search.

This paper addresses the problem of geometry determination of a stereo rig that undergoes general rigid motions. Neither known reference objects nor stereo correspondence are required. With almost no exception, all existing online solutions attempt to recover stereo geometry by first establishing stereo correspondences. We first describe a mathematical framework that allows us to solve for stereo geometry, i.e., the rotation and translation between the two cameras, using only motion correspondence that is far easier to acquire than stereo correspondence. Second, we show how to recover the rotation and present two linear methods, as well as a nonlinear one to solve for the translation. Third, we perform a stability study for the developed methods in the presence of image noise, camera parameter noise, and ego-motion noise. We also address accuracy issues. Experiments with real image data are presented. The work allows the concept of online calibration to be broadened, as it is no longer true that only single cameras can exploit structure-from-motion strategies; even the extrinsic parameters of a stereo rig of cameras can do so without solving stereo correspondence. The developed framework is applicable for estimating the relative three-dimensional (3D) geometry associated with a wide variety of mounted devices used in vision and robotics, by exploiting their scaled ego-motion streams.

This paper addresses the 3D tracking of pose and animation of the human face in monocular image sequences using deformable 3D models. For each frame, the proposed adaptation is split into two consecutive stages: global and local. In the first stage, the 3D pose of the face is recovered using a RANSAC-based technique involving both the consensus measure and the consistency with a statistical model of a face texture. In the second stage, the local motion associated with some facial features is recovered using the concept of the active appearance model search. Adaptation examples demonstrate the feasibility and robustness of the developed framework.

This paper addresses the problem of self-calibration from one unknown motion of an uncalibrated stereo rig. Unlike the existing methods for stereo rig self-calibration, which have been focused on applying the autocalibration paradigm using both motion and stereo correspondences, our method does not require the recovery of stereo correspondences. Our method combines purely algebraic constraints with implicit geometric constraints. Assuming that the rotational part of the stereo geometry has two unknown degrees of freedom and the principle point of each camera is known, we show that the computation of the intrinsic and extrinsic parameters of the stereo rig can be recovered from the motion correspondences only, i.e. the monocular fundamental matrices. We provide a stability study for the method in the presence of image noise. Synthetic and real experiments demonstrate the feasibility and robustness of the proposed method.

We present a system to determine human hand pose from a single monochrome image in real-time. We formulate the hand pose problem as determining the position and orientation of the palm as well as the joint angles of the fingers. The 3D pose of the palm and each joint angle are computed simultaneously using feature points on the human hand. We exploit an existing fast object pose algorithm developed by DeMenthon and Davis (1995) together with a heuristic minimization of a 1D function. We make use of a special glove to remove noise caused by the deformation of the hand and to help find correspondence between 3D object points and their images. We demonstrate our system by using a mock-up cardboard hand with one finger. Experimental results with a real human hand wearing the special glove are presented.

This paper introduces a new framework for stereo correspondence recovery using one motion of a stereo rig. Both the stereo correspondence and the motion of the stereo rig are unknown. By combining the stereo geometry and the motion correspondence we are able to infer the stereo correspondence from motion correspondence without having to systematically use the intensity-based stereo matching algorithms. The stereo correspondence recovery consists of two consecutive steps: the first step uses metric data associated with the stereo rig while the second step uses feature correspondences only. Experiments involving real stereo pairs indicate the feasibility and robustness of the approach.

Zhuang et al. (1994) proposed a method that allows simultaneous computation of the rigid transformations from world frame to robot base frame and from hand frame to camera frame. Their method attempts to solve a homogeneous matrix equation of the for, AX=ZB. They use quaternions to derive explicit linear solution for X and Z. In this paper, we present two new solutions that attempt to solve the homogeneous matrix equation mentioned above: 1) a closed-form method which uses quaternion algebra and a positive quadratic error function associated with this representation; 2) a method based on nonlinear constrained minimization and which simultaneously solves for rotations and translations. These results may be useful to other problems that can be formulated in the same mathematical form. We perform a sensitivity analysis for both our two methods and the linear method developed by Zhuang et al. This analysis allows the comparison of the three methods. In the light of this comparison, the nonlinear optimization method, which solves for rotations and translations simultaneously, seems to be the most stable one with respect to noise and to measurement errors.

We present a visual serving approach to the problem of object grasping and more generally, to the problem of aligning an end-effector with an object. First, we extend the method proposed by Espiau et al. (1992) to the case of a camera which is not mounted onto the robot being controlled, and we stress the importance of the real-time estimation of the image Jacobian. Next, we show how to represent a grasp or more generally, an alignment between two solids in 3D projective space using an uncalibrated stereo rig. Such a 3D projective representation is view-invariant in the sense that it can be easily mapped into an image set-point without any knowledge about the camera parameters. Finally, we perform an analysis of the performances of the visual servoing algorithm and of the grasping precision that can be expected from this type of approach.

D.F. Dementhon and L.S. Davis (1995) proposed a method for determining the pose of a 3D object with respect to a camera from 3D to 2D point correspondences. The method consists of iteratively improving the pose computed with a weak perspective camera model to converge at the limit, to a pose estimation computed with a perspective camera model. We show that the method of Dementhon and Davis can be extended to paraperspective. The iterative paraperspective pose algorithm that we describe in detail has interesting properties both in terms of speed and rate of convergence. Moreover, we introduce a simple way of taking into account the orthogonality constraint associated with the rotation matrix and we define the optimal experimental setup to be used in the presence of camera calibration errors.>