In this paper, we present our DML (Dependant Multi-Label) Mask R-CNN, a novel architecture designed for defect detection and severity estimation in manufacturing processes. Our approach aims to enhance automated visual inspection systems, addressing critical challenges of multi-label defect classification. The DML Mask R-CNN incorporates a modified Mask R-CNN framework with a dedicated classification head that effectively handles dependent multi-labels. We also introduced a simple geometric-based feature extraction module for defects severity classification, alongside an innovative inference algorithm involving a multi-threshold inference strategy and a modified Non-Maximum Suppression (NMS). Our architecture demonstrates significant improvements, with an average F1-score increase of 6.4% compared to our baseline across two self-collated tire datasets and the public Severstal dataset. This work emphasizes the architecture’s versatility and potential applicability to various domains requiring detailed instance segmentation and classification with multiple labels.

Self-learning has demonstrated remarkable performance in computer vision tasks. However, dealing with unlabeled data that needs to be recognized based on previously acquired knowledge is crucial. Specifically, in Continual Learning where new data is incrementally learned, without the need for training from scratch, models often suffer from Catastrophic forgetting. In this study, we investigate the potential of continual self-learning, using LeNet architecture, to address the issue of learning and recognizing unlabeled historical digits by transferring the knowledge gained from the Split-MNIST dataset. The application of Continual self-learning to unlabeled historical digits unlocks the potential for machines to gain a deeper understanding of our digit-based history and provide increasingly insightful interpretations. To the best of our knowledge, this is the first work that investigates Continual Learning in the context of historical handwritten text recognition.

Over the past few decades, handwriting recognition has made tremendous progress based on deep learning architectures. Most commonly, static learning has achieved impressive results on various datasets. However, this approach relies predominantly on fixed datasets and stationary environments, making it challenging to handle continuous streams of data without forgetting previously learned knowledge. Conversely, Continual learning offers a more efficient and flexible framework for learning from a stream of data over time. However, it can be challenging to compare the performance of these approaches due to differences in evaluation protocols and a lack of a common framework. Moreover, prior research has primarily focused on updating models to perform well on specific task or domain scenarios. Yet, the Class-Incremental Learning scenario, which involves learning new classes incrementally over time, has not been as well addressed as it is considered a more challenging task. To address this gap, this paper provides a comprehensive empirical comparison of some of the currently used continual learning strategies, specifically focusing on the case where the model must learn to recognize continuously new classes while the task-ID is unknown. We conducted a deep overview of recent research from different perspectives and show that well-known approaches such as EWC, LWF, and SI are not ideal for the class-IL scenarios. We discuss the results in more detail and highlight some of the limitations that we believe require further investigation.

Sequence metric learning is becoming a widely adopted approach for various applications dealing with sequential multi-variate data such as activity recognition or natural language processing. It is most of the time tackled with sequence alignment approaches or representation learning. In this paper, we propose to study this subject from the point of view of dynamical system theory by drawing the analogy between synchronized trajectories produced by dynamical systems and the distance between similar sequences processed by a siamese recurrent neural network. Indeed, a siamese recurrent network comprises two identical sub-networks, two identical dynamical systems which can theoretically achieve complete synchronization if a coupling is introduced between them. We therefore propose a new neural network model that implements this coupling with a new gate integrated into the classical Gated Recurrent Unit architecture. This model is thus able to simultaneously learn a similarity metric and the synchronization of unaligned multi-variate sequences in a weakly supervised way. Our experiments show that introducing such a coupling improves the performance of the siamese Gated Recurrent Unit architecture on two datasets: one dedicated to activity recognition and another to transportation recognition.

Anomaly detection is a critical element in the design of secure and reliable networks. Networks are vulnerable to diverse anomalies, including malicious attacks, traffic congestion, operational problems, and hardware failures. In addition, novel unknown anomalies are constantly emerging, which challenges existing knowledge-based anomaly detectors. In this paper, we propose RADON, a robust algorithm for unsupervised anomaly detection, which does not require anomaly signatures or any labeled training data. RADON consists in a novel robust training strategy applied to an autoencoder. First, a filtering rule is applied on the reconstruction scores of traffic metadata to identify potential anomalies contaminating the training data. Then, RADON simultaneously learns to minimize the reconstruction scores of nominal instances, and to maximize that of the filtered anomalies. Finally, the trained model is leveraged to detect anomalous observations. Extensive experiments on the MNIST, NSL-KDD and MedBIoT benchmark datasets illustrate the effectiveness of our approach for anomaly detection, both in image and network traffic analyses.

In this paper, we present an extensive study of different neural network-based approaches and loss functions applied to the Multiple Instance Learning (MIL) problem and binary classification. In the MIL setting, training is performed on small sets of instances called bags, where each positive bag contains at least one positive instance and each negative bag contains only negative instances. We propose a new loss function based on the generalised mean and an effective training strategy particularly suited to this setting and to problems where the instances of one class contain a considerable amount of label noise. Furthermore, we present a probabilistic approach to dynamically estimate the label noise in this unbalanced binary classification setting and utilise it to automatically modulate the hyper-parameter of our proposed loss function. We experimentally evaluated our approach on a number of standard benchmarks for binary classification and showed that it outperforms standard neural network optimisation algorithms as well as most state-of-the-art MIL methods, both on numerical/categorical vector data with MLP architectures and images with Convolutional Neural Networks.

Despite a huge leap in performance of face recognition systems in recent years, some cases remain challenging for them while being trivial for humans. This is because a human brain is exploiting much more information than the face appearance to identify a person. In this work, we aim at capturing the social context of unlabeled observed faces in order to improve face retrieval. In particular, we propose a framework that substantially improves face retrieval by exploiting the faces occurring simultaneously in a query's context to infer a multi-dimensional social context descriptor. Combining this compact structural descriptor with the individual visual face features in a common feature vector considerably increases the correct face retrieval rate and allows to disambiguate a large proportion of query results of different persons that are barely distinguishable visually.

Activities of Daily Living (ADL) classification is a key part of assisted living systems as it can be used to assess a person autonomy. We present in this paper an activity classification pipeline using Gated Recurrent Units (GRU) and inertial sequences. We aim to take advantage of the feature extraction properties of neural networks to free ourselves from defining rules or manually choosing features. We also investigate the advantages of resampling input sequences and personalizing GRU models to improve the performances. We evaluate our models on two datasets: a dataset containing five common postures: sitting, lying, standing, walking and transfer and a dataset named MobiAct V2 providing ADL and falls. Results show that the proposed approach could benefit eHealth services and particularly activity monitoring.

Multiple object tracking is an important but challenging computer vision task. Thanks to the significant progress in object detection field, tracking-by-detection becomes a trending paradigm for tracking multiple objects at the same time. Appearance models are also widely used for associating detection results. In this paper, we combine cosine similarity metric learning with very deep convolutional neural network, yielding a robust appearance pairwise matching model: a deep Siamese network capable of re-identifying the same object after a long time and dealing with partial and complete occlusion. Embedded in existing tracking algorithms, our model is a lightweight but powerful module for decision-making among track hypotheses. Experiments on MOT Challenge 2016 benchmark [1] demonstrate the effectiveness of our model, which achieves state-of-the-art performance without delving into extensive hyper-parameter tuning.

This paper presents a study of metric learning systems on pairwise identity verification, including pairwise face verification and pairwise speaker verification, respectively. These problems are challenging because the individuals in training and testing are mutually exclusive, and also due to the probable setting of limited training data. For such pairwise verification problems, we present a general framework of metric learning systems and employ the stochastic gradient descent algorithm as the optimization solution. We have studied both similarity metric learning and distance metric learning systems, of either a linear or shallow nonlinear model under both restricted and unrestricted training settings. Extensive experiments demonstrate that with limited training pairs, learning a linear system on similar pairs only is preferable due to its simplicity and superiority, i.e., it generally achieves competitive performance on both the labeled faces in the wild face dataset and the NIST speaker dataset. It is also found that a pretrained deep nonlinear model helps to improve the face verification results significantly.

In this paper, we present an approach for minimizing the computational complexity of the trained convolutional neural networks (ConvNets). The idea is to approximate all elements of a given ConvNet and replace the original convolutional filters and parameters (pooling and bias coefficients; and activation function) with an efficient approximations capable of extreme reductions in computational complexity. Low-complexity convolution filters are obtained through a binary (zero and one) linear programming scheme based on the Frobenius norm over sets of dyadic rationals. The resulting matrices allow for multiplication-free computations requiring only addition and bit-shifting operations. Such low-complexity structures pave the way for low power, efficient hardware designs. We applied our approach on three use cases of different complexities: 1) a "light" but efficient ConvNet for face detection (with around 1000 parameters); 2) another one for hand-written digit classification (with more than 180 000 parameters); and 3) a significantly larger ConvNet: AlexNet with million matrices. We evaluated the overall performance on the respective tasks for different levels of approximations. In all considered applications, very low-complexity approximations have been derived maintaining an almost equal classification performance.

In this paper, we present a new algorithm for real-time single-object tracking in videos in unconstrained environments. The algorithm comprises two different components that are trained "in one shot" at the first video frame: a detector that makes use of the generalized Hough transform with color and gradient descriptors and a probabilistic segmentation method based on global models for foreground and background color distributions. Both components work at pixel level and are used for tracking in a combined way adapting each other in a co-training manner. Moreover, we propose an adaptive shape model as well as a new probabilistic method for updating the scale of the tracker. Through effective model adaptation and segmentation, the algorithm is able to track objects that undergo rigid and non-rigid deformations and considerable shape and appearance variations. The proposed tracking method has been thoroughly evaluated on challenging benchmarks, and outperforms the state-of-the-art tracking methods designed for the same task. Finally, a very efficient implementation of the proposed models allows for extremely fast tracking.

The separation between texts and graphics when they are overlapped is a challenging problem for digitization companies. In a previous work [1], we presented the first unsupervised fully automatic segmentation system adapted for colour business document with significant colour complexity and dithered background. The system achieves several operations to segment automatically colour images, separate text from noise and graphics and provides colour information about text colour. After split overlapped characters and separates characters from graphics, characters are broken. The OCR system becomes unable to recognize successfully broken characters and its efficiency is thus seriously affected. This paper presents the first Character Reconstruction System through a new PDE (Partial Differential Equation)-based approach. Our approach takes benefit of the combination of the anisotropic morphology proposed by Breuß and the Weickert Coherence enhancing shock filter diffusion. We introduce and present a continuous anisotropic morphology method driven by the main direction of the first order tensors applied in the neighborhood of the missing part left by the separation between text and graphics. It reconstructs the missing part even when the left area is larger than the strokes width. The coherency of the orientation of the tensors around missing parts overcomes the problem of image noises. The application of the ABBY FineReader OCR engine proves an important reduction in OCR errors. Our experiments show that our proposition compared to the existing state of the art requires no training steps and outperforms both of anisotropic morphology and the Weickert Coherence enhancing shock filter diffusion applied separately.

Super‐resolution (SR) task has become an important research area due to the rapidly growing interest for high quality images in various computer vision and pattern recognition applications. This has led to the emergence of various SR approaches. According to the number of input images, two kinds of approaches could be distinguished: single or multi‐input based approaches. Certainly, processing multiple inputs could lead to an interesting output, but this is not the case mainly for textual image processing. This study focuses on single image‐based approaches. Most of the existing methods have been successfully applied on natural images. Nevertheless, their direct application on textual images is not enough efficient due to the specificities that distinguish these particular images from natural images. Therefore, SR approaches especially suited for textual images are proposed in the literature. Previous overviews of SR methods have been concentrated on natural images application with no real application on the textual ones. Thus, this study aims to tackle this lack by surveying methods that are mainly designed for enhancing low‐resolution textual images. The authors further criticise these methods and discuss areas which promise improvements in such task. To the best of the authors’ knowledge, this survey is the first investigation in the literature.

In this paper, we propose an algorithm for online, real-time tracking of arbitrary objects in videos from unconstrained environments. The method is based on a particle filter framework using different visual features and motion prediction models. We effectively integrate a discriminative online learning classifier into the model and propose a new method to collect negative training examples for updating the classifier at each video frame. Instead of taking negative examples only from the surroundings of the object region, or from specific background regions, our algorithm samples the negatives from a contextual motion density function in order to learn to discriminate the target as early as possible from potential distracting image regions. We experimentally show that this learning scheme improves the overall performance of the tracking algorithm. Moreover, we present quantitative and qualitative results on four challenging public data sets that show the robustness of the tracking algorithm with respect to appearance and view changes, lighting variations, partial occlusions, as well as object deformations. Finally, we compare the results with more than 30 state-of-the-art methods using two public benchmarks, showing very competitive results.

In this paper, we propose a new method for estimating the visual focus of attention (VFOA) in a video stream captured by a single distant camera and showing several persons sitting around a table, like in formal meeting or video conferencing settings. The visual targets for a given person are automatically extracted online using an unsupervised algorithm that incrementally learns the different appearance clusters from low-level visual features computed from face patches provided by a face tracker without the need of an intermediate error-prone step of head pose estimation as in classical approaches. The clusters learned in that way can then be used to classify the different visual attention targets of the person during a tracking run, without any prior knowledge on the environment and the configuration of the room or the visible persons. The experiments on public datasets containing almost 2 h of annotated videos from meetings and video conferencing show that the proposed algorithm produces state-of-the-art results and even outperforms a traditional supervised method that is based on head orientation estimation and that classifies VFOA using Gaussian mixture models.

This paper presents a new method for similarity metric learning, called Logistic Similarity Metric Learning (LSML), where the cost is formulated as the logistic loss function, which gives a probability estimation of a pair of faces being similar. Especially, we propose to shift the similarity decision boundary gaining significant performance improvement. We test the proposed method on the face verification problem using four single face descriptors: LBP, OCLBP, SIFT and Gabor wavelets. Extensive experimental results on the LFW-a data set demonstrate that the proposed method achieves competitive state-of-the-art performance on the problem of face verification.

This paper presents the first international competition on Text Image Super-Resolution (SR) and the ICDAR2015-TextSR dataset. We describe the core of the competition: interest, dataset generation and evaluation procedure, together with participating teams and their respective methods. The obtained results, along with baseline image upscaling schemes and state-of-the-art SR approaches are reported and commented. The main conclusion of this competition is that SR systems may improve OCR performances by up to 16.55 points in accuracy compared with bicubic interpolation for the proposed low resolution images.

Current issues on textual image magnification have been focused on noise-free low-resolution images. Nevertheless, real circumstances are far from these assumptions and existing systems are generally confronted with noisy images; limiting thus the efficiency of the magnification process. The scope of this study is to propose a joint denoising and magnification system based on sparse coding to tackle such a problem. The underlying idea suggests the representation of an image patch by a linear combination of few elements from a suitable dictionary. The proposed system uses both online and offline learned dictionaries that are selected adaptively for each image patch of the input Low-Resolution (LR) noisy image to generate its corresponding noise-free High-Resolution (HR) version. In fact, the online learned dictionaries are trained on a clustered dataset of the image patches selected from the input image and used for the denoising purpose in order to take benefit of the non-local self-similarity assumption in textual images. For the offline learned dictionaries, they are trained on an external LR/HR image patch pair dataset and employed for the magnification purpose. The performance of the proposed system is evaluated visually and quantitatively on different LR noisy textual images and promising results are achieved when compared with other existing systems and conventional approaches dealing with such kind of images.

This paper focuses on recognizing Arabic embedded text in videos. The proposed methods proceed without applying any prior pre-processing operations or character segmentation. Difficulties related to the video or text properties are faced using a learned robust representation of the input text image. This is performed using Convolutional Neural Networks and Deep Auto-Encoders. Features are computed using a multi-scale sliding window scheme. A connectionist recurrent approach is then used. It is trained to predict correct transcriptions of the input image from the associated sequence of features. Proposed methods are extensively evaluated on a large video database recorded from several Arabic TV channels.

This paper proposes a dataset, called ALIF, for Arabic embedded text recognition in TV broadcast. The dataset is publicly available for a non-commercial use. It is composed of a large number of manually annotated text images that were extracted from Arabic TV broadcast. It is the first public dataset dedicated to the development and the evaluation of video Arabic OCR techniques. Text images in the dataset are highly variable in terms of text characteristics (fonts, sizes, colors...) and acquisition conditions (background complexity, low resolution, non-uniform luminosity and contrast...). Moreover, an important part of the dataset is finely annotated, i.e. the text in an image is segmented into characters, paws and words, and each segment is labeled. The dataset can hence be used for both segmentation-based and segmentation-free text recognition techniques. In order to illustrate how the ALIF dataset can be used, the results of an evaluation study that we have conducted on different techniques for Arabic text recognition are also presented.

Effective information retrieval on handwritten document images has always been a challenging task. In this paper, we propose a novel handwritten word spotting approach based on graph representation. The presented model comprises both topological and morphological signatures of handwriting. Skeleton-based graphs with the Shape Context labelled vertexes are established for connected components. Each word image is represented as a sequence of graphs. In order to be robust to the handwriting variations, an exhaustive merging process based on DTW alignment result is introduced in the similarity measure between word images. With respect to the computation complexity, an approximate graph edit distance approach using bipartite matching is employed for graph matching. The experiments on the George Washington dataset and the marriage records from the Barcelona Cathedral dataset demonstrate that the proposed approach outperforms the state-of-the-art structural methods.

In this paper, we present an approach that classifies 3D gestures using jointly accelerometer and gyroscope signals from a mobile device. The proposed method is based on a convolutional neural network with a specific structure involving a combination of 1D convolution, averaging, and max-pooling operations. It directly classifies the fixed-length input matrix, composed of the normalised sensor data, as one of the gestures to be recognises. Experimental results on different datasets with varying training/testing configurations show that our method outperforms or is on par with current state-of-the-art methods for almost all data configurations.

Sparse coding has shown to be an effective technique in solving various reconstruction tasks such as denoising, in painting, and resolution enhancement of natural images. In this paper, we explore the use of this technique specifically to deal with low-resolution and degraded textual images. Firstly, we propose a sparse coding based resolution enhancement approach to recover a textual image with higher resolution than the input low-resolution one. It is based on the use of multiple coupled dictionaries which are learned from a clustered training low-resolution/high-resolution patch-pair database. A reconstruction scheme is then suggested in order to adaptively select the appropriate dictionaries that are useful for better recovering each local patch. This approach can be applied for the magnification of both printed and handwritten characters. Secondly, we propose to integrate the magnification in a restoration framework specifically to denoise and reconstruct at the same time degraded characters. The performances of these propositions are evaluated on various types of degraded printed and handwritten textual images where loss of details and background noise exist. Promising results are achieved when compared with results of other existing approaches.

In this paper, we develop a comprehensive representation model for handwriting, which contains both morphological and topological information. An adapted Shape Context descriptor built on structural points is employed to describe the contour of the text. Graphs are first constructed by using the structural points as nodes and the skeleton of the strokes as edges. Based on graphs, Topological Node Features (TNFs) of n-neighbourhood are extracted. Bag-of-Words representation model based on the TNFs is employed to depict the topological characteristics of word images. Moreover, a novel approach for word spotting application by using the proposed model is presented. The final distance is a weighted mixture of the SC cost, and the TNF distribution comparison. Linear Discriminant Analysis (LDA) is used to learn the optimal weight for each part of the distance with the consideration of writing styles. The evaluation of the proposed approach shows the significance of combining the properties of the handwriting from different aspects.

Text detection in videos is a primary step in any semantic-based video analysis systems. In this work, we propose and compare three machine learning-based methods for embedded Arabic text detection. These methods are able to detect Arabic text regions without any prior knowledge and without any pre-processing. The first method relies on a convolution neural network. The two other methods are based on a multi-exit asymmetric boosting cascade. The proposed methods have been extensively evaluated on a large database of Arabic TV channel videos. Experiments highlight a good detection rate of all methods even though neural network-based method outperforms the other ones in terms of recall/precision and computation time.

Sparse coding is widely known as a methodology where an input signal can be sparsely represented from a suitable dictionary. It was successfully applied on a wide range of applications like the textual image Super-Resolution. Nevertheless, its complexity limits enormously its application. Looking for a reduced computational complexity, a coupled dictionary learning approach is proposed to generate dual dictionaries representing coupled feature spaces. Under this approach, we optimize the training of a first dictionary for the high-resolution image space and then a second dictionary is simply deduced from the latter for the low-resolution image space. In contrast with the classical dictionary learning approaches, the proposed approach allows a noticeable speedup and a major simplification of the coupled dictionary learning phase both in terms of algorithm architecture and computational complexity. Furthermore, the resolution enhancement results achieved by applying the proposed approach on poorly resolved textual images lead to image quality improvements.

Effective information retrieval on handwritten document images has always been a challenging task, especially historical ones. In the paper, we propose a coarse-to-fine handwritten word spotting approach based on graph representation. The presented model comprises both the topological and morphological signatures of the handwriting. Skeleton-based graphs with the Shape Context labelled vertexes are established for connected components. Each word image is represented as a sequence of graphs. Aiming at developing a practical and efficient word spotting approach for large-scale historical handwritten documents, a fast and coarse comparison is first applied to prune the regions that are not similar to the query based on the graph embedding methodology. Afterwards, the query and regions of interest are compared by graph edit distance based on the Dynamic Time Warping alignment. The proposed approach is evaluated on a public dataset containing 50 pages of historical marriage license records. The results show that the proposed approach achieves a compromise between efficiency and accuracy.

In this paper, we propose a novel approach for estimating visual focus of attention in video streams. The method is based on an unsupervised algorithm that incrementally learns the different appearance clusters from low-level visual features extracted from face patches provided by a face tracker. The clusters learnt in that way can then be used to classify the different visual attention targets of a given person during a tracking run, without any prior knowledge on the environment and the configuration of the room or the visible persons. Experiments on public datasets containing almost two hours of annotated videos from meetings and video-conferencing show that the proposed algorithm produces state-of-the-art results and even outperforms a traditional supervised method that is based on head orientation estimation and that classifies visual focus of attention using Gaussian Mixture Models.

This article presents a new method aiming at automatically learning a visual similarity between two images from a class model. This kind of problem is present in many research domains such as object tracking, image classification, signing identification, etc. We propose a new method for facial recognition with a system based on non-linear projection and metric learning. To achieve this objective, we feed a “Bag of Features” representation of the face images into a specific neural network that learns a mapping to a more compact and discriminant representation. This learning process aims at non-linearly projecting the facial features into a reduced space where two images belonging to the same category (i.e. a person) are “close” according to a given similarity metric and “distant” otherwise. The proposed method gives very promising results for face identification in adverse conditions like expression, illumination and facial pose variations. Experimental results give 97% correct recognition rate on the CMU PIE database containing 68 individuals, under vary variable pose and illumination conditions.

In this paper, we present a novel algorithm for fast tracking of generic objects in videos. The algorithm uses two components: a detector that makes use of the generalised Hough transform with pixel-based descriptors, and a probabilistic segmentation method based on global models for foreground and background. These components are used for tracking in a combined way, and they adapt each other in a co-training manner. Through effective model adaptation and segmentation, the algorithm is able to track objects that undergo rigid and non-rigid deformations and considerable shape and appearance variations. The proposed tracking method has been thoroughly evaluated on challenging standard videos, and outperforms state-of-the-art tracking methods designed for the same task. Finally, the proposed models allow for an extremely efficient implementation, and thus tracking is very fast.

This paper addresses the problem of generating a super-resolved version of a low-resolution textual image by using Sparse Coding (SC) which suggests that image patches can be sparsely represented from a suitable dictionary. In order to enhance the learning performance and improve the reconstruction ability, we propose in this paper a multiple learned dictionaries based clustered SC approach for single text image super resolution. For instance, a large High-Resolution/Low-Resolution (HR/LR) patch pair database is collected from a set of high quality character images and then partitioned into several clusters by performing an intelligent clustering algorithm. Two coupled HR/LR dictionaries are learned from each cluster. Based on SC principle, local patch of a LR image is represented from each LR dictionary generating multiple sparse representations of the same patch. The representation that minimizes the reconstruction error is retained and applied to generate a local HR patch from the corresponding HR dictionary. The performance of the proposed approach is evaluated and compared visually and quantitatively to other existing methods applied to text images. In addition, experimental results on character recognition illustrate that the proposed method outperforms the other methods, involved in this study, by providing better recognition rates.

In this paper, we propose an original representation model for handwriting document images. Most state-of-the-art handwriting representation models only use separately textural properties, selective dominant features (such as stroke orientation or gradient orientation) or structural properties. To avoid the drawbacks of using the properties from a single aspect, we design a comprehensive model that contains both morphological and topological information of handwriting. After interest points (the starting/ending points, branch points and high-curved points) are selected, an adapted version of Shape Context (SC) descriptor built on the interest points is employed to describe the contour of the text. In order to model the structural characteristics of the handwritten text, a graph is constructed based on the interest points and the skeleton of the text. With the graph, loops and specific strokes in the handwriting are detected and analyzed. Based on this model, a coarse-to-fine approach for word spotting application is introduced. Without segmenting texts into words, a group of regions of interest are selected by comparing textural features (orientation, projection profile, upper and lower border projection) using the DTW method. Afterwards, regions of interest and queries are represented by the proposed model. The final similarity measure is a weighted mixture of the SC cost, loop difference, stroke analysis and texture comparison with different weights. The validation of the model shows the significance of combining the various properties of the handwriting envisaged in its different aspects.

International audience

Understanding text captured in real-world scenes is a challenging problem in the field of visual pattern recognition and continues to generate a significant interest in the OCR (Optical Character Recognition) community. This paper proposes a novel method to recognize scene texts avoiding the conventional character segmentation step. The idea is to scan the text image with multi-scale windows and apply a robust recognition model, relying on a neural classification approach, to every window in order to recognize valid characters and identify non valid ones. Recognition results are represented as a graph model in order to determine the best sequence of characters. Some linguistic knowledge is also incorporated to remove errors due to recognition confusions. The designed method is evaluated on the ICDAR 2003 database of scene text images and outperforms state-of-the-art approaches.

International audience

This work aims at helping multimedia content understanding by deriving benefit from textual clues embedded in digital videos. For this, we developed a complete video Optical Character Recognition system (OCR), specifically adapted to detect and recognize embedded texts in videos. Based on a neural approach, this new method outperforms related work, especially in terms of robustness to style and size variabilities, to background complexity and to low resolution of the image. A language model that drives several steps of the video OCR is also introduced in order to remove ambiguities due to a local letter by letter recognition and to reduce segmentation errors. This approach has been evaluated on a database of French TV news videos and achieves an outstanding character recognition rate of 95%, corresponding to 78% of words correctly recognized, which enables its incorporation into an automatic video indexing and retrieval system.

This paper presents an embedded facial image analysis framework based on Convolutional Neural Networks (ConvNets). This robust framework has been proposed by Garcia, Delakis and Duffner on general purpose workstations without any constraints on computational and memory resources. We show that ConvNets, which consist of a pipeline of convolution and subsampling operations followed by a Multi Layer Perceptron, are particularly well suited for implementation on embedded processors. We present a set of high-level optimizations, such as automatic fractional transformation, convolution and subsampling fusion and memory requirement optimizations that can be applied to these algorithms without any loss in performance, leading to a speedup factor up to 700 compared to the reference implementation. This work leads to a face processing library able to handle the complete framework and its applications on mobile phones.

This paper presents a graph based scheme for color text recognition in images and videos, which is particularly robust to complex background, low resolution or video coding artifacts. This scheme is based on a novel method named the image text recognition graph (iTRG) composed of five main modules: an image text segmentation module, a graph connection builder module, a character recognition module, a graph weight calculator module and an optimal path search module. The first two modules are based on convolutional neural networks so that the proposed system automatically learns how to robustly perform segmentation and recognition. The proposed method is evaluated on the public ICDAR 2003 test word dataset.

Current object detection systems reach high detection rates, at the expense of requiring a large training database. This paper presents a new method for object detection, that gives state-of-the-art results, while using a reduced training database. The proposed system relies on a new local feature extraction approach inspired by Convolutional Neural Networks, Principal Component Analysis and Multilayer Perceptrons. We show that the proposed scheme improves robustness and generalization on the specific problem of face detection, with a very reduced set of exemplar face images.

This article presents a method aiming at quantifying the visual similarity between an image and a class model. This kind of problem is recurrent in many applications such as object recognition, image classification, etc. In this paper, we propose to label a self-organizing map (SOM) to measure image similarity. To manage this goal, we feed local signatures associated to the regions of interest into the neural network. At the end of the learning step, each neural unit is tuned to a particular local signature prototype. During the labeling process, each image signature presented to the network generates an activity vote for its referent neuron. Facial recognition is then performed by a probabilistic decision rule. This scheme offers very promising results for face identification dealing with illumination variation and facial poses and expressions.

We present a face recognition technique based on a special type of convolutional neural network that is trained to extract characteristic features from face images and reconstruct the corresponding reference face images which are chosen beforehand for each individual to recognize. The reconstruction is realized by a so-called "bottle-neck" neural network that learns to project face images into a low-dimensional vector space and to reconstruct the respective reference images from the projected vectors. In contrast to methods based on the Principal Component Analysis (PCA), the Linear Discriminant Analysis (LDA) etc., the projection is non-linear and depends on the choice of the reference images. Moreover, local and global processing are closely interconnected and the respective parameters are conjointly learnt. Having trained the neural network, new face images can then be classified by comparing the respective projected vectors. We experimentally show that the choice of the reference images influences the final recognition performance and that this method outperforms linear projection methods in terms of precision and robustness.

We present a novel approach for face recognition based on salient singularity descriptors. The automatic feature extraction is performed thanks to a salient point detector, and the singularity information selection is performed by a SOM region-based structuring. The spatial singularity distribution is preserved in order to activate specific neuron maps and the local salient signature stimuli reveals the individual identity. This proposed method appears to be particularly robust to facial expressions and facial poses, as demonstrated in various experiments on well-known databases.

This paper presents an automatic binarization method for color text areas in images or videos, which is robust to complex background, low resolution or video coding artefacts. Based on a specific architecture of convolutional neural networks, the proposed system automatically learns how to perform binarization, from a training set of synthesized text images and their corresponding desired binary images, without making any assumptions or using tunable parameters. The proposed method is compared to state-of-the-art binarization techniques, with respect to Gaussian noise and contrast variations, demonstrating the robustness and the efficiency of our method. Text recognition experiments on a database of images extracted from video frames and web pages, with two classical OCRs applied on the obtained binary images show a strong enhancement of the recognition rate by more than 40%.

In this paper, a high-level optimization methodology is applied for the implementation of the well-known convolutional face finder (CFF) algorithm for real-time applications on cellular phone, such as teleconferencing, advanced user interfaces, pictures indexing and security access control. This face detector is based on a feature extraction and classification technique which consists in a pipeline of convolutions and subsampling operations. Design of embedded systems must find a good trade off between performance and code size due to the limited amount of resource available. We propose a methodology to cope with the main drawbacks of the CFF original implementation like floating point computation and memory allocation, to allow parallelism exploitation and perform algorithm optimizations. Results show that our embedded face detection system can accurately locate faces with less computational load and memory cost. It runs on a 275 MHz Starcore DSP at 9 QCIF images/s with state-of-the-art detection rates and very low false alarm rates

This paper presents an approach to model the gaze behavior of an Embodied Conversational Agent in a real time multimodal dialogue interaction with users. The ECA's gaze control results from the fusion of a rational dialogue engine based on natural language interaction and a multi-users face tracker.

The aim of this work is to investigate a way of enhancing the performance of face recognition from video sequences by selecting only well-framed face images from those extracted from video sequences. It is known that noisy face images (e.g. not well-centered, non-frontal poses...) significantly reduce the performance of face recognition methods, and therefore, need to be filtered out during the training and the recognition. The proposed method is based on robust statistics, and more precisely, a recently proposed robust high-dimensional data analysis method, RobPCA. Experiments show that this filtering procedure improves the recognition rate by 10 to 20%.

In this paper, we present a novel approach for face recognition, using a new subspace method called bilinear discriminant analysis (BDA) and normalized radial basis function networks (NRBFNs). In a first step, BDA extracts the features that enhance separation between classes by using a generalized bilinear projection-based Fisher criterion, computed from image matrices directly. In a second step, the features are fed into a NRBFN that learns class conditional probabilities. This results in an efficient and computationally simple open-world identification process. Experimental results assess the performance and robustness of the proposed algorithm compared to other subspace methods combined with NRBFNs, in the presence of variations in head poses, facial expressions, and partial occlusions.

The ability to verify automatically and with great accuracy the identity of a person has become crucial in everyday life. Biometrics is an emerging topic in the field of signal processing. Our research on biometrics aims at developing a complete framework useful to control access. This technical demo shows the latest image processing techniques for face detection developed at France Telecom and for face recognition developed at Eurécom. Using only one computer and one standard webcam, our biometric system detects the user face and the recognition algorithm uses this image to enable the access to a resource, a service or a location.

In this paper, we present a novel face detection approach based on a convolutional neural architecture, designed to robustly detect highly variable face patterns, rotated up to +/-20 degrees in image plane and turned up to +/-60 degrees, in complex real world images. The proposed system automatically synthesizes simple problem-specific feature extractors from a training set of face and nonface patterns, without making any assumptions or using any hand-made design concerning the features to extract or the areas of the face pattern to analyze. The face detection procedure acts like a pipeline of simple convolution and subsampling modules that treat the raw input image as a whole. We therefore show that an efficient face detection system does not require any costly local preprocessing before classification of image areas. The proposed scheme provides very high detection rate with a particularly low level of false positives, demonstrated on difficult test sets, without requiring the use of multiple networks for handling difficult cases. We present extensive experimental results illustrating the efficiency of the proposed approach on difficult test sets and including an in-depth sensitivity analysis with respect to the degrees of variability of the face patterns.

Recovering 3D motion parameters from 2D displacements is a difficult task, given the influence of noise contained in these data, which correspond at best to a crude approximation of the real motion field. The need for stability of the system of equations to be solved is therefore essential. We present a novel method based on an unbiased estimator that aims at enhancing this stability and strongly reduces the influence of noise contamination. Experimental results using synthetic and real optical flows are presented to demonstrate the effectiveness of our method in comparison to a set of selected methods.

Text is a very powerful index in content-based image and video indexing. We propose a new text detection and segmentation algorithm that is especially designed for being applied to color images with complicated background. Our goal is to minimize the number of false alarms and to binarize efficiently the detected text areas so that they can be processed by standard OCR software. First, potential areas of text are detected by enhancement and clustering processes, considering most of constraints related to the texture of words. Then, classification and binarization of potential text areas are achieved in a single scheme performing color quantization and characters periodicity analysis. We report a high rate of good detection results with very few false alarms and reliable text binarization.

Detecting and recognizing human faces automatically in digital images strongly enhance content-based video indexing systems. In this paper, a novel scheme for human faces detection in color images under nonconstrained scene conditions, such as the presence of a complex background and uncontrolled illumination, is presented. Color clustering and filtering using approximations of the YCbCr and HSV skin color subspaces are applied on the original image, providing quantized skin color regions. A merging stage is then iteratively performed on the set of homogeneous skin color regions in the color quantized image, in order to provide a set of potential face areas. Constraints related to shape and size of faces are applied, and face intensity texture is analyzed by performing a wavelet packet decomposition on each face area candidate in order to detect human faces. The wavelet coefficients of the band filtered images characterize the face texture and a set of simple statistical deviations is extracted in order to form compact and meaningful feature vectors. Then, an efficient and reliable probabilistic metric derived from the Bhattacharrya distance is used in order to classify the extracted feature vectors into face or nonface areas, using some prototype face area vectors, acquired in a previous training stage.

Today's technology and advances in networking and telecommunications stimulate a change in the way business is carried out, making it a globally distributed process, in which communication and collaboration of geographically dispersed groups is of vital importance.Virtual Reality systems are adapting accordingly, by providing not only a better man-machine interface, but also by facilitating human to human interaction and collaboration over distance.The approach presented in this paper, creates an environment where remote participants not only meet as if face to face, but also share the same virtual space and perform common tasks.Live stereo-video and audio, from a projectionbased VR system are transmitted and integrated into the virtual space of another participant at a distant VR system, allowing geographically separated groups to meet in a common virtual space, while maintaining eyecontact, gaze awareness and body language.The scientific approach involves stereo-camera calibration and rectification, and use of the camera parameters for integrating the stereovideo into a virtual environment, while maintaining the stereo-effect and correct perspective for each participant.A prototype environment in CyberStage, is also presented in detail in this paper.Virtual Reality is widely accepted as a promising approach to a better man-machine interface, overcoming the present limitations of desktop systems and adapting more closely to the user needs.Projection-based VR systems are using metaphors, such as the blackboard or the desk for creating shared working environments that provide a more natural man-machine communication [ 14][ 191.Today's technology and advances in telecommunications lead to sophisticated multimedia systems which combined with virtual reality can provide a high degree of co-presence and co-working for geographically dispersed groups [2] [ 161.Therefore, new challenges are introduced in terms of multimedia integration in distributed virtual reality environments and interaction.It is not only a question of solving the technical problems of gathering and transmitting multimedia datastreams with sufficient quality and speed, but also a question of addressing the specific needs of human communication.For example, facial expression, body language and eye contact are an integral part of this communication.