Artificial intelligence-based face recognition solutions are becoming increasingly popular. Therefore, it is crucial to fully understand and explain how these technologies work in order to make them more effective and acceptable to society. This is the goal of the CHIST-ERA project XAIface, the final results of which are reported in this article: a framework and toolkit for improving AI decision explainability, in the context of automated face recognition, through several novel methods are presented. These methods are integrated into an end-to-end face recognition demonstrator system, which facilitates studying the impact of various influencing factors and system processes on recognition performance. By doing so, we can visually explain the decisions made by the face verification pipeline for specific instances in our test set using heatmaps and locally interpretable features. Furthermore, we offer a comprehensive explanation of the end-to-end model by examining the relationship between verification failures and misclassifications of soft biometric facial traits.

We present a novel LSTM cell architecture capable of learning both intra- and inter-perspective relationships available in visual sequences captured from multiple perspectives. Our architecture adopts a novel recurrent joint learning strategy that uses additional gates and memories at the cell level. We demonstrate that by using the pro-posed cell to create a network, more effective and richer visual representations are learned for recognition tasks. We validate the performance of our proposed architecture in the context of two multi-perspective visual recognition tasks namely lip reading and face recognition. Three relevant datasets are considered and the results are compared against fusion strategies, other existing multi-input LSTM architectures, and alternative recognition solutions. The experiments show the superior performance of our solution over the considered benchmarks, both in terms of recognition accuracy and complexity. We make our code publicly available at https://github.com/arsm/MPLSTM.

Long Short-Term Memory (LSTM) is a prominent recurrent neural network for extracting dependencies from sequential data such as time-series and multi-view data, having achieved impressive results for different visual recognition tasks. A conventional LSTM network, hereafter referred only as LSTM network, can learn a model to posteriorly extract information from one input sequence. However, if two or more dependent sequences of data are simultaneously acquired, the LSTM networks may only process those sequences consecutively, not taking benefit of the information carried out by their mutual dependencies. In this context, this paper proposes two novel LSTM cell architectures that are able to jointly learn from multiple sequences simultaneously acquired, targeting to create richer and more effective models for recognition tasks. The efficacy of the novel LSTM cell architectures is assessed by integrating them into deep learning-based methods for face recognition with multi-view, light field images. The new cell architectures jointly learn the scene horizontal and vertical parallaxes available in a light field image, to capture richer spatio-angular information from both directions. A comprehensive evaluation, with the IST-EURECOM LFFD dataset using three challenging evaluation protocols, shows the advantage of using the novel LSTM cell architectures for face recognition over the state-of-the-art light field-based methods. These results highlight the added value of the novel cell architectures when learning from correlated input sequences.

Light field (LF) cameras provide rich spatio-angular visual representations by sensing the visual scene from multiple perspectives and have recently emerged as a promising technology to boost the performance of human-machine systems such as biometrics and affective computing. Despite the significant success of LF representation for constrained facial image analysis, this technology has never been used for face and expression recognition in the wild. In this context, this paper proposes a new deep face and expression recognition solution, called CapsField, based on a convolutional neural network and an additional capsule network that utilizes dynamic routing to learn hierarchical relations between capsules. CapsField extracts the spatial features from facial images and learns the angular part-whole relations for a selected set of 2D sub-aperture images rendered from each LF image. To analyze the performance of the proposed solution in the wild, the first in the wild LF face dataset, along with a new complementary constrained face dataset captured from the same subjects recorded earlier have been captured and are made available. A subset of the in the wild dataset contains facial images with different expressions, annotated for usage in the context of face expression recognition tests. An extensive performance assessment study using the new datasets has been conducted for the proposed and relevant prior solutions, showing that the CapsField proposed solution achieves superior performance for both face and expression recognition tasks when compared to the state-of-the-art.

Gait, or the way of walking, of a person can be affected by several pathologies. Automatic gait analysis methods have recently been proposed to identify and distinguish different types of pathological gait. Vision-based methods are often adopted, using some representation of the gait video sequence as input to feature extraction and classification algorithms. The Gait Energy Image (GEI), a mean image of the person's silhouettes along a gait cycle, is one of the most used gait representations for biometric recognition purposes and also for pathology identification. In this paper we propose an alternative compact gait representation, the Skeleton Energy Image (SEI), computed based on skeleton images instead of silhouettes, to remove from the representation some characteristics mostly useful for biometric recognition purposes, like the body shape, while preserving and highlighting the movement information. A method for feature extraction and classification is developed, and a new pathological gait dataset is acquired to test the system, which is made available to the research community. The proposed SEI representation allows obtaining better gait pathology classification results than the GEI. A method combining both representations further improves the performance of the system.

Light field cameras are able to capture the intensity of light rays coming from multiple directions, thus representing the visual scene from multiple viewpoints. This paper exploits the rich spatio-angular information available in light field images for facial emotion recognition. In this context, a new deep network is proposed that first extracts spatial features using a VGG16 convolutional neural network. Then, a Bidirectional Long Short-Term Memory (Bi-LSTM) recurrent neural network is used to learn spatio-angular features from viewpoint feature sequences, exploring both forward and backward angular relationships. Additionally, an attention mechanism allows our model to selectively focus on the most important spatio-angular features, thus enabling a more effective learning outcome. Finally, a fusion scheme is adopted to obtain the emotion recognition classification results. Comprehensive experiments have been conducted on the IST-EURECOM Light Field Face database using two challenging evaluation protocols, showing the superiority of our method over the state-of-the-art.

Face recognition has attracted increasing attention due to its wide range of applications, but it is still challenging when facing large variations in the biometric data characteristics. Lenslet light field cameras have recently come into prominence to capture rich spatio-angular information, thus offering new possibilities for advanced biometric recognition systems. This paper proposes a double-deep spatio-angular learning framework for light field-based face recognition, which is able to model both the intra-view/spatial and inter-view/angular information using two deep networks in sequence. This is a novel recognition framework that has never been proposed in the literature for face recognition or any other visual recognition task. The proposed double-deep learning framework includes a long short-term memory (LSTM) recurrent network, whose inputs are VGG-Face descriptions, computed using a VGG-16 convolutional neural network (CNN). The VGG-Face spatial descriptions are extracted from a selected set of 2D sub-aperture (SA) images rendered from the light field image, corresponding to different observation angles. A sequence of the VGG-Face spatial descriptions is then analyzed by the LSTM network. A comprehensive set of experiments has been conducted using the IST-EURECOM light field face database, addressing varied and challenging recognition tasks. The results show that the proposed framework achieves superior face recognition performance when compared to the state of the art.

Light field cameras capture the intensity of light rays coming from multiple directions, thus allowing a set of 2D images, named sub-aperture (SA) images, to be rendered. These images correspond to observations of the scene from slightly different angles. The rich spatio-angular information obtained using these cameras is exploited in this paper, for the first time, in the context of facial emotion recognition. A deep learning spatio-angular fusion framework is adopted which is able to model both the intra-view/spatial and inter-view/angular information, using a VGG-16 convolutional neural network and a long short-term memory (LSTM) recurrent network. The proposed solution, based on the adopted deep spatio-angular fusion framework, creates two view sequences, horizontal and vertical, with selected SA images, for which VGG-Face descriptions are extracted. The resulting descriptions are fed to two LSTM networks, with the aim of independently learning horizontal and vertical classification models. The softmax classifier scores obtained for the horizontal and vertical descriptors are then fused to obtain the final emotion recognition labels. A comprehensive set of experiments has been conducted on the IST-EURECOM light field face database using two assessment protocols. The adopted framework achieves superior emotion recognition performance when compared with state-of-the-art benchmarking methods.

Underwater imaging is being used increasingly by marine biologists as a means to assess the abundance of marine resources and their biodiversity. Previously, we developed the first automatic approach for estimating the abundance of Norway lobsters and counting their burrows in video sequences captured using a monochrome camera mounted on trawling gear. In this paper, an alternative framework is proposed and tested using deep-water video sequences acquired via a remotely operated vehicle. The proposed framework consists of four modules: (1) preprocessing, (2) object detection and classification, (3) object-tracking, and (4) quantification. Encouraging results were obtained from available test videos for the automatic video-based abundance estimation in comparison with manual counts by human experts (ground truth). For the available test set, the proposed system achieved 100% precision and recall for lobster counting, and around 83% precision and recall for burrow detection.

Vulnerability of face recognition systems to presentation attacks has attracted increasing attention from the biometrics and forensics communities. Moreover, the recent availability of light field cameras is opening new possibilities for designing improved face presentation attack detection solutions. In this context, this paper provides the first review and benchmarking study in the literature on light field-based face presentation attack detection solutions. State-of-the-art solutions are assessed in terms of accuracy, generalization and complexity, using a common, representative evaluation framework. This paper also proposes a novel face presentation attack detection solution, based on a histogram of oriented gradients descriptor, which exploits the disparity information available in light field imaging. The evaluation of the proposed face presentation attack detection solution for different presentation attack types shows a very effective and stable performance, notably performing better than the state-of-the-art alternatives.

Light field cameras are emerging as powerful sensor devices to capture the full spatio-angular visual information in a viewing range. As more information should allow better analysis performance, this paper proposes a simple, yet effective descriptor, named Light Field Local Binary Patterns (LFLBP), able to exploit the richer information available in light field images for face recognition. The LFLBP descriptor combines two main components, the spatial, local LBP and the angular LBP, to capture not only the usual spatial information but also the light field angular information associated to the set of sub-aperture images, corresponding to different viewpoints. Experiments were conducted with the novel IST-EURECOM light field face database. When compared with competing methods, the proposed descriptor has shown superior face recognition performance under varied and challenging acquisition conditions. Moreover, the proposed light field angular LBP descriptor can be flexibly combined with any available spatial descriptor to derive combined descriptors for enhanced light field based face recognition performance.

This paper presents a comprehensive set of image processing algorithms for detection and characterization of road pavement surface crack distresses, which is being made available to the research community. The toolbox, in the Matlab environment, includes algorithms to preprocess images, to detect cracks and characterize them into types, based on image processing and pattern recognition techniques, as well as modules devoted to the performance evaluation of crack detection and characterization solutions. A sample database of 84 pavement surface images taken during a traditional road survey is provided with the toolbox, since no pavement image databases are publicly available for crack detection and characterization evaluation purposes. Results achieved applying the proposed toolbox to the sample database are discussed, illustrating the potential of the available algorithms.

A fully integrated system for the automatic detection and characterization of cracks in road flexible pavement surfaces, which does not require manually labeled samples, is proposed to minimize the human subjectivity resulting from traditional visual surveys. The first task addressed, i.e., crack detection, is based on a learning from samples paradigm, where a subset of the available image database is automatically selected and used for unsupervised training of the system. The system classifies nonoverlapping image blocks as either containing crack pixels or not. The second task deals with crack type characterization, for which another classification system is constructed, to characterize the detected cracks' connect components. Cracks are labeled according to the types defined in the Portuguese Distress Catalog, with each different crack present in a given image receiving the appropriate label. Moreover, a novel methodology for the assignment of crack severity levels is introduced, computing an estimate for the width of each detected crack. Experimental crack detection and characterization results are presented based on images captured during a visual road pavement surface survey over Portuguese roads, with promising results. This is shown by the quantitative evaluation methodology introduced for the evaluation of this type of system, including a comparison with human experts' manual labeling results.

Holoscopic imaging, also known as integral imaging, provides a solution for glassless 3D, and is promising to change the market for 3D television. To start, this paper briefly describes the general concepts of holoscopic imaging, focusing mainly on the spatial correlations inherent to this new type of content, which appear due to the micro-lens array that is used for both acquisition and display. The micro-images that are formed behind each micro-lens, from which only one pixel is viewed from a given observation point, have a high cross-correlation between them, which can be exploited for coding. A novel scheme for spatial prediction, exploring the particular arrangement of holoscopic images, is proposed. The proposed scheme can be used for both still image coding and intra-coding of video. Experimental results based on an H.264/AVC video codec modified to handle 3D holoscopic images and video are presented, showing the superior performance of this approach.

Distributed source coding (DSC) is typically used to compress information from multiple correlated sources that do not communicate with each other. In this paper, DSC principles are used to secure biometric data in a multimodal hand-based recognition system, introducing a novel approach for Log-Likelihood Ratio initialization and a new binarization technique. The proposed biometric recognition system relies on a new architecture using three hand-based biometric traits: palmprint, finger surface and hand geometry, being the latter used as a soft biometric to accelerate the identification process. Promising results are achieved in terms of recognition accuracy, speed and security when performing identification on large databases, like the UST Hand Image Database.

A novel unsupervised strategy to detect cracks on flexible road pavement images, acquired by laser imaging systems, is proposed. It explores the UINTA entropy reduction filter in an innovative way. A two stage approach is followed, after a pre-processing stage, aimed at reducing the variance of image pixel intensities. First, a one-class clustering, using Parzen density estimation, is applied to select image areas likely to contain cracks, exploiting a simple two dimensional feature space which includes the mean and standard deviation of pixel intensities computed for non-overlapping image blocks. Second, the selected blocks are filtered using the UINTA entropy reduction properties and later automatically labeled as containing cracks, or not. Encouraging experimental crack detection results are presented based on real images captured along Canadian roads.

This paper proposes a system capable of identifying and retrieving distress images from a road pavement survey image database. A database of images acquired during road pavement surface surveys along Portuguese roads is considered. Regions corresponding to cracks are detected over the acquired images, based on a subdivision of the images into a set of non-overlapping windows, which may be classified as containing cracks, or not. Crack detection results, represented by binary images where windows containing crack pixels are set to one, undergo a second classification stage to distinguish several crack types. This classification follows the structure proposed by the Portuguese Distresses Catalog, produced by the national entity in charge of road maintenance. Three crack types are identified at this stage: longitudinal cracks, transversal cracks and miscellaneous cracks. The experimental results, obtained by processing real survey imagery over Portuguese roads, present encouraging results for automating the process of identifying road distresses from images.

Video analysis can be used in the context of a wide variety of applications and therefore a multiplicity of techniques has been proposed in the literature. Each of those techniques is usually devoted to solving a specific part of the complete analysis problem, unless the problem is rather simple. Typically, to be able to propose meaningful analysis solutions, the analysis problem must first be appropriately constrained, taking into account the relevant application environment. Then, complementary types of analysis techniques may have to be used in combination to achieve the desired results. This paper proposes a classification of segmentation applications into a set of scenarios, according to the different application constraints and goals. This allows an easier selection of the appropriate video segmentation solution for each specific application. Examples of segmentation solutions for the most relevant scenarios identified are presented.

Video segmentation assumes a major role in the context of object-based coding and description applications. Evaluating the adequacy of a segmentation result for a given application is a requisite both to allow the appropriate selection of segmentation algorithms as well as to adjust their parameters for optimal performance. Subjective testing, the current practice for the evaluation of video segmentation quality, is an expensive and time-consuming process. Objective segmentation quality evaluation techniques can alternatively be used; however, it is recognized that, so far, much less research effort has been devoted to this subject than to the development of segmentation algorithms. This paper discusses the problem of video segmentation quality evaluation, proposing evaluation methodologies and objective segmentation quality metrics for individual objects as well as for complete segmentation partitions. Both stand alone and relative evaluation metrics are developed to cover the cases for which a reference segmentation is missing or available for comparison.

When working in image and video segmentation, the major objective is to design an algorithm producing the appropriate segmentation results for the particular goals of the application addressed. Therefore, the assessment of the segmentation quality assumes a crucial importance to evaluate the likeliness that the application targets are met. Since no well-established methods for objective segmentation quality evaluation are currently available, this paper's major goal is to propose objective metrics for the evaluation of relative segmentation quality for both individual objects and the overall segmentation partition. The paper presents a methodology for performing objective evaluation of relative segmentation quality, identifies the relevant features to be compared against those of the reference segmentation, and proposes appropriate objective quality metrics. These metrics build on the existing knowledge of segmentation quality evaluation and also on some relevant aspects from the video quality evaluation field.

This paper discusses the usefulness of a partition-based image representation in the context of user-assisted segmentation, and compares it with the two other most common image representations in this framework: pixel-based and transition-based. Partition-based image representations allow user-interaction at the region level, which is a very natural and friendly manner to interact with images. The paper describes various region-based implementations of the most common types of user interaction; namely, initial object selection, object refinement, imposing object characteristics, selection of specific objects, selection of objects that are similar to a previously selected one. In all cases, partition-based image representations, and their associated region-based tools for user interaction, show up to be very suitable for interactive image segmentation.

In this paper we describe an interactive video object segmentation tool developed in the framework of the ACTS-AC098 MOMUSYS project. The Video Object Generator with User Environment (VOGUE) combines three different sets of automatic and semi-automatic-tool (spatial segmentation, object tracking and temporal segmentation) with general purpose tools for user interaction. The result is an integrated environment allowing the user-assisted segmentation of any sort of video sequences in a friendly and efficient manner.

This paper proposes two description schemes (DSs) to describe the visual information of an audio-visual (AV) document. The first one, is devoted to still images. It describes the image visual appearance and its structure with regions as well as its semantic content in terms of objects. The second DS is devoted to video sequences. It describes the sequence structure as well as its semantic content in terms of events. Features such as motion, camera activity, etc. are included in this DS. Moreover, it involves static visual representations such as key-frames, background mosaics and key-regions. These elements are considered as still images and are described by the first DS.

The analysis of video data targeting the identification of relevant objects and the extraction of associated descriptive characteristics will be the enabling factor for a number of multimedia applications. This process has intrinsic difficulties, and since semantic criteria are difficult to express, usually only a part of the desired analysis results can be automatically achieved. For many applications, the automatic tools can be complemented with user guidance to improve performance. This paper proposes an integrated framework for video analysis, addressing the video segmentation and feature extraction problems. The framework includes a set of modules that can be combined following specific application needs. It includes both automatic (more objective) and user interaction (more semantic) analysis modules. The paper also proposes a specific segmentation solution to one of the most relevant application scenarios considered-off-line applications requiring precise segmentation.