Accurate 3D mouth localization from a single camera remains challenging in smart spaces, where speakers appear at low resolution and under unconstrained head poses. Most existing audiovisual tracking methods approximate the mouth position using a fixed location within the detected face bounding box, implicitly assuming frontal pose and leading to large errors under head rotations. In addition, monocular 3D estimation suffers from scale ambiguity, i.e., the inherent uncertainty between object size and distance in 2D image projections. We investigate whether head pose estimation (HPE) can overcome these limitations. Two monocular approaches are proposed: LPoseVL , combining facial landmarks with geometric pose recovery using a rigid 3D head model, and LFreeVL , based on landmark-free 6DoF pose estimation with calibration. Experiments on the AV16.3 and CAV3D datasets show that explicitly modeling head pose significantly improves accuracy, achieving up to 46.9% error reduction over bounding-box-based state-of-the-art methods.

We address the task of identifying distracted driving by analyzing in-car videos using efficient transformers. Although transformer models have achieved outstanding performance in human action recognition tasks, their high computational costs limit their application onboard a vehicle. We introduce PO-GUISE+, a multi-task video transformer that, given an input clip, predicts the distracted driving action, the driver’s pose, and the interacting object. Our enhanced features for token selection are specifically adapted to driver actions by leveraging information about object interaction and the driver’s pose. With PO-GUISE+, we significantly reduce the model’s computational demands while maintaining or improving baseline accuracy across various computational budgets. Additionally, to evaluate our model’s performance in real-world scenarios, we have developed benchmarks on a Jetson computing platform, demonstrating its effectiveness across different configurations and computational budgets. Our model outperforms current state-of-the-art results on the Drive&Act, 100-Driver, and 3MDAD datasets, while having superior efficiency compared to existing video transformer-based methods.

Deepfake detection has progressively become a topic of interest in recent years due to the proliferation of automated facial forgery generation techniques that are able to produce manipulated media indistinguishable for the human eye. One of the most difficult aspects in deepfake detection is generalization to unseen manipulation techniques, which is a key factor to make a method useful in real world applications. In this paper, we propose a new multi-task network termed SFA, which leverages spatiotemporal features extracted from video inputs to provide more robust predictions compared to image-only models, as well as a face alignment task that helps the network to identify anomalous facial movements in the temporal dimension. We show that this multi-task approach improves generalization compared to the single-task baseline, and succeeds in producing results on par with the current state-of-the-art using different cross-dataset and cross-manipulation benchmarks.

Large pre-trained video transformers are becoming the standard architecture for video processing due to their exceptional accuracy. However, their computational complexity has been a major obstacle to their practical application in problems that require the recognition of precise motion patterns in video, such as in the recognition of Activities of Daily Living (ADL). Techniques like token pruning help mitigate their computational cost, but overlook some specific aspects of this task such as the actor movement. To address this we propose an improved token selection method that integrates semantic information from the ADL recognition task with that of human motion. Our model relies on a multi-task architecture that infers human pose and activity classification from RGB videos. We show that guiding token pruning with motion information significantly improves the trade-off between higher efficiency, obtained by reducing the number of tokens, and accuracy of the classification task. We evaluate our model on three popular ADL recognition benchmarks with their respective cross-subject and cross-view setups. In our experiments, a video transformer modified with our proposed modules sets a new state-of-the-art on the ADL recognition task whilst achieving significant reductions in computational cost. • PO-GUISE is a human motion and ADL-guided token selection for video transformers. • The resulting model improves the accuracy-GFLOPs trade-off during inference. • Our model integrates heatmap tokens for temporal and multi-actor prediction. • Sets new state-of-the-art results on ADL benchmarks at a reduced computational cost.

Recent advancements in video transformers have significantly impacted the field of human action recognition. Leveraging these models for distracted driver action recognition could potentially revolutionize road safety measures and enhance Human-Machine Interaction (HMI) technologies. A factor that limits their potential use is the need for extensive data for model training. In this paper, we propose DRVMon-VM, a novel approach for the recognition of distracted driver actions. This is based on a large pre-trained video transformer called VideoMaeV2 (backbone) and a classification head as decoder, which are fine-tuned using a dual learning rate strategy and a medium-sized driver actions database complemented by various data augmentation techniques. Our proposed model exhibits a substantial improvement, exceeding previous results by 7.34% on the challenging Drive&Act dataset, thereby setting a new benchmark in this field.

Head pose estimation (HPE) is a problem of interest in computer vision to improve the performance of face processing tasks in semi-frontal or profile settings. Recent applications require the analysis of faces in the full 360° rotation range. Traditional approaches to solve the semi-frontal and profile cases are not directly amenable for the full rotation case. In this paper we analyze the methodology for short- and wide-range HPE and discuss which representations and metrics are adequate for each case. We show that the popular Euler angles representation is a good choice for short-range HPE, but not at extreme rotations. However, the Euler angles’ gimbal lock problem prevents them from being used as a valid metric in any setting. We also revisit the current cross-data set evaluation methodology and note that the lack of alignment between the reference systems of the training and test data sets negatively biases the results of all articles in the literature. We introduce a procedure to quantify this misalignment and a new methodology for cross-data set HPE that establishes new, more accurate, SOTA for the 300W-LP/Biwi benchmark. We also propose a generalization of the geodesic angular distance metric that enables the construction of a loss that controls the contribution of each training sample to the optimization of the model. Finally, we introduce a wide range HPE benchmark based on the CMU Panoptic data set.

The paper proposes a surrogate model for liquid moulding of structural composites. A methodology is presented to simulate the dual-phase Darcy’s flow in a heterogeneous porous medium. The approach is an encoder–decoder that receives as input a matrix of permeabilities and produces two scalar fields that represent the pressure and front flow. This model is trained with synthetic data generated with a computer fluid dynamics simulator. In this context, the lack of robustness of models trained with the popular L2 and L1 losses is highlighted and several enhancements to these baseline approaches are introduced. First, the study provides a piece-wise power-logarithmic loss that improves training in the presence of the bimodal distribution of error residuals produced by the dual-phase flow predictions. A non-uniform sampling strategy for the selection of time training snapshots is also included, which contributes to improve the prediction accuracy. The estimation of the front flow field is further refined with a multi-task training strategy. The introduction of these improvements in the baseline models reduce the relative error of the pressure and front flow fields by more than 50%, performing these simulations in a record time of 50 ms. The surrogate model is further evaluated as a digital twin to predict – in a real experiment – the location and spatial extent of race-tracking channels and regions with dissimilar degrees of permeability.

Detecting local features, such as corners, segments or blobs, is the first step in the pipeline of many Computer Vision applications. Its speed is crucial for real-time applications. In this paper we present ELSED, the fastest line segment detector in the literature. The key for its efficiency is a local segment growing algorithm that connects gradient-aligned pixels in presence of small discontinuities. The proposed algorithm not only runs in devices with very low end hardware, but may also be parametrized to foster the detection of short or longer segments, depending on the task at hand. We also introduce new metrics to evaluate the accuracy and repeatability of segment detectors. In our experiments with different public benchmarks we prove that our method accounts the highest repeatability and it is the most efficient in the literature.1 In the experiments we quantify the accuracy traded for such gain.

We present a deep learning-based multi-task approach for head pose estimation in images. We contribute with a network architecture and training strategy that harness the strong dependencies among face pose, alignment and visibility, to produce a top performing model for all three tasks. Our architecture is an encoder-decoder CNN with residual blocks and lateral skip connections. We show that the combination of head pose estimation and landmark-based face alignment significantly improve the performance of the former task. Further, the location of the pose task at the bottleneck layer, at the end of the encoder, and that of tasks depending on spatial information, such as visibility and alignment, in the final decoder layer, also contribute to increase the final performance. In the experiments conducted the proposed model outperforms the state-of-the-art in the face pose and visibility tasks. By including a final landmark regression step it also produces face alignment results on par with the state-of-the-art.

Line extraction is a preliminary step in various visual robotic tasks performed in low textured scenes such as city and indoor settings. Several efficient line segment detection algorithms such as LSD and EDLines have recently emerged. However, the state of the art segment grouping methods are not robust enough or not amenable for detecting lines in real-time. In this paper we present FSG, a fast and robust line detection algorithm. It is based on two independent components. A proposer that greedily cluster segments suggesting plausible line candidates and a probabilistic model that decides if a group of segments is an actual line. In the experiments we show that our procedure is more robust and faster than the best methods in the literature and achieves state-of-the art performance in a high level robot localization task such as vanishing points detection.

BACKGROUND: Serial block face scanning electron microscopy (SBFEM) is becoming a popular technology in neuroscience. We have seen in the last years an increasing number of works addressing the problem of segmenting cellular structures in SBFEM images of brain tissue. The vast majority of them is designed to segment one specific structure, typically membranes, synapses and mitochondria. Our hypothesis is that the performance of these algorithms can be improved by concurrently segmenting more than one structure using image descriptions obtained at different scales. RESULTS: We consider the simultaneous segmentation of two structures, namely, synapses with mitochondria, and mitochondra with membranes. To this end we select three image stacks encompassing different SBFEM acquisition technologies and image resolutions. We introduce both a new Boosting algorithm to perform feature scale selection and the Jaccard Curve as a tool compare several segmentation results. We then experimentally study the gains in performance obtained when simultaneously segmenting two structures with properly selected image descriptor scales. The results show that by doing so we achieve significant gains in segmentation accuracy when compared to the best results in the literature. CONCLUSIONS: Simultaneously segmenting several neuronal structures described at different scales provides voxel classification algorithms with highly discriminating features that significantly improve segmentation accuracy.

Full understanding of the architecture of the brain is a long term goal of neuroscience. To achieve it, advanced image processing tools are required, that automate the the analysis and reconstruction of brain structures. Synapses and mitochondria are two prominent structures with neurological interest for which various automated image segmentation approaches have been recently proposed. In this work we present a comparative study of several image feature descriptors used for the segmentation of synapses and mitochondria in stacks of electron microscopy images.

We introduce new results connecting differential and morphological operators that provide a formal and theoretically grounded approach for stable and fast contour evolution. Contour evolution algorithms have been extensively used for boundary detection and tracking in computer vision. The standard solution based on partial differential equations and level-sets requires the use of numerical methods of integration that are costly computationally and may have stability issues. We present a morphological approach to contour evolution based on a new curvature morphological operator valid for surfaces of any dimension. We approximate the numerical solution of the curve evolution PDE by the successive application of a set of morphological operators defined on a binary level-set and with equivalent infinitesimal behavior. These operators are very fast, do not suffer numerical stability issues, and do not degrade the level set function, so there is no need to reinitialize it. Moreover, their implementation is much easier since they do not require the use of sophisticated numerical algorithms. We validate the approach providing a morphological implementation of the geodesic active contours, the active contours without borders, and turbopixels. In the experiments conducted, the morphological implementations converge to solutions equivalent to those achieved by traditional numerical solutions, but with significant gains in simplicity, speed, and stability.

In this paper we describe a collection of course materials designed to be used in an undergraduate introduction to artificial intelligence (AI) course. These materials include three programming assignments, each touching upon core AI topics, which require that the students build the main functionalities of a guide robot. These assignments were carefully designed to allow the same solution to work both with a robot simulator and an inexpensive web-cam as well as with real robots. An overview of the course and the assignments is given along with references to online versions of the resources developed to teach the course.

Emerging applications of computer vision and pattern recognition in mobile devices and networked computing require the development of resource-limited algorithms. Linear classification techniques have an important role to play in this context, given their simplicity and low computational requirements. The paper reviews the state-of-the-art in gender classification, giving special attention to linear techniques and their relations. It discusses why linear techniques are not achieving competitive results and shows how to obtain state-of-the-art performances. Our work confirms previous results reporting very close classification accuracies for Support Vector Machines (SVMs) and boosting algorithms on single-database experiments. We have proven that Linear Discriminant Analysis on a linearly selected set of features also achieves similar accuracies. We perform cross-database experiments and prove that single database experiments were optimistically biased. If enough training data and computational resources are available, SVM's gender classifiers are superior to the rest. When computational resources are scarce but there is enough data, boosting or linear approaches are adequate. Finally, if training data and computational resources are very scarce, then the linear approach is the best choice.

We introduce a morphological approach to curve evolution. The differential operators used in the standard PDE snake models can be approached using morphological operations on a binary level set. By combining the morphological operators associated to the PDE components we achieve a new snakes evolution algorithm. This new solution is based on numerical methods which are very simple, fast and stable. Moreover, since the level set is just a binary piecewise constant function, this approach does not require to estimate a contour distance function. To illustrate the results obtained we present some numerical experiments on real images.

We present an efficient algorithm for fitting a morphable model to an image sequence. It is built on a projective geometry formulation of perspective projection, which results in a linear mapping from 3D shape to the projective plane, and a factorisation of this mapping into matrices that can be partially computed off-line. This algorithm can cope with full 360 degrees object rotation and linear deformations. We validate our approach using synthetically generated and real sequences. Compared to a plain Lucas-Kanade implementation, we achieve a six fold increase in performance for a rigid object and two fold for a non-rigid face.

Pixel brightness variations in an image sequence depend both on the objects ‘surface reflectance and on the motion of the camera and object. In the case of rigid shapes some proposed models have been very successful explaining the relation among these strongly coupled components. On the other hand, shapes which deform pose new challenges since the relation between pixel brightness variation with non-rigid motion is not yet clear. In this paper, we introduce a new model which describes brightness variations with two independent components represented as linear basis shapes. Lighting influence is represented in terms of Spherical Harmonics and non-rigid motion as a linear model which represents image coordinates displacement. We then propose an efficient procedure for the estimation of this image model in two distinct steps. First, shape normal’s and albedo are estimated using standard photometric stereo on a sequence with varying lighting and no deformable motion. Then, given the knowledge of the object’s shape normal’s and albedo, we efficiently compute the 2D coordinates bases by minimizing image pixel residuals over an image sequence with constant lighting and only non-rigid motion. Experiments on real tests show the effectiveness of our approach in a face modelling context.

We introduce a multi-view direct procedure for efficiently tracking 3D objects. It is an extension of Hager and Belhumeur’s factorisation approach to the case of three-dimensional objects and multi-camera setup. By tracking a 3-D object we mean to estimate the pose and location of the object through a video sequence. A novel parameterisation of the object texture allow us to compute the Jacobian that emerges in the minimisation in a way it has a large constant part. The pixels viewed by each camera determine the rows of the Jacobian used for tracking. We perform qualitative and quantitative experiments confirming the validity of the approach.

We present a real-time user-independent computer vision system that processes a sequence of images of a front-facing human face and recognises a set of facial expressions at 30 fps. We track the face using an efficient appearance-based face tracker. We model changes in illumination with a user-independent appearance-based model. In our approach to facial expression classification, the image of a face is represented by a low dimensional vector that results from projecting the illumination corrected image onto a low dimensional expression manifold. In the experiments conducted we show that the system is able to recognise facial expressions in image sequences with large facial motion and illumination changes.

Facial expression recognition is a topic of interest both in industry and academia. Recent approaches to facial expression recognition are based on mapping expressions to low dimensional manifolds. In this paper we revisit various dimensionality reduction algorithms using a graph-based paradigm. We compare eight dimensionality reduction algorithms on a facial expression recognition task. For this task, experimental results show that although Linear Discriminant Analysis (LDA) is the simplest and oldest supervised approach, its results are comparable to more flexible recent algorithms. LDA, on the other hand, is much simpler to tune, since it only depends on one parameter.

We introduce a subspace representation of face appearance which separates facial expressions from illumination variations. The appearance of a face is represented by the addition of two approximately independent linear subspaces modelling facial expressions and illumination respectively. The independence assumption notably simplifies the training of the system. We only require two image sequences. One in which one facial expression is subject to all possible illuminations and another in which the face, under one illumination, performs all facial expressions. This simple model enables us to train the system with no manual intervention. We also introduce an efficient procedure for fitting this model, which can be used for tracking a human face in real-time. 1

Efficient incremental image alignment is a topic of renewed interest in the computer vision community because of its applications in model fitting and model-based object tracking. Successful compositional procedures for aligning 2D and 3D models under weak-perspective imaging conditions have already been proposed. Here we present a mixed compositional and additive algorithm which is applicable to the full projective camera case.

Tracking planar patches is a subject of interest in computer vision. We present a method to estimate, in real-time, the position and orientation of a previously viewed planar patch. The algorithm is based on minimising the sum of squared differences (SSD) between a selected set of pixels obtained from a previously stored image of the patch and the current image of it. First a linear model for projectively tracking a planar patch is introduced; then, a procedure to accelerate the tracking process is presented. This procedure is based on using a small set of informative points in the tracking process. In experiments, we show the gain in performance and compare different procedures for selecting the set of points used in tracking.

We consider the motion estimation problem in the case of very closely spaced views. We revisit the differential epipolar equation providing an interpretation of it. On the basis of this interpretation we introduce a cost function to estimate the parameters of the differential epipolar equation, which enables us to compute the camera extrinsics and some intrinsics. In the synthetic tests performed we compare this continuous method with traditional discrete motion estimation and, contrary to previous findings by Vieville et al. (1996), show that the continuous method did not perceive any computational advantage.