Accurate 3D mouth localization from a single camera remains challenging in smart spaces, where speakers appear at low resolution and under unconstrained head poses. Most existing audiovisual tracking methods approximate the mouth position using a fixed location within the detected face bounding box, implicitly assuming frontal pose and leading to large errors under head rotations. In addition, monocular 3D estimation suffers from scale ambiguity, i.e., the inherent uncertainty between object size and distance in 2D image projections. We investigate whether head pose estimation (HPE) can overcome these limitations. Two monocular approaches are proposed: LPoseVL , combining facial landmarks with geometric pose recovery using a rigid 3D head model, and LFreeVL , based on landmark-free 6DoF pose estimation with calibration. Experiments on the AV16.3 and CAV3D datasets show that explicitly modeling head pose significantly improves accuracy, achieving up to 46.9% error reduction over bounding-box-based state-of-the-art methods.

The world’s population has grown in recent decades, increasing social events and leading to more crowd situations with potential issues, such as bottlenecks, stampedes, or falls. In this context, this paper presents an approach for stampede detection from image sequences in low- and medium-crowd. It is based on a feature vector extracted from the dense optical flow , using the Gunner-Farneback method, and a deep learning-based classification model capable of determining, frame by frame, whether a stampede is happening. It has been evaluated on four different datasets: two widely used in the state-of-the-art— University of Minnesota (UMN) and Performance Evaluation of Tracking and Surveillance (PETS-2009)— and two new labeled datasets, Geintra-Behaviour-Analysis (GBA-Stampedes) and Geintra-Santander Multiple Actions Dataset in Cruises (GSMADC), which include realistic indoor and outdoor scenarios, as well as diverse crowd types and sizes (up to 6 people in GSMADC and a minimum of 15 in GBA). Both datasets have been made publicly available to increase the limited number of sequences for validating stampede detection in videos, with more than 43000 frames. The proposed method was evaluated across various training scenarios to test its adaptability to new environments. In the most challenging scenario, using a limited training set, our system achieved average metrics of around 99% on UMN and PETS-2009, 95% on GBA, and 91% on GSMADC. In comparison, other models achieved only 90% on UMN and PETS-2009, 80% on GBA, and below 80% on GSMADC, demonstrating the accuracy and robustness of the stampede detector across scenarios.

Gaze Object Prediction (GOP) is a novel task that aims to evaluate which object is being looked at by humans. There are many applications for this task within the image understanding topic, but unfortunately, there are not many works that address it. In this paper, we propose a model named Gaze Estimation and Object Prediction (GEOP) that performs GOP, fusing the object detection and gaze following tasks. Specifically, we use object detection features to carry out both tasks, avoiding the extra computational cost added by using different branches. In addition, we include a dual gaze regression approach to improve the gaze following performance. A deeply analysis of the influence of auxiliary prediction heads to improve the detection accuracy has been carried out to optimize the proposed architecture. In this work, we also propose a novel metric called Average Precision Bounding Box Looked at (AP BL ) that explores further the GOP task. Extensive experiments on the Gaze On Object (GOO) dataset prove the performance of the proposed model in object detection, gaze following, and GOP tasks, demonstrating that the proposal outperforms other state-of-the-art approaches on both average error (Distance and Angle) and GOP metrics.

This work presents a novel proposal aimed at automating the assessment of human functional limitations. It represents an interdisciplinary effort to develop and clinically evaluate an automated system for objective functional assessment, and is being carried out within the EYEFUL research project. Functional evaluation is a complex task that requires a holistic approach, considering various components like cognition, motor skills, and executive functions. The EYEFUL project has created an intelligent space within a fully equipped apartment, which includes the deployment of advanced technology (cameras, microphones, and wearable devices), to capture data from subjects undergoing assessment. In this paper, we provide an extensive overview of the project's architecture, including the selection of Activities of Daily Living (ADLs), the recruitment of subjects, the validation methodology, and the experimental setup. We describe the specialized apartment and sensor hardware deployed for data collection, along with the initial machine learning systems for data analysis. Additionally, we analyze the existing state-of-the-art research in automatic functional performance evaluation and present our proposal. This encompasses ADL selection, database design, recording considerations, and an overview of technological modules to be implemented. We also discuss the current status of select modules, including quantitative and qualitative results and references to ongoing work. In conclusion, the EYEFUL project is pioneering automated functional assessment, integrating advanced technology and interdisciplinary collaboration. It aims to provide healthcare professionals with accurate, comprehensive, and clinically validated evaluations of human functionality during ADLs, addressing the complex nature of functional limitations.

Nowadays, video surveillance systems are commonly found in most public and private spaces. These systems typically consist of a network of cameras that feed into a central node. However, the processing aspect is evolving toward distributed approaches, leveraging edge-computing. These distributed systems are capable of effectively addressing the detection of people or events at each individual node. Most of these systems, rely on the use of deep-learning and segmentation algorithms which enable them to achieve high performance, but usually with a significant computational cost, hindering real-time execution. This paper presents an approach for people detection and action recognition in the wild, optimized for running on the edge, and that is able to work in real-time, in an embedded platform. Human Action Recognition (HAR) is performed by using a Recurrent Neural Network (RNN), specifically a Long Short-Term Memory (LSTM). The input to the LSTM is an ad-hoc, lightweight feature vector obtained from the bounding box of each detected person in the video surveillance image. The resulting system is highly portable and easily scalable, providing a powerful tool for real-world video surveillance applications (in the wild and real-time action recognition). The proposal has been exhaustively evaluated and compared against other state-of-the-art (SOTA) proposals in five datasets, including four widely used (KTH, WEIZMAN, WVU, IXMAX) and a novel one (GBA) recorded in the wild, that includes several people performing different actions simultaneously. The obtained results validate the proposal, since it achieves SOTA accuracy within a much more complicated video surveillance real scenario, and using a lightweight embedded hardware.

Abstract This work describes an end-to-end approach for real-time human action recognition from raw depth image-sequences. The proposal is based on a 3D fully convolutional neural network, named 3DFCNN, which automatically encodes spatio-temporal patterns from raw depth sequences. The described 3D-CNN allows actions classification from the spatial and temporal encoded information of depth sequences. The use of depth data ensures that action recognition is carried out protecting people’s privacy, since their identities can not be recognized from these data. The proposed 3DFCNN has been optimized to reach a good performance in terms of accuracy while working in real-time. Then, it has been evaluated and compared with other state-of-the-art systems in three widely used public datasets with different characteristics, demonstrating that 3DFCNN outperforms all the non-DNN-based state-of-the-art methods with a maximum accuracy of 83.6% and obtains results that are comparable to the DNN-based approaches, while maintaining a much lower computational cost of 1.09 seconds, what significantly increases its applicability in real-world environments.

This work proposes a new method for face and mouth tracking using video cameras. This method has been proposed to be used in intelligent spaces as an initial stage to provide information to higher level applications. The method incorporates the classifiers of the Viola and Jones detection method, modified to provide a probabilistic output previously proposed by the authors, useful for tracking with particle filter. The method combines classifiers trained to detect specific poses (frontal and side face views), building an independent likelihood model of pose changes. We also propose to combine the model in several cameras to allow the mouth tracking in a three-dimensional space. The system has been evaluated on the AV16.3 database sequences showing good results in both precision and recall when using a single camera (in a bidimensional space), and an error below 3cm when using three cameras (in a three-dimensional space).

This paper is focused in the inclusion of open software tools and standard hardware devices into advanced mobility aids. Its urgency is due to the incorporation need of the numerous advances in mobile robotics to the area of the Assistive Technologies, and more specifically in the field of the so called Intelligent Wheelchairs (IW). In this paper we propose a common core for IWs based both on an open robotic platform and an open and a well extended software for robotics (ROS “Robotic Operating System”). Such an IW core has got a distributed architecture whose processing nodes are linked through a CAN standard bus, which connects the low level control functions based on low cost and highly portable Cortex platforms. This paper describes also the autonomous navigation algorithm developed for the Intelligent Wheelchair by using an Extended Kalman Filter. In the chosen approach the absolute position of the IW (a mobile robot) is estimated using sensor fusion which combines data from two of the sensory systems included in the described open platform: the dead-reckoning system and a visual system based on QR landmarks located on the ceiling. Within this context, the paper demonstrates the potential of the platform proposal in relation to the quick and easy incorporation of recent advances in mobile robotics.

During the last decades topics such as video analysis and image understanding techniques have experimented an important evolution due to its inclusion in applications such as surveillance, intelligent spaces and assisted living. In order to validate all related works different datasets have been distributed within the research community: CAVIAR, KTH, Weizmann, INRIA or MuHAVI are some of the most well-known examples, but in most cases these datasets have not been created neither specifically for the mentioned applications, nor in realistic scenarios. Within this context, in this paper we present a work that implements a solution for falling detection from monocular video sequences acquired with an standard video-camera. It includes, both the multi-person detector and tracker in realistic scenarios, and the action classifier for each of the detected persons. Besides, it is also presented a newly created dataset with realistic scenes specifically designed for surveillance applications. Scientific soundness and development of the proposed algorithm and its results and validation, both within well-known datasets as CAVIAR and KTH and within the one ad-hoc generated for the applications of interest, are discussed in the paper.

Employment interviews are relevant scenarios for the study of social interaction. In this setting, social skills play an important role, even though the interactions between potential employers and candidates are often limited. One fundamental aspect of social interaction is the use of nonverbal communication , which affects how we are socially perceived. We present a method to automatically extract body communicative cues from one-on-one conversations recorded with Kinect devices. First, we find the three-dimensional position of hands and head of the subject, and, aided by training data, we infer the upper body pose. Then, we use the inferred poses to perform action recognition and build person-specific activity descriptors. We evaluate our system with both domain-specific and public, generic datasets, and show competitive performance.

We address the problem of body communication retrieval and measuring in seated conversations by means of markerless motion capture. In psychological studies, the use of automatic methods is key to reduce the subjectivity present in manual behavioral coding used to extract these cues. These studies usually involve hundreds of subjects with different clothing, non-acted poses, or different distances to the camera in uncalibrated, RGB-only video. However, range cameras are not yet common in psychology research, especially in existing recordings. Therefore, it becomes highly relevant to develop a fast method that is able to work in these conditions. Given the known relationship between depth and motion estimates, we propose to robustly integrate highly appearance-invariant image motion features in a machine learning approach, complemented with an effective tracking scheme. We evaluate the method's performance with existing databases and a database of upper body poses displayed in job interviews that we make public, showing that in our scenario it is comparable to that of Kinect without using a range camera, and state-of-the-art w.r.t. the HumanEva and ChaLearn 2011 evaluation datasets.

We present a robust method to automatically detect blinks in video sequences of conversations, aimed to discovering stress. Psychological studies have shown a relationship between blink frequency and dopamine levels, which in turn are affected by stress. Task performance correlates through an inverted U shape to both dopamine and stress levels. This shows the importance of automatic blink detection as a way of reducing human coding burden. We use an off-the-shelf face tracker in order to extract the eye region. Then, we perform per-pixel classification of the extracted eye images to later identify blinks through their dynamics. We evaluate the performance of our system with a job interview database with annotations of psychological variables, and show statistically significant correlation between perceived stress resistance and the automatically detected blink patterns.

Hand gestures and body posture are intimately linked to speech as they are used to enrich the vocal content, and are therefore inherently multimodal. As an important part of nonverbal behavior, body communication carries relevant information that can reveal social constructs as diverse as personality, internal states, or job interview outcomes. In this work, we analyze body communication cues in real dyadic employment interviews, where the protagonists of the interaction are seated. We use a mixture of body communicative features based on manual annotations and automated extraction methods to successfully predict two key organizational constructs, namely personality and job interview ratings. Our work also confirms the multimodal nature of body communication and shows that the speaking status can be used to improve the prediction performance of personality and hirability.

The Viola and Jones face detectors and Particle Filters are great algorithms for face detections and target tracking. However Viola outputs a binary result, while Particle Filters work with probabilistic inputs. This is the reason why there are not so many works that combine both algorithms. A probabilistic model or likelihood functions to transform Viola and Jones output to probabilistic data are needed to allow linking both methods. In this work we explore some Viola and Jones based likelihood functions presented in literature, and propose new strategies. We also extend the evaluation of the likelihood functions in position, scale and pose. One of our proposed functions shows better characteristics to be used in intelligent spaces in three dimensional face tracking applications.

In this paper a ring of calibrated and synchronized cameras is used for achieving robot and obstacle localization inside a common observed area. To avoid complex appearance matching derived from the wide-baseline arrangement of cameras, a metric occupancy grid is obtained by intersection of silhouettes projected onto the floor. A particle filter is proposed for tracking multiple objects by using the grid as observation data. A clustering algorithm is included in the filter to increase the robustness and adaptability of the multimodal estimation task. To preserve identity of the robot from the set of tracked objects, odometry readings are used to compute a maximum likelihood (ML) global trajectory identification. As a proof of concept, real results are obtained in a long sequence with a mobile robot moving in a human-cluttered scene.

A hardware solution is presented to obtain the eigenvalues and eigenvectors of a real and symmetrical matrix using field-programmable gate arrays (FPGAs). Currently, this system is used to compute the eigenvalues and eigenvectors in covariance matrices for applications in digital image processing that make use of the principal component analysis (PCA) technique. The proposed solution in this paper is based on the Jacobi method, but in comparison with other related works, it presents a different architecture that remarkably improves execution time, while reducing the number of consumed resources of the FPGA.

The work described in this paper explores a new solution for tracking multiple and dynamic objects in complex environments. An XPF (extended particle filter) is used to implement a multimodal distribution that represents the most probable estimation for each object position and velocity. A standard PF (particle filter) cannot be used with a variable number of obstacles; some other solutions have been tested in different previous works, but most of them require heavy computational resources at least for a high number of obstacles to be tracked. The solution described here includes a clustering procedure that increases the robustness of the probabilistic process in order to provide on-line adaptation to the variable number of clusters. The result is the XPFCP: extended particle filter with clustering process. The presented algorithm has been tested using stereovision measurements; the results included in the paper show the efficiency of the proposed system.

This paper presents the development of a probabilistic algorithm based on an Extended Kalman Filter (EKF), used to estimate the absolute position of an indoor autonomous robot. With EKF it is possible to fuse relative and absolute positioning data, including some kind of uncertainty related to sensory systems. To reach this objective it is necessary to do an important model analysis to enable the on-line adaptation of the estimation algorithm. The development presented in this paper has been designed for an autonomous wheelchair, whose real-time and reliability constraints have to be taken into account in the algorithm.