Meetings are a common activity that provides certain challenges when creating systems that assist them. Such is the case of the Active speaker detection, which can provide useful information for human interaction modeling, or human-robot interaction. Active speaker detection is mostly done using speech, however, certain visual and contextual information can provide additional insights. In this paper we propose an active speaker detection framework that integrates audiovisual features with social information, from the meeting context. Visual cue is processed using a Convolutional Neural Network (CNN) that captures the spatio-temporal relationships. We analyze several CNN architectures with both cues: raw pixels (RGB images) and motion (estimated with optical flow). Contextual reasoning is done with an original methodology, based on the gaze of all participants. We evaluate our proposal with a public benchmark in state-of-art: AMI corpus. We show how the addition of visual and context information improves the performance of the active speaker detection.

The activities we do in our daily-life are generally carried out as a succession of atomic actions, following a logical order. During a video sequence, actions usually follow a logical order. In this paper, we propose a hybrid approach resulting from the fusion of a deep learning neural network with a Bayesian-based approach. The latter models human-object interactions and transition between actions. The key idea is to combine both approaches in the final prediction. We validate our strategy in two public datasets: CAD-120 and Watch-n-Patch. We show that our fusion approach yields performance gains in accuracy of respectively +4% and +6% over a baseline approach. Temporal action recognition performances are clearly improved by the fusion, especially when classes are imbalanced.

Knowing the exact number of passengers among the city bus fleets allows public transport operators to optimally distribute their vehicles into the traffic. However, interpreting overcrowded scenarios, at rush hour, with day/night illumination changes can be tricky. Based on the visual trackingby-detection paradigm, we benefit from video stream information provided by cameras placed above doors to infer people trajectories and deduce the number of enterings/leavings at every bus stop. In this way a person detector estimates the location of the passengers in each image, a tracker matches detections between successive frames based on different cues such as appearance or motion, and infers trajectories over time. This paper proposes a fast and embeddable framework that performs person detection using relevant state-of-the-art CNN detectors, and couple the best one (in our applicative context) with a newly designed Siamese network for real-time tracking/data association purposes. Evaluations on our own large scale in-situ dataset are very promising in terms of performances and real-time constraint expected for on-board processing.

Intelligent surveillance systems in human-centered environments require people behavioral monitoring. In this paper, we propose a new Bayesian framework to recognize actions on RGB-D videos by two different observations: the human pose and objects in its vicinity. We design a model for each action that integrates these observations and a probabilistic sequencing of actions performed during activities. We validate our approach on two public video datasets: CAD-120 and Watch-n-Patch. We show a performance gain of 4% in action detection on the fly on CAD-120 videos. Our approach is competitive to 2D image features and skeleton-based methods, as we present an improvement of 16% on Watch-n-Patch. Action recognition performance is clearly improved by our Bayesian and joint human-object perception.

This work presents a method that incorporates 2D and 3D cues for the estimation of head pose. We propose the use of the concept of Key-Frames (KF), a set of frames where the position and orientation of the head is automatically calculated off-line, to improve the precision of pose estimation and detection rate. Each KF consists of: 2D information, encoded by SURF descriptors; 3D information from a depth image (both acquired by an RGB-D sensor); and a generic 3D model that corresponds to the head localization and orientation in the real world. Our algorithm compares a new frame against all KFs and selects the most relevant one. The 3D transformation between both, selected KF and current frame, can be estimated using the depth image and the Iterative Closest Point algorithm in an online framework. Compared to reference approaches, our system can handle partial occlusions and extreme rotations even with noisy depth data. We evaluate the proposal using two challenging datasets: (1) an dataset acquired by us where the ground-truth information is given by a commercial Motion Capture system and (2) the public benchmark Biwi Kinect Head Pose Database.

This paper evaluates and compares different hyperparameters optimization tools that can be used in any vision applications for tuning their underlying free parameters. We focus in the problem of multiple object tracking, as it is widely studied in the literature and offers several parameters to tune. The selected tools are freely available or easy to implement. In this paper we evaluate the impact of parameter optimization tools over the tracking performances using videos from public datasets. Also, we discuss differences between the tools in term of performances, stability, documentation, etc.

We propose a methodology for learning and using a multiple-goal probabilistic motion model within a particle filter-based target tracking on video streams. In a set of training video sequences, we first extract the locations (coined as “goals”) where the pedestrians either leave the scene or often change directions. Then, we learn one motion prior model per detected goal. Each of these models is learned statistically based on the local motion observed by the camera during the training phase. Given that the initial, empirical distribution may be incomplete or noisy, we regularize it in a second phase. These priors are then used in an Interactive Multiple Model (IMM) scheme for target tracking and goal estimation. We demonstrate the relevance of this methodology with tracking experiments and comparisons done on standard datasets.

Even though pedestrian motion may look chaotic in most of the cases, recent studies have shown that this motion is mainly ruled by environment and social aspects. In this paper, we propose an interacting multiple model pedestrian tracking framework that incorporates these semantic considerations as a prior knowledge about intentions and interactions between targets. We consider 4 cases of motion for pedestrians: going straight, finding one's way, walking around and standing still. Those models are competing within an Interacting Multiple Model Particle Filter strategy. Targets interactions are handled with social forces, included as potential functions in the weighting process of the Particle Filter (PF). We use different social force models in each motion model to handle high level behaviors (collision avoidance, flocking...). We evaluate our algorithm on challenging datasets and demonstrate that such semantic information improves the tracker performance.

Multiple targets tracking is a challenging problem due to occlusions or identity switching. Although the use of prior information about the motion of the targets improves the tracking results, a single motion model may not capture the complex dynamic of the targets. This is a common situation with pedestrians, as each person moves in its own way, making tracking a difficult task. In this paper, this problem is faced by using a proposal based on the Interacting Multiple Model (IMM) and implemented in a Bayesian scheme through a particle filter. The core of this approach is to leave the filter choose the motion model that fits better the motion of the targets. The algorithm is evaluated, under several combinations of motion models, with middle-dense crowded scenes from the PETS 2009 dataset.

We present a novel method for multi-object tracking that tracks target both in the video streams and in a reference ground frame. This allows to remove ambiguities created by occlusions in one view. Our system takes as a base a recently proposed collaborative scheme and makes it handle multiple targets. We use a fast, simple solution for data association in the ground plane based on principal axis and a partly joint probabilistic model with MCMC sampling to ensure that tracked targets are kept separated whenever groups of targets appear. Results are presented on several popular databases of multi-camera, multi-target videos.