Common person re-identification (re-ID) approaches focus on finding person signature similarities extracted in pairs of cameras without explicitly maintaining consistency of the re-ID results across the camera network. In contrast, the so-called NCR [2] approach considers all person detections in the networked cameras as a weighted graph, and then partitions the graph into clusters aiming at maximizing the sum of the cluster profits. NCR clustering uses a parameter K ∈ [0, 1] to offset similarity scores, making them negative. The partitioning is highly sensitive to the tuning of this value, which significantly impacts it. To counter this drawback, a new D(Directed)-NCR approach is proposed in this paper. It exploits a directed graph structure that explicitly incorporates temporal and topological information, and also employs a clustering approach to estimate the number of paths associated with different identities in the network. Evaluations on the well-known public datasets HDA and RPIfield demonstrate that D-NCR outperforms the NCR approach.

We propose two light-weight and specialized Spatio-Temporal Graph Convolutional Networks (ST-GCNs): one for actions characterized by the motion of the human body and a novel one we especially design to recognize particular objects configurations during human actions execution. We propose a late-fusion strategy of the predictions of both graphs networks to get the most out of the two and to clear out ambiguities in the action classification. This modular approach enables us to reduce memory cost and training times. Moreover we also propose the same late fusion mechanism to further improve the performance using a Bayesian approach. We show results on 2 public datasets: CAD-120 and Watch-n-Patch. Our late-fusion mechanism yields performance gains in accuracy of respectively + 21 percentage points (pp), + 7 pp on Watch-n-Patch and CAD-120 compared to the individual graphs. Our approach outperforms most of the significant existing approaches.

Following the tracking-by-detection paradigm, multiple object tracking deals with challenging scenarios, occlusions or even missing detections; the priority is often given to quality measures instead of speed, and a good trade-off between the two is hard to achieve. Based on recent work, we propose a fast, lightweight tracker able to predict targets position and reidentify them at once, when it is usually done with two sequential steps. To do so, we combine a bounding box regressor with a target-oriented appearance learner in a newly designed and unified architecture. This way, our tracker can infer the targets' image pose but also provide us with a confidence level about target identity. Most of the time, it is also common to filter out the detector outputs with a preprocessing step, throwing away precious information about what has been seen in the image. We propose a tracks management strategy able to balance efficiently between detection and tracking outputs and their associated likelihoods. Simply put, we spotlight a full siamese based single object tracker able to predict both position and appearance features at once with a light-weight and all-in-one architecture, within a balanced overall multi-target management strategy. We demonstrate the efficiency and speed of our system w.r.t the literature on the well-known MOT17 challenge benchmark, and bring to the fore qualitative evaluations as well as state-of-the-art quantitative results.

Meetings are a common activity that provides certain challenges when creating systems that assist them. Such is the case of the Active speaker detection, which can provide useful information for human interaction modeling, or human-robot interaction. Active speaker detection is mostly done using speech, however, certain visual and contextual information can provide additional insights. In this paper we propose an active speaker detection framework that integrates audiovisual features with social information, from the meeting context. Visual cue is processed using a Convolutional Neural Network (CNN) that captures the spatio-temporal relationships. We analyze several CNN architectures with both cues: raw pixels (RGB images) and motion (estimated with optical flow). Contextual reasoning is done with an original methodology, based on the gaze of all participants. We evaluate our proposal with a public benchmark in state-of-art: AMI corpus. We show how the addition of visual and context information improves the performance of the active speaker detection.

The activities we do in our daily-life are generally carried out as a succession of atomic actions, following a logical order. During a video sequence, actions usually follow a logical order. In this paper, we propose a hybrid approach resulting from the fusion of a deep learning neural network with a Bayesian-based approach. The latter models human-object interactions and transition between actions. The key idea is to combine both approaches in the final prediction. We validate our strategy in two public datasets: CAD-120 and Watch-n-Patch. We show that our fusion approach yields performance gains in accuracy of respectively +4% and +6% over a baseline approach. Temporal action recognition performances are clearly improved by the fusion, especially when classes are imbalanced.

Knowing the exact number of passengers among the city bus fleets allows public transport operators to optimally distribute their vehicles into the traffic. However, interpreting overcrowded scenarios, at rush hour, with day/night illumination changes can be tricky. Based on the visual trackingby-detection paradigm, we benefit from video stream information provided by cameras placed above doors to infer people trajectories and deduce the number of enterings/leavings at every bus stop. In this way a person detector estimates the location of the passengers in each image, a tracker matches detections between successive frames based on different cues such as appearance or motion, and infers trajectories over time. This paper proposes a fast and embeddable framework that performs person detection using relevant state-of-the-art CNN detectors, and couple the best one (in our applicative context) with a newly designed Siamese network for real-time tracking/data association purposes. Evaluations on our own large scale in-situ dataset are very promising in terms of performances and real-time constraint expected for on-board processing.

Intelligent surveillance systems in human-centered environments require people behavioral monitoring. In this paper, we propose a new Bayesian framework to recognize actions on RGB-D videos by two different observations: the human pose and objects in its vicinity. We design a model for each action that integrates these observations and a probabilistic sequencing of actions performed during activities. We validate our approach on two public video datasets: CAD-120 and Watch-n-Patch. We show a performance gain of 4% in action detection on the fly on CAD-120 videos. Our approach is competitive to 2D image features and skeleton-based methods, as we present an improvement of 16% on Watch-n-Patch. Action recognition performance is clearly improved by our Bayesian and joint human-object perception.

This paper introduces a new raw-audio, environmental, kitchen-related, non-vocal dataset to fill what we consider as a gap in the context of audio datasets. Our so-called Kitchen20 dataset is compared to other datasets such as ESC-50 and shown that both datasets can be merged together in what we call ESC-70. A human quantitative appreciation of the audio samples contained in Kitchen20 is provided as well as several machine learning benchmarks on both Kitchen20 and ESC-70.

In this paper, we present a comparative evaluation of various multi-person tracking-by-detection approaches on public data sets. This paper investigates five popular trackers coupled with six relevant visual people detectors evaluated on seven public data sets. The evaluation emphasizes on exhibited performance variation depending on tracker-detector choices. Our experimental results show that the overall performance depends on how challenging the data set is, the performance of the detector on the specific data set, and the tracker-detector combination. Some trackers are more sensitive to the choice of a detector and some detectors to the choice of a tracker than others. Based on our results, two of the trackers demonstrate the best performances consistently across different data sets, whereas the best performing detectors vary per data set. This underscores the need for careful application context specific evaluation when choosing a detector.

This paper presents a novel framework for learning a soft-cascade detector with explicit computation time considerations. Classically, training techniques for softcascade detectors select a set of weak classifiers and their respective thresholds, solely to achieve the desired detection performance without any regard to the detector response time. Nevertheless, since computation time performance is of utmost importance in many time-constrained applications, this work divulges an optimization approach that aims to minimize the mean cascade response time, given a desired detection performance, fixed beforehand. The resulting problem is NP-Hard, therefore finding an optimal threshold vector can be very time-consuming, especially when building a soft-cascade detector of long length. An efficient local search procedure is presented that deals with long-length detectors. Our evaluations on two challenging public datasets confirm that a faster cascade detector can be learned while maintaining similar detection performances.

This work presents a method that incorporates 2D and 3D cues for the estimation of head pose. We propose the use of the concept of Key-Frames (KF), a set of frames where the position and orientation of the head is automatically calculated off-line, to improve the precision of pose estimation and detection rate. Each KF consists of: 2D information, encoded by SURF descriptors; 3D information from a depth image (both acquired by an RGB-D sensor); and a generic 3D model that corresponds to the head localization and orientation in the real world. Our algorithm compares a new frame against all KFs and selects the most relevant one. The 3D transformation between both, selected KF and current frame, can be estimated using the depth image and the Iterative Closest Point algorithm in an online framework. Compared to reference approaches, our system can handle partial occlusions and extreme rotations even with noisy depth data. We evaluate the proposal using two challenging datasets: (1) an dataset acquired by us where the ground-truth information is given by a commercial Motion Capture system and (2) the public benchmark Biwi Kinect Head Pose Database.

This paper evaluates and compares different hyperparameters optimization tools that can be used in any vision applications for tuning their underlying free parameters. We focus in the problem of multiple object tracking, as it is widely studied in the literature and offers several parameters to tune. The selected tools are freely available or easy to implement. In this paper we evaluate the impact of parameter optimization tools over the tracking performances using videos from public datasets. Also, we discuss differences between the tools in term of performances, stability, documentation, etc.

Perceptual systems deployed in embedded intelligent systems are handicapped with limited computation resources. As a result, they require explicit computation time consideration during design time. In this paper, we investigate the impact a visual person detector explicitly optimized to minimize computational time has on an exemplar robotic application - navigation in human shared environments. The experimental results attest the improved perceptual frame rate leads to significant improvements in vision system performance and robotic mission success rate.

We present a multi-modal perception based architecture to realize a non-intrusive domestic assistant robot. The realized robot is non-intrusive in that it only starts interaction with a user when it detects the user's intention to do so automatically. All the robot's actions are based on multi-modal perceptions, which include: user detection based on RGB-D data, user's intention-for-interaction detection with RGB-D and audio data, and communication via speech recognition. The utilization of multi-modal cues in different parts of the robotic activity paves the way to successful robotic runs.

In this paper, the problem of minimizing the mean response-time of a soft-cascade detector is addressed. A soft-cascade detector is a machine learning tool used in applications that need to recognize the presence of certain types of object instances in images. Classical soft-cascade learning methods select the weak classifiers that compose the cascade, as well as the classification thresholds applied at each cascade level, so that a desired detection performance is reached. They usually do not take into account its mean response-time, which is also of importance in time-constrained applications. To overcome that, we consider the threshold selection problem aiming to minimize the computation time needed to detect a target object in an image (i.e., by classifying a set of samples). We prove the NP-hardness of the problem and propose a mathematical model that takes benefit from several dominance properties, which are put into evidence. On the basis of computational experiments, we show that we can provide a faster cascade detector, while maintaining the same detection performances.

In this paper, we investigate the notion of incorporating feature computation time (CT) measures during feature selection in a boosted cascade people detector utilizing heterogeneous pool of features. We present various approaches based on pareto-front analysis, CT weighted adaboost, and Binary Integer Programming (BIP) with comparative evaluations. The novel feature selection method proposed based on BIP — the main contribution — mines heterogeneous features taking both detection performance and CT explicitly into consideration. The results demonstrate that the detector using this feature selection scheme exhibits low miss rates (MRs) with significant boost in frame rate. For example, it achieves a [Formula: see text] less MR at [Formula: see text] FPPW compared to Dalal and Triggs HOG detector with a [Formula: see text]x speed improvement. The presented extensive experimental results clearly highlight the improvements the proposed framework brings to the table.

Multi-person tracking is a very difficult problem in Computer Vision as a tracking algorithm is facing several issues, such as appearance changes, targets' occlusions and similar appearances between people. In an online tracking-by-detection algorithm, robust and discriminative specific appearance models help handling these difficulties. As done in single object tracking, we use sparse representations to extract local features of the targets and study how these representations can be specifically employed for multi-person tracking. Experiments on several datasets show that considering spatial information is crucial in order to improve the tracking performances with local descriptions compared to holistic features. Using large collaborative representations also improve the tracking results by naturally discarding irrelevant local patches.

In this work, we present a comparative evaluation of various `tracking-by-detection' approaches on public datasets. The work investigates popular sequential Monte Carlo and template ensemble based trackers coupled with relevant visual people detectors with emphasis on exhibited performance variation depending on tracker-detector choice. Extensive experimental results are provided on public dataset and results indicate the choice of a detector can significantly vary the performance of a tracker. Our experimental results show, depending on the choice of the detector, average tracking accuracy across three public datasets could exhibit a 45% standard deviation with only, on average, a 6.8% and 11.1% standard deviation in detector recall and precision respectively.

Multi-person tracking is still a challenging problem due to recurrent occlusion, pose variation and similar appearances between people. Inspired by the success of sparse representations in single object tracking and face recognition, we propose in this paper an online tracking by detection framework based on collaborative sparse representations. We argue that collaborative representations can better differentiate people compared to target-specific models and therefore help to produce a more robust tracking system. We also show that despite the size of the dictionaries involved, these representations can be efficiently computed with large-scale optimization techniques to get a near real-time algorithm. Experiments show that the proposed approach compares well to other recent online tracking systems on various datasets.

Understanding people's intention, be it action or thought, plays a fundamental role in establishing coherent communication amongst people, especially in non-proactive robotics, where the robot has to understand explicitly when to start an interaction in a natural way. In this work, a novel approach is presented to detect people's intention-for-interaction. The proposed detector fuses multimodal cues, including estimated head pose, shoulder orientation and vocal activity detection, using a probabilistic discrete state Hidden Markov Model. The multimodal detector achieves up to 80% correct detection rates improving purely audio and RGB-D based variants.

Visual tracking is dynamic optimization where time and object state simultaneously influence the problem. In this paper, we intend to show that we built a tracker from an evolutionary optimization approach, the PSO (Particle Swarm optimization) algorithm. We demonstrated that an extension of the original algorithm where system dynamics is explicitly taken into consideration, it can perform an efficient tracking. This tracker is also shown to outperform SIR (Sampling Importance Resampling) algorithm with random walk and constant velocity model, as well as a previously PSO inspired tracker, SPSO (Sequential Particle Swarm Optimization). Experiments were performed both on simulated data and real visual RGB-D information. Our PSO inspired tracker can be a very effective and robust alternative for visual tracking.

Even though pedestrian motion may look chaotic in most of the cases, recent studies have shown that this motion is mainly ruled by environment and social aspects. In this paper, we propose an interacting multiple model pedestrian tracking framework that incorporates these semantic considerations as a prior knowledge about intentions and interactions between targets. We consider 4 cases of motion for pedestrians: going straight, finding one's way, walking around and standing still. Those models are competing within an Interacting Multiple Model Particle Filter strategy. Targets interactions are handled with social forces, included as potential functions in the weighting process of the Particle Filter (PF). We use different social force models in each motion model to handle high level behaviors (collision avoidance, flocking...). We evaluate our algorithm on challenging datasets and demonstrate that such semantic information improves the tracker performance.

In this paper we present a novel people detector that employs discrete optimization for feature selection. Specifically, we use binary integer programming to mine heterogeneous features taking both detection performance and computation time explicitly into consideration. The final trained detector exhibits low Miss Rates with significant boost in frame rate. For example, it achieves a 2.6% less Miss Rate at 10-4 FPPW compared to Dalal and Triggs HOG detector with a 9.22x speed improvement.

In this paper, we present a fast Histogram of Oriented Gradients (HOG) based person detector. The detector adopts a cascade of rejectors framework by selecting discriminant features via a new proposed feature selection framework based on Binary Integer Programming. The mathematical programming explicitly formulates an optimization problem to select discriminant features taking detection performance and computation time into account. The learning of the cascade classifier and its detection capability are validated using a proprietary dataset acquired using the Ladybug2 spherical camera and the public INRIA person detection dataset. The final detector achieves a comparable detection performance as Dalal and Triggs [2] detector while achieving on average more than 2.5×-8× speed up depending on the training dataset.

Human motion is a critical aspect of interacting, even between people. It has become an interesting field to exploit in human-robot interaction. Even with today's computing power, it remains a difficult task to successfully follow the human's motion from image processing alone. New sensors were introduced, bringing depth sensing at low or no cost. Using this new technology, this paper presents a new methodology to see space with multiple depth sensors, using machine-learning technique, and features in voxel space to learn to reconstruct humans' joints in single, fused acquisitions. We back up and validate the procedure with ground truth acquired from commercial Motion Capture, and prove the approach to perform particularly well on an expansive set of motion and poses, and compare with current standard software on single depth sensors.

In this paper, a boosted cascade person detection framework with heterogeneous pool of features is presented. The framework unveils a new feature selection scheme based on Pareto-Front analysis and AdaBoost. At each cascade node, Pareto-Front analysis is used to select dominant features thereby reducing the total number of features to a size easily manageable by AdaBoost. The final detector achieves a very low Miss Rate of 0.07 at 10-4 False Positives Per Window on the INRIA public dataset.

Camera networks are an important component of modern complex systems, be it for surveillance, human/machine interaction or healthcare. Having smart cameras that can, by themselves, perform part of the data processing improves scalability both in processing and network resources. In this paper, we present the HYBRID algorithm for multiple person tracking intended for implementation on a smart camera platform, along with the development methodology to implement said algorithm in an FPGA-based smart camera. The HYBRID strategy outperforms the well-known Markov Chain Monte Carlo based particle filter (MCMC-PF) in terms of (i) parallelization capabilities as the MCMC-PF sequentially processes the particles, and (ii) tracking performances (i.e., robustness and precision).

An important challenge in service robotics is to design user interfaces and interaction capabilities that can be used intuitively to enable the potential users to benefit from the robot's functionalities and to allow effective collaborative task execution. The Interactive Behavior Operated shopping Trolley (InBOT) addresses this issue at the example of supporting complex shopping tasks in large supermarkets. The actual control of the robot is shared or traded between the robot and it's dedicated user ranging from closely coupled haptic-based interaction via observation-based interaction up to loosely coupled command-based interaction. The four modes of operation of InBOT, i.e. the manual steering, following, guiding and autonomous mode, the transitions between them and the relevant modalities of interaction are introduced. Finally the paper is completed by describing the results of user studies.

Visual Human Motion Capture (HMC) is a motivating challenge in the Computer Vision community as it enables lots of applications. Many methods have been proposed among which Particle Filters (PF) meet a great success. In this paper, we propose a new algorithm, mixing advantages of the PARTITIONED scheme and quasi random methods. We use a trinocular visual system to propose a comparative study of this particle filter against four other classical ones with respect to a ground truth provided by a commercial HMC system.

Particle filters (PF) are widely used in the Vision literature for visual object tracking. However, the selection and the tuning of the observation pdf (or likelihood function) involved in the particle weighting stage are often eclipsed. These considerations have a strong influence on the tracking performance, especially for human motion capture (HMC) due to the high number of degrees of freedom and the presence of local extrema in the state space. The proposed method is illustrated in the HMC context on a predefined set of likelihoods and assessed w.r.t. a ground truth provided by a commercial HMC system. This paper highlights the influence of their associated free parameters as well as their combination in order to characterize the optimal unified likelihood function. These insights lead to some heuristics to tackle the difficult problem of the likelihood function tuning.

This paper deals with the tracking of persons in a human cluttered environment. It is performed by an active perception system, consisting of a camera mounted on a pan-tilt unit and a 360° RFID detection system which are embedded on a mobile robot. Particle filters enable the fusion of heterogeneous data into the proposal distribution from which the particles are sampled. The information provided by the tracker is then used to build sensor-based dedicated control laws in order to make the robot follow the RFID tagged person. Finally, experiments on our mobile robot are presented in order to highlight the relevance and complementarity of the developed perceptual functions.

With personal robotics and assistance to dependant people, robots are in continuous interface with humans. To enable a more natural communication, based on speech and gesture, robots must be endowed with auditive and visual perception capacities. This paper describes a modular multimodal interface based on speech and gestures in order to control an interactive robot called Jido. In this paper we describe how the speech recognition and understanding module processes deictic and anaphoric utterances in a robust way and how gesture is taken into account at the fusion level to complement speech utterances. These two other modules, namely gesture interpretation and probabilistic multimodal fusion are then briefly depicted. Their integration and the associated robotics experiments are finally reported. These experiments were carried out in the context of a multimodal interactive manipulation task involving successive local motion and handling commands. Our object exchange scenario was successfully run several times with different users.

In this paper, we present a visual human motion capture (HMC) system embedded on a mobile robot. The estimation of the 3D location and configuration of our 14 DOF model of the upper human body is performed with a particle filter. We use a stereoscopic camera to derive appearance based visual cues together with 3D measurements computed from a sparse 3D reconstruction. A comparative study is carried out in order to achieve the best association between measurement cues and filtering strategies in our robotics context. The system performs in real-time in various indoor environments.

This paper presents a thorough study of some particle filter (PF) strategies dedicated to human motion capture from a trinocular vision surveillance setup. An experimental procedure is used, based on a commercial motion capture ring to provide ground truth. Metrics are proposed to assess performances in terms of accuracy, robustness, but also estimator dispersion which is often neglected elsewhere. Relative performances are discussed through some quantitative and qualitative evaluations on a video database. PF strategies based on Quasi Monte Carlo sampling, a scheme which is surprisingly seldom exploited in the Vision community, provide an interesting way to explore. Future works are finally discussed.

This paper deals with video-based face recognition and tracking from a camera mounted on a mobile robot companion. All persons must be logically identified before being authorized to interact with the robot while continuous tracking is compulsory in order to estimate the person's approximate position. A first contribution relates to experiments of still-image-based face recognition methods in order to check which image projection and classifier associations give the highest performance of the face database acquired from our robot. Our approach, based on Principal Component Analysis (PCA) and Support Vector Machines (SVM) improved by genetic algorithm optimization of the free-parameters, is found to outperform conventional appearance-based holistic classifiers (eigenface and Fisherface) which are used as benchmarks. Relative performances are analyzed by means of Receiver Operator Characteristics which systematically provide optimized classifier free-parameter settings. Finally, for the SVM-based classifier, we propose a non-dominated sorting genetic algorithm to obtain optimized free-parameter settings. The second and central contribution is the design of a complete still-to-video face recognition system, dedicated to the previously identified person, which integrates face verification, as intermittent features, and shape and clothing color, as persistent cues, in a robust and probabilistically motivated way. The particle filtering framework, is well-suited to this context as it facilitates the fusion of different measurement sources. Automatic target recovery, after full occlusion or temporally disappearance from the field of view, is provided by positioning the particles according to face classification probabilities in the importance function. Moreover, the multi-cue fusion in the measurement function proves to be more reliable than any other individual cues. Evaluations on key-sequences acquired by the robot during long-term operations in crowded and continuously changing indoor environments demonstrate the robustness of the tracker against such natural settings. Mixing all these cues makes our video-based face recognition system work under a wide range of conditions encountered by the robot during its movements. The paper concludes with a discussion of possible extensions.

In this article we present a two ambiance video camera system dedicated to markerless human motion capture. We introduce a new particle filter algorithm which entails an importance function enabling auto-(re)initialisation, and takes account of the global curvature of the likelihood so as to guide the search along poorly observable directions of the state space. The system robustness is improved by fusing different visual cues. Performances are nearly real time.

This paper presents a Markov Chain Monte Carlo (MCMC) based particle filter to track multiple persons dedicated to video surveillance applications. This hybrid tracker, devoted to networked intelligent cameras, takes benefit from the best properties of both MCMC and joint particle filter. A saliency map-based proposal distribution is shown to limit the well-known burst in terms of particles and MCMC iterations. Qualitative and quantitative results for real-world video data are presented.

Among the cognitive abilities a robot companion must be endowed with, human perception and speech understanding are both fundamental in the context of multimodal human-robot interaction. First, we propose a multiple object visual tracker which is interactively distributed and dedicated to two-handed gestures and head location in 3D. An on-board speech understanding system is also developed in order to process deictic and anaphoric utterances. Characteristics and performances for each of the two components are presented. Finally, integration and experiments on a robot companion highlight the relevance and complementarity of our multimodal interface. Outlook to future work is finally discussed.

This paper presents a new algorithm for human motion three-dimensional tracking based on a stereo camera system embedded on a mobile robot. The approach mixes advantages of the well-known ICONDENSATION and annealed particle filters into a more reliable "I-Annealed" particle filter based tracker. Data fusion is also studied to show that a wide variety of visual cues must be used so that the system can adapt to various backgrounds. A complete implementation of the proposed tracker is described as well as some results on indoor sequences. Finally, evolutions and future work are discussed.

This paper deals with visual recognition and tracking of people and gestures from a camera mounted on a tour-guide robot in a human, cluttered, environment. The particle filtering framework enables the fusion of visual cues, both into an importance function from which the particles are sampled, and into a measurement model used for weights definition. The multi-cues associations prove to be more robust than any of the cues individually. For the purpose of gestures recognition, a tracker is proposed which handles multiple hand configurations templates. Finally, implementation and experiments on a tour-guide robot are presented in order to highlight the relevance and complementarity of the developed visual functions. Extensions are finally discussed.

This article presents the development of a human-robot interaction mechanism based on vision. The functionalities required for such mechanism range from user detection and recognition, to gesture tracking. Particle filters, which are extensively described in the literature, are well suited to this context as they enable a straight combination of several visual cues like colour, shape or motion. Additionally, different algorithms can be considered for a better handling of the particles depending of the context. This article presents the visual functionalities developed namely user recognition and following, and 3D gestures tracking. The challenge is to find which algorithms and visual cues fulfil the best, the requirements of the considered functionalities for our companion robot. The employed methods to attain these required functionalities and their results are presented

Rackham is an interactive robot-guide that has been used in several places and exhibitions. This paper presents its design and reports on results that have been obtained after its deployment in a permanent exhibition. The project is conducted so as to incrementally enhance the robot functional and decisional capabilities based on the observation of the interaction between the public and the robot. Besides robustness and efficiency in the robot navigation abilities in a dynamic environment, our focus was to develop and test a methodology to integrate human-robot interaction abilities in a systematic way. We first present the robot and some of its key design issues. Then, we discuss a number of lessons that we have drawn from its use in interaction with the public and how that will serve to refine our design choices and to enhance robot efficiency and acceptability

The interaction between men and machines has become an important topic for the robotics community as it can generalize the use of robots. In this context, advanced robots must integrate capabilities to interpret humans motion as well as persons gestures in order to perform tasks for the humans or in synergy with them. The purpose of this paper is to show a real-time system for face/hand tracking and hand gesture recognition in the particle filtering framework. We introduce mechanisms for visual data fusion within particle filtering to develop trackers combining in a novel way color and shape cues, skin blobs or frontal face detection. For the purpose of face tracking, the fusion of modalities based on color and shape allows to avoid noticeable drift, even possible subsequent loss in the worst case. For gestures interpretation, an extension is proposed to achieve in the tracking loop the recognition of the current hand posture and of its motion in the video stream. In both tracking scenarios, the combination or fusion of cues proves to be more robust in cluttered environments than any of the cues individually. The global performances of the proposed trackers and future works are also discussed.

The interaction between man and machines has become an important topic for the robotic community as it can generalise the use of robots. For active H/R interaction scheme, the robot needs to detect human faces in its vicinity and then interpret canonical gestures of the tracked person assuming this interlocutor has been beforehand identified. In this context, we depict functions suitable to detect and recognise faces in video stream and then focus on face or hand tracking functions. An efficient colour segmentation based on a watershed on the skin-like coloured pixels is proposed. A new measurement model is proposed to take into account both shape and colour cues in the particle filter to track face or hand silhouettes in video stream. An extension of the basic condensation algorithm is proposed to achieve recognition of the current hand posture and automatic switching between multiple templates in the tracking loop. Results of tracking and recognition using are illustrated in the paper and show the process robustness in cluttered environments and in various light conditions. The limits of the method and future works are also discussed.

This article describes visual functions dedicated to the extraction and recognition of planar quadrangles detected from a single camera. Extraction is based on a relaxation scheme with constraints between image segments, while the characterization we propose allows recognition to be achieved from different view-points and viewing conditions. We defined and evaluated several metrics on this representation space - a correlation-based one and another one based on sets of interest points.

This article describes the integration in a whole navigation system of visual functions dedicated to the extraction and recognition of visual landmarks, i.e. planar quadrangles detected from a single camera. The extraction of these landmarks is based on a relaxation scheme, used to satisfy constraints between image segments. During the exploration of an indoor environment, according to the current topology, there are several ways to take advantage of these visual landmarks. Two representations are considered, relying both on laser data and or our visual landmarks system: a GVG-based model (generalized Voronoi graph) for corridors, and a composite stochastic map for open spaces. In the particular case of corridors, the corresponding modules - laser GVG construction and visual landmarks extraction and recognition - can cooperate to complete each other, as image processing can be enhanced by some structural knowledge about the scene, whereas the GVG is annotated, even as far as its edges are concerned, by qualitative visual information.

Presents vision functions needed on a mobile robot to deal with landmark-based navigation in buildings. Landmarks are planar, quadrangular surfaces, which must be distinguished from the background, typically a poster on a wall or a door-plate. In a first step, these landmarks are detected and their positions with respect to a global reference frame are learned; this learning step is supervised so that only the best landmarks are memorized, with an invariant representation based on a set of interest points. Then, when the robot looks for visible landmarks, the recognition procedure takes advantage of the partial Hausdorff distance to compare the landmark model and the detected quadrangles. The paper presents the landmark detection and recognition procedures, and discusses their performances.

This article describes the integration in a complete navigation system of an environment modeling method based on a Generalized Voronoi Graph (GVG), relying on laser data, on the one hand, and of a localization method based on monocular vision landmark learning and recognition framework, on the other hand. Such a system is intended to work in structured environments. It is shown that the two corresponding modules - laser GVG construction and visual landmarks learning and recognition - can cooperate to complete each other, as image processing can be enhanced by some structural knowledge about the scene, whereas the GVG is annotated, even as far as its edges are concerned, by qualitative visual information.

A machine vision system based on a CCD camera combined with a light beam matrix is developed to output the 3D representation of a scene. This paper focuses on the problem of image spot labelling: matching between image spots and light beams is required for the triangulation procedure. A previous part of our work addressed the labelling problem through discrete relaxation. Each frame is analyzed independently and the labelling process considers all the possible combinations at each time. A major contribution, focused in this paper, concerns the insertion of another constraint based on image spot motion over time in order to improve the robustness of the labelling process. Motion prediction of image spots is deduced from previous images and allowed to reduce the possible labelling combinations in the current frame. Beam tracking results are discussed and 3D reconstructions are then presented.

A machine vision system based on a CCD camera combined with a light beams matrix is developed to output the 3D surface shape of vehicle cockpit occupancy. This paper focuses on the problem of image spots labeling. An initial set of possible spot/beam matching is deduced from epipolar constraints provided by the prior calibration of the relative camera/projector position. Then, using the topological constraints in the 2D mesh of illuminated dots, a constraint propagation process, based on relaxation or maximal cliques search, eliminates most of the possible combinations from the initial set of matching. Results of the labeling process, obtained by the two methods, are then discussed.

Describes the localization function integrated in a landmark-based navigation system. It relies on the use of planar landmarks (typically, posters) to localize the robot. It is based on two periodic processes running at different frequencies. One of them performs the poster tracking (based on the partial Hausdorff distance) and the active control of the camera. The other one runs on a lower frequency and localizes the robot thanks to the tracked landmarks, the positions of which have been learnt during an offline exploration step. The system has been embedded on our indoor Hilare mobile robot and works in real time. Experiments, illustrated in the paper, demonstrate the validity of the approach.