Smart tangible toys, designed for hand manipulation, can transform fine motor skills assessment into enjoyable activities which are engaging for children to play (partially) unsupervised. Such toys can support school teachers and parents for early detection of deficiencies in motor skills development of children, as well as objectively monitor the progress of skills development over time. To make a game enjoyable for children with different skills level, these smart toys could offer an adaptive game play. In this paper we describe the design and deployment of a digital board game, equipped with sensors, which we use to explore the potential of using smart toys for fine motor skills assessment in children.

Design-based inquiries into the networked products of the Internet of Things (IoT) lack a coherent understanding of the effect of such products on society. This paper proposes a new taxonomy for networked products, which would allow articulation on their current state and future, and provide insights to designers for creating meaningful and aesthetic products of IoT. Central to this framework is the proposition that our current product-scape should be understood as a distribution of material agencies and best analyzed through the metaphor of "agency". We identify three types of agencies, i.e., the Collector, the Actor, and the Creator, and discuss how this approach could create new design methodologies to create more meaningful networked products that would empower people in their everyday lives.

Human activity recognition system is of great importance in robot-care scenarios. Typically, training such a system requires activity labels to be both completely and accurately annotated. In this paper, we go beyond such restriction and propose a learning method that allow labels to be incomplete and uncertain. We introduce the idea of soft labels which allows annotators to assign multiple, and weighted labels to data segments. This is very useful in many situations, e.g., when the labels are uncertain, when part of the labels are missing, or when multiple annotators assign inconsistent labels. We formulate the activity recognition task as a sequential labeling problem. Latent variables are embedded in the model in order to exploit sub-level semantics for better estimation. We propose a max-margin framework which incorporate soft labels for learning the model parameters. The model is evaluated on two challenging datasets. To simulate the uncertainty in data annotation, we randomly change the labels for transition segments. The results show significant improvement over the state-of-the-art approach.

Sensor systems can be deployed in the homes of older adults living alone for functional health assessments. Their information is very useful for health care specialists. The problem lies in developing person independent models while facing a large variability in behavior. We address this problem by, first, proposing a new feature extraction method for data from ambient motion sensors. The method uses functional similarities between houses and daily structure to extract meaningful features. Second, we propose a change-based approach for analyzing data, taking difference scores of both the sensor features and health metrics. To evaluate our approach, experiments on longitudinal data were conducted, where the relationship between sensor data and health measurements was modeled with linear regression and (nonlinear) regression forests. These experiments show that the change-based approach yields better results and that the resulting models can be used as a reliable metric for (functional) health. In addition, feature analysis can help health care specialists understand relevant aspects of behavior. Prediction of health metrics is possible even with simple sensors. With such sensors, it is possible to detect problems and health decline in an early stage. This will have great impact on clinical practice.

The performance of human-robot collaboration tasks can be improved by incorporating predictions of the human collaborator's movement intentions. These predictions allow a collaborative robot to both provide appropriate assistance and plan its own motion so it does not interfere with the human. In the specific case of human reach intent prediction, prior work has divided the task into two pieces: recognition of human activities and prediction of reach intent. In this work, we propose a joint model for simultaneous recognition of human activities and prediction of reach intent based on skeletal pose. Since future reach intent is tightly linked to the action a person is performing at present, we hypothesize that this joint model will produce better performance on the recognition and prediction tasks than past approaches. In addition, our approach incorporates a simple human kinematic model which allows us to generate features that compactly capture the reachability of objects in the environment and the motion cost to reach those objects, which we anticipate will improve performance. Experiments using the CAD-120 benchmark dataset show that both the joint modeling approach and the human kinematic features give improved F1 scores versus the previous state of the art.

Human activity recognition is an essential task for robots to effectively and efficiently interact with the end users. Many machine learning approaches for activity recognition systems have been proposed recently. Most of these methods are built upon a strong assumption that the labels in the training data are noise-free, which is often not realistic. In this paper, we incorporate the uncertainty of labels into a max-margin learning algorithm, and the algorithm allows the labels to deviate over iterations in order to find a better solution. This is incorporated with a hierarchical approach where we jointly estimate activities at two different levels of granularity. The model is tested on two datasets, i.e., the CAD-120 dataset and the Accompany dataset, and the proposed model shows outperforming results over the state-of-the-art methods.

We present a novel hierarchical model for human activity recognition. In contrast with approaches that successively recognize actions and activities, our approach jointly models actions and activities in a unified framework, and their labels are simultaneously predicted. The model is embedded with a latent layer that is able to capture a richer class of contextual information in both state-state and observation-state pairs. Although loops are present in the model, the model has an overall linear-chain structure, where the exact inference is tractable. Therefore, the model is very efficient in both inference and learning. The parameters of the graphical model are learned with a structured support vector machine. A data-driven approach is used to initialize the latent variables; therefore, no manual labeling for the latent states is required. The experimental results from using two benchmark datasets show that our model outperforms the state-of-the-art approach, and our model is computationally more efficient.

We describe a novel framework that combines an overhead camera and a robot RGB-D sensor for real-time people finding. Finding people is one of the most fundamental tasks in robot home care scenarios and it consists of many components, e.g. people detection, people tracking, face recognition, robot navigation. Researchers have extensively worked on these components, but as isolated tasks. Surprisingly, little attention has been paid on bridging these components as an entire system. In this paper, we integrate the separated modules seamlessly, and evaluate the entire system in a robot-care scenario. The results show largely improved efficiency when the robot system is aided by the localization system of the overhead cameras.

We present a novel latent discriminative model for human activity recognition. Unlike the approaches that require conditional independence assumptions, our model is very flexible in encoding the full connectivity among observations, latent states, and activity states. The model is able to capture richer class of contextual information in both state-state and observation-state pairs. Although loops are present in the model, we can consider the graphical model as a linear-chain structure, where the exact inference is tractable. Thereby the model is very efficient in both inference and learning. The parameters of the graphical model are learned with the Structured-Support Vector Machine (Structured-SVM). A data-driven approach is used to initialize the latent variables, thereby no hand labeling for the latent states is required. Experimental results on the CAD-120 benchmark dataset show that our model outperforms the state-of-the-art approach by over 5% in both precision and recall, while our model is more efficient in computation.

Automated human activity recognition is an essential task for Human Robot Interaction (HRI). A successful activity recognition system enables an assistant robot to provide precise services. In this paper, we present a two-layered approach that can recognize sub-level activities and high-level activities successively. In the first layer, the low-level activities are recognized based on the RGB-D video. In the second layer, we use the recognized low-level activities as input features for estimating high-level activities. Our model is embedded with a latent node, so that it can capture a richer class of sub-level semantics compared with the traditional approach. Our model is evaluated on a challenging benchmark dataset. We show that the proposed approach outperforms the single-layered approach, suggesting that the hierarchical nature of the model is able to better explain the observed data. The results also show that our model outperforms the state-of-the-art approach in accuracy, precision and recall.

This paper proposes a novel probabilistic approach for appearance-based person reidentification in non-overlapping camera networks.It accounts for varying illumination, varying camera gain and has low computational complexity.More specifically, we present a graphical model where we model the person's appearance in addition to camera illumination and gain.We analytically derive the solutions for the person's appearance and camera properties, and use a novel constant time Gibbs sampling scheme to estimate the identification labels.We validate our algorithm on two indoor datasets and perform a comparative analysis with existing algorithms.We demonstrate significantly increased re-identification accuracy in addition to significantly reducing the computational complexity on our datasets.

In this paper we outline the design process of the TaSST (Tactile Sleeve for Social Touch), a touch-sensitive vibrotactile arm sleeve. The TaSST was designed to enable two people to communicate different types of touch over a distance. The touch-sensitive surface of the sleeve consists of a grid of 4×3 sensor compartments filled with conductive wool. Each compartment controls the vibration intensity of a vibration motor, located in a grid of 4×3 motors beneath the touch-sensitive layer. An initial evaluation of the TaSST revealed that it was mainly suitable for communicating protracted (e.g. pressing), and simple (e.g. poking) touches.

With changes in life expectancy across the world, technologies enhancing well-being of individuals, specifically for older people, are subject to a new stream of research and development. In this paper we present the ACCOMPANY project, a pan-European project which focuses on home companion technologies. The projects aims to progress beyond the state of the art in multiple areas such as empathic and social human-robot interaction, robot learning and memory visualisation, monitoring persons and chores at home, and technological integration of these multiple approaches on an existing robotic platform, Care-O-Bot®3 and in the context of a smart-home environment utilising a multitude of sensor arrays. The resulting prototype from integrating these developments undergoes multiple formative cycles and a summative evaluation cycle towards identifying acceptable behaviours and roles for the robot for example role as a butler or a trainer. Furthermore, the evaluation activities will use an evaluation grid in order to assess achievement of the identified user requirements, formulated in form of distinct scenarios. Finally, the project considers ethical concerns and by highlighting principles such as autonomy, independence, enablement, safety and privacy, it embarks on providing a discussion medium where user views on these principles and the existing tension between some of these principles for example tension between privacy and autonomy over safety, can be captured and considered in design cycles and throughout project developments.

We address the problem of person re-identification in colour-depth camera using the height temporal information of people. Our proposed gait-based feature corresponds to the frequency response of the height temporal information. We demonstrate that the discriminative periodic motion associated with human gait is encoded within the height temporal information. Additionally, we also investigate the discriminative ability of a novel feature vector obtained by the integration of the height temporal information with a color and height-based appearance model. Given the proposed features we adopt a feature selection scheme for each person, based on KL-divergence, to identify the discriminative subset of frequency bins for the height-based gait feature that enhances the overall identification accuracy. To identify the test person, we formulate a probabilistic matching framework incorporating the selected frequency bins. We validate our algorithm on the publicly available TUM-GAID dataset and our studio datasets and report over 80% accuracy for 75 people with our combined feature, significantly better than standard colour-based features. Additionally, we also observe that height-based gait features reports over 90% for smaller population and are rotation invariant, robust to appearance noise and occlusion.

We describe a system that recognizes human postures with heavy self-occlusion. In particular, we address posture recognition in a robot assisted-living scenario, where the environment is equipped with a top-view camera for monitoring human activities. This setup is very useful because top-view cameras lead to accurate localization and limited inter-occlusion between persons, but conversely they suffer from body parts being frequently self-occluded. The conventional way of posture recognition relies on good estimation of body part positions, which turns out to be unstable in the top-view due to occlusion and foreshortening. In our approach, we learn a posture descriptor for each specific posture category. The posture descriptor encodes how well the person in the image can be `explained' by the model. The postures are subsequently recognized from the matching scores returned by the posture descriptors. We select the state-of-the-art approach of pose estimation as our posture descriptor. The results show that our method is able to correctly classify 79.7% of the test sample, which outperforms the conventional approach by over 23%.

We present a novel probabilistic framework that fuses information coming from the audio and video modality to perform speaker diarization. The proposed framework is a Dynamic Bayesian Network (DBN) that is an extension of a factorial Hidden Markov Model (fHMM) and models the people appearing in an audiovisual recording as multimodal entities that generate observations in the audio stream, the video stream, and the joint audiovisual space. The framework is very robust to different contexts, makes no assumptions about the location of the recording equipment, and does not require labeled training data as it acquires the model parameters using the Expectation Maximization (EM) algorithm. We apply the proposed model to two meeting videos and a news broadcast video, all of which come from publicly available data sets. The results acquired in speaker diarization are in favor of the proposed multimodal framework, which outperforms the single modality analysis results and improves over the state-of-the-art audio-based speaker diarization.

The first step towards analysing social interactive behaviour in crowded environments is to identify who is interacting with whom. This paper presents a new method for detecting focused encounters or F-formations in a crowded, real-life social environment. An F-formation is a specific instance of a group of people who are congregated together with the intent of conversing and exchanging information with each other. We propose a new method of estimating F-formations using a graph clustering algorithm by formulating the problem in terms of identifying dominant sets. A dominant set is a form of maximal clique which occurs in edge weighted graphs. As well as using the proximity between people, body orientation information is used; we propose a socially motivated estimate of focus orientation (SMEFO), which is calculated with location information only. Our experiments show significant improvements in performance over the existing modularity cut algorithm and indicates the effectiveness of using a local social context for detecting F-formations.

An activity monitoring system allows many applications to assist in care giving for elderly in their homes. In this paper we present a wireless sensor network for unintrusive observations in the home and show the potential of generative and discriminative models for recognizing activities from such observations. Through a large number of experiments using four real world datasets we show the effectiveness of the generative hidden Markov model and the discriminative conditional random fields in activity recognition.

The human robot interaction community is multidisciplinary by nature and has members from social science to engineering backgrounds. In this paper we aim to provide human robot developers with a straightforward toolkit to evaluate users' acceptance of assistive social robots they are designing or developing for elderly care environments. We will explain how we developed the measures for this analysis, provide do's and don'ts in designing the experiments, demonstrate the application of the measures we have developed for this purpose and the analysis and interpretation of the data. As such we hope to engage human robot interaction developers in evaluating the acceptability of their own robot to inform the development process and improve the final robot's design.

In this paper we explore the concept of enjoyment as a possible factor influencing acceptance of robotic technology by elderly people. We describe an experiment with a conversational robot and elderly users (n=30) that incorporates both a test session and a long term user observation. The experiment did confirm the hypothesis that perceived enjoyment has an effect on the intention to use a robotic system. Furthermore, findings show that the general assumption in technology acceptance models that intention to use predicts actual use is also applicable to this specific technology used by elderly people.

A sensor system capable of automatically recognizing activities would allow many potential ubiquitous applications. In this paper, we present an easy to install sensor network and an accurate but inexpensive annotation method. A recorded dataset consisting of 28 days of sensor data and its annotation is described and made available to the community. Through a number of experiments we show how the hidden Markov model and conditional random fields perform in recognizing activities. We achieve a timeslice accuracy of 95.6% and a class accuracy of 79.4%.

Summary form only given. The possibilities of using robots in eldercare have inspired a growing number of research projects. Not only can robotic technologies be used for rehabilitation, also they could facilitate the work of caregivers and provide social company for elders. In our project we are interested in the acceptance of robotic technology by elders and particularly in the influence of a robots social abilities on acceptance. Several studies on interaction with robots stress the importance of social intelligence and even more so in a health- and eldercare environment. It shows a more social intelligent robot will be more effective in its communication and it can therefore be expected to be easier and more pleasant to interact with and therefore would be indeed accepted easier.

When using a robot or a screen agent, elderly users might feel more enjoyment if they experience a stronger social presence. In two experiments with a robotic agent and a screen agent (both n=30) this relationship between these two concepts could be established. Besides, both studies showed that social presence correlates with the intention to use the system, although there were some differences between the agents. This implicates that factors that influence social presence are relevant when designing assistive agents for elderly people.

An annotated data set is presented meant to help researchers in developing, evaluating, and comparing various approaches in robotics for building space representations appropriate for communicating with humans. The data consist of omnidirectional images, laser range scans, sonar readings, and robot odometry. A set of base-level human spatial concepts is used to annotate the data.

This paper presents a novel framework that utilizes multi-modal information to achieve speaker diarization. We use dynamic Bayesian networks to achieve on-line results. We progress from a simple observation model to a complex multi-modal one as more data becomes available. We present an efficient way to guide the learning procedure of the complex model using the early results achieved with the simple model. We present the results achieved in various real-world situations, including videos coming from webcameras, human computer interaction and video conferences.

Vision systems are used more and more in 'personal' robots interacting with humans, since semantic information about objects and places can be derived from the rich sensory information. Visual information is also used for building appearance based topological maps, which can be used for localization. In this paper we describe a system capable of using this appearance based topological map for navigation. The system is made robust by using the epipolar geometry and a planar floor constraint in computing the necessary heading information. Using this method the robot is able to drive robustly in a large environment. We tested the method on real data under varying environment conditions and compared performance with a human-controlled robot.

This paper addresses the problem of people detection using 2D range data and omnidirectional vision. The range data is often used in robotics to detect person legs. First, we will develop a reliable 2D range data leg detector using the AdaBoost algorithm. Second, instead of using the leg detector to detect people, we propose a more reliable method that takes into account the spatial arrangement of the detected legs. The method is inspired by the latest results on the "part-based representations" from the computer vision area. Finally, we show how the part-based representations can be used to combine the leg detections from the 2D range data and upper body, lower body and full body detections from omnidirectional camera. The experimental results show highly reliable people detection when the two sensors are combined. We use Haar-like features and the AdaBoost again to construct the omnicam body parts detectors.

Content analysis of clips containing people speaking involves processing informative cues coming from different modalities. These cues are usually the words extracted from the audio modality, and the identity of the persons appearing in the video modality of the clip. To achieve efficient assignment of these cues to the person that created them, we propose a Bayesian network model that utilizes the extracted feature characteristics, their relations and their temporal patterns. We use the EM algorithm in which the E-step estimates the expectation of the complete-data log-likelihood with respect to the hidden variables - that is the identity of the speakers and the visible persons. In the M-step , the person models that maximize this expectation are computed. This framework produces excellent results, exhibiting exceptional robustness when dealing with low quality data.

Mobile robot localization and navigation requires a map - the robot's internal representation of the environment. A common problem is that path planning becomes very inefficient for large maps. In this paper we address the problem of segmenting a base-level map in order to construct a higher-level representation of the space which can be used for more efficient planning. We represent the base-level map as a graph for both geometric and appearance based space representations. Then we use a graph partitioning method to cluster nodes of the base-level map and in this way construct a high-level map, which is also a graph. We apply a hierarchical path planning method for stochastic tasks based on Markov decision processes (MDPs) and investigate the effect of choosing different numbers of clusters

In appearance based robot localization a new image is matched with every image in the database. In this paper we describe how to reduce the number of images in this database with minimal loss of information and thereby increasing the efficiency of localization significantly. First we build an appearance based model consisting of a graph in which the nodes denote images and links denote relations between images. This graph is then pruned using the connected dominating set algorithm. The method is tested on an image set acquired by a mobile robot equipped with an omnidirectional camera driving around in an office environment, as well as sets of images taken while driving through a real furnished home environment

An ambitious goal in modern robotic science is to build mobile robots that are able to interact as companions in real world environments. Especially for caretaking of elderly people a system robustly working at private homes is essential, requiring a very natural and human oriented way of communication. Since home environments are usually very individual a first task for a newly acquired robot is to get familiar with its new environment. This paper gives a short overview on how we integrated a vision based localization using the advantages of a very modular architecture and extending a spoken dialog system for online labeling and interaction about different locations. We present results from the integrated system working in a real, fully furnished home environment where it was able to learn the names of different rooms. This system enables us to perform real user studies in future without the need to fall back to Wizard-of-Oz experiments. Ongoing work aims at enabling the robot to take initiative by asking for unknown locations. A future extension is the ability to generalize over features of known rooms to make predictions when encountering unknown rooms

This study examines the influence of perceived social abilities of a robot on user's attitude towards and acceptance of the robot. An interface robot with simulated conversational capabilities was used in a Wizard of Oz experiment with two conditions: a more socially communicative (the robot made use of a larger set of social abilities in interaction) and a less socially communicative interface. Participants (n=40) were observed in 5 minute interaction sessions and were asked to answer questions on perceived social abilities and technology acceptance. Results show that participants who were confronted with the more socially communicative version of the robot felt more comfortable and were more expressive in communicating with it. This suggests that the more socially communicative condition would be more likely to be accepted as a conversational partner. However, the findings did not show a significant correlation between perceived social abilities and technology acceptance

When determining subject preferences for human-robot interaction, an important issue is the interpretation of the subjects' responses during the trials. Employing a non-intrusive approach, this paper discusses the methodological issues for annotating vision data by allowing the subjects to indicate their comfort using a handheld comfort level device during the trials. In previous research, the analysis of collected comfort and vision data was made difficult due to problems concerning the manual synchronization of different modalities. In the current paper, we overcome this issue by real-time integration of the subject's feedback on subjective comfort into the video stream. The implications for more efficient analysis of human-robot interaction data, as well as possible future developments of this approach are discussed

This paper shows that the common task of interest region matching using local descriptors can be improved using a new similarity measure. The similarity measure is motivated by the information theoretic image alignment that maximize mutual information between images. A property of the mutual information metric is that it does not only depend on how similar the signals are but also how complex they are. We present how similar logic can be applied to the standard SIFT descriptor. The results show improvement at almost no additional computational costs.

We consider a visual scene analysis scenario where objects (e.g. people, cars) pass through the viewing field of a static camera and need to be detected and segmented from the background. For this purpose, we introduce a hybrid dynamic Bayesian network and derive an expectation propagation (EP) algorithm for robust estimation of object shapes and appearance statistics. We demonstrate the viability of the approximation on an object detection task from real videos, where objects' smooth shapes are segmented from the background. The model is readily extendible to multi-object multi-camera scenarios and can be coupled in a transparent and consistent way with a hierarchical model for object identification under uncertainty.

In this paper we describe a system that enables a mobile robot equipped with a color vision system to track humans in indoor environments. We developed a method for tracking humans when they are within the field of view of the camera, based on motion and color cues. However, the robot also has to keep track of humans which leave the field of view and re-enter later. We developed a dynamic Bayesian network for such a global tracking task. Experimental results on real data confirm the viability of the developed method.

This paper addresses the question how robot planning (e.g. for navigation) can be done with hierarchical maps. We present an algorithm for hierarchical path planning for stochastic tasks, based on Markov decision processes (MDPs) and dynamic programming. It is more efficient than standard dynamic programming for "flat" MDPs, because it reduces the state space for all levels in its hierarchy and it allows reuse of previously computed partial policies. This computational advantage comes at the cost of some extra memory and overhead to represent and coordinate the hierarchical system, and in some cases somewhat longer paths to target locations. We demonstrate the method on artificially generated MDP data, and on real robot data from our vision-controlled robot navigating in an office environment.

This paper addresses the problem of automatic construction of a hierarchical map from images. Our approach departs from a large collection of omnidirectional images taken at many locations in a building. First, a low-level map is built that consists of a graph in which relations between images are represented. For this, we use a metric based on visual landmarks (SIFT features) and geometrical constraints. Then, we use a graph partitioning method to cluster nodes and in this way construct the high-level map. Experiments on real data show that meaningful higher and lower level maps are obtained, which can be used for accurate localization and planning.

Visual surveillance in wide areas (e.g. airports) relies on sparsely distributed cameras, that is, cameras that observe nonoverlapping scenes. In this setup, multiobject tracking requires reidentification of an object when it leaves one field of view, and later appears at some other. Although similar association problems are common for multiobject tracking scenarios, in the distributed case one has to cope with asynchronous observations and cannot assume smooth motion of the objects. In this paper, we propose a method for human indoor tracking. The method is based on a Dynamic Bayes Network (DBN) as a probabilistic model for the observations. The edges of the network define the correspondences between observations of the same object. Accordingly, we derive an approximate EM-like method for selecting the most likely structure of DBN and learning model parameters. The presented algorithm is tested on a collection of real-world observations gathered by a system of cameras in an office building.

There is a rapidly growing demand for using smart cameras for various biometric applications in surveillance. Although having a small form-factor, most of these applications demand huge processing performance for real-time processing. Face recognition is one of those applications. In this paper we show that we can run face recognition in real-time by implementing the algorithm on an architecture which combines a parallel pixel processor: with a digital signal processor. The algorithm consists of a cascade of filters for detection, registration and normalization and an RBF neural network with temporal filtering. Everything fits within a digital camera, the size of a normal surveillance camera.

The main drawback of appearance-based robot localization with respect to landmark-based one is that it requires a map including images taken at known positions in the area where the robot is expected to move. In this paper, we describe a concurrent map-building and localization (CML) system developed within the appearance-base robot localization paradigm. This allows us to combine the good features of appearance-base localization, such as simple sensor processing or robustness, without having to deal with its inconveniences. In our CML system, both the robot's position and the map are represented using Gaussian mixtures. Using this kind of representation, we can deal with the global localization problem while being efficient both in memory and in execution time.

We describe a method for estimating the translation and rotation between two subsequent poses of a moving robot from images taken with an omnidirectional vision system. This allows some form of visual odometry. We use two sorts of projections derived from the omnidirectional image. The rotation and translation direction are determined from panoramic projections. After that, a projection on a plane parallel to the ground is used to estimate the length of the translation vector. Experiments on real and simulated data are carried out.

In this paper, we extend our appearance-based localization system moving from a passive approach to an active one where the robot can execute actions with the only purpose of gaining information about its location in the environment. We present a general framework for entropy-based action selection and we describe how this framework can be implemented in our localization system. The result is an action evaluation process more efficient in memory and in execution time than previously existing ones. The experiments we present show that the action selection mechanism effectively decreases the error on localization in environments with a high degree of aliasing. This can be of great help to improve the performance of our localization system in dynamic environments.

In this paper, we enhance appearance-based robot localization by using disparity maps. Disparity maps provide the same type of information as range based sensors (distance to objects) and thus, they are likely to be less sensitive to changes of illumination than plain images, that are the source of information generally used in appearance-based localization. The main drawback of disparity maps is that they can include very noisy depth values: points for which the algorithms can not determine reliable depth information. These noisy values have to be discarded resulting in missing values. The presence of missing values makes principal component analysis (the standard method used to compress images in the appearance-based framework) unfeasible. We describe a novel expectation-maximization algorithm to determine the principal components of a data set including missing values and we apply it to disparity maps. The results we present show that disparity maps are a valid alternative to increase the robustness of appearance-based localization.

This article concerns the greedy learning of gaussian mixtures. In the greedy approach, mixture components are inserted into the mixture one after the other. We propose a heuristic for searching for the optimal component to insert. In a randomized manner, a set of candidate new components is generated. For each of these candidates, we find the locally optimal new component and insert it into the existing mixture. The resulting algorithm resolves the sensitivity to initialization of state-of-the-art methods, like expectation maximization, and has running time linear in the number of data points and quadratic in the (final) number of mixture components. Due to its greedy nature, the algorithm can be particularly useful when the optimal number of mixture components is unknown. Experimental results comparing the proposed algorithm to other methods on density estimation and texture segmentation are provided.

In this paper, we present an efficient multi-baseline stereo algorithm for panoramic image data. We derive a parameterization of epipolar curves in terms of inverse depth. As a result, the search for image correspondences across multiple images can be performed efficiently. Furthermore, depth estimates are obtained directly, thus bypassing the need to perform explicit stereoscopic triangulation. We apply our method to obtain a three-dimensional reconstruction of an environment from a set of panoramic images. The images are acquired by a single omnidirectional vision sensor mounted on top of our mobile robot during navigation. Experimental results demonstrate the effectiveness of our approach.

We describe a method for the reconstruction of a driven trajectory of a mobile robot if the begin and end states of the trajectory are known, and intermediate readings from odometry are available. Our method uses a Kalman filter to combine a forward and backward dead-reckoning trajectory. We show that our method is more reliable for long trajectories than just combining the dead-reckoning trajectories independently.

We apply the auxiliary particle filter algorithm of Pitt and Shephard (1999) to the problem of robot localization. To deal with the high-dimensional sensor observations (images) and an unknown observation model, we propose the use of an inverted nonparametric observation model computed by nearest neighbor conditional density estimation. We show that the proposed model can lead to a fully adapted optimal filter, and is able to successfully handle image occlusion and robot kidnap. The proposed algorithm is very simple to implement and exhibits a high degree of robustness in practice. We report experiments involving robot localization from omnidirectional vision in an indoor environment.

We propose a method for making globally consistent pose estimates using vision. Lu and Milios (1997) described an approach where the links between poses were estimated using range scanner data. When using point correspondences the length of the links cannot be estimated and the approach of Lu and Milios has to be modified. First, we use the nonlinear orientation part of the pose differences to obtain a reference trajectory. Then the reference trajectory is used to scale and orientate the linear spatial part of the pose differences, such that the positions can be estimated as well. We show the results of an experiment of navigating a robot equipped with an omnidirectional camera on the corridor.

High-dimensional data generated by a system with limited degrees of freedom are often constrained in low-dimensional manifolds in the original space. In this article, we investigate dimension-reduction methods for such intrinsically low-dimensional data through linear projections that preserve the manifold structure of the data. For intrinsically one dimensional data, this implies projecting to a curve on the plane with as few intersections as possible. We are proposing a supervised projection pursuit method that can be regarded as an extension of the single-index model for nonparametric regression. We show results from a toy and two robotic applications.

We address the problem of recovering range information from a pair of images obtained from camera viewpoints whose relative poses are known (stereo vision). The images we deal with are cylindrical re-projections of images captured by an omnidirectional vision sensor. Images obtained by such a device have a relatively low spatial resolution in comparison to standard camera images. We analyze the sensitivity of range estimation from image correspondences with respect to errors caused by discretization. The analysis reveals the significance of obtaining sub-pixel accuracy for range estimation. We present our method and experimental results. The results indicate that high accuracy range estimates can be obtained using our method.

Feature extraction from robot sensor data is a standard way to deal with the high dimensionality and redundancy of such data. In order to get optimal task-relevant features, PCA must be replaced by a supervised projection method. In this paper we extend our previously proposed supervised linear feature extraction method (2000) in two ways: 1) the projection matrix is optimized simultaneously over all columns under the constraint of orthonormality; and 2) a Jacobi parametrization of the matrix allows the use of unconstrained nonlinear optimization algorithms. The new algorithm is more efficient and many times faster than the old version. We show experimental results in extracting features from panoramic images of a mobile robot. The results compare favorably to the PCA solutions.

We introduce a way of modeling distributions with a low latent dimensionality our method allows for a strict control of the properties of the mapping between the latent and the feature space. Usually, as in for example generative topographic mapping, this mapping is constructed through the maximization of the log likelihood of the data set. However, if the data set is supervised, in the sense that we know the corresponding latent vector value for each feature vector; it is more sensible to use same regression method for finding the mapping in advance. The mapping is then fixed during optimization of the log likelihood of the data set. It is concluded that in terms of log likelihood the methods are comparable. The advantages however lie in the better understanding of the properties of the mapping and a clear interpretation of the latent variables.

We are seeking linear projections of supervised high-dimensional robot observations and an appropriate environment model that optimize the robot localization task. We show that an appropriate risk function to minimize is the conditional entropy of the robot positions given the projected observations. We propose a method of iterative optimization through a probabilistic model based on kernel smoothing. To obtain good starting optimization solutions we use canonical correlation analysis. We apply our method on a real experiment involving a mobile robot equipped with an omnidirectional camera in an office setup.

In order to do useful things a mobile robot needs some sort of global information about the environment it is operating in. In the paper an approach is described where the global information is not cast in a model of the geometry of the environment but in a model of all sensory data of the robot. As a primary sensing system we use computer vision. The model gives a probability distribution over the learned locations given an observation. We developed an active vision strategy to increase the performance and tested the method on real image data from our robot.

A key issue in mobile robot applications involves building a map of the environment to be used by the robot for localization and path planning. We propose a framework for robot map building which is based on principal component regression, a statistical method for extracting low-dimensional dependencies between a set of input and target values. A supervised set of robot positions (inputs) and associated high-dimensional sensor measurements (targets) are assumed. A set of globally uncorrelated features of the original sensor measurements are obtained by applying principal component analysis on the target set. A parametrized model of the conditional density function of the sensor features given the robot positions is built based on an unbiased estimation procedure that fits interpolants for both the mean and the variance of each feature independently. The simulation results show that the average Bayesian localization error is an increasing function of the principal component index.

If a camera moves on a straight line, the optic flow field is a diverging vector field, of which the singularity is called "focus of expansion" (FOE). An object which is seen in this FOE is located on the future path of the camera. If the camera is also rotating, the future path is no longer a point in the image domain, but a line. All objects which are on the future path (and thus will cause collisions) are projected on this line. However, the reverse is not necessary true - not all points on the line are collision points. In this paper we derive how the optic flow can be used to compute which points in the image are projections of collision points.

The robot navigation task presented in this paper is to drive through the center of a corridor, based on a sequence of images from an on-board camera. Our measurements of the system state the distance to the wall and orientation of the wall, are derived from the optic flow. Whereas the structure of the environment is usually computed from the spatial derivatives of the optic flow, we used the structure contained in the temporal derivatives of the optic flow to compare the environment structure and hence the system state. The algorithm is used to control a 'remote brain' robot and results on the accuracy of the state estimates are presented.

The paper describes a sensor based navigation scheme which makes use of a global representation of the environment by means of a self-organizing map or Kohonen network. In contrast to existing methods for self-organizing environment representation, this discrete map is not represented in the world domain or in the configuration space of the vehicle, but in the sensor domain. The map is built by exploration. A conventional path planning technique now gives a path from current state to a desired state in the sensor domain, which can be followed using sensor based control. Collisions with obstacles are detected and used in the path planning. Results from a simulation show that the learned representation gives correct paths from an arbitrary starting point to an arbitrary end point.>

The paper describes a self-learning control system for a mobile robot. Based on sensor information the control system has to provide a steering signal in such a way that collisions are avoided. Since in our case no `examples' are available, the system learns on the basis of an external reinforcement signal which is negative in case of a collision and zero otherwise. Rules from Temporal Difference learning are used to find the correct mapping between the (discrete) sensor input space and the steering signal. We describe the algorithm for learning the correct mapping from the input (state) vector to the output (steering) signal, and the algorithm which is used for a discrete coding of the input state space. keywords: reinforcement learning, neural networks, state-space quantisation, mobile robot navigation. I. Introduction Control modules which map information from external sensors into motor signals are often used with mobile robots to accomplish a reflex-like collision avoidance beha...

A method is presented to learn 3D grasping of objects with unknown dimensions using a monocular eye-in-hand manipulator. From a sequence of images a motion profile is generated to approach the object of unknown size. It is shown that monocular visual information suffices to control the deceleration of the robot manipulator. A strategy for generating learning samples is presented, and simulation results demonstrate the effectiveness of the method. I. Introduction Sensor based robot control systems can overcome many of the difficulties which are caused by unknown or uncertain models of the environment. Also, conventional sensor based control systems require explicit knowledge of the kinematics and dynamics of the robot arm and a careful calibration of the sensor system. We are interested in self-learning and adaptive systems, where an implicit model of the arm and sensor system is learned from the behaviour of the robot. Neurocomputational techniques have been successfully applied in t...