Shepherding is a skill which can take sheepdogs anywhere from three months to three years to learn, and can cost shepherds tens of thousands of pounds. This study explores the potential of robot sheepdogs, which could be as low as £1600. Using expert observations from farm managers the study assesses the efficacy of BostonDynamics Spot at shepherding. Observations from the three field trials showed potential for the robot in maintaining the flock whilst in motion, dynamically adjusting the flock's flight distance with indirect eye contact, and eliciting pressure in collaboration with a sheepdog, to move the sheep into their pens.

Deploying robots in human-shared environments requires a deep understanding of how nearby agents and objects interact. Employing causal inference to model cause-and-effect relationships facilitates the prediction of human behaviours and enables the anticipation of robot interventions. However, a significant challenge arises due to the absence of implementation of existing causal discovery methods within the ROS ecosystem, the standard de-facto framework in robotics, hindering effective utilisation on real robots. To bridge this gap, in our previous work we proposed ROS-Causal, a ROS-based framework designed for onboard data collection and causal discovery in human-robot spatial interactions. In this work, we present an experimental evaluation of ROS-Causal both in simulation and on a new dataset of human-robot spatial interactions in a lab scenario, to assess its performance and effectiveness. Our analysis demonstrates the efficacy of this approach, showcasing how causal models can be extracted directly onboard by robots during data collection. The online causal models generated from the simulation are consistent with those from lab experiments. These findings can help researchers to enhance the performance of robotic systems in shared environments, firstly by studying the causal relations between variables in simulation without real people, and then facilitating the actual robot deployment in real human environments. ROS-Causal: https://lcastri.github.io/roscausal

Reasoning on the context of human beings is crucial for many real-world applications especially for those deploying autonomous systems (e.g. robots). In this paper, we present a new approach for context reasoning to further advance the field of human motion prediction. We therefore propose a neuro-symbolic approach for human motion prediction (NeuroSyM), which weights differently the interactions in the neighbourhood by leveraging an intuitive technique for spatial representation called Qualitative Trajectory Calculus (QTC). The proposed approach is experimentally tested on medium and long term time horizons using two architectures from the state of art, one of which is a baseline for human motion prediction and the other is a baseline for generic multivariate time-series prediction. Six datasets of challenging crowded scenarios, collected from both fixed and mobile cameras, were used for testing. Experimental results show that the NeuroSyM approach outperforms in most cases the baseline architectures in terms of prediction accuracy.

Deploying service robots in our daily life, whether in restaurants, warehouses or hospitals, calls for the need to reason on the interactions happening in dense and dynamic scenes. In this paper, we present and benchmark three new approaches to model and predict multi-agent interactions in dense scenes, including the use of an intuitive qualitative representation. The proposed solutions take into account static and dynamic context to predict individual interactions. They exploit an input- and a temporal-attention mechanism, and are tested on medium and long-term time horizons. The first two approaches integrate different relations from the so-called Qualitative Trajectory Calculus (QTC) within a stateof-the-art deep neural network to create a symbol-driven neural architecture for predicting spatial interactions. The third approach implements a purely data-driven network for motion prediction, the output of which is post-processed to predict QTC spatial interactions. Experimental results on a popular robot dataset of challenging crowded scenarios show that the purely data-driven prediction approach generally outperforms the other two. The three approaches were further evaluated on a different but related human scenarios to assess their generalisation capability.

Recent developments in agriculture have high-lighted the potential of as well as the need for the use of robotics. Various processes in this field can benefit from the proper use of state of the art technology [1], in terms of efficiency as well as quality. One of these areas is the harvesting of ripe fruit.In order to be able to automate this process, a robotic harvester needs to be aware of the full poses of the crop/fruit to be collected in order to perform proper path- and collision-planning. The current state of the art mainly considers problems of detection and segmentation of fruit with localisation limited to the 3D position only. The reliable and real-time estimation of the respective orientations remains a mostly unaddressed problem.In this paper, we present a compact and efficient network architecture for estimating the orientation of soft fruit such as strawberries from colour and, optionally, depth images. The proposed system can be automatically trained in a realistic simulation environment. We evaluate the system’s performance on simulated datasets and validate its operation on publicly available images of strawberries to demonstrate its practical use. Depending on the amount of training data used, coverage of state space, as well as the availability of RGB-D or RGB data only, mean errors of as low as 11° could be achieved.

We develop a taxonomy that categorizes HRI failure types and their impact on trust to structure the broad range of knowledge contributions. We further identify research gaps in order to support fellow researchers in the development of trustworthy robots. Studying trust repair in HRI has only recently been given more interest and we propose a taxonomy of potential trust violations and suitable repair strategies to support researchers during the development of interaction scenarios. The taxonomy distinguishes four failure types: Design, System, Expectation, and User failures and outlines potential mitigation strategies. Based on these failures, strategies for autonomous failure detection and repair are presented, employing explanation, verification and validation techniques. Finally, a research agenda for HRI is outlined, discussing identified gaps related to the relation of failures and HR-trust.

Progress in autonomous mobile robotics has seen significant advances in the development of many algorithms for motion control and path planning. However, robust performance from these algorithms can often only be expected if the parameters controlling them are tuned specifically for the respective robot model, and optimised for specific scenarios in the environment the robot is working in. Such parameter tuning can, depending on the underlying algorithm, amount to a substantial combinatorial challenge, often rendering extensive manual tuning of these parameters intractable. In this paper, we present a framework that permits the use of different navigation actions and/or parameters depending on the spatial context of the navigation task. We consider the respective navigation algorithms themselves mostly as a "black box", and find suitable parameters by means of an iterative optimisation, improving for performance metrics in simulated environments. We present a genetic algorithm incorporated into the framework, and empirically show that the resulting parameter sets lead to substantial performance improvements in both simulated and real-world environments in the domain of agricultural robots.

In this paper, enhancement to the novel grasp planning algorithm based on gripper workspace spheres is presented. Our development requires a registered point cloud of the target from different views, assuming no prior knowledge of the object, nor any of its properties. This work features a new set of metrics for grasp pose candidates evaluation, as well as exploring the impact of high object sampling on grasp success rates. In addition to gripper position sampling, we now perform orientation sampling about the x, y, and z-axes, hence the grasping algorithm no longer require object orientation estimation. Successful experiments have been conducted on a simple jaw gripper (Franka Panda gripper) as well as a complex, high Degree of Freedom (DoF) hand (Allegro hand) as a proof of its versatility. Higher grasp success rates of 76% and 85.5% respectively has been reported by real world experiments.

Global navigation satellite system (GNSS) has been considered as a panacea for positioning and tracking since the last decade. However, it suffers from severe limitations in terms of accuracy, particularly in highly cluttered and indoor environments. Though real-time kinematics (RTK) supported GNSS promises extremely accurate localisation, employing such services are expensive, fail in occluded environments and are unavailable in areas where cellular base stations are not accessible. It is, therefore, necessary that the GNSS data is to be filtered if high accuracy is required. Thus, this article presents a GNSS-based particle filter that exploits the spatial constraints imposed by the environment. In the proposed setup, the state prediction of the sample set follows a restricted motion according to the topological map of the environment. This results in the transition of the samples getting confined between specific discrete points, called the topological nodes, defined by a topological map. This is followed by a refinement stage where the full set of predicted samples goes through weighting and resampling, where the weight is proportional to the predicted particle's proximity with the GNSS measurement. Thus, a discrete space continuous-time Bayesian filter is proposed, called the Topological Particle Filter (TPF).The proposed TPF is put to test by localising and tracking fruit pickers inside polytunnels. Fruit pickers inside polytunnels can only follow specific paths according to the topology of the tunnel. These paths are defined in the topological map of the polytunnels and are fed to TPF to tracks fruit pickers. Extensive datasets are collected to demonstrate the improved discrete tracking of strawberry pickers inside polytunnels thanks to the exploitation of the environmental constraints.

Robotic technology is increasingly considered the major mean for fruit picking. However, picking fruits in a dense cluster imposes a challenging research question in terms of motion/path planning as conventional planning approaches may not find collision-free movements for the robot to reach-and-pick a ripe fruit within a dense cluster. In such cases, the robot needs to safely push unripe fruits to reach a ripe one. Nonetheless, existing approaches to planning pushing movements in cluttered environments either are computationally expensive or only deal with 2-D cases and are not suitable for fruit picking, where it needs to compute 3-D pushing movements in a short time. In this work, we present a path planning algorithm for pushing occluding fruits to reach-and-pick a ripe one. Our proposed approach, called Interactive Probabilistic Movement Primitives (I-ProMP), is not computationally expensive (its computation time is in the order of 100 milliseconds) and is readily used for 3-D problems. We demonstrate the efficiency of our approach with pushing unripe strawberries in a simulated polytunnel. Our experimental results confirm I-ProMP successfully pushes table top grown strawberries and reaches a ripe one.

In this paper, we present a novel grasp planning algorithm for unknown objects given a registered point cloud of the target from different views. The proposed methodology requires no prior knowledge of the object, nor offline learning. In our approach, the gripper kinematic model is used to generate a point cloud of each finger workspace, which is then filled with spheres. At run-time, first the object is segmented, its major axis is computed, in a plane perpendicular to which, the main grasping action is constrained. The object is then uniformly sampled and scanned for various gripper poses that assure at least one object point is located in the workspace of each finger. In addition, collision checks with the object or the table are performed using computationally inexpensive gripper shape approximation. Our methodology is both time efficient (consumes less than 1.5 seconds in average) and versatile. Successful experiments have been conducted on a simple jaw gripper (Franka Panda gripper) as well as a complex, high Degree of Freedom (DoF) hand (Allegro hand).

The long-term deployment of autonomous robots co-located with humans in real-world scenarios remains a challenging problem. In this paper, we present the “Lindsey” tour guide robot system in which we attempt to increase the social capability of current state-of-the-art robotic technologies. The robot is currently deployed at a museum displaying local archaeology where it is providing guided tours and information to visitors. The robot is operating autonomously daily, navigating around the museum and engaging with the public, with on-site assistance from roboticists only in cases of hardware/software malfunctions. In a deployment lasting seven months up to now, it has travelled nearly 300km and has delivered more than 2300 guided tours. First, we describe the robot framework and the management interfaces implemented. We then analyse the data collected up to now with the goal of understanding and modelling the visitors' behavior in terms of their engagement with the technology. These data suggest that while short-term engagement is readily gained, continued engagement with the robot tour guide is likely to require more refined and robust socially interactive behaviours. The deployed system presents us with an opportunity to empirically address these issues.

This paper presents a novel 3DOF pedestrian trajectory prediction approach for autonomous mobile service robots. While most previously reported methods are based on learning of 2D positions in monocular camera images, our approach uses range-finder sensors to learn and predict 3DOF pose trajectories (i.e. 2D position plus 1D rotation within the world coordinate system). Our approach, T-Pose-LSTM (Temporal 3DOF-Pose Long-Short-Term Memory), is trained using long-term data from real-world robot deployments and aims to learn context-dependent (environment- and time-specific) human activities. Our approach incorporates long-term temporal information (i.e. date and time) with short-term pose observations as input. A sequence-to-sequence LSTM encoder-decoder is trained, which encodes observations into LSTM and then decodes the resulting predictions. On deployment, the approach can perform on-the-fly prediction in real-time. Instead of using manually annotated data, we rely on a robust human detection, tracking and SLAM system, providing us with examples in a global coordinate system. We validate the approach using more than 15 km of pedestrian trajectories recorded in a care home environment over a period of three months. The experiments show that the proposed T-Pose-LSTM model outperforms the state-of-the-art 2D-based method for human trajectory prediction in long-term mobile robot deployments.

Adapting to users' intentions is a key requirement for autonomous robots in general, and in care settings in particular. In this paper, a comprehensive long-term study of a mobile robot providing information services to residents, visitors, and staff of a care home is presented, with a focus on adapting to the when and where the robot should be offering its services to best accommodate the users' needs. Rather than providing a fixed schedule, the presented system takes the opportunity of long-term deployment to explore the space of possibilities of interaction while concurrently exploiting the model learned to provide better services. But in order to provide effective services to users in a care home, not only then when and where are relevant, but also the way how the information is provided and accessed. Hence, also the usability of the deployed system is studied specifically, in order to provide a most comprehensive overall assessment of a robotic info-terminal implementation in a care setting. Our results back our hypotheses, (i) that learning a spatio-temporal model of users' intentions improves efficiency and usefulness of the system, and (ii) that the specific information sought after is indeed dependent on the location the info-terminal is offered.

We present a lifelong mapping and localisation system for long-term autonomous operation of mobile robots in changing environments. The core of the system is a spatio-temporal occupancy grid that explicitly represents the persistence and periodicity of the individual cells and can predict the probability of their occupancy in the future. During navigation, our robot builds temporally local maps and integrates then into the global spatio-temporal grid. Through re-observation of the same locations, the spatio-temporal grid learns the long-term environment dynamics and gains the ability to predict the future environment states. This predictive ability allows to generate time-specific 2d maps used by the robot's localisation and planning modules. By analysing data from a long-term deployment of the robot in a human-populated environment, we show that the proposed representation improves localisation accuracy and the efficiency of path planning. We also show how to integrate the method into the ROS navigation stack for use by other roboticists.

Even though research on autonomous robots and human-robot interaction accomplished great progress in recent years, and reusable soft- and hardware components are available, many of the reported findings are only hardly reproducible by fellow scientists. Usually, reproducibility is impeded because required information, such as the specification of software versions and their configuration, required data sets, and experiment protocols are not mentioned or referenced in most publications. In order to address these issues, we recently introduced an integrated tool chain and its underlying development process to facilitate reproducibility in robotics. In this contribution we instantiate the complete tool chain in a unique user study in order to assess its applicability and usability. To this end, we chose three different robotic systems from independent institutions and modeled them in our tool chain, including three exemplary experiments. Subsequently, we asked twelve researchers to reproduce one of the formerly unknown systems and the associated experiment. We show that all twelve scientists were able to replicate a formerly unknown robotics experiment using our tool chain.

In this work, we propose the combination of a state-of-the-art sampling-based local planner with so-called Velocity Costmaps to achieve human-aware robot navigation. Instead of introducing humans as “special obstacles” into the representation of the environment, we restrict the sample space of a “Dynamic Window Approach” local planner to only allow trajectories based on a qualitative description of the future unfolding of the encounter. To achieve this, we use a Bayesian temporal model based on a Qualitative Trajectory Calculus to represent the mutual navigation intent of human and robot, and translate these descriptors into sample space constraints for trajectory generation. We show how to learn these models from demonstration and evaluate our approach against standard Gaussian cost models in simulation and in real-world using a non-holonomic mobile robot. Our experiments show that our approach exceeds the performance and safety of the Gaussian models in pass-by and path crossing situations.

This article presents an integrated robot system capable of interactive learning in dialogue with a human. Such a system needs to have several competencies and must be able to process different types of representations. In this article, we describe a collection of mechanisms that enable integration of heterogeneous competencies in a principled way. Central to our design is the creation of beliefs from visual and linguistic information, and the use of these beliefs for planning system behaviour to satisfy internal drives. The system is able to detect gaps in its knowledge and to plan and execute actions that provide information needed to fill these gaps. We propose a hierarchy of mechanisms which are capable of engaging in different kinds of learning interactions, e.g. those initiated by a tutor or by the system itself. We present the theory these mechanisms are build upon and an instantiation of this theory in the form of an integrated robot system. We demonstrate the operation of the system in the case of learning conceptual models of objects and their visual properties.

In planning for deliberation or navigation in real-world robotic systems, one of the big challenges is to cope with change. It lies in the nature of planning that it has to make assumptions about the future state of the world, and the robot's chances of successively accomplishing actions in this future. Hence, a robot's plan can only be as good as its predictions about the world. In this paper, we present a novel approach to specifically represent changes that stem from periodic events in the environment (e.g. a door being opened or closed), which impact on the success probability of planned actions. We show that our approach to model the probability of action success as a set of superimposed periodic processes allows the robot to predict action outcomes in a long-term data obtained in two real-life offices better than a static model. We furthermore discuss and showcase how this knowledge gathered can be successfully employed in a probabilistic planning framework to devise better navigation plans. The key contributions of this paper are (i) the formation of the spectral model of action outcomes from non-uniform sampling, the (ii) analysis of its predictive power using two long-term datasets, and (iii) the application of the predicted outcomes in an MDP-based planning framework.

We present a pilot study to identify hesitation signals in Human-Robot Spatial Interaction which we aim to employ to evaluate the quality of the robots executed behaviour. The presented study focuses on head-on encounters between a human and a robot in pass-by scenarios. Our results indicate that these hesitation signals can be found and therefore present a form a implicit feedback.

This paper presents a new approach for topological localisation of service robots in dynamic indoor environments. In contrast to typical localisation approaches that rely mainly on static parts of the environment, our approach makes explicit use of information about changes by learning and modelling the spatio-temporal dynamics of the environment where the robot is acting. The proposed spatio-temporal world model is able to predict environmental changes in time, allowing the robot to improve its localisation capabilities during long-term operations in populated environments. To investigate the proposed approach, we have enabled a mobile robot to autonomously patrol a populated environment over a period of one week while building the proposed model representation. We demonstrate that the experience learned during one week is applicable for topological localization even after a hiatus of three months by showing that the localization error rate is significantly lower compared to static environment representations.

In this paper we propose to augment a wellestablished Qualitative Trajectory Calculus (QTC) by incorporating social distances into the model to facilitate a richer and more powerful representation of Human-Robot Spatial Interaction (HRSI). By combining two variants of QTC that implement different resolutions and switching between them based on distance thresholds we show that we are able to both reduce the complexity of the representation and at the same time enrich QTC with one of the core HRSI concepts: proxemics. Building on this novel integrated QTC model, we propose to represent the joint spatial behaviour of a human and a robot employing a probabilistic representation based on Hidden Markov Models. We show the appropriateness of our approach by encoding different HRSI behaviours observed in a human-robot interaction study and show how the models can be used to represent and classify these behaviours using social distance-augmented QTC.

Facial communicative signals (FCSs) such as head gestures, eye gaze, and facial expressions can provide useful feedback in conversations between people and also in human-robot interaction. This paper presents a pattern recognition approach for the interpretation of FCSs in terms of valence, based on the selection of discriminative subsequences in video data. These subsequences capture important temporal dynamics and are used as prototypical reference subsequences in a classification procedure based on dynamic time warping and feature extraction with active appearance models. Using this valence classification, the robot can discriminate positive from negative interaction situations and react accordingly. The approach is evaluated on a database containing videos of people interacting with a robot by teaching the names of several objects to it. The verbal answer of the robot is expected to elicit the display of spontaneous FCSs by the human tutor, which were classified in this work. The achieved classification accuracies are comparable to the average human recognition performance and outperformed our previous results on this task.

The analysis and understanding of human-robot joint spatial behaviour (JSB) - such as guiding, approaching, departing, or coordinating movements in narrow spaces - and its communicative and dynamic aspects are key requirements on the road towards more intuitive interaction, safe encounter, and appealing living with mobile robots. This endeavours demand for appropriate models and methodologies to represent JSB and facilitate its analysis. In this paper, we adopt a qualitative trajectory calculus (QTC) as a formal foundation for the analysis and representation of such spatial behaviour of a human and a robot based on a compact encoding of the relative trajectories of two interacting agents in a sequential model. We present this QTC together with a distance measure and a probabilistic behaviour model and outline its usage in an actual JSB study. We argue that the proposed QTC coding scheme and derived methodologies for analysis and modelling are flexible and extensible to be adapted for a variety of other scenarios and studies.

This paper gives a brief overview of the challenges for multi-model perception and generation applied to robot companions located in human social environments. It reviews the current position in both perception and generation and the immediate technical challenges and goes on to consider the extra issues raised by embodiment and social context. Finally, it briefly discusses the impact of systems that must function continually over months rather than just for a few hours.

In this paper we present an account of the problems faced by a mobile robot given an incomplete tour of an unknown environment, and introduce a collection of techniques which can generate successful behaviour even in the presence of such problems. Underlying our approach is the principle that an autonomous system must be motivated to act to gather new knowledge, and to validate and correct existing knowledge. This principle is embodied in Dora, a mobile robot which features the aforementioned techniques: shared representations, non-monotonic reasoning, and goal generation and management. To demonstrate how well this collection of techniques work in real-world situations we present a comprehensive analysis of the Dora system's performance over multiple tours in an in door environment. In this analysis Dora successfully completed 18 of 21 attempted runs, with all but 3 of these successes requiring one or more of the integrated techniques to recover from problems.

Robots must perform tasks efficiently and reliably while acting under uncertainty. One way to achieve efficiency is to give the robot commonsense knowledge about the structure of the world. Reliable robot behaviour can be achieved by modelling the uncertainty in the world probabilistically. We present a robot system that combines these two approaches and demonstrate the improvements in efficiency and reliability that result. Our first contribution is a probabilistic relational model integrating common-sense knowledge about the world in general, with observations of a particular environment. Our second contribution is a continual planning system which is able to plan in the large problems posed by that model, by automatically switching between decision-theoretic and classical procedures. We evaluate our system on object search tasks in two different real-world indoor environments. By reasoning about the trade-offs between possible courses of action with different informational effects, and exploiting the cues and general structures of those environments, our robot is able to consistently demonstrate efficient and reliable goal-directed behaviour. 1

In this paper we demonstrate the online applicability of the fault detection and diagnosis approach which we previously developed and published in [1]. In our former work we showed that a purely data driven fault detection approach can be successfully built based on monitored inter-component communication data of a robotic system and used for a-posteriori fault detection. Here we propose an extension to this approach which is capable of online learning of the fault model as well as for online fault detection. We evaluate the application of our approach in the context of a RoboCup task executed by our service robot BIRON in corporation with an expert user.

In this paper we present representations and mechanisms that facilitate continuous learning of visual concepts in dialogue with a tutor and show the implemented robot system. We present how beliefs about the world are created by processing visual and linguistic information and show how they are used for planning system behaviour with the aim at satisfying its internal drive - to extend its knowledge. The system facilitates different kinds of learning initiated by the human tutor or by the system itself. We demonstrate these principles in the case of learning about object colours and basic shapes.

In this paper, we present a calibration-free head gesture recognition system using a motion-sensor-based approach. For data acquisition we conducted a comprehensive study with 10 subjects. We analyzed the resulting head movement data with regard to separability and transferability to new subjects. Ordered means models (OMMs) were used for classification, since they provide an easy-to-use, fast, and stable approach to machine learning of time series. In result, we achieved classification rates of 85-95% for nodding, head shaking and tilting head gestures and good transferability. Finally, we show first promising attempts towards online recognition.

In order to address the problem of failure detection in the robotics domain, we present in this contribution a so-called self-awareness model, based on the system's internal data exchange and the inherent dynamics of inter-component communication. The model is strongly data driven and provides an anomaly detector for robotics systems both applicable in-situ at runtime as well as a-posteriori in post-mortem analysis. Current architectures or methods for failure detection in autonomous robots are either implementations of watch dog concepts or are based on excessive amounts of domain-specific error detection code. The approach presented in this contribution provides an avenue for the detection of more subtle anomalies originating from external sources such as the environment itself or system failures such as resource starvation. Additionally, developers are alleviated from explicitly modeling and foreseeing every exceptional situation, instead training the presented probabilistic model with the known normal modes within the specification of the robot system. As we developed and evaluated the self-awareness model on a mobile robot platform featuring an event-driven software architecture, the presented method can easily be applied in other current robotics software architectures.

Mobile robots that are employed in people's homes need to safely navigate their environment. And natural human-inhabited environments still pose significant challenges for robots despite the impressive progress that has been achieved in the field of path planning and obstacle avoidance. These challenges mostly arise from the fact that (i) the perceptual abilities of a robot are limited, thus sometimes impeding its ability to see relevant obstacles (e.g. transparent objects), and (ii) the environment is highly dynamic being populated by humans. In this contribution we are making a case for an integrated solution to these challenges that builds upon the analysis and use of implicit human knowledge in path planning and a cascade of replanning approaches. We combine state of the art path planning and obstacle avoidance algorithms with the knowledge about how humans navigate in their very own environment. The approach results in a more robust and predictable navigation ability for domestic robots as is demonstrated in a number of experimental runs.

Recent developments in robotics enable advanced human-robot interaction. Especially interactions of novice users with robots are often unpredictable and, therefore, demand for novel methods for the analysis of the interaction in systemic ways. We propose Systemic Interaction Analysis (SInA) as a method to jointly analyze system level and interaction level in an integrated manner using one tool. The approach allows us to trace back patterns that deviate from prototypical interaction sequences to the distinct system components of our autonomous robot. In this paper, we exemplarily apply the method to the analysis of the follow behavior of our domestic robot BIRON. The analysis is the basis to achieve our goal of improving human-robot interaction iteratively.

We present an Augmented Reality (AR) system to support collaborative tasks in a shared real-world interaction space by facilitating joint attention. The users are assisted by information about their interaction partner's field of view both visually and acoustically. In our study, the audiovisual improvements are compared with an AR system without these support mechanisms in terms of the participants' reaction times and error rates. The participants performed a simple object-choice task we call the "gaze game" to ensure controlled experimental conditions. Additionally, we asked the subjects to fill in a questionnaire to gain subjective feedback from them. We were able to show an improvement for both dependent variables as well as positive feedback for the visual augmentation in the questionnaire.

In scenarios that require a close collaboration and knowledge transfer between inexperienced users and robots, the ldquolearning by interactingrdquo paradigm goes hand in hand with appropriate representations and learning methods. In this paper we discuss a mixed initiative strategy for robotic learning by interacting with a user in a joint map acquisition process. We propose the integration of an environment representation approach into our interactive learning framework. The environment representation and mapping system supports both user driven and data driven strategies for the acquisition of spatial information, so that a mixed initiative strategy for the learning process is realised. We evaluate our system with test runs according to the scenario of a guided tour, extending the area of operation from structured laboratory environment to less predictable domestic settings.

Navigation and obstacle avoidance in robotics using planar laser scans has matured over the last decades. They basically enable robots to penetrate highly dynamic and populated spaces, such as people's home, and move around smoothly. However, in an unconstrained environment the two-dimensional perceptual space of a fixed mounted laser is not sufficient to ensure safe navigation. In this paper, we present an approach that pools a fast and reliable motion generation approach with modern 3D capturing techniques using a time-of-flight camera. Instead of attempting to implement full 3D motion control, which is computationally more expensive and simply not needed for the targeted scenario of a domestic robot, we introduce a ldquovirtual laserrdquo. For the originally solely laser-based motion generation the technique of fusing real laser measurements and 3D point clouds into a continuous data stream is 100% compatible and transparent. The paper covers the general concept, the necessary extrinsic calibration of two very different types of sensors, and exemplarily illustrates the benefit which is to avoid obstacles not being perceivable in the original laser scan.

In everyday conversation besides speech people also communicate by means of nonverbal cues. Facial expressions are one important cue, as they can provide useful information about the conversation, for instance, whether the interlocutor seems to understand or appears to be puzzled. Similarly, in human-robot interaction facial expressions also give feedback about the interaction situation. We present a Wizard of Oz user study in an object-teaching scenario where subjects showed several objects to a robot and taught the objects' names. Afterward, the robot should term the objects correctly. In a first evaluation, we let other people watch short video sequences of this study. They decided by looking at the face of the human whether the answer of the robot was correct (unproblematic situation) or incorrect (problematic situation). We conducted the experiments under specific conditions by varying the amount of temporal and visual context information and compare the results with related experiments described in the literature.

This paper presents a new method for detecting and merging redundant points in registered range data. Given a global representation from sequences of 3D points, the points are projected onto a virtual image plane computed from the intrinsic parameters of the sensor. Candidates for redundancy are collected per pixel which then are clustered locally via region growing and replaced by the clusterpsilas mean value. As data is provided in a certain manner defined by camera characteristics, this processing step preserves the structural information of the data. For evaluation, our approach is compared to two other algorithms. Applied to two different sequences, it is shown that the presented method gives smooth results within planar regions of the point clouds by successfully reducing noise and redundancy and thus improves registered range data.

In order to provide personalized services and to develop human-like interaction capabilities robots need to recognize their human partner. Face recognition has been studied in the past decade exhaustively in the context of security systems and with significant progress on huge datasets. However, these capabilities are not in focus when it comes to social interaction situations. Humans are able to remember people seen for a short moment in time and apply this knowledge directly in their engagement in conversation. In order to equip a robot with capabilities to recall human interlocutors and to provide user- aware services, we adopt human-human interaction schemes to propose a face memory on the basis of active appearance models integrated with the active memory architecture. This paper presents the concept of the interactive face memory, the applied recognition algorithms, and their embedding into the robot's system architecture. Performance measures are discussed for general face databases as well as scenario-specific datasets.

Despite increasing efforts in the field of social robotics and interactive systems integrated and fully autonomous robots which are capable of learning from interaction with inexperienced and non-expert users are still a rarity. However, in order to tackle the challenge of learning by interaction robots need to be equipped with a set of basic behaviors and abilities which have to be coupled and combined in a flexible manner. This paper presents how a recently proposed information-driven integration concept termed ldquoactive memoryrdquo is adopted to realize learning-enabling behaviors for a domestic robot. These behaviors enable it to (i) learn about its environment, (ii) interact with several humans simultaneously, and (iii) couple learning and interaction tightly. The basic interaction strategies on the basis of information exchange through the active memory are presented. A brief discussion of results obtained from live user trials with inexperienced users in a home tour scenario underpin the relevance and appropriateness of the described concepts.

Human-robot interaction (HRI) research is here presented into social robots that have to be able to interact with inexperienced users. In the design of these robots many research findings of human-human interaction and human-computer interaction are adopted but the direct applicability of these theories is limited because a robot is different from both humans and computers. Therefore, new methods have to be developed in HRI in order to build robots that are suitable for inexperienced users. In this paper we present a video study we conducted employing our robot BIRON (Bielefeld robot companion) which is designed for use in domestic environments. Subjects watched the system during the interaction with a human and rated two different robot behaviours (extrovert and introvert). The behaviours differed regarding verbal output and person following of the robot. Aiming to improve human-robot interaction, participantspsila ratings of the behaviours were evaluated and compared.

An important goal for research on service robots is the cooperation of a human and a robot as team. A service robot in a domestic environment needs to build a representation of its future workspace that corresponds to the human user's understanding of these surroundings. But it also needs to apply this model about the "where" and "what" in its current interaction to allow communication about objects and places in a human-adequate way. In this paper we present the integration of a hierarchical robotic mapping system into an interactive framework controlled by a dialog system. The goal is to use interactively acquired environment models to implement a robot with interaction aware behaviors. A major contribution of this work is a three-level hierarchy of spatial representation affecting three different communication dimensions. This hierarchy is consequently applied in the design of the grounding-based dialog, laser-based topological mapping, and an objects attention system. We demonstrate the benefits of this integration for learning and tour guiding in a human- comprehensible interaction between a robot and its user in a home-tour scenario. The enhanced interaction capabilities are crucial for developing a new generation of robots that will be accepted not only as service robots but also as robot companions.

A very important aspect in developing robots capable of human-robot interaction (HRI) is the research in natural, human-like communication, and subsequently, the development of a research platform with multiple HRI capabilities for evaluation. Besides a flexible dialog system and speech understanding, an anthropomorphic appearance has the potential to support intuitive usage and understanding of a robot, e.g., human-like facial expressions and deictic gestures can as well be produced and also understood by the robot. As a consequence of our effort in creating an anthropomorphic appearance and to come close to a human- human interaction model for a robot, we decided to use human-like sensors, i.e., two cameras and two microphones only, in analogy to human perceptual capabilities too. Despite the challenges resulting from these limits with respect to perception, a robust attention system for tracking and interacting with multiple persons simultaneously in real time is presented. The tracking approach is sufficiently generic to work on robots with varying hardware, as long as stereo audio data and images of a video camera are available. To easily implement different interaction capabilities like deictic gestures, natural adaptive dialogs, and emotion awareness on the robot, we apply a modular integration approach utilizing XML-based data exchange. The paper focuses on our efforts to bring together different interaction concepts and perception capabilities integrated on a humanoid robot to achieve comprehending human-oriented interaction.

With the increasing interest in computer vision for interactive systems, the challenges of the development process involving many researchers are becoming more prominent. Issues like reuse of algorithms, modularity, and distributed processing are getting more important in the endeavor of building complex vision systems. We present a framework that allows independent development of enclosed components and supports interactive optimization of algorithmic parameters in an online fashion. The communication between components is performed nearly without any slow down compared to a monolithic system. Through the modular concept, all components can be flexibly distributed and reused in other application domains. The suitability of the approach is demonstrated with an example system.

In this paper, we present a case study that exemplifies general ideas of system integration and coordination. The application field of assistant technology provides an ideal test bed for complex computer vision systems including real-time components, human-computer interaction, dynamic 3-d environments, and information retrieval aspects. In our scenario the user is wearing an augmented reality device that supports her/him in everyday tasks by presenting information that is triggered by perceptual and contextual cues. The system integrates a wide variety of visual functions like localization, object tracking and recognition, action recognition, interactive object learning, etc. We show how different kinds of system behavior are realized using the Active Memory Infrastructure that provides the technical basis for distributed computation and a data- and eventdriven integration approach.

As wearable sensors and computing hardware are becoming a reality, new and unorthodox approaches to seamless human-computer interaction can be explored. This paper presents the prototype of a wearable, head-mounted device for advanced human-machine interaction that integrates speech recognition and computer vision with head gesture analysis based on inertial sensor data. We will focus on the innovative idea of integrating visual and inertial data processing for interaction. Fusing head gestures with results from visual analysis of the environment provides rich vocabularies for human-machine communication because it renders the environment into an interface: if objects or items in the surroundings are being associated with system activities, head gestures can trigger commands if the corresponding object is being looked at. We will explain the algorithmic approaches applied in our prototype and present experiments that highlight its potential for assistive technology. Apart from pointing out a new direction for seamless interaction in general, our approach provides a new and easy to use interface for disabled and paralyzed users in particular.