This tutorial aims to equip researchers with the knowledge and skills to leverage multimodal data in human-robot interaction (HRI) studies. It covers the HRI study cycle, from sensor selection to data analysis, introducing commonly used sensors, pre-processing data, feature extraction techniques, fusion techniques and analysis techniques: both frequentist and Bayesian. Hands-on exercises using public datasets are designed to provide practical experience. The concluding panel discussion on ethics and bias in HRI is focused on fostering broader ethical considerations of HRI studies. Website tutorial is found online at https://sites.google.com/monash.edu/multimodal-hri-study-tutorial.

A predictive model for mobility systems capable of understanding the trajectory a user intends to follow in the environment is proposed. Understanding user intention is paramount for any shared-control navigation strategy between a user and an active robotic agent. Equally important however is being able to go beyond simple sample generation to assign probabilistic meaning to the set of possible future trajectories, so most likely scenarios can be assumed. The framework estimates a distribution over possible intentions, proposing a novel generative model predicated on Normalizing Flows which accounts for past behaviours, as traditionally reported in the literature, but also incorporates visual scene information. As the model permits trajectories to be assigned exact likelihoods, tractable density estimates can be readily exploited to finalize an executable intention. Baseline comparisons with the publicly available and widely used KITTI navigational dataset show significant improvements (up to 11.08%) with respect to traditional metrics such as Average and Final Displacement Errors. A novel metric that stands independent of the number of samples is also proposed as a more fitting comparison for future works.

In this paper, we present a novel approach that predicts spatially and temporally crowd behaviour for robotic social navigation. Integrating mobile robots into human society involves the fundamental problem of navigation in crowds. A robot should attempt to navigate in a way that is minimally invasive to the humans in its environment. However, planning in a dynamic environment is difficult as the environment must be predicted into the future. This problem has been thoroughly studied considering the behaviour of pedestrians at the level of individuals. Instead, we represent a pedestrian crowd by its macroscopic properties over space, such as density and velocity. With this spatial representation, we propose to learn a convolutional recurrent model to predict these properties into the future. The key design of a probabilistic loss function capturing the crowd's macroscopic properties empowers the spatio-temporal crowd prediction. Using a social invasiveness metric defined on these properties predicted by our convolutional recurrent model, we develop a framework that produces globally-optimal plans in expectation. Extensive results using a realistic pedestrian simulator show the validity and performance of the proposed social navigation approach.

A multi-modal framework to generate user intention distributions when operating a mobile vehicle is proposed in this work. The model learns from past observed trajectories and leverages traversability information derived from the visual surroundings to produce a set of future trajectories, suitable to be directly embedded into a perception-action shared control strategy on a mobile agent, or as a safety layer to supervise the prudent operation of the vehicle. We base our solution on a conditional Generative Adversarial Network with Long-Short Term Memory cells to capture trajectory distributions conditioned on past trajectories, further fused with traversability probabilities derived from visual segmentation with a Convolutional Neural Network. The proposed data-driven framework results in a significant reduction in error of the predicted trajectories (versus the ground truth) from comparable strategies in the literature (e.g. Social-GAN) that fail to account for information other than the agent’s past history. Experiments were conducted on a dataset collected with a custom wheelchair model built onto the open-source urban driving simulator CARLA, proving also that the proposed framework can be used with a small, unannotated dataset.

Advancements in technology propose a future where systems work collaboratively sharing the same workspace as humans. Navigation is one such crucial aspect of daily life where collaborative technologies can offer major assistance. Ageing population dictates a likely increase in personal mobility devices (PMDs), whilst autonomous cars are bringing intelligent vehicles to the road today. However, in such scenarios the expected assistance can only be given if the device is aware of its user's intention, so that controls can be applied in a tightly collaborative manner. Moreover, they should be robust to different environments, users and mobile platforms. A user driven navigation framework is proposed in this work to complement end-to-end sensing-only solutions to estimate controls as joint intention from vehicle states and user inputs. The solution is proven to be an improvement over similar strategies that rely on exteroceptive data and omit inputs from the driving agent. Furthermore, the developed framework is proven capable of transferring the learning into different environments and mobility platforms using a small amount of training data. Data from the autonomous driving community (Udacity dataset) and other obtained in-house with an instrumented power wheelchair are given to demonstrate the validity of the proposed approach.

This paper presents an algorithm for calibrating a “3D range sensor” constructed using a two-dimensional laser range finder (LRF), that is rotated about an axis using a motor to obtain a three-dimensional point cloud. The sensor assembly is modelled as a two degree of freedom open kinematic chain, with one joint corresponding to the axis of the internal mirror in the LRF and the other joint set along the axis of the motor used to rotate the body of the LRF. In the application described in this paper, the sensor unit is mounted on a robot arm used for infrastructure inspection. The objective of the calibration process is to obtain the coordinate transform required to compute the locations of the 3D points with respect to the robot coordinate frame. Proposed strategy uses observations of a set of markers arbitrarily placed in the environment. Distances between these markers are measured and a metric multidimensional scaling is used to obtain the coordinates of the markers with respect to a local coordinate frame. Intensity associated with each beam point of a laser scan is used to locate the reflective markers in the 3D point cloud and a least squares problem is formulated to compute the relationship between the robot coordinate frame, LRF coordinate frame and the marker coordinate frame. Results from experiments using the robot, LRF combination to map a cavity inside a steel bridge structure are presented to demonstrate the effectiveness of the calibration process.