For a robot to perform personalised human support in a home, it is essential to uniquely identify the people it is working for. Each individual has their own attributes, knowledge of the environment, and experiences which the robot should learn and keep in its memory. One of the current challenges is for a robot to identify and re-identify an individual while dealing with visual occlusion, changes in appearance, and identity switching. This paper presents a novel approach to address these challenges in robot vision by extending current state-of-the-art solutions to enhance person memorisation by taking multiple viewpoints to extract features for matching and maintaining this information in a world model of the robot.

Human Activity Recognition is an important task in many human-computer collaborative scenarios, with various practical applications. Although uni-modal approaches have been extensively studied, they suffer from data quality issues and require modality-specific feature engineering, making them neither robust nor effective enough for real-world deployment. By utilizing various sensors, Multi-modal Human Activity Recognition can leverage complementary information to build models that generalize well. While deep learning methods have shown promising results, their potential in extracting salient multi-modal spatial-temporal features and better fusing complementary information has not been fully explored. Additionally, reducing the complexity of the multi-modal approach for edge deployment is another unresolved issue. To address these issues, a Knowledge Distillation-based Multi-modal Mid-Fusion approach, DMFT, is proposed to facilitate informative feature extraction and fusion for efficiently solving the Multi-modal Human Activity Recognition task. DMFT first encodes the multi-modal input data into a unified representation. The DMFT teacher model then applies an attentive multi-modal spatial-temporal transformer module that extracts the salient spatial-temporal features. A temporal mid-fusion module is also proposed to further fuse the temporal features. Subsequently, the Knowledge Distillation method is applied to transfer the learned representation from the teacher model to a simpler DMFT student model, which consists of a lite version of the multi-modal spatial-temporal transformer module, to produce the results. Evaluation of DMFT was conducted on two public multi-modal human activity recognition datasets alongside various state-of-the-art approaches. The experimental results demonstrate that the model achieves competitive performance in terms of effectiveness, scalability, and robustness.

Trajectory prediction is fundamental to various intelligent technologies, such as autonomous driving and robotics. The motion prediction of pedestrians and vehicles helps emergency braking, reduces collisions, and improves traffic safety. Current trajectory prediction research faces problems of complex social interactions, high dynamics and multi-modality. Especially, it still has limitations in long-time prediction. We propose Attention-aware Social Graph Transformer Networks for multi-modal trajectory prediction. We combine Graph Convolutional Networks and Transformer Networks by generating stable resolution pseudo-images from Spatio-temporal graphs through a designed stacking and interception method. Furthermore, we design the attention-aware module to handle social interaction information in scenarios involving mixed pedestrian-vehicle traffic. Thus, we maintain the advantages of the Graph and Transformer, i.e., the ability to aggregate information over an arbitrary number of neighbors and the ability to perform complex time-dependent data processing. We conduct experiments on datasets involving pedestrian, vehicle, and mixed trajectories, respectively. Our results demonstrate that our model minimizes displacement errors across various metrics and significantly reduces the likelihood of collisions. It is worth noting that our model effectively reduces the final displacement error, illustrating the ability of our model to predict for a long time.

Teaching robots can be challenging, particularly for novice human users who struggle to understand the robot’s learning process. Current research in interactive robot learning lacks effective methods for assessing a user’s interpretation of the robot’s learning state, which makes it difficult to compare different teaching approaches. To address these issues, we propose and demonstrate a method for assessing the user’s interpretation of the robot’s learning state in an interactive learning scenario with a robotic manipulator. Additionally, we draw on existing literature to categorise types of interface interventions that can enhance the human-robot teaching process for novice users – both pragmatically and hedonically. In a user study (N=30), we implement two of these interventions and show how they improve robot performance, teaching efficiency and interpretability. These findings provide preliminary insights into the design of effective human-robot teaching interfaces and can be used to assist the development of future teaching approaches.

Human activity recognition has become an important challenge yet to resolve while also having promising benefits in various applications for years. Existing approaches have made great progress by applying deep-learning and attention-based methods. However, the deep learning-based approaches may not fully exploit the features to resolve multimodal human activity recognition tasks. Also, the potential of attention-based methods still has not been fully explored to better extract the multimodal spatial-temporal relationship and produce robust results. In this work, we propose Multi-agent Transformer Network (MATN), a multi-agent attention-based deep learning algorithm, to address the above issues in multimodal human activity recognition. We first design a unified representation learning layer to encode the multimodal data, which preprocesses the data in a generalized and efficient way. Then we develop a multimodal spatial-temporal transformer module that applies the attention mechanism to extract the salient spatial-temporal features. Finally, we use a multi-agent training module to collaboratively select the informative modalities and predict the activity labels. We have extensively conducted experiments to evaluate MATN's performance on two public multimodal human activity recognition datasets. The results show that our model has achieved competitive performance compared to the state-of-the-art approaches, which also demonstrates scalability, effectiveness, and robustness.

Pedestrian trajectory prediction is essential for many modern applications, such as abnormal motion analysis and collision avoidance for improved traffic safety. Previous studies still face challenges in embracing high social interaction, dynamics, and multi-modality for achieving high accuracy with long-time predictions. We propose Social Graph Transformer Networks for multi-modal prediction of pedestrian trajectories, where we combine Graph Convolutional Network and Transformer Network by generating stable resolution pseudo-images from Spatio-temporal graphs through a designed stacking and interception method. Specifically, we adopt adjacency matrices to obtain Spatio-temporal features and Transformer for long-time trajectory predictions. As such, we retrain the advantages of both, i.e., the ability to aggregate information over an arbitrary number of neighbors and to conduct complex time-dependent data processing. Our experimental results show that our model reduces the final displacement error and achieves state-of-the-art in multiple metrics. The module's effectiveness is demonstrated through ablation experiments.

In interactive agent learning, the human may teach in a collaborative or adversarial manner. Past research has been focusing on collaborative teaching styles as these are common in human education settings, while overlooking adversarial ones despite promising results in recent research. Moreover, agent performance has been the main focal point while neglecting the perspective of the human teacher, who is crucial to the instructional process. In this work, we examine the impact of competitive and collaborative teaching styles on agent learning and human perception. We conducted a study (N=40) for participants to demonstrate a task in different interaction modes for teaching a computer agent: collaboratively, competitively, or without interacting with the agent. Most participants reported that they preferred competing against the computer agent to the other two modes. Despite smaller numbers of demonstrations given from the user, the agent performance from the interactive modes (collaborative and competitive) was comparable to the non-interactive mode (solo). The agent was perceived as being more competent in the competitive mode than the collaborative mode despite the marginally worse in-task performance. These preliminary findings suggest that competitive types of interaction, when agents or robots learn from humans, lead to better human perception of the agent’s learning when compared to collaborative, and better user engagement when compared to non-interactive learning from demonstrations.

Augmented reality (AR) technologies are today more frequently being introduced to Human-Robot Interaction (HRI) to mediate the interaction between human and robot. Indeed, better technical support and improved framework integration allow the design and study of novel scenarios augmenting interaction with AR. While some literature reviews have been published, so far no classifications have been devised for the role of AR in HRI. AR constitutes a vast field of research in HCI, and as it is picking up in HRI, it is timely to articulate the current knowledge and information about the functionalities of AR in HRI. Here we propose a multidimensional taxonomy for AR in HRI that distinguishes the type of perception augmentation, the functional role of AR, and the augmentation artifact type. We place sample publications within the taxonomy to demonstrate its utility. Lastly, we derive from the taxonomy some research gaps in current AR-for-HRI research and provide suggestions for exploration beyond the current state-of-the-art.

With the rising use of Augmented Reality (AR) technologies in Human-Robot Interaction (HRI), it is crucial that HRI research examines the role of AR in HRI to better define AR-HRI systems and identify potential areas for future research. A taxonomy for AR in HRI has recently been proposed for the field. However, it was limited to the definition of the framework, and exemplifying its use was missing. In this paper, we perform a demonstration of how the aforementioned taxonomy of AR in HRI can be used to analyse an existing AR-HRI system and come up with questions for alternative ways AR-HRI could be designed and further extended.

The feature analysis of point clouds, a popular representation of three-dimensional (3D) objects, is rising as a hot research topic nowadays. Point cloud data bear a sparse and unordered nature, making many commonly used feature extraction methods, for example, Convolutional Neural Networks (CNNs) inapplicable, while previous models suitable for the task are usually complex. We aim to reduce model complexity by reducing the number of parameters while achieving better (or at least comparable) performance. We propose an Interpolation Graph Convolutional Network (IGCN) for extracting features of point clouds. IGCN uses the point cloud graph structure and a specially designed Interpolation Convolution Kernel to mimic the operations of CNN for feature extraction. On the basis of weight postfusion and multilevel-resolution aggregation, IGCN not only reduces the cost of calculating the interpolation operation but also improves the model's performance. We validate the performance of IGCN on both point cloud classification and segmentation tasks and explore the contribution of each module of our model through ablation experiments. Furthermore, we embed the IGCN point cloud feature extraction module as a plug-and-play module into other frameworks and perform point cloud registration experiments.

We present two contributions built upon on a previous footstep planner based on the ARA* search. Firstly, we have developed an improved foothold selection method using support polygons, to increase foothold availability in rough terrain. Secondly, we present a footstep classification method using the C5.0 algorithm, that takes advantage of cost similarity between adjacent steps. This is intended to learn feasibility and approximate transition costs for the ARA*planner.These contributions extend capabilities of the planner by increasing footstep availability and allowing to generate more complex plans, without compromising safety.

Statistical machine learning is widely used in image classification. However, most techniques (1) require many images to achieve high accuracy and (2) do not provide support for reasoning below the level of classification, and so are unable to support secondary reasoning, such as the existence and position of light sources and other objects outside the image. This paper describes an Inductive Logic Programming approach called Logical Vision which overcomes some of these limitations. LV uses Meta-Interpretive Learning (MIL) combined with low-level extraction of high-contrast points sampled from the image to learn recursive logic programs describing the image. In published work LV was demonstrated capable of high-accuracy prediction of classes such as regular polygon from small numbers of images where Support Vector Machines and Convolutional Neural Networks gave near random predictions in some cases. LV has so far only been applied to noise-free, artificially generated images. This paper extends LV by (a) addressing classification noise using a new noise-telerant version of the MIL system Metagol, (b) addressing attribute noise using primitive-level statistical estimators to identify sub-objects in real images, (c) using a wider class of background models representing classical 2D shapes such as circles and ellipses, (d) providing richer learnable background knowledge in the form of a simple but generic recursive theory of light reflection. In our experiments we consider noisy images in both natural science settings and in a RoboCup competition setting. The natural science settings involve identification of the position of the light source in telescopic and microscopic images, while the RoboCup setting involves identification of the position of the ball. Our results indicate that with real images the new noise-robust version of LV using a single example (i.e. one-shot LV) converges to an accuracy at least comparable to a thirty-shot statistical machine learner on both prediction of hidden light sources in the scientific settings and in the RoboCup setting. Moreover, we demonstrate that a general background recursive theory of light can itself be invented using LV and used to identify ambiguities in the convexity/concavity of objects such as craters in the scientific setting and partial obscuration of the ball in the RoboCup setting.

This paper presents a robust 3D position tracking and map generation algorithm that combines data from a Laser Rangefinder and a RGB-D camera. Our algorithm is capable of running in real time on resource constrained robots. The novelty of this paper is the fusing of 2D-ICP from a Laser Rangefinder directly into a 3D position tracking, map building and SLAM system. Our approach takes advantage of the wide field of view and precision of laser scans to significantly improve the efficiency and accuracy of 3D alignment. We introduce a regularisation term to restrict movement in degrees of freedom that are not constrained by surface geometry. With efficiency in mind, we mirror a 2D local map based Graph SLAM approach in 3D to reliably detect loops if the robot comes near to an area it has previously visited. The mesh of each local map is warped after a graph of local maps is optimised to maintain a globally fused map. We provide experimental results to show that our algorithm is able to generate dense, globally consistent, mesh-based maps on resource constrained robots.

The Robocup Standard Platform League competition is a highly competitive league, with very little separating the top teams. Winning the competition in consecutive years is particularly challenging as other teams look to counter the tactics and game play of the previous champions. As the reigning champions from 2014, team UNSW Australia was able to overcome this challenge and win the competition for a second consecutive year. Although this success is not only related to developments from this year, this paper focuses on the new innovations and development by team UNSW Australia for the 2015 Robocup Competition. These innovations include white goal detection, whistle detection, foot detection and avoidance, improved path planning and new odometry.

This paper resolves previous problems in the Multi-Strategy architecture for online learning of robotic behaviours. The hybrid method includes a symbolic qualitative planner that constructs an approximate solution to a control problem. The approximate solution provides constraints for a numerical optimisation algorithm, which is used to refine the qualitative plan into an operational policy. Introducing quantitative constraints into the planner gives previously unachievable domain independent reasoning. The method is demonstrated on a multi-tracked robot intended for urban search and rescue.

Qualitative Simulation (QSIM) reasons about the behaviour of dynamic physical systems as they evolve over time. The system is represented by a coarse qualitative model rather than precise numerical models. However, for large complex domains, such as robotics for Urban Search and Rescue, existing QSIM implementations are inefficient. ASPQSIM is a novel formulation of the QSIM algorithm in Answer Set Programming that takes advantage of the similarities between qualitative simulation and constraint satisfaction problems. ASPQSIM is compared against an existing QSIM implementation on a variety of domains that demonstrate ASPQSIM provides a significant improvement in efficiency especially on complex domains, and producing simulations in domains that are not solvable by the procedural implementation.

Learning a map of an unknown environment and localising a robot in it is a common problem in robotics, with solutions usually requiring an estimate of the robot's motion. In scenarios such as Urban Search and Rescue, motion encoders can be highly inaccurate, and weight and battery requirements often limit computing power. We have developed a GPU based algorithm using Iterative Closest Point position tracking and Graph SLAM that can accurately generate a map of an unknown environment without the need for motion encoders and requiring minimal computational resources. The algorithm is able to correct for drift in the position tracking by rapidly identifying loops and optimising the map. We present a method for refining the existing map when revisiting areas to increase the accuracy of the existing map and bound the run-time to the size of the environment.

We present experiments on the application of machine learning to predicting slip. The sensing information is provided by a force/torque sensor and an artificial finger, which has randomly distributed strain gauges and polyvinylidene fluoride (PVDF) films embedded in silicone resulting in multidimensional time-series data on the finger-object contact. An incipient slip is detected by studying temporal patterns in the data. The data is analysed using probabilistic clustering that transforms the data into a sequence of symbols, which is used to train a hidden Markov model (HMM) classifier. Experimental results show that the classifier can predict a slip, at least 100ms before a slip takes place, with an accuracy of 96% on the validation set.

Advances in sensing technology and algorithm design make it possible for a robot equipped with a laser range-finder to generate a map and localise itself within the map as the robot explores its environment.We describe a system for mapping and virtual reconstruction developed as part of a robot for urban search and rescue. The process of mapping and, at the same time, localising the robot within the map, is called Simultaneous Localisation and Mapping (SLAM). In many applications, such as urban search and rescue, information from wheel encoders is inaccurate and cannot be used for odometry to obtain a position estimate. However, iterative closest point scan matching algorithms make it possible for a robot to perform accurate positioning in unstructured environments where wheel slip is common. When this positioning is combined with a mapping algorithm such as FastSLAM, the robot can construct an accurate map in real-time as it moves. Given the generated map and the robot's position within it, a variety of exploration algorithms allow the robot to autonomously explore its environment. The robot is also equipped with an RGB-D camera. The 3D information as well as the colour video images are incorporated into the map to produce a 3D virtual reconstruction of the environment as the robot explores. This robot won the award for best autonomous robot in three successive RoboCup Rescue Robot competitions [1], 2009 - 2011.

We presents a method for a robot to autonomously learn hidden properties of an object using active interaction and outcome prediction. Using a simulator we generate hypotheses about an object's properties and predictions of the outcomes of robot actions. To determine which hypothesis model most accurately describes the object, we match the result of a real world action to the simulated outcomes. The simulation is also used to find the most informative action, minimising the total number of actions the robot needs to perform to model the object. The end result is a model accurately describing the physical properties of the real world object.

Many applications for robotics require that the robot know its current position in the environment. While there exist several solutions for localizing a robot, even in a previously unknown environment, they often require an estimate of the robot's motion. However, in many situations, a robot may not have motion encoders, or its encoders may be highly inaccurate. We have developed an algorithm for tracking the position of a robot, based on a rangeflnder device, that is robust to temporary errors in the range scan. By aligning each scan to an occupancy grid of prior scan data, we can find the robot's position more accurately than current techniques which only align to the previous scan. In addition, our solution can track the position of the robot based on three dimensional scan data, instead of requiring that the range sensor be fixed in a level plane.

Controllers for autonomous mobile robots that operate in rough terrain must consider the shape of the surrounding terrain and its impact on the robot's movements. For complex terrain, these interactions are extremely difficult to model in a way that allows traditional controllers to be built. We have used Behavioural Cloning, a type of learning by imitation that produces rules that clone the skills of an expert human operator. We have also developed an autonomous instructor in simulation and used it to generate training data from which we have cloned controllers. The resulting controllers perform at a level comparable to that of a human expert. The controllers behave similarly both in simulation, where they were developed, and on the physical robot without the need for further modification or training.

In this paper, we present an application of machine learning to distinguish between different materials based on their surface texture. Such a system can be used for the estimation of surface friction during manipulation tasks; quality assurance in the textile, cosmetics, and harvesting industries; and other applications requiring tactile sensing. Several machine learning algorithms, such as naive Bayes, decision trees, and naive Bayes trees, have been trained to distinguish textures sensed by a biologically inspired artificial finger. The finger has randomly distributed strain gauges and polyvinylidene fluoride (PVDF) films embedded in silicone. Different textures induce different intensities of vibrations in the silicone. Consequently, textures can be distinguished by the presence of different frequencies in the signal. The data from the finger are preprocessed, and the Fourier coefficients of the sensor outputs are used to train classifiers. We show that the classifiers generalize well for unseen datasets with performance exceeding previously reported algorithms. Our classifiers can distinguish between different materials, such as carpet, flooring vinyls, tiles, sponge, wood, and polyvinyl-chloride (PVC) woven mesh with an accuracy of on unseen test data.

We present a method for matching image local features, specifically SIFT features, to a database of learned object features for the purpose of object recognition and localisation. Our approach differs from existing methods by taking into account the geometric consistency of matched features concurrently with their description vector similarity. As a result we do not need to over-constrain the description vector matching criteria (description vectors of matching features do not need to be nearest neighbours). The outcome of our approach is a greater number of feature matches between a scene image and a database image, as well an improvement in matching speed under certain circumstances.

In this paper we describe an application of machine learning to distinguish between seven different materials, based on their surface texture. Applications of such a system includes quality assurance and estimating surface friction during manipulation tasks. A naive Bayes classifier is used to distinguish textures sensed by a bio-inspired artificial finger. The finger has randomly distributed strain gauges and Polyvinylidene Fluoride (PVDF) films embedded in silicone. Different textures induce different intensity of vibrations in the silicone. Textures can be distinguished by the presence of different frequencies in the signal. The data from the finger is pre-processed and the Fourier coefficients of the sensor outputs are used to learn a classifier for different textures. The performance of the classifier is evaluated against a naive time domain based learner. Preliminary results show that our classifier performs better.

This paper describes the implementation of a novel real time robot audition system which combines a 3D sound localisation system and a voice characterisation (VC) system. The localisation system employs a 4 microphone array and uses the time delay estimation method. Accuracy is improved through the use of a correlation confidence threshold and a median filter. The VC system, which classifies between speech, non speech and silence, uses a decision tree classifier and a feature set comprising MFCCs, mean MFCCs and variance in MFCCs. The complete system has a processing time of 0.73x real time, and a range of up to 3 m. The compact design, high accuracy, and real time processing ability makes the system and the approach well suited to robotics.

Until robots are able to autonomously navigate, carry out a mission and report back to base, effective human-robot interfaces will be an integral part of any practical mobile robot system. This is especially the case for robot-assisted Urban Search and Rescue (USAR). Unfamiliar and unstructured environments, unreliable communications and many sensors combine to make the job of a human operator, and hence the interface designer challenging.This paper presents the design, implementation and deployment of a human-robot interface for the teleoperated USAR research robot, textsfCASTER. Proven HCI-based user interface design principles were adopted in order to produce an interface that was intuitive and minimised learning time while maximising effectiveness.The human-robot interface was deployed by Team CASualty in the 2005 RoboCup Rescue Robot League competition. This competition allows a wide variety of approaches to USAR research to be evaluated in a realistic environment. Despite the operator having less than one month of experience, Team CASualty came 3rd, beating teams that had far longer to train their operators. In particular, the ease with which the robot could be driven and high quality information gathered played a crucial part in Team CASualty's success. Further empirical evaluations of the system on a group of twelve users as well as members of the public further reinforce our belief that this interface is quick to learn, easy to use and effective.

Robot-assisted Urban Search and Rescue (USAR) operations benefit from having multiple robots search an area, especially if doing so does not require additional operators. However, designing a user interface that facilitates a single operator controlling many robots is challenging. In particular, the problems of situation awareness and cognitive load are amplified. This is especially the case when the robots concerned have a large number of degrees of freedom. We present a preliminary design and implementation of a user interface for a team of heterogeneous, potentially autonomous USAR robots with many degrees of freedom for both sequential and parallel operation. It extends our earlier design for a successfully deployed single-robot interface. Our design is inspired by Real-Time Strategy computer games, which must address many similar issues. The design seeks to maximise situational awareness and reduce cognitive load while allowing the operator to monitor and, if necessary, control all of the robots. Our user interface was deployed during the 2006 RoboCup Rescue Robot League where it played an important role in achieving the highest single-run scores in the preliminary rounds of the competition.

Reinforcement learning deals with learning optimal or near optimal policies while inter-acting with the environment. Application domains with many continuous variables are dicult to solve with existing reinforcement learning methods due to the large search space. In this paper, we use a relational rep-resentation to dene powerful abstractions that allow us to incorporate domain knowl-edge and re-use previously learned policies in other similar problems. We also describe how to learn useful actions from human traces us-ing a behavioural cloning approach combined with an exploration phase. Since several con-icting actions may be induced for the same abstract state, reinforcement learning is used to learn an optimal policy over this reduced space. It is shown experimentally how a com-bination of behavioural cloning and reinforce-ment learning using a relational representa-tion is powerful enough to learn how to \ny an aircraft through dierent points in space and dierent turbulence conditions. 1.

Recognition of Chinese characters has been an area of great interest for many years, and a large number of research papers and reports have already been published in this area. There are several major problems with Chinese character recognition: Chinese characters are distinct and ideographic, the character size is very large and a lot of structurally similar characters exist in the character set. Thus, classification criteria are difficult to generate. This paper presents a new technique for the recognition of hand-printed Chinese characters using machine learning C4.5. Conventional methods have relied on hand-constructed dictionaries which are tedious to construct and difficult to make tolerant to variation in writing styles. The paper also discusses Chinese character recognition using dominant point feature extraction and C4.5. The system was tested with 900 characters (each character has 40 samples) and the rate of recognition obtained was 84%.

Recognition of Chinese characters has been a major interest of researchers for many years, and a large number of research papers and reports have already been published in this area. There are several major problems: Chinese characters are distinct and ideographic, the character size is very large and a lot of structurally similar characters exist in the character set. Thus, classification criteria are difficult to find. This paper presents a new technique for the recognition of hand-printed Chinese characters using machine learning. Conventional methods have relied on hand-constructed dictionaries which are tedious to construct and difficult to make tolerant to variations in writing styles. The advantages of machine learning are twofold: it can generalize over the large degree of variations between writing styles and recognition rules can be constructed by example. The paper also describes three methods of feature extraction for Chinese character recognition: regular expression, dominant point and modified Hough transform. These methods are then compared in terms of accuracy and efficiency.

This paper presents a new technique for the recognition of hand-printed Latin characters using machine learning. Conventional methods have relied on manually constructed dictionaries which are tedious to construct and difficult to make tolerant to variation in writing styles. The advantages of machine learning are that it can generalise over a large degree of variation between writing styles and recognition rules can be constructed by example. Characters are scanned into the computer and preprocessing techniques transform the bit-map representation of the characters into set of primitives which can be represented in an attribute base form. A set of such representations for each character is then input to C4.5 which produces a decision tree for classifying each character.