Human-robot collaboration frequently requires extensive communication, e.g., using natural language and gesture. Augmented reality (AR) has provided an alternative way of bridging the communication gap between robots and people. However, most current AR-based human-robot communication methods are unidirectional, focusing on how the human adapts to robot behaviors, and are limited to single-robot domains. In this paper, we develop AR for Robots Collaborating with a Human (ARROCH), a novel algorithm and system that supports bidirectional, multi-turn, human-multi-robot communication in indoor multi-room environments. The human can see through obstacles to observe the robots’ current states and intentions, and provide feedback, while the robots’ behaviors are then adjusted toward human-multi-robot teamwork. Experiments have been conducted with real robots and human participants using collaborative delivery tasks. Results show that ARROCH outperformed a standard non-AR approach in both user experience and teamwork efficiency. In addition, we have developed a novel simulation environment using Unity (for AR and human simulation) and Gazebo (for robot simulation). Results in simulation demonstrate ARROCH’s superiority over AR-based baselines in human-robot collaboration.

In the past few years, numerous Deep Neural Network (DNN) models and frameworks have been developed to tackle the problem of real-time object detection from RGB images. Ordinary object detection approaches process information from the images only, and they are oblivious to the camera pose with regard to the environment and the scale of the environment. On the other hand, mobile Augmented Reality (AR) frameworks can continuously track a camera's pose within the scene and can estimate the correct scale of the environment by using Visual-Inertial Odometry (VIO). In this paper, we propose a novel approach that combines the geometric information from VIO with semantic information from object detectors to improve the performance of object detection on mobile devices. Our approach includes three components: (1) an image orientation correction method, (2) a scale-based filtering approach, and (3) an online semantic map. Each component takes advantage of the different characteristics of the VIO-based AR framework. We implemented the AR-enhanced features using ARCore and the SSD Mobilenet model on Android phones. To validate our approach, we manually labeled objects in image sequences taken from 12 room-scale AR sessions. The results show that our approach can improve on the accuracy of generic object detectors by 12% on our dataset.

Although mobile robots are increasingly being used in real-world applications, the ability to robustly sense and interact with the environment is still missing. A key requirement for the widespread deployment of mobile robots is the ability to operate autonomously by learning desired environmental models and revising the learned models in response to environmental changes. This paper presents an approach that enables a mobile robot to autonomously learn layered models for environmental objects using temporal, local and global visual cues. A temporal assessment of image gradient features is used to detect candidate objects, which are then modeled using color distribution statistics and a spatial representation of gradient features. The robot incrementally revises the learned models and uses them for object recognition and tracking based on a matching scheme comprising a spatial similarity measure and second order distribution statistics. All algorithms are implemented and tested on a wheeled robot platform in dynamic indoor environments.

Mobile robots are increasingly being used in real-world applications due to the ready availability of high-fidelity sensors and the development of sophisticated information processing algorithms. However, one key challenge to the widespread deployment of mobile robots equipped with multiple sensors and processing algorithms is the ability to autonomously tailor sensing and information processing to the task at hand. This paper poses this challenge as the task of planning under uncertainty, and more specifically as an instance of probabilistic sequential decision-making. A novel hierarchy of partially observable Markov decision processes (POMDPs) is incorporated, which uses constrained-convolutional policies and automatic belief propagation to achieve efficient and reliable operation on mobile robots. All algorithms are implemented and evaluated on simulated and physical robot platforms for the task of searching for target objects in dynamic indoor environments.