While numerous deep approaches to the problem of vision-aided localization have been recently proposed, systems operating in the real world will undoubtedly experience novel sensory states previously unseen even under the most prodigious training regimens. We address the localization problem with online error correction (OEC) modules that are trained to correct a vision-aided localization network's mistakes. We demonstrate the generalizability of the OEC modules and describe our unsupervised deep neural network approach to the fusion of RGB-D imagery with inertial measurements for absolute trajectory estimation. Our network, dubbed the Visual-Inertial-Odometry Learner (VIOLearner), learns to perform visual-inertial odometry (VIO) without inertial measurement unit (IMU) intrinsic parameters or the extrinsic calibration between an IMU and camera. The network learns to integrate IMU measurements and generate hypothesis trajectories which are then corrected online according to the Jacobians of scaled image projection errors with respect to spatial grids of pixel coordinates. We evaluate our network against state-of-the-art (SoA) VIO, visual odometry (VO), and visual simultaneous localization and mapping (VSLAM) approaches on the KITTI Odometry dataset as well as a micro aerial vehicle (MAV) dataset that we collected in the AirSim simulation environment. We demonstrate better than SoA translational localization performance against comparable SoA approaches on our evaluation sequences.

Recent developments in artificial intelligence enabled training of autonomous robots without human supervision. Even without human supervision during training, current models have yet to be human-engineered and have neither guarantees to match human expectation nor perform within safety bounds. This paper proposes CyberSteer to leverage human-robot interaction and align goals between humans and robotic intelligent agents. Based on human demonstration of the task, CyberSteer learns an intrinsic reward function used by the human demonstrator to pursue the goal of the task. The learned intrinsic human function shapes the robotic behavior during training through deep reinforcement learning algorithms, removing the need for environment-dependent or hand-engineered reward signal. Two different hypotheses were tested, both using non-expert human operators for initial demonstration of a given task or desired behavior: one training a deep neural network to classify human-like behavior and other training a behavior cloning deep neural network to suggest actions. In this experiment, CyberSteer was tested in a high-fidelity unmanned air system simulation environment, Microsoft AirSim. The simulated aerial robot performed collision avoidance through a clustered forest environment using forward-looking depth sensing. The performance of CyberSteer is compared to behavior cloning algorithms and reinforcement learning algorithms guided by handcrafted reward functions. Results show that the human-learned intrinsic reward function can shape the behavior of robotic systems and have better task performance guiding reinforcement learning algorithms compared to standard human-handcrafted reward functions.

Adstract- We present an unsupervised deep neural network approach to the fusion of RGB-D imagery with inertial measurements for absolute trajectory estimation. Our network, dubbed the Visual-Inertial-Odometry Learner (VIOLearner), learns to perform visual-inertial odometry (VIO) without inertial measurement unit (IMU) intrinsic parameters (corresponding to gyroscope and accelerometer bias or white noise) or the extrinsic calibration between an IMU and camera. The network learns to integrate IMU measurements and generate hypothesis trajectories which are then corrected online according to the Jacobians of scaled image projection errors with respect to a spatial grid of pixel coordinates. We evaluate our network against state-of-the-art (SOA) visual-inertial odometry, visual odometry, and visual simultaneous localization and mapping (VSLAM) approaches on the KITTI Odometry dataset [1] and demonstrate competitive odometry performance.

A novel template matching algorithm that can incorporate the concept of deformable parts, is presented in this paper. Unlike the deformable part model (DPM) employed in object recognition, the proposed template-matching approach called Deformable Template Matching (DTM) does not require a training step. Instead, deformation is achieved by a set of predefined basic rules (e.g. the left sub-patch cannot pass across the right patch). Experimental evaluation of this new method using the PASCAL VOC 07 dataset demonstrated substantial performance improvement over conventional template matching algorithms. Additionally, to confirm the applicability of DTM, the concept is applied to the generation of a rotation-invariant SIFT descriptor. Experimental evaluation employing deformable matching of SIFT features shows an increased number of matching features compared to a conventional SIFT matching.

The different strategies for feature extraction and synthesis employed by humans and computers are often complementary, hence combining the two into an integrated object recognition system may considerably improve performance over either used in isolation. Rapid Serial Visual Presentation (RSVP) is one well-established technique that has shown promise integrating human perception into a machine perception system. In this paper, we apply computer vision techniques to image data filtered through human RSVP. We introduce “task conversions” to integrate the two modalities, applying the precise localization capabilities of computer vision with the detection capabilities of RSVP. We employ naive Bayesian fusion and a novel method, dynamic belief fusion (DBF), in a joint scheme as fusion approaches. Preliminary experiments demonstrate that DBF extracts complementary information from both human and machine sources to improve performance for both target classification and object detection.

While autonomous capabilities have proliferated across a wide range of commercial and domestic applications, some tasks require intermittent aid from a human operator. Guaranteeing the safety of these intermittently-teleoperated systems requires stability guarantees that hold in the presence of switching. In this paper, we consider the problem of controlling a robotic vehicle using both a human controller and an autonomous controller. The strategy is to allow the human operator to switch between manual control and autonomous control as needed. The feedback loop is analyzed and shown to be stable using a notion of passivity from nonlinear system analysis. Finally, an example is provided to demonstrate the approach.

This paper investigates the effects of integrating automation into the various stages of information processing in a military command and control scenario. Command and control (C2) is an extreme decision-making paradigm characterized by high uncertainty, high risk, and severe time pressure. We introduce a principled approach to decision support system (DSS) design that specifically addresses these issues. Our approach establishes the principles of communicating confidence in sensor estimates and consequence of actions in an intuitive, timely manner. We hypothesize that automation designed to communicate confidence and/or consequence will improve task performance over systems that neglect these concepts. Toward this end, human-subjects experiments were conducted to compare the effects of displaying confidence/consequence information in a C2 target-tracking and interdiction scenario. Four variations of a decision support interface were designed, each with a distinct “degree of automation”: (i) an instantaneous sensor measurement visualization (baseline), (ii) a confidence-based visualization, (iii) a confidence- and consequence-based visualization, and (iv) a confidence- and consequence-based visualization with explicit decision recommendations. While increasing automation generally improved results, the inclusion of consequence information did not have a major effect, perhaps because the scenario was overly-simplified.

A novel approach for the fusion of heterogeneous object detection methods is proposed. In order to effectively integrate the outputs of multiple detectors, the level of ambiguity in each individual detection score is estimated using the precision/recall relationship of the corresponding detector. The main contribution of the proposed work is a novel fusion method, called Dynamic Belief Fusion (DBF), which dynamically assigns probabilities to hypotheses (target, non-target, intermediate state (target or non-target)) based on confidence levels in the detection results conditioned on the prior performance of individual detectors. In DBF, a joint basic probability assignment, optimally fusing information from all detectors, is determined by the Dempster's combination rule, and is easily reduced to a single fused detection score. Experiments on ARL and PASCAL VOC 07 datasets demonstrate that the detection accuracy of DBF is considerably greater than conventional fusion approaches as well as individual detectors used for the fusion.

Human-autonomy sensor fusion is an emerging technology with a wide range of applications, including object detection/recognition, surveillance, collaborative control, and prosthetics. For object detection, humans and computer-vision-based systems employ different strategies to locate targets, likely providing complementary information. However, little effort has been made in combining the outputs of multiple autonomous detectors and multiple human-generated responses. This paper presents a method for integrating several sources of human- and autonomy-generated information for rapid object detection tasks. Human electroencephalography (EEG) and button-press responses from rapid serial visual presentation (RSVP) experiments are fused with outputs from trained object detection algorithms. Three fusion methods—Bayesian, Dempster-Shafer, and Dynamic Dempster-Shafer—are implemented for comparison. Results demonstrate that fusion of these human classifiers with computer-vision-based detectors improves object detection accuracy over purely computer-vision-based detection (5% relative increase in mean average precision) and the best individual computer vision algorithm (28% relative increase in mean average precision). Computer vision fused with button press response and/or the XDAWN + Bayesian Linear Discriminant Analysis neural classifier provides considerable improvement, while computer vision fused with other neural classifiers provides little or no improvement. Of the three fusion methods, Dynamic Dempster-Shafer Theory (DDST) Fusion exhibits the greatest performance in this application.

Controls frameworks for human-autonomy integration (HAI) often treat human sources of information as highly reliable [1]. However, data from human sensing are quite variable, both because of human nature and limitations on sensor technologies. This paper focuses on estimating the degree of uncertainty in human sensed data, i.e., developing confidence metrics, and implementing those metrics in a HAI for target recognition. Here we demonstrate that applying such confidence estimates to sensed human data can mitigate effects of variability in human sensing and improve HAI performance.