Representing articulated objects remains a difficult problem within the field of robotics. Objects such as pliers, clamps, or cabinets require representations that capture not only geometry and color information, but also part seperation, connectivity, and joint parametrization. Furthermore, learning these representations becomes even more difficult with each additional degree of freedom. Complex articulated objects such as robot arms may have seven or more degrees of freedom, and the depth of their kinematic tree may be notably greater than the tools, drawers, and cabinets that are the typical subjects of articulated object research. To address these concerns, we introduce SPLATART - a pipeline for learning Gaussian splat representations of articulated objects from posed images, of which a subset contains image space part segmentations. SPLATART disentangles the part separation task from the articulation estimation task, allowing for post-facto determination of joint estimation and representation of articulated objects with deeper kinematic trees than previously exhibited. In this work, we present data on the SPLATART pipeline as applied to the syntheic Paris dataset objects [1], and qualitative results on a real-world object under spare segmentation supervision. We additionally present on articulated serial chain manipulators to demonstrate usage on deeper kinematic tree structures. Further media and information can be found at the project website here: https://progress.eecs.umich.edu/projects/splatart/

The Michigan Robotics MBot is a low-cost mobile robot platform that has been used to train over 1,400 students in autonomous navigation since 2014 at the University of Michigan and our collaborating colleges. The MBot platform was designed to meet the needs of teaching robotics at scale to match the growth of robotics as a field and an academic discipline. Transformative advancements in robot navigation over the past decades have led to a significant demand for skilled roboticists across industry and academia. This demand has sparked a need for robotics courses in higher education, spanning all levels of undergraduate and graduate experiences. Incorporating real robot platforms into such courses and curricula is effective for conveying the unique challenges of programming embodied agents in real-world environments and sparking student interest. However, teaching with real robots remains challenging due to the cost of hardware and the development effort involved in adapting existing hardware for a new course. In this paper, we describe the design and evolution of the MBot platform, and the underlying principals of scalability and flexibility which are keys to its success.

Particle filtering is a common technique for six degree of freedom (6D) pose estimation due to its ability to tractably represent belief over object pose. However, the particle filter is prone to particle deprivation due to the high-dimensional nature of 6D pose. When particle deprivation occurs, it can cause mode collapse of the underlying belief distri-bution during importance sampling. If the region surrounding the true state suffers from mode collapse, recovering its belief is challenging since the area is no longer represented in the probability mass formed by the particles. Previous methods mitigate this problem by randomizing and resetting particles in the belief distribution, but determining the frequency of reinvigoration has relied on hand-tuning abstract heuristics. In this paper, we estimate the necessary reinvigoration rate at each time step by introducing a Counter-Hypothetical likelihood function, which is used alongside the standard likelihood. Inspired by the notions of plausibility and implausibility from Evidential Reasoning, the addition of our Counter-Hypothetical likelihood function assigns a level of doubt to each particle. The competing cumulative values of confidence and doubt across the particle set are used to estimate the level of failure within the filter, in order to determine the portion of particles to be reinvigorated. We demonstrate the effectiveness of our method on the rigid body object 6D pose tracking task.

Advancements in autonomous agents have led to an increasingly ubiquitous presence of robots in human environments where social and physical interaction is expected. Such environments are often composed of heterogeneous agents with disparate action capabilities, intentions, and motivations. Intra- and inter-agent dissimilarity often prevents enacting effective behavioral skills (e.g., collaboration, communication, coordination) towards dynamic task allocation objectives. We propose a Bayesian probabilistic inference approach, MultiRobot Belief Propagation with Identity constraints (MRPBI), for 1) decentralized task allocation in multi-agent systems and 2) modeling task affinity using personal identities. MRPBI leverages competing costs of individual and group capabilities that result in less error-prone convergence to steady state, scalability without loss of accuracy, and sensitivity to environmental dynamics. An implementation of MRBP-I as a distributed algorithm that weighs factors of both individual and cooperative perception in an energy-minimizing task allocation scheme is presented.

Perceiving the position and orientation of objects (i.e., pose estimation) is a crucial prerequisite for robots acting within their natural environment. We present a hardware acceleration approach to enable real-time and energy efficient articulated pose estimation for robots operating in unstructured environments. Our hardware accelerator implements Nonparametric Belief Propagation (NBP) to infer the belief distribution of articulated object poses. Our approach is on average, 26× more energy efficient than a high-end GPU and 11× faster than an embedded low-power GPU implementation. Moreover, we present a Monte-Carlo Perception Library generated from high-level synthesis to enable reconfigurable hardware designs on FPGA fabrics that are better tuned to user-specified scene, resource, and performance constraints.

We present a new technique that enables manifold learning to accurately embed data manifolds that contain holes, without discarding any topological information. Manifold learning aims to embed high-dimensional data into a lower dimensional Euclidean space by learning a coordinate chart, but it requires that the entire manifold can be embedded in a single chart. This is impossible for manifolds with holes. In such cases, it is necessary to learn an atlas: a collection of charts that collectively cover the entire manifold. We begin with many small charts, and combine them in a bottom-up approach, where charts are only combined if doing so will not introduce problematic topological features. When it is no longer possible to combine any charts, each chart is individually embedded with standard manifold learning techniques, completing the construction of the atlas. We show the efficacy of our method by constructing atlases for challenging synthetic manifolds; learning human motion embeddings from motion capture data; and learning kinematic models of articulated objects.

Assembly tasks are challenging for robot manipulation because the robot must reason over the composed effects of actions and execute multi-objective behaviors. Robots typically use predefined priorities provided by users to determine how to compose controller behaviors, but we want the robot to autonomously select these compositions based on their composed effects within the task. We present Composable Causality in Semantic Robot Programming to allow robots to reason over the composed effects of controllers when executing multi-objective actions and autonomously compose controllers without predefined priorities. Our proposed causal control basis combines controller behaviors with causal information about how the behaviors can be used to execute high-level symbolic actions. The robot uses the causal control basis to predict the transition probability of achieving the composed effects of a multi-objective action. The composed causality estimates are used to select which action to execute within the context of a furniture assembly task. We evaluate the robot's transition probability estimates in different furniture assembly trials in simulation on the Baxter robot. The robot's ability to assemble furniture using different multi-objective connection actions demonstrates the usefulness of the composed causality estimates from our causal control basis.

For robots to successfully execute tasks as-signed to them, they must be capable of planning the right sequence of actions. These actions must be both optimal with respect to a specified objective and satisfy whatever constraints exist in their world. We propose an approach for robot task planning that is capable of planning the optimal sequence of grounded actions to accomplish a task given a specific objective function while satisfying all specified numerical constraints. Our approach accomplishes this by encoding the entire task planning problem as a single mixed integer convex program, which it then solves using an off-the-shelf Mixed Integer Program-ming solver. We evaluate our approach on several mobile manipulation tasks in both simulation and on a physical humanoid robot. Our approach is able to consistently produce optimal plans while accounting for all specified numerical constraints in the mobile manipulation tasks. Open-source implementations of the components of our approach as well as videos of robots executing planned grounded actions in both simulation and the physical world can be found at this url: https://adubredu.github.io/gtpmip

Visual perception tasks often require vast amounts of labelled data, including 3D poses and image space segmen-tation masks. The process of creating such training data sets can prove difficult or time-intensive to scale up to efficacy for general use. Consider the task of pose estimation for rigid objects. Deep neural network based approaches have shown good performance when trained on large, public datasets. However, adapting these networks for other novel objects, or fine-tuning existing models for different environments, requires significant time investment to generate newly labelled instances. Towards this end, we propose ProgressLabeller as a method for more efficiently generating large amounts of 6D pose training data from color images sequences for custom scenes in a scalable manner. ProgressLabeller is intended to also support transparent or translucent objects, for which the previous methods based on depth dense reconstruction will fail. We demonstrate the effectiveness of ProgressLabeller by rapidly create a dataset of over 1M samples with which we fine-tune a state-of-the-art pose estimation network in order to markedly improve the downstream robotic grasp success rates. Progresslabeller is open-source at https://github.com/huijieZH/ProgressLabeller

Articulated objects pose a unique challenge for robotic perception and manipulation. Their increased number of degrees-of-freedom makes tasks such as localization computationally difficult, while also making the process of realworld dataset collection unscalable. With the aim of addressing these scalability issues, we propose Neural Articulated Radiance Fields (NARF22), a pipeline which uses a fully-differentiable, configuration-parameterized Neural Radiance Field (NeRF) as a means of providing high quality renderings of articulated objects. NARF22 requires no explicit knowledge of the object structure at inference time. We propose a two-stage partsbased training mechanism which allows the object rendering models to generalize well across the configuration space even if the underlying training data has as few as one configuration represented. We demonstrate the efficacy of NARF22 by training configurable renderers on a real-world articulated tool dataset collected via a Fetch mobile manipulation robot. We show the applicability of the model to gradient-based inference methods through a configuration estimation and 6 degree-of-freedom pose refinement task.

Benefiting from language flexibility and compositionality, humans naturally intend to use language to command an embodied agent for complex tasks such as navigation and object manipulation. In this work, we aim to fill the blank of the last mile of embodied agents---object manipulation by following human guidance, e.g., “move the red mug next to the box while keeping it upright.” To this end, we introduce an Automatic Manipulation Solver (AMSolver) system and build a Vision-and-Language Manipulation benchmark (VLMbench) based on it, containing various language instructions on categorized robotic manipulation tasks. Specifically, modular rule-based task templates are created to automatically generate robot demonstrations with language instructions, consisting of diverse object shapes and appearances, action types, and motion constraints. We also develop a keypoint-based model 6D-CLIPort to deal with multi-view observations and language input and output a sequence of 6 degrees of freedom (DoF) actions. We hope the new simulator and benchmark will facilitate future research on language-guided robotic manipulation.

As the first robotics journal offered by ACM, the Transactions on Human-Robot Interaction (ACM THRI) made considerable progress to advance the field of HRI since its inaugural offering in 2018.ACM THRI remains the flagship journal for leading research in the burgeoning area of Human-Robot Interaction (HRI).The interdisciplinary field of HRI aims to understand and advance our knowledge of human-robot systems towards advancing usable and useful robotic technology.ACM THRI features leading thought into how people interact with robots and robotic technologies, how to improve these interactions and make new kinds of interaction possible, and the effects of such interactions on organizations or society.HRI brings together researchers and practitioners with diverse expertise, including but not limited to robotics, artificial intelligence, human factors, psychology, design, haptics, and mechatronics.We officially began our term in January 2017 as Editors-in-Chief of the Journal of Human-Robot Interaction, the predecessor to ACM THRI.We were fortunate to build on the foundation of JHRI provided by its founding editors, Profs.Sara Kiesler and Michael Goodrich.Our leadership of ACM THRI would not be possible without the tremendous effort and dedication of our core editorial team, including our amazing Managing Editor (Prof.James Young), Journal Administrator (Gita Delsing) and Information Director (Dr.Daniel Rea) as well as their predecessors (Prof.David Feil-Seifer, Ms. Jeanie Lyubelsky, and Prof. Monica Anderson).Although it has presented unique challenges and new opportunities, we remain grateful for the opportunity to lead the transition of JHRI to ACM ownership and further extend ACM THRI.Our focus as Editors-in-Chief has been to establish commitment to excellence, sustainability, and inclusion in the scholarship and review processes of ACM THRI, which has proved successful to this point.As stated in our introductory editorial, our primary aims for the Journal have been three fold: (1) increase the sustainability and impact of HRI as a field (both quantitatively and qualitatively), (2) enable timely and productive review feedback, and (3) cultivate new and leading ideas in both robotics and the human-centered sciences.ACM THRI has remained true to this scope and made significant strides towards realizing these values across the practices of the journal.For our review processes, our goal has been to provide useful feedback to our authors that enables them to improve their work and make contributions worthy of intellectual leadership.We aim to return substantive review feedback to authors within 105 days of the date of submission, and 70 days for revisions.The total average for ACM THRI returning decisions for submitted manuscripts is currently 142 days in 2021, 122 days from April 2020 to December 2020, 114.7 days in the first three months of 2020 (submissions just prior to the COVID-19 pandemic), and 99 days in 2019.Our review process includes significant deliberation and discussion among our editorial board of every manuscript and decision, which occurs during our weekly editorial teleconferences.A summary of our editorial consensus with a reasoned and supported argument is provided with

We aim for mobile robots to function in a variety of common human environments, which requires them to efficiently search previously unseen target objects. We can exploit background knowledge about common spatial relations between landmark objects and target objects to narrow down search space. In this paper, we propose an active visual object search strategy method through our introduction of the Semantic Linking Maps (SLiM) model. SLiM simultaneously maintains the belief over a target object's location as well as landmark objects' locations, while accounting for probabilistic inter-object spatial relations. Based on SLiM, we describe a hybrid search strategy that selects the next best view pose for searching for the target object based on the maintained belief. We demonstrate the efficiency of our SLiM-based search strategy through comparative experiments in simulated environments. We further demonstrate the real-world applicability of SLiM-based search in scenarios with a Fetch mobile manipulation robot.

For autonomous service robots to successfully perform long horizon tasks in the real world, they must act intelligently in partially observable environments. Most Task and Motion Planning approaches assume full observability of their state space, making them ineffective in stochastic and partially observable domains that reflect the uncertainties in the real world. We propose an online planning and execution approach for performing long horizon tasks in partially observable domains. Given the robot’s belief and a plan skeleton composed of symbolic actions, our approach grounds each symbolic action by inferring continuous action parameters needed to execute the plan successfully. To achieve this, we formulate the problem of joint inference of action parameters as a Hybrid Constraint Satisfaction Problem (H-CSP) and solve the H-CSP using Belief Propagation. The robot executes the resulting parameterized actions, updates its belief of the world and replans when necessary. Our approach is able to efficiently solve partially observable tasks in a realistic kitchen simulation environment. Our approach outperformed an adaptation of the state-of-the-art method across our experiments.

Reliable, efficient shared autonomy requires balancing human operation and robot automation on complex tasks, such as dexterous manipulation. Adding to the difficulty of shared autonomy is a robot’s limited ability to perceive the 6 degree-of-freedom pose of objects, which is essential to perform manipulations those objects afforded. Inspired by Monte Carlo Localization, we propose a generative human-in-the-loop approach to estimating object pose. We characterize the performance of our mixed-initiative 3D registration approach using 2D pointing devices via a user study. Seeking an analog for Fitts’s Law for 3D registration, we introduce a new evaluation framework that takes the entire registration process into account instead of only the outcome. When combined with estimates of registration confidence, we posit that mixed-initiative registration will reduce the human workload while maintaining or even improving final pose estimation accuracy.

We aim for mobile robots to function in a variety of common human environments. Such robots need to be able to reason about the locations of previously unseen target objects. Landmark objects can help this reasoning by narrowing down the search space significantly. More specifically, we can exploit background knowledge about common spatial relations between landmark and target objects. For example, seeing a table and knowing that cups can often be found on tables aids the discovery of a cup. Such correlations can be expressed as distributions over possible pairing relationships of objects. In this paper, we propose an active visual object search strategy method through our introduction of the Semantic Linking Maps (SLiM) model. SLiM simultaneously maintains the belief over a target object's location as well as landmark objects' locations, while accounting for probabilistic inter-object spatial relations. Based on SLiM, we describe a hybrid search strategy that selects the next best view pose for searching for the target object based on the maintained belief. We demonstrate the efficiency of our SLiM-based search strategy through comparative experiments in simulated environments. We further demonstrate the realworld applicability of SLiM-based search in scenarios with a Fetch mobile manipulation robot.

Robots working in human environments must be able to perceive and act on challenging objects with articulations, such as a pile of tools. Articulated objects increase the dimensionality of the pose estimation problem, and partial observations under clutter create additional challenges. To address this problem, we present a generative-discriminative parts-based recognition and localization method for articulated objects in clutter. We formulate the problem of articulated object pose estimation as a Markov Random Field (MRF). Hidden nodes in this MRF express the pose of the object parts, and edges express the articulation constraints between parts. Localization is performed within the MRF using an efficient belief propagation method. The method is informed by both part segmentation heatmaps over the observation, generated by a neural network, and the articulation constraints between object parts. Our generative-discriminative approach allows the proposed method to function in cluttered environments by inferring the pose of occluded parts using hypotheses from the visible parts. We demonstrate the efficacy of our methods in a tabletop environment for recognizing and localizing hand tools in uncluttered and cluttered configurations.

Robots working in human environments often encounter a wide range of articulated objects, such as tools, cabinets, and other jointed objects. Such articulated objects can take an infinite number of possible poses, as a point in a potentially high-dimensional continuous space. A robot must perceive this continuous pose in order to manipulate the object to a desired pose. This problem of perception and manipulation of articulated objects remains a challenge due to its high dimensionality and multi-modal uncertainty. In this paper, we propose a factored approach to estimate the poses of articulated objects using an efficient non-parametric belief propagation algorithm. We consider inputs as geometrical models with articulation constraints, and observed 3D sensor data. The proposed framework produces object-part pose beliefs iteratively. The problem is formulated as a pairwise Markov Random Field (MRF) where each hidden node (continuous pose variable) models an observed object-part's pose and each edge denotes an articulation constraint between a pair of parts. We propose articulated pose estimation by a Pull Message Passing algorithm for Nonparametric Belief Propagation (PMPNBP) and evaluate its convergence properties over scenes with articulated objects.

Robotic Learning from Demonstration (LfD) allows anyone, not just experts, to program a robot for an arbitrary task. Many LfD methods focus on low level primitive actions such as manipulator trajectories. Complex multistep task with many primitive actions must be learned from demonstration if LfD is to encompass the full range of task a user may desire. Existing methods represent the high level task in various forms including, finite state machines, decision trees, formal logic, among others. Behavior trees are proposed as an alternative representation of high level task. Behavior trees are an execution model for the control of a robot designed for real time execution, modularity, and, consequently, transparency. Real time execution allows the robot to reactively perform the task. Modularity allows the reuse of learned primitive actions and high level task in new situations, speeding up the process of learning in new scenarios. Transparency allows users to understand and interactively modify the learned model. Behavior trees are used to represent high level tasks by building on the relationship it has with decision trees. We demonstrate a human teaching our Fetch robot a household cleaning task.

Recent advancements have led to a proliferation of machine learning systems used to assist humans in a wide range of tasks. However, we are still far from accurate, reliable, and resource-efficient operations of these systems. For robot perception, convolutional neural networks (CNNs) for object detection and pose estimation are recently coming into widespread use. However, neural networks are known to suffer from overfitting during the training process and are less robust under unforeseen conditions (which makes them especially vulnerable to adversarial scenarios). In this work, we propose Generative Robust Inference and Perception (GRIP) as a two-stage object detection and pose estimation system that aims to combine the relative strengths of discriminative CNNs and generative inference methods to achieve robust estimation. Our results show that a second stage of sample-based generative inference is able to recover from false object detections by CNNs, and produce robust estimations in adversarial conditions. We demonstrate the efficacy of GRIP robustness through comparison with state-of-the-art learning-based pose estimators and pick-and-place manipulation in dark and cluttered environments.

Transparent objects are prevalent across many environments of interest for dexterous robotic manipulation. Such transparent material leads to considerable uncertainty for robot perception and manipulation, and remains an open challenge for robotics. This problem is exacerbated when multiple transparent objects cluster into piles of clutter. In household environments, for example, it is common to encounter piles of glassware in kitchens, dining rooms, and reception areas, which are essentially invisible to modern robots. We present the GlassLoc algorithm for grasp pose detection of transparent objects in transparent clutter using plenoptic sensing. GlassLoc classifies graspable locations in space informed by a Depth Likelihood Volume (DLV) descriptor. We extend the DLV to infer the occupancy of transparent objects over a given space from multiple plenoptic viewpoints. We demonstrate and evaluate the GlassLoc algorithm on a Michigan Progress Fetch mounted with a first generation Lytro. The effectiveness of our algorithm is evaluated through experiments for grasp detection and execution with a variety of transparent glassware in minor clutter.

No abstract available.

No abstract available.

No abstract available.

Indoor robots hold the promise of automatically handling mundane daily tasks, helping to improve access for people with disabilities, and providing on-demand access to remote physical environments. Unfortunately, the ability to understand never-before-seen objects in scenes where new items may be added (e.g., purchased) or altered (e.g., damaged) on a regular basis remains an open challenge for robotics. In this paper, we introduce EURECA, a mixed-initiative system that leverages online crowds of human contributors to help robots robustly identify 3D point cloud segments corresponding to user-referenced objects in near real-time. EURECA allows robots to understand multi-object 3D scenes on-the-fly (in ~40 seconds) by providing groups of non-expert crowd workers with intelligent tools that can segment objects more quickly (~70% faster) and more accurately (6% higher F1 score) than individuals. More broadly, EURECA introduces the first real-time crowdsourcing tool that addresses the challenge of learning about new objects in real-world settings, creating a new source of data for training robots online, as well as a platform for studying mixed-initiative crowdsourcing workflows for understanding 3D scenes.

Convolutional neural networks (CNNs) are of increasing widespread use in robotics, especially for object recognition. However, such CNNs still lack several critical properties necessary for robots to properly perceive and function autonomously in uncertain, and potentially adversarial, environments. In this paper, we investigate factors for accurate, reliable, and resource-efficient object and pose recognition suitable for robotic manipulation in adversarial clutter. Our exploration is in the context of a three-stage pipeline of discriminative CNN-based recognition, generative probabilistic estimation, and robot manipulation. This pipeline proposes using a SAmpling Network Density filter, or SAND filter, to recover from potentially erroneous decisions produced by a CNN through generative probabilistic inference. We present experimental results from SAND filter perception for robotic manipulation in tabletop scenes with both benign and adversarial clutter. These experiments vary CNN model complexity for object recognition and evaluate levels of inaccuracy that can be recovered by generative pose inference. This scenario is extended to consider adversarial environmental modifications with varied lighting, occlusions, and surface modifications.

We introduce an interactive system for extracting the geometries of generalized cylinders and cuboids from single-or multiple-view point clouds. Our proposed method is intuitive and only requires the object's silhouettes to be traced by the user. Leveraging the user's perceptual understanding of what an object looks like, our proposed method is capable of extracting accurate models, even in the presence of occlusion, clutter or incomplete point cloud data, while preserving the original object's details and scale. We demonstrate the merits of our proposed method through a set of experiments on a public RGB-D dataset. We extracted 16 objects from the dataset using at most two views of each object. Our extracted models represent a high degree of visual similarity to the original objects. Further, we achieved a mean normalized Hausdorff distance of 5.66% when comparing our extracted models with the dataset's ground truths.

We present the Semantic Robot Programming (SRP) paradigm as a convergence of robot programming by demonstration and semantic mapping. In SRP, a user can directly program a robot manipulator by demonstrating a snapshot of their intended goal scene in workspace. The robot then parses this goal as a scene graph comprised of object poses and inter-object relations, assuming known object geometries. Task and motion planning is then used to realize the user's goal from an arbitrary initial scene configuration. Even when faced with different initial scene configurations, SRP enables the robot to seamlessly adapt to reach the user's demonstrated goal. For scene perception, we propose the Discriminatively-Informed Generative Estimation of Scenes and Transforms (DIGEST) method to infer the initial and goal states of the world from RGBD images. The efficacy of SRP withDIGESTperception is demonstrated for the task of tray-setting with a Michigan Progress Fetch robot. Scene perception and task execution are evaluated with a public household occlusion dataset and our cluttered scene dataset.

Affordances provide a natural means for a robot to describe its agency as actions it can perform on objects. Further, affordances can enable robots to reason complicated, multi-step tasks that involve proper use of a diversity of objects. This paper proposes the concept of affordance wayfields for representing manipulation affordances as objective functions in configuration space. Affordance wayfields quantify how well a path, or sequence of motions, will accomplish an afforded action on an object. Paths that enact affordances can be located by performing a randomized form of gradient descent over affordance wayfields. Incorporating obstacles, or other constraints into wayfields allows our method to adaptively generate valid motions for executing afforded actions. We demonstrate that affordance wayfields can enable robots, such as the Michigan Progress Fetch mobile manipulator, to solve complex real-world tasks such as assembling a table, or loading and unloading objects from a storage chest.

We present a filtering-based method for semantic mapping to simultaneously detect objects and localize their 6 degree-of-freedom pose. For our method, called Contextual Temporal Mapping (or CT-Map), we represent the semantic map as a belief over object classes and poses across an observed scene. Inference for the semantic mapping problem is then modeled in the form of a Conditional Random Field (CRF). CT-Map is a CRF that considers two forms of relationship potentials to account for contextual relations between objects and temporal consistency of object poses, as well as a measurement potential on observations. A particle filtering algorithm is then proposed to perform inference in the CT-Map model. We demonstrate the efficacy of the CT-Map method with a Michigan Progress Fetch robot equipped with a RGB-D sensor. Our results demonstrate that the particle filtering based inference of CT-Map provides improved object detection and pose estimation with respect to baseline methods that treat observations as independent samples of a scene.

In order to fully function in human environments, robot perception needs to account for the uncertainty caused by translucent materials. Translucency poses several open challenges in the form of transparent objects (e.g., drinking glasses), refractive media (e.g., water), and diffuse partial occlusions (e.g., objects behind stained glass panels). This paper presents Plenoptic Monte Carlo Localization (PMCL)as a method for localizing object poses in the presence of translucency using plenoptic (light-field)observations. We propose a new depth descriptor, the Depth Likelihood Volume (DLV), and its use within a Monte Carlo object localization algorithm. We present results of localizing and manipulating objects with translucent materials and objects occluded by layers of translucency. Our PMCL implementation uses observations from a Lytro first generation light field camera to allow a Michigan Progress Fetch robot to perform grasping.

No abstract available.

No abstract available.

In order to perform autonomous sequential manipulation tasks, perception in cluttered scenes remains a critical challenge for robots. In this paper, we propose a probabilistic approach for robust sequential scene estimation and manipulation - Sequential Scene Understanding and Manipulation (SUM). SUM considers uncertainty due to discriminative object detection and recognition in the generative estimation of the most likely object poses maintained over time to achieve a robust estimation of the scene under heavy occlusions and unstructured environment. Our method utilizes candidates from discriminative object detector and recognizer to guide the generative process of sampling scene hypothesis, and each scene hypotheses is evaluated against the observations. Also SUM maintains beliefs of scene hypothesis over robot physical actions for better estimation and against noisy detections. We conduct extensive experiments to show that our approach is able to perform robust estimation and manipulation.

Dear colleagues in the HRI community, JHRI needs you!

This experiment demonstrated that automated pace guidance generated from real-time physiological monitoring allowed less stressful completion of a timed (60 minute limit) 5 mile treadmill exercise. An optimal pacing policy was estimated from a Markov decision process that balanced the goals of the movement task and the thermal-work strain safety constraints. The machine guided pace was based on current physiological strain index (PSI), the time, and the distance already completed. Fourteen healthy and fit young subjects participated in the study (9 men, 5 women). Each participated in an unguided exercise session followed by a guided one. In the unguided session, they were instructed to complete 5 miles in 60 minutes and to try to finish at the lowest body temperature possible; in the guided sessions, participants were instructed to match machine-provided pacing guidance provided every 2 minutes. Continuous real-time measures of heart rate and core body temperature were obtained from a wearable Hidalgo EquivitalTMEQ-02 and the MiniMitter Jonah thermometer pill. Of the fourteen subjects, 13 completed the 5 miles in one hour for the unguided session; at least three different self-pacing strategies were observed, with an alternating speed proving to be most effective. In the guided sessions, 6 subjects were stopped by the machine guidance for exceeding the algorithms PSI “safety” limit. Eight subjects were guided to complete the task with significantly lower PSIs. The results indicate that machine guided advice shows promise for preventing hyperthermia and improving outcomes for performers of an unfamiliar task.

Manipulation tasks involving sequential pick-and-place actions in human environments remains an open problem for robotics. Central to this problem is the inability for robots to perceive in cluttered environments, where objects are physically touching, stacked, or occluded from the view. Such physical interactions currently prevent robots from distinguishing individual objects such that goal-directed reasoning over sequences of pick-and-place actions can be performed. Addressing this problem, we introduce the Axiomatic Particle Filter (APF) as a method for axiomatic state estimation to simultaneously perceive objects in clutter and perform sequential reasoning for manipulation. The APF estimates state as a scene graph, consisting of symbolic spatial relations between objects in the robot's world. Assuming known object geometries, the APF is able to infer a distribution over possible scene graphs of the robot's world and produce the maximally likely state estimate of each object's pose and spatial relationships between objects. We present experimental results using the APF to infer scene graphs from depth images of scenes with objects that are touching, stacked, and occluded.

Since its official introduction in 2012, the Robot Web Tools project has grown tremendously as an open-source community, enabling new levels of interoperability and portability across heterogeneous robot systems, devices, and front-end user interfaces. At the heart of Robot Web Tools is the rosbridge protocol as a general means for messaging ROS topics in a client-server paradigm suitable for wide area networks, and human-robot interaction at a global scale through modern web browsers. Building from rosbridge, this paper describes our efforts with Robot Web Tools to advance: 1) human-robot interaction through usable client and visualization libraries for more efficient development of front-end human-robot interfaces, and 2) cloud robotics through more efficient methods of transporting high-bandwidth topics (e.g., kinematic transforms, image streams, and point clouds). We further discuss the significant impact of Robot Web Tools through a diverse set of use cases that showcase the importance of a generic messaging protocol and front-end development systems for human-robot interaction.

Recent developments in human-robot interaction brings about higher requirements for robot navigation. Existing Simultaneous Localization and Mapping (SLAM) algorithms face open challenges for navigation in complex dynamic environments due to presumptions of static environments or exceeding computational limitations. In this paper, we propose a robust graph SLAM formuation exploring Expectation Maximization algorithms to characterize landmark mobility while establishing the estimations of robot trajectory and the map. We evaluate the performance of existing robust SLAM algorithms as baselines, and validate the improvement of our new framework against datasets of dynamic environments with moving landmarks.

Intelligent behavior in robots is implemented through algorithms. Historically, much of algorithmic robotics research strives to compute outputs that achieve mathematically rigid conditions, such as minimizing path length. But today's robots are increasingly being used to empower the daily lives of people, and experience shows that traditional algorithmic approaches are poorly suited for the unpredictable, idiosyncratic, and adaptive nature of human-robot interaction. This raises a need for entirely new computational, mathematical, and technical approaches for robots to better understand and react to humans. The human-friendly robots of the future will need new algorithms, informed from the ground up by HRI research, to generate interpretable, ethical, socially-acceptable behavior, ensure safety around humans, and execute tasks of value to society.

Emergency workers engaged in strenuous work in hot environments risk overheating and mission failure. We describe a real-time application that would reduce these risks in terms of a real-time thermal-work strain index (SI) estimator; and a Markov Decision Process (MDP) to compute optimal work rate policies. We examined the thermo-physiological responses of 14 experienced U.S. Army Ranger students (26±4 years 1.77±0.04 m; 78.3±7.3 kg) who participated in a strenuous 8 mile time-restricted pass/fail road march conducted under thermally stressful conditions. A thermoregulatory model was used to derive SI state transition probabilities and model the students’ observed and policy driven movement rates. We found that policy end-state SI was significantly lower than SI when modeled using the student’s own movement rates (3.94±0.88 vs. 5.62±1.20, P<0.001). We also found an inverse relationship between our policy impact and maximum SI (r=0.64 P<0.05). These results suggest that modeling real world missions as an MDP can provide optimal work rate policies that improve thermal safety and allow students to finish in a “fresher” state. Ultimately, SI state estimation and MDP models incorporated into wearable physiological monitoring systems could provide real-time work rate guidance, thus minimizing thermal work-strain while maximizing the likelihood of accomplishing mission tasks.

We propose a simulation-based dynamical motion prior for tracking human motion from video in presence of physical ground-person interactions. Most tracking approaches to date have focused on efficient inference algorithms and/or learning of prior kinematic motion models; however, few can explicitly account for the physical plausibility of recovered motion. Here, we aim to recover physically plausible motion of a single articulated human subject. Toward this end, we propose a full-body 3D physical simulation-based prior that explicitly incorporates a model of human dynamics into the Bayesian filtering framework. We consider the motion of the subject to be generated by a feedback “control loop” in which Newtonian physics approximates the rigid-body motion dynamics of the human and the environment through the application and integration of interaction forces, motor forces, and gravity. Interaction forces prevent physically impossible hypotheses, enable more appropriate reactions to the environment (e.g., ground contacts), and are produced from detected human-environment collisions. Motor forces actuate the body, ensure that proposed pose transitions are physically feasible, and are generated using a motion controller. For efficient inference in the resulting high-dimensional state space, we utilize an exemplar-based control strategy that reduces the effective search space of motor forces. As a result, we are able to recover physically plausible motion of human subjects from monocular and multiview video. We show, both quantitatively and qualitatively, that our approach performs favorably with respect to Bayesian filtering methods with standard motion priors.

The advent of ROS, the Robot Operating System, has finally made it possible to implement and use state-of-the-art navigation and manipulation algorithms on widely-available, inexpensive standard robot platforms. With the addition of the Rosbridge application programming interface, interface designers and applications programmers can create robot interfaces and behaviors without venturing into the specialized world of robotics engineers. This tutorial introduces ROS and Rosbridge, and shows how quickly and easily these tools can be used to design and conduct large-scale online HRI experiments, access algorithms for autonomous robot behavior, and leverage the huge ecosystem of general-purpose web-based and application-oriented software engineering for robotics and HRI research. Tutorial attendees will learn the basics of autonomous and teleoperated navigation and manipulation, as well as interface design for online interaction with robots. During the tutorial they will design and write their own remote presence application, as well as develop strategies for incorporating autonomy and dealing with data collection.

In this paper, we describe a remote lab system that allows remote groups to access a shared PR2. This lab will enable a larger and more diverse group of researchers to participate directly in state-of-the-art robotics research and will improve the reproducibility and comparability of robotics experiments. We identify a set of requirements that apply to all web-based remote laboratories and focus on solutions to these requirements. Specifically, we present solutions to interface, control and design difficulties in the client and server-side software when implementing a remote laboratory architecture. The combination of shared physical hardware and shared middleware software allows for experiments that build upon and compare against results on the same platform and in the same environment for common tasks. We describe how researchers can interact with the PR2 and its environment remotely through a web interface, as well as develop similar interfaces to visualize and run experiments remotely.

Advancements in robotics have led to an ever-growing repertoire of software capabilities (e.g., recognition, mapping, and object manipulation). However, robotic capabilities grow, the complexity of operating and interacting with such robots increases (such as through speech, gesture, scripting, or programming). Language-based communication can offer users the ability to work with physically and computationally complex robots without diminishing the robot's inherent capability. However, it remains an open question how to build a common ground between natural language and goal-directed robot actions, particularly in a way that scales with the growth of robot capabilities. We examine using semantic frames - a linguistics concept which describes scenes being acted out - as a conceptual stepping stone between natural language and robot action. We examine the scalability of this solution through the development of RoboFrameNet, a generic language-to-action pipeline for ROS (the Robot Operating System) that abstracts verbs and their dependents into semantic frames, then grounds these frames into actions. We demonstrate the framework through experiments with the PR2 and Turtlebot robot platforms and consider the future scalability of the approach.

Marker-less motion capture is a challenging problem, particularly when only monocular video is available. We estimate human motion from monocular video by recovering three-dimensional controllers capable of implicitly simulating the observed human behavior and replaying this behavior in other environments and under physical perturbations. Our approach employs a state-space biped controller with a balance feedback mechanism that encodes control as a sequence of simple control tasks. Transitions among these tasks are triggered on time and on proprioceptive events ( e.g ., contact). Inference takes the form of optimal control where we optimize a high-dimensional vector of control parameters and the structure of the controller based on an objective function that compares the resulting simulated motion with input observations. We illustrate our approach by automatically estimating controllers for a variety of motions directly from monocular video. We show that the estimation of controller structure through incremental optimization and refinement leads to controllers that are more stable and that better approximate the reference motion. We demonstrate our approach by capturing sequences of walking, jumping, and gymnastics.

We present a study of using a robotic learning from demonstration system capable of collecting large amounts of human-robot interaction data through a web-based interface. We examine the effect of different perceptual mappings between the human teacher and robot on the learning from demonstration. We show that humans are significantly more effective at teaching a robot to navigate a maze when presented with information that is limited to the robot's perception of the world, even though their task performance measurably suffers when contrasted with users provided with a natural and detailed raw video feed. Robots trained on such demonstrations learn more quickly, perform more accurately and generalize better. We also demonstrate a set of software tools for enabling internet-mediated human-robot interaction and gathering the large datasets that such crowdsourcing makes possible.

We describe our efforts to create infrastructure to enable web interfaces for robotics. Such interfaces will enable researchers and users to remotely access robots through the internet as well as expand the types of robotic applications available to users with web-enabled devices. This paper centers on rosjs, a lightweight Javascript binding for ROS, Willow Garage's robot middleware framework, rosjs exposes many of the capabilities of ROS, allowing application developers to write controllers that are executed through a web browser. We discuss how rosjs extends ROS and briefly overview some of the features it provides, rosjs has been instrumental in the creation of remote laboratories featuring the iRobot Create and the PR2. These facilities will be available to the community as experimental resources. We describe the overall goals of this project as well as provide a brief description of how rosjs was used to help create web interfaces for these facilities.

We evaluated a Kalman filter (KF) approach to modeling the physiology of internal temperature viewed through “noisy” non-invasive observations of heart rate. Human core body temperature (Tcore) is an important measure of thermal state, e.g., hypo- or hyperthermia, but is difficult to measure using non-invasive wearable sensors. We estimated parameters for a discrete KF model from data collected during several Military training events and from distance runners (n=38). Model performance was evaluated in 25 physically-active subjects who participated in various laboratory and field studies involving exercise of 2-to-8 h duration at ambient temperatures of 20 to 40°C. Overall, the KF model’s estimate of Tcore had a root mean square error of 0.30±0.13 ºC from the observed Tcore, and was within ± 0.5 ºC over 85% of the time. The benefit of the KF approach is that it requires only one input while current state of the art models typically require multiple inputs including individual anthropometrics, metabolic rate, clothing characteristics, and environmental conditions. This state estimation problem in computational physiology illustrates the potential for collaboration between the artificial intelligence and ambulatory physiological monitoring communities.

We propose to incrementally learn the segmentation of a demonstrated task into subtasks and the individual subtask policies themselves simultaneously. Previous robot learning from demonstration techniques have either learned the individual subtasks in isolation, combined known subtasks, or used knowledge of the overall task structure to perform segmentation. Our infinite mixture of experts approach instead automatically infers an appropriate partitioning (number of subtasks and assignment of data points to each one) directly from the data. We illustrate the applicability of our technique by learning a suitable set of subtasks from the demonstration of a finite-state machine robot soccer goal scorer.

We demonstrate that structured light-based depth sensing with standard perception algorithms can enable mobile peer-to-peer interaction between humans and robots. We posit that the use of recent emerging devices for depth-based imaging can enable robot perception of non-verbal cues in human movement in the face of lighting and minor terrain variations. Toward this end, we have developed an integrated robotic system capable of person following and responding to verbal and non-verbal commands under varying lighting conditions and uneven terrain. The feasibility of our system for peer-to-peer HRI is demonstrated through two trials in indoor and outdoor environments.

The Wizard of Oz experiment method has a long tradition of acceptance and use within the field of human-robot interaction. The community has traditionally downplayed the importance of interaction evaluations run with the inverse model: the human simulated to evaluate robot behavior, or Oz of Wizard. We argue that such studies play an important role in the field of human-robot interaction. We differentiate between methodologically rigorous human modeling and placeholder simulations using simplified human models. Guidelines are proposed for when Oz of Wizard results should be considered acceptable. This paper also describes a framework for describing the various permutations of Wizard and Oz states.

Human motion tracking is an important problem in computer vision. Most prior approaches have concentrated on efficient inference algorithms and prior motion models; however, few can explicitly account for physical plausibility of recovered motion. The primary purpose of this work is to enforce physical plausibility in the tracking of a single articulated human subject. Towards this end, we propose a full-body 3D physical simulation-based prior that explicitly incorporates motion control and dynamics into the Bayesian filtering framework. We consider the humanpsilas motion to be generated by a ldquocontrol looprdquo. In this control loop, Newtonian physics approximates the rigid-body motion dynamics of the human and the environment through the application and integration of forces. Collisions generate interaction forces to prevent physically impossible hypotheses. This allows us to properly model human motion dynamics, ground contact and environment interactions. For efficient inference in the resulting high-dimensional state space, we introduce exemplar-based control strategy to reduce the effective search space. As a result we are able to recover the physically-plausible kinematic and dynamic state of the body from monocular and multi-view imagery. We show, both quantitatively and qualitatively, that our approach performs favorably with respect to standard Bayesian filtering methods.

We are interested in transferring control policies for arbitrary tasks from a human to a robot. Using interactive demonstration via teleoperation as our transfer scenario, we cast learning as statistical regression over sensor-actuator data pairs. Our desire for interactive learning necessitates algorithms that are incremental and realtime. We examine locally weighted projection regression, a popular robotic learning algorithm, and sparse online Gaussian processes in this domain on one synthetic and several robot-generated data sets. We evaluate each algorithm in terms of function approximation, learned task performance, and scalability to large data sets.

Human control of high degree-of-freedom robotic systems, e.g. anthropomorphic robot hands, is often difficult due to the overwhelming number of variables that need to be specified. Previous work has addressed this sparse control problem by learning a high-dimensional manifold of robot poses to provide low-dimensional control subspaces. Such subspaces allow cursor control, or eventually decoding of neural activity, to drive a robotic hand. Considering previously identified problems related to noise in manifold learning, we introduce a method for denoising neighborhood graphs in order to embed hand motion into 2D spaces. We present results demonstrating our approach in the case of a synthetic swissroll as well as in the embeddings for interactive sparse control for several grasping tasks.

A method is presented for kinematic pose estimation and action recognition from monocular robot vision through the use of dynamical human motion vocabularies. We propose the utilization of dynamical motion vocabularies towards bridging the decision making of observed humans and information from robot sensing. Our motion vocabulary is comprised of learned primitives that structure the action space for decision making and describe human movement dynamics. Given image observations over time, each primitive infers on pose independently using its prediction density on movement dynamics in the context of a particle filter. Pose estimates from a set of primitives inferencing in parallel are arbitrated to estimate the action being performed. The efficacy of our approach is demonstrated through tracking and action recognition over extended motion trials. Results evidence the robustness of the algorithm with respect to unsegmented multi-action movement, movement speed, and camera viewpoint.

Ubiquitous robots need the ability to adapt their behaviour to the changing situations and demands they will encounter during their lifetimes. In particular, non-technical users must be able to modify a robot's behaviour to enable it to perform new, previously unknown tasks. Learning from demonstration is a viable means to transfer a desired control policy onto a robot and mixed-initiative control provides a method for smooth transitioning between learning and acting. We present a learning system (dogged learning) that combines learning from demonstration and mixed initiative control to enable lifelong learning for unknown tasks. We have implemented dogged learning on a Sony Aibo and successfully taught it behaviours such as mimicry and ball seeking

A critical challenge in robot learning from demonstration is the ability to map the behavior of the trainer onto the robot's existing repertoire of basic/primitive capabilities. Following a behavior-based approach, we aim to express a teacher's demonstration as a linear combination (or fusion) of the robot's primitives. We treat this problem as a state estimation problem over the space of possible linear fusion weights. We consider this fusion state to be a model of the teacher's control policy expressed with respect to the robot's capabilities. Once estimated under various sensory preconditions, fusion state estimates are used as a coordination policy for online robot control to imitate the teacher's decision making. A particle filter is used to infer fusion state from control commands demonstrated by the teacher and predicted by each primitive. The particle filter allows for inference under the ambiguity over a large space of likely fusion combinations and dynamic changes to the teacher's policy over time. We present results of our approach in a simulated and real world environments with a Pioneer 3DX mobile robot

We present an extension of Isomap nonlinear dimension reduction (Tenenbaum et al., 2000) for data with both spatial and temporal relationships. Our method, ST-Isomap, augments the existing Isomap framework to consider temporal relationships in local neighborhoods that can be propagated globally via a shortest-path mechanism. Two instantiations of ST-Isomap are presented for sequentially continuous and segmented data. Results from applying ST-Isomap to real-world data collected from human motion performance and humanoid robot teleoperation are also presented.

We present a unified methodology for humanoid robot control and activity, classification using motor primitives (Mataric, M, 2002), computationally efficient behaviors capable of perception and control. These primitives constitute a vocabulary for humanoid control capable of generating a large variety of complex movement through sequencing and superposition. We demonstrate how such primitives can be automatically derived from human motion-capture data, how they can be used to construct upperbody controllers, and how they can be applied to classification of observed humanoid behavior in real time.

An approach for model-free markerless motion capture of articulated kinematic structures is presented. This approach is centered our method for generating underlying nonlinear axes (or a skeleton curve) from the volume of an arbitrary rigid-body model. We describe the use of skeleton curves for deriving a kinematic model and motion (in the form of joint angles over time) from a captured volume sequence. Our motion capture method uses a skeleton curve, found in each frame of a volume sequence, to automatically determine kinematic postures. These postures are then aligned to determine a common kinematic model for the volume sequence. The derived kinematic model is then reapplied to each frame in the volume sequence to find the motion suited to this model. We demonstrate our method for several types of motion from synthetically generated volume sequences with arbitrary kinematic topology and human volume sequences captured from a set of multiple calibrated cameras.

In this paper, we present a method for motion generation from automatically derived behaviors for a humanoid robot. Behaviors are derived automatically by using the underlying spatio-temporal structure in motion. The derived behaviors are stored in a robot's long-term (or procedural) memory. New motions are generated from the derived ones with a search mechanism. In our approach, vision, speech recognition, short-term memory and decision-making operate in parallel with long-term memory in a unique architecture. This organization is intended for autonomous robot control and learning.

We address the problem of creating basis behaviors for modularizing humanoid robot control and representing human activity. These behaviors, called perceptual-motor primitives, serve as a substrate for linking a system's perception of human activities and the ability to perform those activities. We present a data-driven method for deriving perceptual-motor action and behavior primitives from human motion capture data. In order to find these primitives, we employ a spatio-temporal non-linear dimension reduction technique on a set of motion segments. From this transformation, motions representing the same action can be clustered and generalized. Further dimension reduction iterations are applied to derive extended-duration behaviors.