Imitation learning enables robots to learn new tasks from human examples. One fundamental limitation while learning from humans is causal confusion. Causal confusion occurs when the robot’s observations include both task-relevant and extraneous information: for instance, a robot’s camera might see not only the intended goal, but also clutter and changes in lighting within its environment. Because the robot does not know which aspects of its observations are important a priori, it often misinterprets the human’s examples and fails to learn the desired task. To address this issue, we highlight that — while the robot learner may not know what to focus on — the human teacher does. In this paper we propose that the human proactively marks key parts of their task with small, lightweight beacons. Under our framework (RECON) the human attaches these beacons to task-relevant objects before providing demonstrations: as the human shows examples of the task, beacons track the position of marked objects. We then harness this offline beacon data to train a task-relevant state embedding. Specifically, we embed the robot’s observations to a latent state that is correlated with the measured beacon readings: in practice, this causes the robot to autonomously filter out extraneous observations and make decisions based on features learned from the beacon data. Our simulations and a real robot experiment suggest that this framework for human-placed beacons mitigates causal confusion. Indeed, we find that using RECON significantly reduces the number of demonstrations needed to convey the task, lowering the overall time required for human teaching. See videos here: https://youtu.be/oy85xJvtLSU

Assistive robot arms can help humans by partially automating their desired tasks. Consider an adult with motor impairments controlling an assistive robot arm to eat dinner. The robot can reduce the number of human inputs — and how precise those inputs need to be — by recognizing what the human wants (e.g., a fork) and assisting for that task (e.g., moving towards the fork). Prior research has largely focused on learning the human’s task and providing meaningful assistance. But as the robot learns and assists, we also need to ensure that the human understands the robot’s intent (e.g., does the human know the robot is reaching for a fork?). In this paper, we study the effects of communicating learned assistance from the robot back to the human operator. We do not focus on the specific interfaces used for communication. Instead, we develop experimental and theoretical models of a) how communication changes the way humans interact with assistive robot arms, and b) how robots can harness these changes to better align with the human’s intent. We first conduct online and in-person user studies where participants operate robots that provide partial assistance, and we measure how the human’s inputs change with and without communication. With communication, we find that humans are more likely to intervene when the robot incorrectly predicts their intent, and more likely to release control when the robot correctly understands their task. We then use these findings to modify an established robot learning algorithm so that the robot can correctly interpret the human’s inputs when communication is present. Our results from a second in-person user study suggest that this combination of communication and learning outperforms assistive systems that isolate either learning or communication. See videos here: https://youtu.be/BET9yuVTVU4

When humans interact with robots influence is inevitable. Consider an autonomous car driving near a human: the speed and steering of the autonomous car will affect how the human drives. Prior works have developed frameworks that enable robots to influence humans towards desired behaviors. But while these approaches are effective in the short-term (i.e., the first few human-robot interactions), here we explore long-term influence (i.e., repeated interactions between the same human and robot). Our central insight is that humans are dynamic: people adapt to robots, and behaviors which are influential now may fall short once the human learns to anticipate the robot's actions. With this insight, we experimentally demonstrate that a prevalent game-theoretic formalism for generating influential robot behaviors becomes less effective over repeated interactions. Next, we propose three modifications to Stackelberg games that make the robot's policy both influential and unpredictable. We finally test these modifications across simulations and user studies: our results suggest that robots which purposely make their actions harder to anticipate are better able to maintain influence over long-term interaction. See videos here: https://youtu.be/ydO83cgjZ2Q

What we feel from handling liquids in vessels produces unmistakably fluid tactile sensations. These stimulate essential perceptions in home, laboratory, or industrial contexts. Feeling fluid interactions from virtual fluids would similarly enrich experiences in virtual reality. We introduce Geppetteau, a novel string-driven weight shifting mechanism capable of providing perceivable tactile sensations of handling virtual liquids within a variety of vessel shapes. These mechanisms widen the range of augmentable shapes beyond the state-of-the-art of existing mechanical systems. In this work, Geppetteau is integrated into conical, spherical, cylindrical, and cuboid shaped vessels. Variations of these shapes are often used for fluid containers in our day-to-day. We studied the effectiveness of Geppetteau in simulating fine and coarse-grained tactile sensations of virtual liquids across three user studies. Participants found Geppetteau successful in providing congruent physical sensations of handling virtual liquids in a variety of physical vessel shapes and virtual liquid volumes and viscosities.

Mixed-reality haptic devices introduce a gateway to otherwise intangible virtual content, creating a life-like immersive experience. Congruent haptic sensation requires faithful integration of visual stimuli and perceived tactile sensation. Unfortunately, current commercial mixed-reality systems are unable to reproduce the physical sensation of fluid vessels, due to the shifting nature of fluid motion. To this end, we introduce SWISH, a novel type of ungrounded mixed-reality system, capable of affording the users a realistic haptic sensation of fluid behavior. We also present solutions to prominent challenges of rendering haptic fluid behavior, especially in coordinate translation and virtual adaptation to physical limitation. Our virtual-to-physical coupling uses Nvidia Flex's Unreal Engine integration, wirelessly controlling a motorized mechanical actuation system housed in a plastic "vessel''. In this paper we discuss the current state of SWISH and present results from our preliminary user study, followed by a description of our future planned phases.

Current VR/AR systems are unable to reproduce the physical sensation of fluid vessels, due to the shifting nature of fluid motion. To this end, we introduce SWISH, an ungrounded mixed-reality interface, capable of affording the users a realistic haptic sensation of fluid behaviors in vessels. The chief mechanism behind SWISH is in the use of virtual reality tracking and motor actuation to actively relocate the center of gravity of a handheld vessel, emulating the moving center of gravity of a handheld vessel that contains fluid. In addition to solving challenges related to reliable and efficient motor actuation, our SWISH designs place an emphasis on reproducibility, scalability, and availability to the maker culture. Our virtual-to-physical coupling uses Nvidia Flex's Unity integration for virtual fluid dynamics with a 3D printed augmented vessel containing a motorized mechanical actuation system. To evaluate the effectiveness and perceptual efficacy of SWISH, we conduct a user study with 24 participants, 7 vessel actions, and 2 virtual fluid viscosities in a virtual reality environment. In all cases, the users on average reported that the SWISH bucket generates accurate tactile sensations for the fluid behavior. This opens the potential for multi-modal interactions with programmable fluids in virtual environments for chemistry education, worker training, and immersive entertainment.