Figure 1: In this paper, we present NarraGuide, an LLM-based narrative mobile robot that guides users to explore a remote physical place in real-time and uses dialogue to deliver rich location-based information to enhance the experience.

Robots in real-world environments continuously engage with multiple users and encounter changes that lead to unexpected conflicts in fulfilling user requests. Recent technical advancements (e.g., large-language models (LLMs), program synthesis) offer various methods for automatically generating repair plans that address such conflicts. In this work, we understand how automated repair and explanations can be designed to improve user experience with robot failures through two user studies. In our first, online study (n = 162), users expressed increased trust, satisfaction, and utility with the robot performing automated repair and explanations. However, we also identified risk factors—safety, privacy, and complexity—that require adaptive repair strategies. The second, in-person study (n = 24) elucidated distinct repair and explanation strategies depending on the level of risk severity and type. Using a design-based approach, we explore automated repair with explanations as a solution for robots to handle conflicts and failures, complemented by adaptive strategies for risk factors. Finally, we discuss the implications of incorporating such strategies into robot designs to achieve seamless operation among changing user needs and environments.

Generating feasible robot motions in real-time requires achieving multiple tasks (i.e., kinematic requirements) simultaneously. These tasks can have a specific goal, a range of equally valid goals, or a range of acceptable goals with a preference toward a specific goal. To satisfy multiple and potentially competing tasks simultaneously, it is important to exploit the flexibility afforded by tasks with a range of goals. In this paper, we propose a real-time motion generation method that accommodates all three categories of tasks within a single, unified framework and leverages the flexibility of tasks with a range of goals to accommodate other tasks. Our method incorporates tasks in a weighted-sum multiple-objective optimization structure and uses barrier methods with novel loss functions to encode the valid range of a task. We demonstrate the effectiveness of our method through a simulation experiment that compares it to state-of-the-art alternative approaches, and by demonstrating it on a physical camera-in-hand robot that shows that our method enables the robot to achieve smooth and feasible camera motions.

We investigate how robotic camera systems can offer new capabilities to computer-supported cooperative work through the design, development, and evaluation of a prototype system called Periscope. With Periscope, a local worker completes manipulation tasks with guidance from a remote helper who observes the workspace through a camera mounted on a semi-autonomous robotic arm that is co-located with the worker. Our key insight is that the helper, the worker, and the robot should all share responsibility of the camera view-an approach we call shared camera control. Using this approach, we present a set of modes that distribute the control of the camera between the human collaborators and the autonomous robot depending on task needs. We demonstrate the system's utility and the promise of shared camera control through a preliminary study where 12 dyads collaboratively worked on assembly tasks. Finally, we discuss design and research implications of our work for future robotic camera systems that facilitate remote collaboration.

In telemanipulation, showing the user multiple views of the remote environment can offer many benefits, although such different views can also create a problem for control. Systems must either choose a single fixed control frame, aligned with at most one of the views or switch between view-aligned control frames, enabling view-aligned control at the expense of switching costs. In this paper, we explore the trade-off between these options. We study the feasibility, benefits, and drawbacks of switching the user's control frame to align with the actively used view during telemanipulation. We additionally explore the effectiveness of explicit and implicit methods for switching control frames. Our results show that switching between multiple view-specific control frames offers significant performance gains compared to a fixed control frame. We also find personal preferences for explicit or implicit switching based on how participants planned their movements. Our findings offer concrete design guidelines for future multi-camera interfaces.

Drones can provide a minimally-constrained adapting camera view to support robot telemanipulation. Furthermore, the drone view can be automated to reduce the burden on the operator during teleoperation. However, existing approaches do not focus on two important aspects of using a drone as an automated view provider. The first is how the drone should select from a range of quality viewpoints within the workspace (e.g., opposite sides of an object). The second is how to compensate for unavoidable drone pose uncertainty in determining the viewpoint. In this paper, we provide a nonlinear optimization method that yields effective and adaptive drone viewpoints for telemanipulation with an articulated manipulator. Our first key idea is to use sparse human-in-the-loop input to toggle between multiple automatically-generated drone viewpoints. Our second key idea is to introduce optimization objectives that maintain a view of the manipulator while considering drone uncertainty and the impact on viewpoint occlusion and environment collisions. We provide an instantiation of our drone viewpoint method within a drone-manipulator remote teleoperation system. Finally, we provide an initial validation of our method in tasks where we complete common household and industrial manipulations.

In this paper, we design and evaluate a novel form of visually-simulated haptic feedback cue for communicating weight in robot teleoperation. We propose that a visuo-proprioceptive cue results from inconsistencies created between the user's visual and proprioceptive senses when the robot's movement differs from the movement of the user's input. In a user study where participants teleoperate a six-DoF robot arm, we demonstrate the feasibility of using such a cue for communicating weight in four telemanipulation tasks to enhance user experience and task performance.

Human demonstrations are important in a range of robotics applications, and are created with a variety of input methods. However, the design space for these input methods has not been extensively studied. In this paper, focusing on demonstrations of hand-scale object manipulation tasks to robot arms with two-finger grippers, we identify distinct usage paradigms in robotics that utilize human-to-robot demonstrations, extract abstract features that form a design space for input methods, and characterize existing input methods as well as a novel input method that we introduce, the instrumented tongs. We detail the design specifications for our method and present a user study that compares it against three common input methods: free-hand manipulation, kinesthetic guidance, and teleoperation. Study results show that instrumented tongs provide high quality demonstrations and a positive experience for the demonstrator while offering good correspondence to the target robot.

In this section, we discuss some extensions of Section III and expand on the limitations mentioned in Section VI of the main article.

We present an offline method to generate smooth, feasible motion for robot arms such that end-effector pose goals of a 6-DoF path are matched within acceptable limits specified by the user. Our approach aims to accurately match the position and orientation goals of the given path, and allows deviation from these goals if there is danger of self-collisions, joint-space discontinuities or kinematic singularities. Our method generates multiple candidate trajectories, and selects the best by incorporating sparse user input that specifies what kinds of deviations are acceptable. We apply our method to a range of challenging paths and show that our method generates solutions that achieve smooth, feasible motions while closely approximating the given pose goals and adhering to user specifications.

In this paper, we propose a simple yet effective, computer vision based method for estimating respiration rate in real-time from the thoraco-abdominal video of a subject being monitored. The periodic motion of the chest wall of the subject is captured through the optical flow in the video sequence. The frequency of the chest wall motion is estimated by performing a Fourier analysis on the time sequence of the optical flow vectors. We present how to perform the Fourier analysis recursively leveraging the sequential nature of a video to speed-up our method. Unlike other methods, our method does not require a selection of region of interest because our method aggregates out-of-phase optical flows by factoring out the their relative phase differences. Unlike many existing methods, our method does not require to be reinitialized if the subject changes the posture during the observation. Our method works with different postures and different views (frontal, side, etc.) of the subject. Our method is very simple to implement. We evaluate our method against an impedance pneumograph and demonstrate the high accuracy of our method on thoracoabdominal videos of many subjects wearing a wide variety of clothing.

In this paper, we present a vision based method for respiration rate estimation which can automatically adapt to the scene changes. We capture a video of the subjects thoraco-abdominal region and compute optical flow field at each video frame. The optical flow field changes periodically with the periodic chest wall motion. The pattern of the chest wall motion is captured through the estimation of a principal flow field. The principal flow field is automatically updated with time to cope with the scene changes. Thus, our method can adapt itself to the changes of the posture of a subject. Besides, in our method we do not need to select any region of interest unlike other methods. Yet our method is computationally very inexpensive and simple to implement. We test our method on many human volunteers with a wide variety of their clothing. We compare our method against the gold standard method of impedance pneumography and have found a very high accuracy.

In this manuscript, we demonstrate the estimation of the respiratory signal from a thoraco-abdominal video of a person using an LSTM based learning model. The video is captured with a regular consumer grade camera and the respiratory signal is recorded using an impedance pneumograph simultaneously. The optical flow capturing the motion of the chest wall during an inhalation and exhalation is extracted at each video frame and fed as features to the LSTM model. We then train the LSTM model to estimate the respiratory signal. We fix the design parameters of the LSTM model based on cross-validation. The comparison between the predicted and the ground-truth pneumograph signal shows that the trained LSTM model predicts the respiratory signal quite accurately achieving a strong amplitude correlation of 0.74. Moreover, we estimate the respiration rates from the predicted respiratory signal. The estimated respiration rates have less than ±3 BPM error for more than 95% cases. Also, we achieve a correlation of 0.9 between the ground-truth respiration rates and the estimated respiration rates.