In this paper, we tackle the problem of human-robot coordination in sequences of manipulation tasks. Our approach integrates hierarchical human motion prediction with Task and Motion Planning (TAMP). We first devise a hierarchical motion prediction approach by combining Inverse Reinforcement Learning and short-term motion prediction using a Recurrent Neural Network. In a second step, we propose a dynamic version of the TAMP algorithm Logic-Geometric Programming (LGP) [1]. Our version of Dynamic LGP, replans periodically to handle the mismatch between the human motion prediction and the actual human behavior. We assess the efficacy of the approach by training the prediction algorithms and testing the framework on the publicly available MoGaze dataset [2].

Human movement prediction is difficult as humans naturally exhibit complex behaviors that can change drastically from one environment to the next. In order to alleviate this issue, we propose a prediction framework that decouples short-term prediction, linked to internal body dynamics, and long-term prediction, linked to the environment and task constraints. In this work we investigate encoding short-term dynamics in a recurrent neural network, while we account for environmental constraints, such as obstacle avoidance, using gradient-based trajectory optimization. Experiments on real motion data demonstrate that our framework improves the prediction with respect to state-of-the-art motion prediction methods, as it accounts to beforehand unseen environmental structures. Moreover we demonstrate on an example, how this framework can be used to plan robot trajectories that are optimized to coordinate with a human partner.

Motion prediction in unstructured environments is a difficult problem and is essential for safe and efficient human-robot space sharing and collaboration. In this work, we focus on manipulation movements in environments such as homes, workplaces or restaurants, where the overall task and environment can be leveraged to produce accurate motion prediction. For these cases we propose an algorithmic framework that accounts explicitly for the environment geometry based on a model of affordances and a model of short-term human dynamics both trained on motion capture data. We propose dedicated function networks for graspability and placebility affordances and we make use of a dedicated RNN [1] for short-term motion prediction. The prediction of grasp and placement probability densities are used by a constraint-based trajectory optimizer to produce a full-body motion prediction over the entire horizon. We show by comparing to ground truth data that we achieve similar performance for full-body motion predictions as using oracle grasp and place locations.

It is possible to build highly accurate and mobile robots able to perceive their environment. Such robots can be used to assist humans in everyday tasks to reduce their workload. As a consequence, communication and interaction between humans and robots is becoming more important. In this paper we present a system for human-robot collaboration for on-table tasks. Due to its extensible design our system serves us as a base for further investigations in human-robot collaboration. We used the system to implement five action-selection strategies for the robot:proactive, autonomous, reactive, human-requested and human-commands. We conducted a pilot study to compare the interaction modes during a task in which the human and the robot build a bridge using blocks. The results of the pilot study indicate that for the simple bridge-building task, people prefer to interact with a robot using the proactive action-selection strategy. The completion of the pilot study indicates that the system is useful for human-robot collaboration studies. Several limitations have been identified that will be addressed in future developments.

Large high-resolution displays (LHRDs) present new opportunities for interaction design in areas such as interactive visualization and data analytics. Design processes for graphical interfaces for LHRDs are still challenging. In this paper, we explore the design space of graphical interfaces for LHRDs by engaging in the creation of four prototypes for supporting office work. Specifically, we investigate how users can effectively manage application windows on LHRDs using four window alignment techniques: curved zooming, window grouping, window spinning and side pane navigation. We present the design and implementation of these window alignment techniques in a sample office application. Based on a mixed-methods user study of our prototypes, we contribute insights on designing future graphical interfaces for LHRDs. We show that potential users appreciate techniques, which enhance focus switching without changing the spatial relation between related windows.

Smart devices have arrived in our everyday lives. Being able to notify the user about events is a core feature of these devices. Related work investigated interruptions caused by notifications on single devices. In this paper, we investigate notifications in multi-device environments by analyzing the results of a week-long in-situ study with 16 participants. We used the Experience Sampling Method (ESM) and recorded the participants' interaction with smartphones, smartwatches, tablets and PCs. Disregarding the type or content of notifications, we found that the smartphone is the preferred device on which to be notified. Further, we found that the proximity to the device, whether it is currently being used and the user's current location can be used to predict if the user wants to receive notifications on a device. The findings can be used to design future multi-device aware smart notification systems.