In the not too distant future, assistive humanoid robots will provide versatile assistance for coping with everyday life. In their interactions with humans, not only safety, but also security and privacy issues need to be considered. In this Blue Sky paper, we therefore argue that it is time to bring task planning and execution as a well-established field of robotics with access and usage control in the field of security and privacy closer together. In particular, the recently proposed activity-based view on access and usage control provides a promising approach to bridge the gap between these two perspectives. We argue that humanoid robots provide for specific challenges due to their task-universality and their use in both, private and public spaces. Furthermore, they are socially connected to various parties and require policy creation at runtime due to learning. We contribute first attempts on the architecture and enforcement layer as well as on joint modeling, and discuss challenges and a research roadmap also for the policy and objectives layer. We conclude that the underlying combination of decentralized systems' and smart environments' research aspects provides for a rich source of challenges that need to be addressed on the road to deployment.

Capturing scene dynamics and predicting the future scene state is challenging but essential for robotic manipulation tasks, especially when the scene contains both rigid and deformable objects. In this work, we contribute a simulation environment and generate a novel dataset for task-specific manipulation, involving interactions between rigid objects and a deformable bag. The dataset incorporates a rich variety of scenarios including different object sizes, object numbers and manipulation actions. We approach dynamics learning by proposing an object-centric graph representation and two modules which are Active Prediction Module (APM) and Position Prediction Module (PPM) based on graph neural networks with an encode-process-decode architecture. At the inference stage, we build a two-stage model based on the learned modules for single time step prediction. We combine modules with different prediction horizons into a mixed-horizon model which addresses long-term prediction. In an ablation study, we show the benefits of the two-stage model for single time step prediction and the effectiveness of the mixed-horizon model for long-term prediction tasks. Supplementary material is available at https://github.com/wengzehang/deformable_rigid_interaction_prediction

Understanding the effects of actions is essential for planning and executing robot tasks. By imagining possible action consequences, a robot can choose specific action parameters to achieve desired goal states. We present an approach for parametrizing pushing actions based on learning internal prediction models. These pushing actions must fulfill constraints given by a high-level planner, e. g., after the push the brown box must be to the right of the orange box. In this work, we represent the perceived scenes as object-centric graphs and learn an internal model, which predicts object pose changes due to pushing actions. We train this internal model on a large synthetic data set, which was generated in simulation, and record a smaller data set on the real robot for evaluation. For a given scene and goal state, the robot generates a set of possible pushing action candidates by sampling the parameter space and then evaluating the candidates by internal simulation, i. e., by comparing the predicted effect resulting from the internal model with the desired effect provided by the high-level planner. In the evaluation, we show that our model achieves high prediction accuracy in scenes with a varying number of objects and, in contrast to state-of-the-art approaches, is able to generalize to scenes with more objects than seen during training. In experiments on the humanoid robot ARMAR-6, we validate the transfer from simulation and show that the learned internal model can be used to manipulate scenes into desired states effectively.

Understanding spatial relations is a key element for natural human-robot interaction. Especially, a robot must be able to manipulate a given scene according to a human verbal command specifying desired spatial relations between objects. To endow robots with this ability, a suitable representation of spatial relations is necessary, which should be derivable from human demonstrations. We claim that polar coordinates can capture the underlying structure of spatial relations better than Cartesian coordinates and propose a parametric probability distribution defined in polar coordinates to represent spatial relations. We consider static spatial relations such as left of, behind, and near, as well as dynamic ones such as closer to and other side of, and take into account verbal modifiers such as roughly and a lot. We show that adequate distributions can be derived for various combinations of spatial relations and modifiers in a sample-efficient way using Maximum Likelihood Estimation, evaluate the effects of modifiers on the distribution parameters, and demonstrate our representation's usefulness in a pick-and-place task on a real robot.

Dynamic systems are a practical alternative to motion planning in executing robot actions. They are of particular interest in Learning from Demonstration, as here we aim to carry out actions in a certain fashion, without a model or in-depth knowledge about the world, which might be difficult to achieve with a planner. Using model-based dynamic systems in task space enables robots to flexibly reproduce demonstrated actions. Nevertheless, when dealing with mobile manipulators, we face the challenge of including the kinematic constraints of the robot in the action models. In this paper we propose to couple robot base and end-effector motions generated by arbitrary dynamical systems modulating the base velocity, while respecting the robots kinematic design. To this end we learn an approximation of the inverse reachability in closed form. In real-world robot experiments we demonstrate that we are able to maintain kinematically feasible trajectories in the presence of obstacles and in configurations differing profoundly from the training scene.

Robotic coverage path planning describes the problem of determining a configuration space trajectory for successively covering a specified workspace target area with the robot's end-effector. Performing coverage path planning for mobile robots further requires solving the problem of robot placement, i.e. determining of suitable robot base positions to perform the task. Finding an optimal solution is hard as both problems cannot be solved independently. Combined robot placement and coverage planning is particularly interesting if repositioning of the robot is costly or if simultaneous repositioning and end-effector motion is not desired. In this paper, we present a general approach for combined robot placement and coverage path planning that takes constraints like collision avoidance and static stability into account. In contrast to related approaches, we focus on mobile manipulation tasks that require a fixed placement for executing coverage trajectory segments. The approach is evaluated in two scenarios that exemplify the broad range of possible applications: The coverage of a building facade using a robotic manlift and the coverage of an industrial conveyer belt for maintenance tasks using the humanoid robot ARMAR-III.