The capacity to capture and broadcast sports events with close to real-time volumetric reconstruction techniques opens exciting perspectives in how audiences can consume and interact with these contents. In this work, we propose the design of an real-time cinematography system that is capable of generating qualitative framing and editing of volumetric-captured content, by mimicking real broadcast footage. To illustrate our approach, we focus on the specific problem of cinematography for ring-based events such as Ultimate Fighting Championships (UFC). We start by extracting statistical features from hours of real footage to understand the specific framing and cutting behaviors of real broadcast directors. We then exploit these features in a real-time editing system, not only to replicate the behaviors in existing broadcasting, but also to generalize to novel camera layouts. We demonstrate our approach on the volumetric reconstruction of UFC fights, compare our results with baseline methods, and report different qualitative and quantitative evaluations.

A human handing over an object modulates their grasp and movements to accommodate their partner's capa-bilities, which greatly increases the likelihood of a successful transfer. State-of-the-art robot behavior lacks this level of user understanding, resulting in interactions that force the human partner to shoulder the burden of adaptation. This paper investigates how visual occlusion of the object being passed affects the subjective perception and quantitative performance of the human receiver. We performed an experiment in virtual reality where seventeen participants were tasked with repeatedly reaching to take a tool from the hand of a robot; each of the three tested objects (hammer, screwdriver, scissors) was presented in a wide variety of poses. We carefully analysed the user's hand and head motions, the time to grasp the object, and the chosen grasp location, as well as participants' ratings of the grasp they just performed. Results show that initial visibility of the handle significantly increases the reported holdability and immediate usability of a tool. Furthermore, a robot that offers objects so that their handles are more occluded forces the receiver to spend more time in planning and executing the grasp and also lowers the probability that the tool will be grasped by the handle. Together these findings indicate that robots can more effectively support their human work partners by increasing the visibility of the intended grasp location of objects being passed.

The synthesis of realistic robot grasps in a simulated environment is pivotal in generating datasets that support sim-to-real transfer learning. In a step toward achieving this goal, we propose PrendoSim, an open-source grasp generator based on a proxy-hand simulation that employs NVIDIA’s physics engine (PhysX) and the recently released articulated-body objects developed by Unity (https://prendosim.github.io). We present the implementation details, the method used to generate grasps, the approach to operationally evaluate stability of the generated grasps, and examples of grasps obtained with two different grippers (a parallel jaw gripper and a three-finger hand) grasping three objects selected from the YCB dataset (hammer, screwdriver, and scissors). Compared to simulators proposed in the literature, PrendoSim balances grasp realism and ease of use, displaying an intuitive interface and enabling the user to produce a large and varied dataset of stable grasps.

This article surveys the literature on human–robot object handovers. A handover is a collaborative joint action, where an agent, the giver, gives an object to another agent, the receiver. The physical exchange starts when the receiver first contacts the object held by the giver and ends when the giver fully releases the object to the receiver. However, important cognitive and physical processes begin before the physical exchange, including initiating implicit agreement with respect to the location and timing of the exchange. From this perspective, we structure our review into the two main phases delimited by the aforementioned events: a prehandover phase and the physical exchange. We focus our analysis on the two actors (giver and receiver) and report the state of the art of robotic givers (robot-to-human handovers) and the robotic receivers (human-to-robot handovers). We report a comprehensive list of qualitative and quantitative metrics commonly used to assess the interaction. While focusing our review on the cognitive level (e.g., prediction, perception, motion planning, and learning) and the physical level (e.g., motion, grasping, and grip release) of the handover, we also discuss safety. We compare the behaviors displayed during human-to-human handovers to the state of the art of robotic assistants and identify the major areas of improvement for robotic assistants to reach performance comparable to human interactions. Finally, we propose a minimal set of metrics that should be used in order to enable a fair comparison among the approaches.

This paper presents a path planner, which enables a nonholonomic mobile manipulator to move its end-effector on an observed surface with a constrained orientation, given start and destination points. A partial point cloud of the environment is captured using a vision-based sensor, but no prior knowledge of the surface shape is assumed. We consider the multi-objective optimisation problem of finding robot paths which account for the nonholonomic constraints of the base, maximise the robot's manipulability throughout the motion, while also minimising surface-distance travelled between the two points. This work has application in industrial problems of rough robotic cutting, e.g. demolition of legacy nuclear plants, where dismantling does not require a precise path. We show how our approach embeds the nonholonomic constraints of the mobile platform into an extended Jacobian, while additionally encoding the constraint that the end-effector must remain in contact with the cut surface throughout the motion. We use this constrained Jacobian to plan a time-series of robot configurations. Additionally, we show how our novel cost function is suitable for combining with a variety of well-known path planners, such as RRT*. We present several empirical experiments in different scenarios, where a simulated non-holonomic mobile manipulator follows a trajectory, which is generated on noisy point clouds derived from real depth-camera images of real objects. Our planner (RRT*-CRMM) enables successful task completion by optimising the path over the travelled distance, the manipulability of the arm, and the movements of the mobile base.

This paper presents an assisted telemanipulation approach with integrated grasp planning. It also studies how the human teleoperation performance benefits from the incorporated visual and haptic cues while manipulating objects in cluttered environments. The developed system combines the widely used master-slave teleoperation with our previous model-free and learning-free grasping algorithm by means of a dynamic grasp re-ranking strategy and a semi-autonomous reach-to-grasptrajectory guidance. The proposed re-ranking metric helps in dynamically updating the stable grasps based on the current state of the slave device. The trajectory guidance system assists in maintaining smooth trajectory by controlling the haptic forces. A virtual pose controller has been integrated with the guidance scheme to automatically correct the end-effector orientation while reaching towards the grasp. Various experiments are conducted evaluating the proposed method using a six degrees of freedom (dof) haptic master and a seven dof slave robot. Results obtained with these tests along with the results gathered from the performed human-factor trials demonstrate the efficiency of our method in terms of objective metrics of task completion, and also subjective metrics of user experience.

This paper addresses the problem of grasping arbitrarily shaped objects, observed as partial point-clouds, without requiring: models of the objects, physics parameters, training data, or other a-priori knowledge. A grasp metric is proposed based on Local Contact Moment (LoCoMo). LoCoMo combines zero-moment shift features, of both hand and object surface patches, to determine local similarity. This metric is then used to search for a set of feasible grasp poses with associated grasp likelihoods. LoCoMo overcomes some limitations of both classical grasp planners and learning-based approaches. Unlike force-closure analysis, LoCoMo does not require knowledge of physical parameters such as friction coefficients, and avoids assumptions about fingertip contacts, instead enabling robust contacts of large areas of hand and object surface. Unlike more recent learning-based approaches, LoCoMo does not require training data, and does not need any prototype grasp configurations to be taught by kinesthetic demonstration. We present results of real-robot experiments grasping 21 different objects, observed by a wrist-mounted depth camera. All objects are grasped successfully when presented to the robot individually. The robot also successfully clears cluttered heaps of objects by sequentially grasping and lifting objects until none remain.

This paper represents a step towards vision-based manipulation of plastic materials. Manipulating deformable objects is made challenging by: 1) the absence of a model for the object deformation, 2) the inherent difficulty of visual tracking of deformable objects, 3) the difficulty in defining a visual error and 4) the difficulty in generating control inputs to minimise the visual error. We propose a novel representation of the task of manipulating deformable objects. In this preliminary case study, the shaping of kinetic sand, we assume a finite set of actions: pushing, tapping and incising. We consider that these action types affect only a subset of the state, i.e., their effect does not affect the entire state of the system (specialized actions). We report the results of a user study to validate these hypotheses and release the recorded dataset. The actions (pushing, tapping and incising) are clearly adopted during the task, although it is clear that 1) participants use also mixed actions and 2) actions' effects can marginally affect the entire state, requesting a relaxation of our specialized actions hypothesis. Moreover, we compute task errors and corresponding control inputs (in the image space) using image processing. Finally, we show how machine learning can be applied to infer the mapping from error to action on the data extracted from the user study.

This paper presents a vision-based approach for estimating the configuration of, and providing control signals for, an under-sensored robot manipulator using a single monocular camera. Some remote manipulators, used for decommissioning tasks in the nuclear industry, lack proprioceptive sensors because electronics are vulnerable to radiation. Additionally, even if proprioceptive joint sensors could be retrofitted, such heavy-duty manipulators are often deployed on mobile vehicle platforms, which are significantly and erratically perturbed when powerful hydraulic drilling or cutting tools are deployed at the end-effector. In these scenarios, it would be beneficial to use external sensory information, e.g. vision, for estimating the robot configuration with respect to the scene or task. Conventional visual servoing methods typically rely on joint encoder values for controlling the robot. In contrast, our framework assumes that no joint encoders are available, and estimates the robot configuration by visually tracking several parts of the robot, and then enforcing equality between a set of transformation matrices which relate the frames of the camera, world and tracked robot parts. To accomplish this, we propose two alternative methods based on optimisation. We evaluate the performance of our developed framework by visually tracking the pose of a conventional robot arm, where the joint encoders are used to provide ground-truth for evaluating the precision of the vision system. Additionally, we evaluate the precision with which visual feedback can be used to control the robot's end-effector to follow a desired trajectory.

This paper addresses the problem of constrained motion for a manipulator performing a task while in contact with the environment, and investigates two force control frameworks, one based on projected inverse dynamics, and one based on optimal control. Firstly, we propose a control method based on projected inverse dynamics, which directly exploits the contact constraints to minimise the instantaneous joint torques needed to perform a task. Secondly, we propose an optimal control strategy which provides a tool to minimise the joint torques over an interval of time. We show how contact constraints can be used as optimisation constraints in the definition of the problem, and how to formulate the optimal control problem directly using projected dynamics. Initially we explore a positional control problem, where the robot is required to follow a desired path, and show that both of the proposed methods can satisfy the positional task while significantly reducing the joint torques as compared to simple kinematic control and also classical inverse dynamics control. We also show that the proposed optimal control method outperforms the pure projected inverse dynamics method in terms of minimising the required joint torques. We then show how each method can be extended to follow a desired path while also exerting a desired contact force. Again, the method incorporating optimal control is shown to satisfy the task requirements with significantly smaller commanded torques than the pure projected inverse dynamics method. To confirm the analysis, and demonstrate proof of concept, we present the results of empirical experiments with a simulated 3-degree-of-freedom planar manipulator which is constrained to move while in contact with a rigid surface.