Head-mounted displays (HMDs) in room-scale virtual reality are usually tracked using inside-out visual SLAM algorithms. Alternatively, to track the motion of the HMD with respect to a fixed real-world reference frame, an outside-in instrumentation like a motion capture system can be adopted. However, outside-in tracking systems may temporarily lose tracking as they suffer by occlusion and blind spots. A possible solution is to adopt a hybrid approach where the inside-out tracker of the HMD is augmented with an outside-in sensing system. On the other hand, when the tracking signal of the outside-in system is recovered after a loss of tracking the transition from inside-out tracking to hybrid tracking may generate a discontinuity, i.e a sudden change of the virtual viewpoint, that can be uncomfortable for the user. Therefore, hybrid tracking solutions for HMDs require advanced sensor fusion algorithms to obtain a smooth transition. This work proposes a method for hybrid tracking of a HMD with smooth transitions based on an adaptive complementary filter. The proposed approach can be configured with several parameters that determine a trade-off between user experience and tracking error. A user study was carried out in a room-scale virtual reality environment, where users carried out two different tasks while multiple signal tracking losses of the outside-in sensor system occurred. The results show that the proposed approach improves user experience compared to a standard Extended Kalman Filter, and that tracking error is lower compared to a state-of-the-art complementary filter when configured for the same quality of user experience.

In automated robot assembly and industrial palletization tasks it is crucial to ensure a good accuracy while placing objects given a planned target pose. To achieve this goal post-grasp strategies may be adopted that estimate or correct the displacement error between the expected and the actual grasp pose of an object. Standard in-hand post-grasp strategies require sensors like cameras to estimate the displacement error while the object is grasped. Other approaches are based on object re-grasping using special jigs and fixtures. In this paper a novel post-grasp strategy is proposed, where the displacement error is estimated in-hand by detecting collisions between the grasped object and a fixed peg. The proposed method estimates the displacement error after few collisions. The approach was evaluated on cardboard boxes thanks to the internal forcetorque sensor of a collaborative robot, achieving sub-millimeter and sub-degree residual placement errors.

This work investigates the problem of 6D pose estimation and robot bin picking of non-Lambertian reflecting objects based on a low-cost commercial 3D sensor. In particular, we address the task of estimating the pose of small metal hydraulic components of the same type, randomly placed in a bin. The system consists of a robot arm and an RGB-D sensor in eye-in-hand configuration. The proposed method works in two main phases. In the first phase a Convolutional Neural Network (CNN) extracts the bounding boxes of the objects contained in the bin from a single RGB image of the environment. In the second phase the 6D pose of the objects is estimated using a dense 3D reconstruction of the scene and by applying a template matching algorithm from multiple virtual views of the object CAD model. Experimental results have been carried out on a dataset containing both RGB and depth images. Preliminary experiments are also reported in the real setup.

Depth images usually contain pixels with invalid measurements. This paper presents a deep learning approach that receives as input a partially-known volumetric model of the environment and a camera pose, and it predicts the probability that a pixel would contain a valid depth measurement if a camera was placed at the given pose. The proposed network architecture consists of a 3D Convolutional Neural Network (CNN) module and a 2D CNN module, connected by a deep learning attention-based projection module. The method was integrated into a CNN-based probabilistic Next Best View plan-ner, resulting in a more realistic prediction of the information gain for each possible viewpoint with respect to state of the art approaches. Experiments were carried out in tabletop scenarios using a robot manipulator with an eye-in-hand depth camera.

In this work a multi-robot system is presented for people detection and tracking in automated warehouses. Each Automated Guided Vehicle (AGV) is equipped with multiple RGB cameras that can track the workers’ current locations on the floor thanks to a neural network that provides human pose estimation. Based on the local perception of the environment each AGV can exploit information about the tracked people for self-motion planning or collision avoidance.Additionally, data collected from each robot contributes to a global people detection and tracking system. A warehouse central management software fuses information received from all AGVs into a map of the current locations of workers. The estimated locations of workers are sent back to the AGVs to prevent potential collision. The proposed method is based on two-level hierarchy of Kalman filters. Experiments performed in a real warehouse show the viability of the proposed approach.

This work addresses the task of robot depalletizing by means of a mobile manipulator, taking into account the problem of localizing the boxes to be removed from the pallet and a manipulation strategy that allows to pull the boxes without lifting them with the robot arm. The depalletizing task is of particular interest in the industrial scenario in order to increase efficiency, flexibility and economic affordability of automatic warehouses.The proposed solution makes use of a multi-sensor vision system and a force-controlled collaborative robot in order to detect the boxes on the pallet and to control the robot interaction with the boxes to be removed. The vision system comprises a fixed 3D Time-of-flight camera and an eye-in-hand 2D camera. Preliminary experimental results performed on a laboratory setup with a fixed-based robotic manipulator are reported to show the effectiveness of the perception and control system.

This article presents the first surfel-based method for multi-view 3D reconstruction of the boundary between known and unknown space. The proposed approach integrates multiple views from a moving depth camera and it generates a set of surfels that encloses observed empty space, i.e., it models both the boundary between empty and occupied space, and the boundary between empty and unknown space. One novelty of the method is that it does not require a persistent voxel map of the environment to distinguish between unknown and empty space. The problem is solved thanks to an incremental algorithm that computes the Boolean union of two surfel bounded volumes: the known volume from previous frames and the space observed from the current depth image. A number of strategies were developed to cope with errors in surfel position and orientation. The method, implemented on CPU and GPU, was evaluated on real data acquired in indoor scenarios, and it was compared against state of the art approaches. Results show that the proposed method has a low number of false positive and false negatives, it is faster than a standard volumetric algorithm, it has a lower memory consumption, and it scales better in large environments.

This work presents an approach for humanoid Next Best View (NBV) planning that exploits full body motions to observe objects occluded by obstacles. The task is to explore a given region of interest in an initially unknown environment. The robot is equipped with a depth sensor, and it can perform both 2D and 3D mapping. As main contribution with respect to previous work, the proposed method does not rely on simple motions of the head and it was evaluated in real environments. The robot is guided by two behaviors: a target behavior that aims at observing the region of interest by exploiting body movements primitives, and an exploration behavior that aims at observing other unknown areas. Experiments show that the humanoid is able to peer around obstacles to reach a favourable point of view. Moreover, the proposed approach results in a more complete reconstruction of objects than a conventional algorithm that only changes the orientation of the head.

This letter presents a novel approach for automatic floorplan generation of indoor environments. The floorplan is computed from a large-scale point cloud obtained from registered terrestrial laser scans. In contrast to previous work, the proposed method does not assume either a flat ground, or flat ceiling, or planar walls. Moreover, the method exploits the detection of structural elements, i.e., parts having a constant section over the entire height of the building (such as walls and columns), which is beneficial in cluttered regions to compensate for the lack of information due to occlusions. The evaluation was performed in complex buildings, like industrial warehouses, that include machines and pallet racks, whose layout is included in the generated floorplan. The algorithm achieves a floorplan reconstruction with accuracy comparable to the resolution of the adopted sensor. Results are also compared to a ground truth acquired using a total station.

A novel formulation of the set cover problem is presented to find the optimal placement of the scan stations in a terrestrial laser scanning survey. The problem is formulated in 2-D by including sensor-based constraints such as coverage and overlap. The coverage constraint ensures a minimum density of horizontal scan lines on the ground. The overlap constraint enables automatic scan alignment and registration. The optimization problem takes into account both environment occlusions and a maximum allowed incidence angle of the laser beams. The adopted laser model includes fixed parameters such as laser height, angular resolution, field of view, and minimum and maximum sensor range. The sensor placement problem is solved using a numerical approach implemented on graphics processing unit (GPU). Thanks to the GPU acceleration, experiments have been performed in large-scale environments with internal structures.

Virtual reality (VR) will play an important role in the factory of the future. In this paper, an immersive and interactive VR system is presented for3D visualization of automated guided vehicles (AGVs) moving in a warehouse. The environment model consists of a large scale point cloud obtained through a Terrestrial Laser Scanning (TLS) survey. Realistic AGV animation is achieved thanks to the extraction of an accurate model of the ground. Visualization of AGV safety zones is also supported.Moreover, the system enables real-time collision detection between the 3D vehicle model and the point cloud model of the environment. Collision detection is useful for checking the feasibility of a specified vehicle path. Efficient techniques for dynamic loading of massive point cloud data have been developed to speed up rendering and collision detection. The VR system can be used to assist the design of automated warehouses and to show customers what their future industrial plant would look like.

This paper presents a user-friendly approach for multi-label point cloud annotation. The method requires the user to select sparse control points belonging to the objects through a mouse-based interface. Multiple control points may be assigned to the same label. The software utilizes the selected control points to perform a segmentation algorithm on the neighborhood graph, based on shortest path tree. The user is provided a real-time feedback about the result, and can correct segmentation errors. In contrast to previous work the method supports multi-label annotation of unorganized point clouds. The method has been evaluated by multiple users and compared with a standard rectangle-based selection technique. Results indicate that the proposed method is perceived as easier to use, and that it allows a faster segmentation even in complex scenarios with occlusions.

This paper presents an algorithm for surfel color and position enhancement from RGB-D data acquired across multiple image frames. Surfel-based reconstruction algorithms associate each RGB-D frame pixel to a surfel in the model. As the reconstruction progresses, surfel color and position are the average of all observations. Our proposed algorithm is designed to enhance position discontinuities and to produce sharper colors, to facilitate subsequent segmentation steps on the 3D model. During reconstruction, several colors and positions are tracked for each surfel. Only at the end of reconstruction phase the most frequent value is chosen through a Winner Takes All policy. The result has been compared to the standard averaging policy of reconstruction algorithms. Experiments have been performed using both Flood Fill and Supervoxel-LCCP segmentation and by applying two segmentation evaluation metrics. Results show that the proposed method is suitable to enhance a surfel-based model for object segmentation purposes.

This paper describes the design and evaluation of a vision system conceived to provide perception in Autonomous Underwater Vehicle (AUV) intervention tasks. Due to the accuracy requirements inherent in manipulation tasks, high performance vision systems, enabling adequate perception capabilities, are needed to cope with underwater interventions. However, vision systems are challenged by the difficulties and variability of underwater environments as well as by the need to operate in a sealed canister. The vision system described in this paper addresses design issues like computational performance, energy power consumption, heat dissipation, and network capabilities. Even though the system has been designed to support stereovision, experiments in several underwater contexts have shown that stereovision is seldom applicable, due to the many problems faced by light propagation in water. Developing a system for underwater operation emphasizes the need for tradeoffs between computational performance and power consumption and dissipation, as well as the need for flexibility to support multiple vision processing pipelines and adapt to the specific underwater context.

This work proposes a method for autonomous robot exploration of unknown objects by sensor fusion of 3D range data. The approach aims at overcoming the physical limitation of the minimum sensing distance of range sensors. Two range sensors are used with complementary characteristics mounted in eye-in-hand configuration on a robot arm. The first sensor operates at mid-range and is used in the initial phase of exploration when the environment is unknown. The second sensor, which provides short-range data, is used in the following phase where the objects are explored at close distance through next best view planning. Next best view planning is performed using a volumetric representation of the environment. A complete point cloud model of each object is finally computed by global registration of all object observations including mid-range and short range views. In experiments performed in environments with multiple rigid objects the global registration algorithm has proven more accurate than a standard sequential registration approach.

A method is proposed for gesture recognition and humanoid imitation based on Functional Principal Component Analysis (FPCA). FPCA is a statistical technique of functional data analysis that has never been applied before for humanoid imitation. In functional data analysis data (e.g. gestures) are functions that can be considered as observations of a random variable on a functional space. FPCA is an extension of multivariate PCA that provides functional principal components which describe the modes of variation in the data. In the proposed approach FPCA is used for both unsupervised clustering of training data and gesture recognition. In this work we focus on arm gesture recognition. Human hand paths in Cartesian space are reconstructed from inertial sensors. Recognized gestures are reproduced by a small humanoid robot. The FPCA algorithm has also been compared to a state of the art algorithm for gesture classification based on Dynamic Time Warping (DTW). Results indicate that, in this domain, the FPCA algorithm achieves a comparable recognition rate while it outperforms DTW in terms of efficiency in execution time.

Complex architectural artifacts and industrial systems are increasingly being designed with the help of virtual prototypes. In these virtual environments users can interact with specific features of a simulation to evaluate functional aspects or learn complex operations before system construction or while unaccessible. A key issue for a virtual prototype-based design cycle is the possibility to easily interact with the simulated system to trigger specific functions regardless of the limited technical skills of the user. In this paper, the potential of the Nintendo Wiimote device programmed as a tangible interface to navigate and interact in virtual environments has been assessed in a user study involving multiple subjects. Results with basic 2D point-and-click and select-drag-and-drop tasks, as well as in 3D navigation and interaction scenarios, have shown the effectiveness and reliability of the Wiimote-based interaction. Moreover, the interface was perceived by the subjects as intuitive, not complicated, and requiring little mental effort.

Object category recognition and localization in 3D range data is of great importance in robot manipulation. In this work we propose a novel approach for object categorization and grasping that is focused on topological shape segmentation. The method allows generation of watertight triangulated models of the objects and their shape segmentation into parts. This segmentation provides meaningful information about grasp affordances. An efficient technique for encoding proximity data from range scans is also presented as well as an advanced strategy for manipulation of object sub-parts. Experiments are reported in a real environment using a robot arm equipped with eye-in-hand laser scanner and a parallel gripper.

A method for robot to human object hand-over is presented that takes into account user comfort. Comfort is addressed by serving the object to facilitate user's convenience. The object is delivered so that the most appropriate part is oriented towards the person interacting with the robot. This approach, aimed at contributing to the development of socially aware robots, has not been considered in previous works. The robot system also supports sensory-motor skills like object and people detection, robot grasping and motion planning. The experimental setup consists of a six degrees of freedom robot arm with both an eye-in-hand laser scanner and a fixed range sensor. The user interacting with the robot can assume an arbitrary position in front of the robot. Experiments are reported from a user study.

In this work we introduce a novel approach for robot grasp planning. The proposed method combines the benefits of programming by human demonstration for teaching appropriate grasps with those of automatic 3D shape segmentation for object recognition and semantic modeling. The work is motivated by important studies on human manipulation suggesting that when an object is perceived for grasping it is first parsed in its constituent parts. Following these findings we present a manipulation planning system capable of grasping objects by their parts which learns new tasks from human demonstration. The central advantage over previous approaches is the use of a topological method for shape segmentation enabling both object retrieval and part-based grasp planning according to the affordances of an object. Manipulation tasks are demonstrated in a virtual reality environment using a data glove. After the learning phase, each task is planned and executed in a robot environment that is able to generalize to similar, but previously unknown, objects.

We present an approach for visuo-haptic augmented reality, which is focused on object manipulation tasks. The interactive environment combines three-DOF haptic rendering and physics-based animation. A desktop haptic device with force feedback is driven by the user to interact with movable virtual objects that are superimposed upon a visual representation of the real workspace. Virtual objects coexist with real objects in the augmented reality space and are simulated in a physically plausible manner. The system supports both rigid and deformable objects with arbitrary shape. Accurate algorithms for camera calibration, registration, object manipulation and feedback computation have also been developed. Several experiments have been performed in order to test both single and dual-user collaborative tasks.

Physics-based animation is becoming an essential feature for any advanced simulation software. In this paper we explore potential benefits of physics-based modeling for task programming in virtual reality. Firstly, we show how realistic animation of manipulation tasks can be exploited for learning sequential constraints from user demonstrations. In particular, we propose a method where information about physical interaction is used to discover task precedences and to reason about task similarities at the goal level. A second contribution of the paper is the application of physics-based modeling to the problem of disassembly sequence planning. Experiments have been performed in a desktop virtual reality environment with dataglove and motion tracker.

We address the problem of disassembly planning from a novel perspective. In the proposed method the goal is to find all the physically admissible subassemblies in which a set of objects can be disassembled and to identify feasible disassembly motions. Stability of object configurations under the effect of gravity and friction is computed by relying on a physics-based animation engine. We propose efficient strategies to reduce computational time that take into account precedence relations, arising from user assembly demonstrations as well as geometrical clustering. We have also developed a motion planning technique for generating non-destructive disassembly paths on a query-based approach. Experiments have been performed in an interactive virtual environment including a dataglove that allows realistic object manipulation and grasping.

Biomechanics has strong implications in both medicine and robotics as it is concerned with the study of biological systems from an engineering point of view. In this work we propose a physics-based system for accurate simulation of dogpsilas spine at walking gait. The spinal column has been modeled as a set of rigid bodies, representing the vertebrae, connected by joints. Real-time dynamic simulation has been carried out from motion captured data collected from skin markers. A feedback controller has been developed for error minimization. The main contribution of the paper is the development of a tool for analyzing the forces acting on the spine. We report experiments performed with a boxer dog walking on a treadmill. Potential applications of this study involve bio-inspired robotics and computer graphics, as well as diagnostic applications for live animals.

This paper addresses the issue of how Programming by Demonstration can assist the development of task-related grasping capabilities in a robotic system. Finding a proper quality measure for the evaluation of grasping tasks is a crucial topic for service robots. While classical grasp quality measures do not include task information, we propose a measure which takes into account user experience. Experiments have been performed in a virtual environment that enables real-time human interaction by means of a dataglove and a motion tracker. Also, a local grasp optimization technique is described to amend uncertainties arising from user demonstration. Finally, the grasp quality measure has been applied for synthesizing manipulation tasks with a simulated robot arm.

Automatic environment modeling is an essential requirement for intelligent robots to execute manipulation tasks. Object recognition and workspace reconstruction also enable 3D user interaction and programming of assembly operations. In this paper a novel method for synthesizing robot grasps from demonstration is presented. The system allows learning and classification of human grasps demonstrated in virtual reality as well as teaching of robot grasps and simulation of manipulation tasks. Both virtual grasp demonstration and grasp synthesis take advantage of a topology-preserving approach for automatic workspace modeling with a monocular camera. The method is based on the computation of edge-face graphs. The algorithm works in real-time and shows high scalability in the number of objects thus allowing accurate reconstruction and registration from multiple views. Grasp synthesis is performed mimicking the human hand pre-grasp motion with data smoothing. Experiments reported in the paper have tested the capabilities of both the vision algorithm and the grasp synthesizer.

Sales of entertainment robots are primed to explode in the next few years. Gesture imitation from a human operator demonstration is considered a promising technique that allows unexperienced users to easily interact with a robotic platform. In this paper a Programming by Demonstration system for a humanoid robot is proposed. The robot is a Robosapien V2 toy which is programmed to observe and imitate human gestures. The system combines inputs from multiple sensory devices, including a dataglove, a motion tracker and a monocular vision system. In the proposed imitation approach the sensor data are mapped to the joint space of the robot. Even though the performance of the robot is constrained by the kinematic and dynamic limitations of RSV2 and by the sensor inaccuracies, experiments involving simple gesture imitation of an untrained user show the viability and effectiveness of the method.

This paper describes a virtual reality based programming by demonstration system for grasp recognition in manipulation tasks and robot pregrasp planning. The system classifies the human hand postures taking advantage of virtual grasping and information about the contact points and normals computed in the virtual reality environment. A pregrasp planning algorithm mimicking the human hand motion is also proposed. Reconstruction of human hand trajectories, approaching the objects in the environment, is based on NURBS curves and a data smoothing algorithm. Some experiments involving grasp classification and pregrasp planning, while avoiding obstacles in the workspace, show the viability and effectiveness of the approach

This paper describes the trajectory learning component of a programming by demonstration (PbD) system for manipulation tasks. In case of multiple user demonstrations, the proposed approach clusters a set of hand trajectories and recovers smooth robot trajectories overcoming sensor noise and human motion inconsistency problems. More specifically, we integrate a geometric approach for trajectory clustering with a stochastic procedure for trajectory evaluation based on hidden Markov models. Furthermore, we propose a method for human hand trajectory reconstruction with NURBS curves by means of a best-fit data smoothing algorithm. Some experiments show the viability and effectiveness of the approach.

We investigate the effectiveness of several types of virtual fixtures in a robot programming by demonstration interface. We show that while all types of virtual fixtures examined yield a significant reduction in the number of errors in tight tolerance peg-in-hole tasks, color and sound fixtures generally outperform a tactile fixture in terms of both execution time of successful trials and error rate. We have found also that when users perceive that the task is very difficult but the system is providing some help by means of a virtual fixture, they tend to spend more time trying to achieve a successful task execution. Thus, for difficult tasks the benefits of virtual fixturing are better reflected in a reduction of the error rate than in a decreased execution time. We conjecture that these trends are related to the limitations of currently available interfaces for human-robot interaction through virtual environments and to the different strategies adopted by the users to cope with such limitations in high-accuracy tasks.