Multifingered robotic hands offer stable grasping for a wide variety of objects, yet grasp planning with these hands is more challenging due to the high dimensionality of the search space. In this article, we propose a method for grasping unknown objects from cluttered scenes using a noisy point cloud as an input. Our approach is based on a shape complementarity metric. A fast algorithm for finding a small set of potential grasps is proposed followed by a local shape completion method to infer the occluded parts of the object. Finally, we propose an optimization-based refinement of the hand poses and finger configurations to achieve a power grasp of the target object. The proposed approach is validated extensively both on a simulated and a real world environment. We demonstrate that the proposed grasp planning algorithm produces stable grasps even in heavily dense clutter. Finally, our experiments indicate improved grasp success rate over algorithms that employ precision grasping in the same scene.

Extracting a known target object from a pile of other objects in a cluttered environment is a challenging robotic manipulation task encountered in many robotic applications. In such conditions, the target object touches or is covered by adjacent obstacle objects, thus rendering traditional grasping techniques ineffective. In this paper, we propose a pushing policy aiming at singulating the target object from its surrounding clutter, by means of lateral pushing movements of both the neighboring objects and the target object until sufficient ’grasping room’ has been achieved. To achieve the above goal we employ reinforcement learning and particularly Deep Qlearning (DQN) to learn optimal push policies by trial and error. A novel Split DQN is proposed to improve the learning rate and increase the modularity of the algorithm. Experiments show that although learning is performed in a simulated environment the transfer of learned policies to a real environment is effective thanks to robust feature selection. Finally, we demonstrate that the modularity of the algorithm allows the addition of extra primitives without retraining the model from scratch.

Grasping objects in a cluttered environment is challenging due to the lack of collision free grasp affordances. In such conditions, the target object touches or is covered by other objects in the scene, resulting in a failed grasp. To address this problem, we propose a strategy of singulating the object from its surrounding clutter, which consists of previously unseen objects, by means of lateral pushing movements. We employ reinforcement learning for obtaining optimal push policies given depth observations of the scene. The action-value function(Q-function) is approximated with a deep neural network. We train the robot in simulation and we demonstrate that the transfer of learned policies to the real environment is robust.

Object detection and 6D pose estimation in the crowd (scenes with multiple object instances, severe foreground occlusions and background distractors), has become an important problem in many rapidly evolving technological areas such as robotics and augmented reality. Single shotbased 6D pose estimators with manually designed features are still unable to tackle the above challenges, motivating the research towards unsupervised feature learning and next-best-view estimation. In this work, we present a complete framework for both single shot-based 6D object pose estimation and next-best-view prediction based on Hough Forests, the state of the art object pose estimator that performs classification and regression jointly. Rather than using manually designed features we a) propose an unsupervised feature learnt from depth-invariant patches using a Sparse Autoencoder and b) offer an extensive evaluation of various state of the art features. Furthermore, taking advantage of the clustering performed in the leaf nodes of Hough Forests, we learn to estimate the reduction of uncertainty in other views, formulating the problem of selecting the next-best-view. To further improve pose estimation, we propose an improved joint registration and hypotheses verification module as a final refinement step to reject false detections. We provide two additional challenging datasets inspired from realistic scenarios to extensively evaluate the state of the art and our framework. One is related to domestic environments and the other depicts a bin-picking scenario mostly found in industrial settings. We show that our framework significantly outperforms state of the art both on public and on our datasets.

Perception of garments is a challenging task for robots due to the large variety in shapes, fabric patterns, and materials. We investigate a multi-sensorial approach, making no assumptions about the garments' configuration or properties. We use a robot equipped with RGB-D, tactile, and photometric stereo sensors that interacts with the garment through a combination of different basic actions. By applying machine learning techniques on the autonomously acquired data of different modalities we recognize the manipulated garment's type, fabric pattern, and material. Despite the challenges imposed by the unconstrained environment, promising performances are achieved for the majority of the recognition tasks.

This work presents a complete pipeline for folding a pile of clothes using a dual-armed robot. This is a challenging task both from the viewpoint of machine vision and robotic manipulation. The presented pipeline is comprised of the following parts: isolating and picking up a single garment from a pile of crumpled garments, recognizing its category, unfolding the garment using a series of manipulations performed in the air, placing the garment roughly flat on a work table, spreading it, and, finally, folding it in several steps. The pile is segmented into separate garments using color and texture information, and the ideal grasping point is selected based on the features computed from a depth map. The recognition and unfolding of the hanging garment are performed in an active manner, utilizing the framework of active random forests to detect grasp points, while optimizing the robot actions. The spreading procedure is based on the detection of deformations of the garment's contour. The perception for folding employs fitting of polygonal models to the contour of the observed garment, both spread and already partially folded. We have conducted several experiments on the full pipeline producing very promising results. To our knowledge, this is the first work addressing the complete unfolding and folding pipeline on a variety of garments, including T-shirts, towels, and shorts.

We present a novel approach to the problem of autonomously recognizing and unfolding articles of clothing using a dual manipulator. The problem consists of grasping an article from a random point, recognizing it and then bringing it into an unfolded state. We propose a data-driven method for clothes recognition from depth images using Random Decision Forests. We also propose a method for unfolding an article of clothing after estimating and grasping two key-points, using Hough forests. Both methods are implemented into a POMDP framework allowing the robot to interact optimally with the garments, taking into account uncertainty in the recognition and point estimation process. This active recognition and unfolding makes our system very robust to noisy observations. Our methods were tested on regular-sized clothes using a dual-arm manipulator and an Xtion depth sensor. We achieved 100% accuracy in active recognition and 93.3% unfolding success rate, while our system operates faster compared to the state of the art.

The advent of affordable real-time 3D sensors is expected to have a major impact in face and gesture recognition applications, not excluding facial expression recognition. In this paper we review our latest results, a completely automated facial action and facial expression recognition system based on a real-time 3D sensor and its evaluation in uncontrolled conditions. We also highlight existing challenges and propose future directions.

In this paper, we explore bilinear and elastically deformable models for addressing jointly 3D face and facial expression recognition. An elastically deformable model is built first to allow anatomically valid point-to-point correspondence among face surfaces and then, bilinear models are used to decouple the impact of identity and expression on face appearance. This enables the representation of the surface using two independent sets of control coefficients that can be used for joint face and facial expression recognition. The proposed system was tested on the publicly available BU-3DFE face database where an average facial expression recognition rate of 89.5% and a rank-1 face recognition rate of 85% were achieved.

In this paper, we present a novel approach for 3D facial expression recognition which is inspired by the advances of ant colony and particle swarm optimization (ACO and PSO respectively) in the field of data mining. Anatomical correspondence between faces is first established using a generic 3D face model which is deformed elastically to match the facial surfaces. Surface points are then used as a basis for classification according to a set of classification rules, which are discovered by an ACO/PSO-based rule discovery algorithm. The performance of the proposed algorithm has been evaluated on the BU-3DFEDB facial expression database where a total recognition rate of 92.3% was achieved.

In this paper, we explore bilinear models for jointly addressing 3D face and facial expression recognition. An elastically deformable model algorithm that establishes correspondence among a set of faces is proposed first and then bilinear models that decouple the identity and facial expression factors are constructed. Fitting these models to unknown faces enables us to perform face recognition invariant to facial expressions and facial expression recognition with unknown identity. A quantitative evaluation of the proposed technique is conducted on the publicly available BU-3DFE face database in comparison with our previous work on face recognition and other state-of-the-art algorithms for facial expression recognition. Experimental results demonstrate an overall 90.5% facial expression recognition rate and an 86% rank-1 face recognition rate.

A novel surveillance system integrating 2D and 3D facial data is presented in this paper, based on a low-cost sensor capable of real-time acquisition of 3D images and associated color images of a scene. Depth data is used for robust face detection, localization and 3D pose estimation, as well as for compensating pose and illumination variations of facial images prior to classification . The proposed system was tested under an open-set identification scenario for surveillance of humans passing through a relatively constrained area. Experimental results demonstrate the accuracy and robustness of the system under a variety of conditions usually encountered in surveillance applications.

In this paper, a novel local surface descriptor is proposed and applied to the problem of aligning partial views of a 3D object. The descriptor is based on taking "snapshots" of the surface over each point using a virtual camera oriented perpendicularly to the surface. This representation has the advantage of imposing minimal loss of information be robust to self-occlusions and also be very efficient to compute. Then, we describe an efficient search technique to deal with the rotation ambiguity of our representation and experimentally demonstrate the benefits of our approaches which are pronounced especially when we align views with small overlap.

Advances in image sensors and evolution of digital computation is a strong stimulus for development and implementation of sophisticated methods for capturing, processing and analysis of 3D data from dynamic scenes. Research on perspective time-varying 3D scene capture technologies is important for the upcoming 3DTV displays. Methods such as shape-from-texture, shape-from-shading, shape-from-focus, and shape-from-motion extraction can restore 3D shape information from a single camera data. The existing techniques for 3D extraction from single-camera video sequences are especially useful for conversion of the already available vast mono-view content to the 3DTV systems. Scene-oriented single-camera methods such as human face reconstruction and facial motion analysis, body modeling and body motion tracking, and motion recognition solve efficiently a variety of tasks. 3D multicamera dynamic acquisition and reconstruction, their hardware specifics including calibration and synchronization and software demands form another area of intensive research. Different classes of multiview stereo algorithms such as those based on cost function computing and optimization, fusing of multiple views, and feature-point reconstruction are possible candidates for dynamic 3D reconstruction. High-resolution digital holography and pattern projection techniques such as coded light or fringe projection for real-time extraction of 3D object positions and color information could manifest themselves as an alternative to traditional camera-based methods. Apart from all of these approaches, there also are some active imaging devices capable of 3D extraction such as the 3D time-of-flight camera, which provides 3D image data of its environment by means of a modulated infrared light source.

The performance of automatic 3-D face recognition can be significantly improved by coping with the nonrigidity of the facial surface. In this paper, we propose a geodesic polar parameterization of the face surface. With this parameterization, the intrinsic surface attributes do not change under isometric deformations and, therefore, the proposed representation is appropriate for expression-invariant 3-D face recognition. We also consider the special case of an open mouth that violates the isometry assumption and propose a modified geodesic polar parameterization that also leads to invariant representation. Based on this parameterization, 3-D face recognition is reduced to the classification of expression-compensated 2-D images that can be classified with state-of-the-art algorithms. Experimental results verify theoretical assumptions and demonstrate the benefits of the geodesic polar parameterization on 3-D face recognition.

In this paper, a biometric authentication system based on measurements of the user's three-dimensional (3-D) hand geometry is proposed. The system relies on a novel real-time and low-cost 3-D sensor that generates a dense range image of the scene. By exploiting 3-D information we are able to limit the constraints usually posed on the environment and the placement of the hand, and this greatly contributes to the unobtrusiveness of the system. Efficient, close to real-time algorithms for hand segmentation, localization and 3-D feature measurement are described and tested on an image database simulating a variety of working conditions. The performance of the system is shown to be similar to state-of-the-art hand geometry authentication techniques but without sacrificing the convenience of the user.

This paper presents a complete face authentication system integrating both two-dimensional (color or intensity) and three-dimensional (3-D) range data, based on a low-cost 3-D sensor, capable of real-time acquisition of 3-D and color images. Novel algorithms are proposed that exploit depth information to achieve robust face detection and localization under conditions of background clutter, occlusion, face pose alteration, and harsh illumination. The well-known embedded hidden Markov model technique for face authentication is applied to depth maps and color images. To cope with pose and illumination variations, the enrichment of face databases with synthetically generated views is proposed. The performance of the proposed authentication scheme is tested thoroughly on two distinct face databases of significant size. Experimental results demonstrate significant gains resulting from the combined use of depth and color or intensity information.

The present paper proposes a novel method for content-based search in a database of VRML three-dimensional (3-D) models. The proposed technique is based on a querying-by-3-D-model approach. A set of shape-based descriptors are extracted from the reference 3-D model and compared to the corresponding descriptors of the VRML models contained in the database. The descriptors used vary from simple geometric measurements such as the aspect ratio or a binary 3-D shape mask to more complex and sophisticated shape-based criteria such as the edge paths of each 3-D model. Similarity measures are then introduced for the specific descriptors and introduced into a 3-D model-matching algorithm. Experimental results are presented, evaluating the performance of the proposed method.

The paper describes a face recognition system using a combination of color and depth images. To cope with illumination and pose variations 3D information is used for the normalization of the input images. The proposed pose compensation algorithm is based on a robust 3D face detection and pose estimation technique, while illumination compensation exploits depth data to recover the illumination of the scene and relight the image under frontal lighting. When normalized images, depicting upright orientation and frontal lighting, are used for classification significantly high recognition rates are achieved, as demonstrated on a face database with more than 2000 images.

In this paper a head tracking algorithm using 3D data is described. The system relies on a novel 3D sensor that generates a dense range image of the scene. By not relying on brightness information, the proposed system guarantees robustness under various illumination conditions, and content of the scene. The main novelty of the proposed algorithms, with respect to other head tracking techniques, is the capability for accurate tracking of the 6 degrees of freedom of the head by explicitly utilising 3D head-shoulder geometry. A Bayesian tracking framework is also proposed for continuous 3D head pose estimation. The proposed system has been tested in a real-time application scenario.

This paper addresses the reconstruction of 3D human body models from 2D video sequences. Considering that the input frames are already segmented, the proposed technique consists of three stages. These stages are independently applied over each segmented frame. Firstly, a skeleton of a human figure obtained from the segmented image is extracted by means of a fast algorithm based on a Voronoi diagram of the boundary points. Afterwards, the skeleton is labelled according to the human body parts (e.g. head, upper arm, lower arm, torso, etc). Secondly, an initial 3D model posture is estimated from the labelled skeleton. Finally, an iterative closest point (ICP) implementation is used to refine the initial model posture by maximizing the similarity between the projected 3D model and the segmented image. Experimental results with video sequences are presented.

This paper presents the virtual reality applications developed for the feasibility study tests of the EU funded IST project ENORASI. ENORASI aims at developing a highly interactive and extensible haptic VR training system that allows visually impaired people, especially those blind from birth, to study and interact with various virtual objects. A number of custom applications have been developed based on the interface provided by the CyberGrasp haptic device. Eight test categories were identified and corresponding tests were developed for each category. Twenty-six blind persons conducted the tests and the evaluation results have shown the degree of acceptance of the technology and the feasibility of the proposed approach.

Thepaper proposes a novel method for content-based search in a database of VRML 3D models. The proposed technique is based on a querying-by-3D-model approach. A set of shape-based descriptors are extracted from the reference 3D model and compared to the corresponding descriptors of the VRML models contained in the database. The descriptors used vary from simple geometric measurements such as the aspect ratio or a binary 3D shape mask to more complex and sophisticated shape-based criteria such as the edge paths of each 3D model. Similarity measures are then introduced for the specific descriptors and introduced into a 3D model-matching algorithm. Experimental results are presented, evaluating the performance of the proposed method.

The paper presents several novel 3D image analysis algorithms, applied towards the segmentation and modeling of faces and hands. These are subsequently used to build a face-based authentication system and a system for human-computer interaction based on static and dynamic gestures. The system relies on an active stereo sensor that uses a structured light approach to obtain 3D information. In this paper we demonstrate how the use of 3D information may significantly improve the efficiency of traditional face and gesture recognition techniques that use 2D images only.

Summary form only given, as follows. The transmission of the large store of information contained in 3D medical data sets through limited capacity channels is a critical procedure in many telemedicine applications. Techniques are presented for the compression of visualizations of 3D image data for efficient storage and transmission. Methods are first presented for the transmission of the 3D surface of the objects using contour following methods. Alternately, the visualization at the receiver may be based on a series of depth maps corresponding to some motion of the object, specified by the medical observer. Depth maps may be transmitted by using depth map motion compensated prediction. Alternately, a wire-mesh model of the depth map may be formed and transmitted by encoding the motion of its nodes. All these methods are used for the transmission of the 3D image with visualization carried out at the receiver. Methods are also developed for efficient transmission of the images visualized at the encoder site. These methods allow remote interactive manipulation (rotation, translation, zoom) of the 3D objects, and may be implemented even if the receiver is a relatively simple and inexpensive workstation or a simple monitor. In all these cases, the coding of binocular views of the 3D scene is examined and recommendations are made for the implementation of coders of stereo views of 3D medical data.

Optimal mechanisms are determined for the hierarchical decomposition of wire-frame surfaces generated by box splines. A family of box splines with compact support, suitable for the approximation of wire-frames is first defined, generated by arbitrary sampling matrices with integer eigenvalues. For each such box spline, the optimal positioning of the wire-frame nodes is determined for each level of the hierarchical wire-frame decomposition. Criterion of optimality is the minimization of the variance of the error difference between the original surface and its representation at each resolution level. This is needed so as to ensure that the wire mesh produces at each resolution as close a replica of the original surface as possible. Several such combinations of box spline generated meshes and the corresponding optimal node lattice sequences are examined in detail with a view to practical application. Their specific application to the hierarchical coding of three-dimensional (3-D) wire meshes is experimentally evaluated.

In this paper a temporal learning-filtering procedure is applied to refine the left ventricle (LV) boundary detected by an active-contour model. Instead of making prior assumptions about the LV shape or its motion, this information is incrementally gathered directly from the images and is exploited to achieve more coherent segmentation. A Hough transform technique is used to find an initial approximation of the object boundary at the first frame of the sequence. Then, an active-contour model is used in a coarse-to-fine framework, for the estimation of a noisy LV boundary. The PCA transform is applied to form a reduced ordered orthonormal basis of the LV deformations based on a sequence of noisy boundary observations. Then this basis is used to constrain the motion of the active contour in subsequent frames, and thus provide more coherent identification. Results of epicardial boundary identification in B-mode images are presented.

This paper presents an algorithm for the estimation of the motion of textured objects undergoing nonrigid deformations over a sequence of images. An active mesh model, which is a finite-element deformable membrane, is introduced in order to achieve efficient representation of global and local deformations. The mesh is constructed using an adaptive triangulation procedure that places more triangles over high detail areas. Through robust least squares techniques and modal analysis, efficient estimation of global object deformations is achieved, based on a set of sparse displacement measurements. A local warping procedure is then applied to minimize the intensity matching error between subsequent images, and thus estimate local deformations. Among the major contributions of this paper are novel techniques developed to acquire knowledge of the object dynamics and structure directly from the image sequence, even in the absence of prior intelligence regarding the scene. Specifically, a coarse-to-fine estimation scheme is first developed, which adapts the model to locally deforming features. Subsequently, principal components modal analysis is used to accumulate knowledge of the object dynamics. This knowledge is finally exploited to constrain the object deformation. The problem of tracking the model over time is addressed, and a novel motion-compensated prediction approach is proposed to facilitate this. A novel method for the determination of the dynamical principal axes of deformation is developed. The experimental results demonstrate the efficiency and robustness of the proposed scheme, which has many potential applications in the areas of image coding, image analysis, and computer graphics.

In this paper, a motion estimation algorithm is presented, based on spatiotemporal warping of an image sequence. The motion field function is modeled by isoparametric cubic finite elements, depending on a set of elementary displacement vectors. A model refinement scheme is proposed for the adaptively higher bit allocation to motion information in locations with high-motion activity.

An object-based stereo image coding algorithm is proposed. The algorithm relies on modeling of the object structure using 3-D wire-frame models and motion estimation using globally rigid and locally deformable motion models. Algorithms for the estimation of motion and structure parameters from stereo images are described. Motion parameters are used to construct predicted images by mapping the image texture on the object surface. Coding of object parameters, appearing background regions, and prediction errors are investigated, and experimental results with standard stereo image sequences depicting general scenes are presented. The proposed algorithm is seen to be very efficient for applications such as stereoscopic video transmission. Furthermore, it is especially well suited to advanced applications such as generation and transmission of intermediate views for multiview receiver systems and applications in which object-wise editing of the bit stream is required. The latter include video production using preanalyzed scenes and virtual reality applications.