This paper explores visual pointing gestures for two-way nonverbal communication for interacting with a robot arm. Such non-verbal instruction is common when humans communicate spatial directions and actions while collaboratively performing manipulation tasks. Using 3D RGBD we compare human-human and human-robot interaction for solving a pick-and-place task. In the human-human interaction we study both pointing and other types of gestures, performed by humans in a collaborative task. For the human-robot interaction we design a system that allows the user to interact with a 7DOF robot arm using gestures for selecting, picking and dropping objects at different locations. Bi-directional confirmation gestures allow the robot (or human) to verify that the right object is selected. We perform experiments where 8 human subjects collaborate with the robot to manipulate ordinary household objects on a tabletop. Without confirmation feedback selection accuracy was 70-90% for both humans and the robot. With feedback through confirmation gestures both humans and our vision-robotic system could perform the task accurately every time (100%). Finally to illustrate our gesture interface in a real application, we let a human instruct our robot to make a pizza by selecting different ingredients.

Pointing to indicate direction or position is one of the intuitive communication mechanisms used by humans in all life stages. Our aim is to develop a natural human-robot command interface using pointing gestures for human-robot interaction (HRI). We propose an interface based on the Kinect sensor for selecting by pointing (SEPO) in a 3D real-world situation, where the user points to a target object or location and the interface returns the 3D position coordinates of the target. Through our interface we perform three experiments to study precision and accuracy of human pointing in typical household scenarios: pointing to a “wall”, pointing to a “table”, and pointing to a “floor”. Our results prove that the proposed SEPO interface enables users to point and select objects with an average 3D position accuracy of 9:6 cm in household situations.

This paper is concerned with the use of a spherical-projection model for visual servoing from three points. We propose a new set of six features to control a 6-degree-of-freedom (DOF) robotic system with good decoupling properties. The first part of the set consists of three invariants to camera rotations. These invariants are built using the Cartesian distances between the spherical projections of the three points. The second part of the set corresponds to the angle-axis representation of a rotation matrix measured from the image of two points. Regarding the theoretical comparison with the classical perspective coordinates of points, the new set does not present more singularities. In addition, using the new set inside its nonsingular domain, a classical control law is proven to be optimal for pure rotational motions. The theoretical results and the robustness to points range errors of the new control scheme are validated through simulations and experiments on a 6-DOF robot arm.

This paper deals with visual servoing from three points. Using the geometric properties of the spherical projection of points, a new decoupled set of six visual features is proposed. The main originality lies in the use of the distances between spherical projection of points to define three features that are invariant to camera rotations. The three other features present a linear link with respect to camera rotations. In comparison with the classical perspective coordinates of points, the new decoupled set does not present more singularities. In addition, using the new set in its non-singular domain, a classical control law is proven to be ideal for rotational motions. These theoretical results as well as the robustness to errors of the new decoupled control scheme are illustrated through simulation results.

A catadioptric vision system combines a camera and a mirror to achieve a wide field of view imaging system. This type of vision system has many potential applications in mobile robotics. This paper is concerned with the design of a robust image-based control scheme using a catadioptric vision system mounted on a mobile robot. We exploit the fact that the decoupling property contributes to the robustness of a control method. More precisely, from the image of a point, we propose a minimal and decoupled set of features measurable on any catadioptric vision system. Using the minimal set, a classical control method is proved to be robust in the presence of point range errors. Finally, experimental results with a coarsely calibrated mobile robot validate the robustness of the new decoupled scheme.

This paper is concerned with improvements to visual feature modeling using a spherical projection model. Three spherical targets are considered: a sphere, a sphere marked with a tangent vector to a point on its surface, and a sphere marked with two points on its surface. A new minimal and decoupled set of visual features is proposed for each target using any central catadioptric camera. Using the proposed set for a sphere, a classical control law is proved to be globally asymptotically stable in the presence of modeling errors and locally asymptotically stable in the presence of calibration errors, considering that perspective and paracatadioptric cameras were used. Simulation and experimental results with perspective, paracatadioptric, and fish-eye cameras validate the proposed theoretical results.

This paper focuses on visual servoing from a special target using a vision sensor with an unique projection center. The target is a sphere marked with two points on its surface. Using a spherical projection model, a new minimal set of six features is proposed for this target. Spherical moments are exploited to compute the new set on the image of the sensor. Using the new set, a classical control method is proved to be asymptotically stable even in the presence of modeling errors. Experimental results with a perspective camera and a fish-eye camera validate the proposed theoretical results.

This paper is concerned with the use of a spherical projection model to design optimal visual features for visual servoing. Here two special targets are considered: a sphere marked with two points on its surface and a sphere marked with a tangent vector to a point on its surface. For these targets we propose a new minimal set of six visual features which can be computed on classical perspective cameras. Using the new set, a classical control method is proved to be globally stable even in the presence of modeling error. In comparison with the previous set originally proposed for the second target, the new set draws a better camera trajectory. Finally, simulation and experimental results confirm the validity of the proposed theoretical results.

In this paper, we investigate the use of a spherical projection model to search for optimal visual features for visual servoing. A new minimal set of three visual features is proposed for visual servoing from spheres using any central catadioptric system. Using this set of features, a classical control method is proved to be globally stable even in the presence of modeling error and locally stable to calibration errors on perspective cameras. Using this type of cameras, experimental results are presented and validate the proposed theoretical results.