This work presents a Mixed Reality (MR) teleoperation framework that enables intuitive control of a bimanual robot. The system integrates immersive MR with differential kinematics to map the operator’s movements to the robot in real time, while providing multiple visual feedback modalities (2D video stream, colored point cloud rendering, and stereoscopic stream) for enhanced spatial perception. A key feature of the proposed framework is the Point-and-Grasp mode, which implements a complete grasping pipeline grounded in geometric analysis and scoring criteria. In this mode, the user selects a target object via a ray-cast in the MR interface; the system then segments the object, generates candidate grasps, evaluates them according to geometric suitability, and autonomously executes the most promising one. A user study was conducted to assess usability, task efficiency, and subjective workload across different visual feedback modalities in object-manipulation tasks. To evaluate grasping performance, a custom dataset of forty cluttered tabletop scenes was created to ensure consistent experimental conditions. The proposed grasp selection strategy achieved the highest success rate among the evaluated methods under the target-conditioned protocol, showing consistent behavior in cluttered tabletop scenes. Overall, the results indicate that immersive teleoperation combined with selective grasp autonomy can improve efficiency, usability, and accessibility in remote robot manipulation. To support reproducibility and further research, the source code associated with the grasping pipeline is publicly available at: https://github.com/MiguelGarciaUVa/PointAndGrasp .

In the general field of collaborative robotics, one of the topics of greatest interest to the scientific community is the ability to learn to perform certain actions by imitating humans. If we think about humans, when someone teaches us how to perform a certain action, we often need to be shown just one time how to do it. Likewise, we believe that robotics should follow this line, using models that do not involve the capture of huge data sets or exhaustive training. Furthermore, while general models can typically be pretrained offline, the robot must quickly adapt to new knowledge without requiring an expensive retraining process. In this article we present a flexible neural learning architecture that allows a robot to learn how-to pick-up a given object just by watching how a human does it. Then, the robot will be able to pick up the current object, or other objects previously learned, anywhere in the work field, with a simple audible indication from the user. This is achieved based on continuous incremental learning techniques and generic segmentation networks integrated with Siamese network models according to the recently proposed CP-CVV method. Results are presented for the success rate in grasping a varied set of objects.

Abstract Gaze control represents an important issue in the interaction between a robot and humans. Specifically, deciding who to pay attention to in a multi-party conversation is one way to improve the naturalness of a robot in human-robot interaction. This control can be carried out by means of two different models that receive the stimuli produced by the participants in an interaction, either an on-center off-surround competitive network or a recurrent neural network. A system based on a competitive neural network is able to decide who to look at with a smooth transition in the focus of attention when significant changes in stimuli occur. An important aspect in this process is the configuration of the different parameters of such neural network. The weights of the different stimuli have to be computed to achieve human-like behavior. This article explains how these weights can be obtained by solving an optimization problem. In addition, a new model using a recurrent neural network with LSTM layers is presented. This model uses the same set of stimuli but does not require its weighting. This new model is easier to train, avoiding manual configurations, and offers promising results in robot gaze control. The experiments carried out and some results are also presented.

In this paper an interaction system based on a three dimensional virtual head projected onto a pyramidal display is proposed. The proposed system makes use of a social robot behavioral architecture already developed in our lab, which allows us to interchange developments between our robotic realizations and the 3D avatar. The overall system is divided into two parts: back projection subsystem and expression generator subsystem. The back projection subsystem projects a three-dimensional avatar onto a pyramidal structure in order to achieve a sensation of depth and realism. The expression generator subsystem carries out the avatar animations using shape keys and bones, following the Facial Action Coding System (FACS). The system consists in several nodes that are integrated in ROS middleware (Robotic Operating System), and includes a user interface that makes the avatar teleoperation easier (the package is avaible in github public respository). In order to evaluate the expressiveness of the system, two sets of experiments have been performed: one to analyze the avatar's gestural ability, that is, its capability to perform expressions that can be identified by an observer, and a second experiment to measure the emotion displaying ability in terms of valence and arousal.

As part of a multimodal animated avatar previously presented in Marcos-Pablos et al. ((2010) A realistic, virtual head for human-computer interaction. Interact. Comput., 22, 176–192, ISSN 0953-5438), in this paper we describe a method for dynamic recognition of displayed facial emotions on low-resolution streaming images. First, we address the detection of action units (AUs) of the facial action coding system using active shape models and Gabor filters. Normalized outputs of the AU recognition step are then used as inputs for a neural network that consists of an habituation network plus a competitive network. Both the competitive and the habituation layer use differential equations, thus taking into account the dynamic information of facial expressions through time. Experimental results carried out on live video sequences and on the Cohn-Kanade face database show that the proposed method provides high recognition hit rates. To assess the suitability of the developed emotional recognition system for human–computer interaction applications, it has been successfully integrated in the architecture of an avatar and we have conducted a preliminary experiment on empathy. The experiment showed promising results, as the avatar that made use of the emotional recognition system obtained a clear increase in the positivity of the rating when compared with the same avatar with no emotional response.

In this paper, a comprehensive automatic visual inspection system for detecting pavement cracks, built around a Laser Road Inspection System (LRIS) onboard an instrumented vehicle, is presented. Two inertial profilers, a Differential Global Position System (DGPS), a high-definition camera and a high-speed area scan camera are the additional acquisition equipment. Visual appearance and geometrical information are obtained simultaneously since 3D profiles are obtained by capturing the laser line projected by the LRIS with the external area scan camera. Using AdaBoost algorithm for the combination of the processing results of these two types of data allows us to improve surface crack detection rates.

In this paper, an algorithm for the reconstruction of an outdoor environment using a mobile robot is presented. The focus of this algorithm is making the mapping process efficient by capturing the greatest amount of information on every scan, ensuring at the same time that the overall quality of the resulting 3D model of the environment complies with the specified standards. With respect to existing approaches, the proposed approach is an innovation since there are very few information based methods for outdoor reconstruction that use resulting model quality and trajectory cost estimation as criteria for view planning.

Based on our previous work in the development of a multimodal animated avatar oriented to human-machine interaction and social robotics, a facial expression recognition system is presented in this paper. Both the avatar's architecture and facial recognition system have been oriented to FACS Action Units recognition and generation, so they can be easily integrated. The resulting animated system is able to maintain nonverbal and verbal bidirectional communication, enhancing the interaction with the avatar and making it more natural for the users.

In this paper a low cost mobile robot localization system that uses the popular Wii remote control (WiiMote) is presented. Both, the WiiMote and a specialized IR light source are located on top of the robot pointing upwards. The IR light is reflected on retroreflective tags located on the ceiling of the robot working area. Two solutions that depend on the working environment, amount of tags needed and robot characteristics are presented. The first solution is deterministic and more restrictive in the quality of sensor data and number of tags. The second solution is a stochastic solution that integrates dead-reckoning information and IR tag readings using a particle filter. The cost of the onboard localization device, including the WiiMote is less than $50 while each tag costs less than $0.05. The system has been tested on a real mobile robot system used for security and surveillance tasks.

In order to enable tight interaction and close collaboration between human and service robots it is necessary that robots imitate the cognitive mechanisms of the brain and their role in competent activity control. That goal can only be achieved if robots are capable of observing their environment, reasoning about it, and autonomously deciding the most appropriate action in a given situation. We present a possible application in this broad area: “Sacarino” is a service robot desingned to work as bellboy in a hotel environment as part of the ROSETEL project. In order to tackle the challenge of learning by interaction we define: (i) the scenarios in which our agent have to work, (ii) its prior knowledge in an episodic memory module to be used in a prime dialog interaction, and (iii) the proper architecture that will support intelligent behaviour suitable for the control of our robot. We will use frameworks like JCOLOBRI (Cases and Ontology Libraries Integration for Building Reasoning Infrastructures) to define the agent's memory structure, taking a deeper look into the episodic memory of the system.

In this paper an interactive and realistic virtual head oriented to human–computer interaction and social robotics is presented. It has been designed following a hybrid approach, taking robotic characteristics into account and searching for a convergence between these characteristics, real facial actions and animation techniques. An initial head model is first obtained from a real person using a laser scanner. Then the model is animated using a hierarchical skeleton based procedure. The proposed rig structure is close to real facial muscular anatomy and its behaviour follows the Facial Action Coding System. Speech synthesis and visual human-face tracking capabilities are also integrated for providing the head with further interaction ability. Using the said hybrid approach, the head can be readily linked to a social-robot architecture. The opinions of a number of persons interacting with this social avatar have been evaluated and are reported in the paper, as against their reactions when interacting with a social robot with a mechatronic face. Results show the suitability of the avatar for on-screen, real-time interfacing in human–computer interaction. The proposed technique could also be helpful in the future for designing and parameterizing mechatronic human-like heads for social robots.

This paper describes the motivation and learning subsystems of Arisco which is a mechatronic head with interactive capacity which includes high expressivity through gesturing, voice recognition, text to speech generation, visual tracking, and Internet information retrieval. The general architecture is first described in the paper. Then, the learning capacity of Arisco is addressed. It learns and performs associations between different stimulus responses through several dynamic neural networks, guided by motivational drives. A number of experiments are discussed, covering stimulus competition, habituation, classical and operant conditioning.

In this paper an interactive and realistic facial animation oriented to human-computer interaction and social robotics is proposed. An initial model is obtained from a real face using a range scanner, and is then animated using a hierarchical skeleton oriented procedure. The rig structure is close to that of real facial muscular anatomy and the behavior follows the Facial Action Coding System. The resulting animated face is suitable for both real-time interfacing in human-machine interaction such as a social robot, and for helping in design and parametrization of android heads.

In this paper a method is presented for the three-dimensional reconstruction of environment, using a mobile platform equipped with an onboard laser scanner (for the scene geometry acquisition), and a webcam (for the texture registration). The used algorithm is based on calculating the 3D planes in the scene upon the measured 3D points, through the normal calculation, the projection of the points onto the found planes, and the 2D triangulation of the projected points. This way, the plane regions and the corresponding contours in the model are obtained. Concerning the texture, images are assigned to the plane regions, and the complete scene is built through a stitching algorithm. Images and 3D points are projected onto a sphere, in order the information loss on free form objects, and the distortion caused by points improperly assigned to planes, to be reduced.

A map of the environment map is a model of the surroundings of the robot. In many robot application domains, it is infeasible to obtain a prior map of the environment. It is therefore desirable that robots are able to generate maps by themselves. This problem is challenging, since it involves a simultaneous localization problem of the robot relative to its (incomplete) map. This article presents a new probabilistic algorithm for the simultaneous localization and mapping problem. The algorithm is based on matching of a set of local maps that have been obtained from range data. The main advantage of the proposed method is of computational nature. The use of segmented maps permits to eliminate unnecessary details, leading the method to be very appropriate for dynamical environments.

Describes a neural network model for the reactive behavioral navigation of a mobile robot. From the information received through the sensors the robot can elicit one of several behaviors (e.g., stop, avoid, stroll, wall following), through a competitive neural network. The robot is able to develop a control strategy depending on sensor information and learning operation. Reinforcement learning improves the navigation of the robot by adapting the eligibility of the behaviors and determining the linear and angular robot velocities.

Modeling reflectance properties from real objects is useful for solving many practical problems ranging from industrial inspection to computer graphics. In particular building computer graphics models directly from real objects has received increased attention, because it is useful in many important fields such as CAD, entertainment (3D games, movies) and virtual museums. The paper describes an automatic way to recover actual reflectance parameters, from range and intensity (or color) data acquired from objects. The proposed approach is based on the Phong reflection model, since this model is usually preferred for computer graphics.

The well-known Torrance-Sparrow (1967) reflection model describes the optical properties of objects, by means of the diffuse reflectance of the body material, the Fresnel reflectance of the air-media interface, and the slope roughness. This paper describes an approach for recovering these parameters from range and brightness (or color) data. Some details about the system built to collect the data are also discussed.