This paper presents a system integrating physical human-robot interaction in a dual-arm object manipulation task using multi-modal robot skin. The skin covers the end-effectors, enabling distance and active force control for aligning and grasping unknown objects, and extends over the entire arms, allowing multi-contact interaction by a user to guide the manipulation motion. Additionally, the skin patches on the non-contact area of the end-effectors function as tactile buttons, with gestures like short taps or long presses designed to switch control objectives. The system integrates two main controllers: a direct wrench controller regulating the grasp force and a Cartesian impedance controller governing the end-effector motion. Interaction wrenches on the arm patches are projected into the end-effector frame or object frame for pre-grasp adjustments or generating manipulation motion. Our method is validated on a dual-arm mobile manipulator equipped with robotic skin, demonstrating smooth motion in response to tactile interaction while consistently maintaining a stable grasp. The system's robustness enables it to handle objects of various geometries without prior knowledge. The system's capability is further highlighted through a practical application where, with human guidance, the robot successfully hangs a toolbox on two nails.

Driven by the need for rapid and reliable heavy payload transport in logistics and manufacturing, researchers are increasingly exploring early applications of humanoid robotics in these domains. Although bipedal locomotion excels on challenging terrain, wheeled modes of transportation remain significantly more energy-efficient on flat surfaces. In this work, we develop a control system that enables a humanoid robot to achieve fast transportation - by riding a two-wheeled hoverboard - and robust heavy payload handling through whole-body grasping, where the robot uses its chest and arms to stabilize bulky objects. Our approach models payload-induced disturbances using a Linear Inverted Pendulum Mode extended with external forces and leverages tactile feedback from an integrated robotic skin to estimate the payload’s weight and center of mass. Feeding these estimates into the hoverboard controller reduces drift and enhances stability. Experimental evaluations on a full-sized real humanoid robot show that our system can withstand strong disturbances and autonomously navigate to deliver payloads of up to 20 kg.

The human somatosensory system integrates multimodal sensory feedback, including tactile, proprioceptive, and thermal signals, to enable comprehensive perception and effective interaction with the environment. Inspired by the biological mechanism, we present a sensorized soft anthropomorphic hand equipped with diverse sensors designed to emulate the sensory modalities of the human hand. This system incorporates biologically inspired encoding schemes that convert multimodal sensory data into spike trains, enabling highly-efficient processing through Spiking Neural Networks (SNNs). By utilizing these neuromorphic signals, the proposed framework achieves 97.14% accuracy in object recognition across varying poses, significantly outperforming previous studies on soft hands. Additionally, we introduce a novel differentiator neuron model to enhance material classification by capturing dynamic thermal responses. Our results demonstrate the benefits of multimodal sensory fusion and highlight the potential of neuromorphic approaches for achieving efficient, robust, and human-like perception in robotic systems.

Current methods for human-object interaction segmentation excel in closed-world settings but struggle to generalize to open-world scenarios where novel actions emerge. Since collecting exhaustive training data for all possible dynamic human activities is impractical, a model capable of detecting and segmenting novel, out-of-distribution (OOD) actions without manual annotation is needed. To address this, we formally define the open-world action segmentation problem and propose a novel framework featuring three key components: 1) an Enhanced Pyramid Graph Convolutional Network with a new decoder for robust spatiotemporal upsampling, 2) hybrid-based training synthesizing OOD data to eliminate reliance on manual labels, and 3) a temporal clustering loss that groups in-distribution actions while distancing OOD samplesWe evaluate our framework on two challenging human-object interaction recognition datasets: Bimanual Actions and Two Hands and Object datasets. Experimental results demonstrate significant improvements over state-of-the-art action segmentation models across multiple open-set evaluation metrics, achieving 16.9% and 34.6% relative gains in open-set segmentation (F1@50) and out-of-distribution detection performances (AUROC), respectively. Additionally, we conduct an in-depth ablation study to assess the impact of each proposed component, identifying the optimal framework configuration for open-world action segmentation.

Accurate temporal segmentation of human actions is critical for intelligent robots in collaborative settings, where a precise understanding of sub-activity labels and their temporal structure is essential. However, the inherent noise in both human pose estimation and object detection often leads to over-segmentation errors, disrupting the coherence of action sequences. To address this, we propose a Multi-Modal Graph Convolutional Network (MMGCN) that integrates low-frame-rate (e.g., 1 fps) visual data with high-frame-rate (e.g., 30 fps) motion data (skeleton and object detections) to mitigate fragmentation. Our framework introduces three key contributions. First, a sinusoidal encoding strategy that maps 3D skeleton coordinates into a continuous sin-cos space to enhance spatial representation robustness. Second, a temporal graph fusion module that aligns multi-modal inputs with differing resolutions via hierarchical feature aggregation, Third, inspired by the smooth transitions inherent to human actions, we design SmoothLabelMix, a data augmentation technique that mixes input sequences and labels to generate synthetic training examples with gradual action transitions, enhancing temporal consistency in predictions and reducing over-segmentation artifacts.Extensive experiments on the Bimanual Actions Dataset, a public benchmark for human-object interaction understanding, demonstrate that our approach outperforms state-of-the-art methods, especially in action segmentation accuracy, achieving F1@10: 94.5% and F1@25: 92.8%.

THE sense of touch is an indispensable requirement for humans to effectively interact with the physical world around them and perform dexterous tasks. Similarly, this should be no different for robots. Imagine, for example, a robot that can open a bottle of medicine and dispense pills to an elderly person. Although this might seem a straightforward task for a human, it remains a significant challenge for a robot. Critically, the completion of the task depends on tactile sensing: the robot needs to receive and interpret the feedback from interacting with the bottle, determine the appropriate force based on the size and hardness of the pills, and adjust its pose to safely dispense them. Each step involves contact-rich interactions that can only be effectively deciphered through tactile sensing. Typically, tactile sensing works in conjunction with other modalities, such as vision, enabling the robot to adjust its actions dynamically and complete the task. In response to this vision of robots interacting with the physical world through touch, tactile robotics has now emerged as a key research area. Tactile robots can be defined as intelligent systems equipped with tactile sensors that can extract and process tactile data to guide their operations and interactions. The development of tactile robots presents scientific challenges, ranging from the design and fabrication of tactile sensors to methodologies for processing tactile data, integrating tactile feedback into task execution, and combining it with other sensory modalities to improve robot perception. As a result, tactile robotics demands collaborative efforts across several disciplines, involving material and data scientists …

This work presents a novel method to keep stable contact and balance while stepping over partial footholds for biped humanoid robots with flat feet. We exploit plantar tactile feedback to detect the geometry of the terrain and reconstruct online the new supporting polygon after landing every step. Plantar tactile feedback detects early contacts to stop the swing foot motion. Then we compute the convex hull of the cluster of contact points detected by distributed normal force sensors over the foot soles. The centroid of the supporting polygon is then used for retargeting the reference ZMP and DCM positions. Finally, the supporting polygon is used to define constraints for ZMP balance feedback control. These methods were implemented in two biped humanoid robots running different walking controllers.

Robots that autonomously operate in human living environments require the ability to adapt to unpredictable changes and flexibly handle a variety of tasks. Particularly, coordinated bimanual motions are essential for enabling tasks that are difficult with just one hand, such as grasping bulky objects, transporting heavy loads, and precision work. Traditional methods of generating robot motions typically involve executing pre-programmed motions, making it challenging to adapt to complex and unpredictable environmental changes. To address this issue, our research focuses on generating diverse motions that can flexibly adapt to environmental changes based on Deep Predictive Learning from a small amount of real-world data. Previous Deep Predictive Learning models have generated the motions of a robot’s left and right arms by a single LSTM, making it difficult to operate them independently. Therefore, we propose a new Hierarchical Deep Predictive Learning model specialized for generating coordinated bimanual motions. This model comprises three components: a Left-LSTM, which learns the body and visual information on the robot’s left side, a Right-LSTM that performs a similar function for the right side, and a Union-LSTM which integrates this information at a higher level. To verify the effectiveness of the proposed model, we conducted bimanual grasping experiments with multiple different objects using two different robots. The experimental results showed that independent of hardware, our model demonstrated a higher success rate compared to the traditional approach, indicating its enhanced capability in coordinating bimanual motions.

This paper presents an optimization of an admittance control model for gait balance assistance offered by a walker-type assistive robot. We previously introduced the notion of quasi-passive physical Human-Robot Interaction (pHRI) where a non-wearable assistive device adaptively achieves supportability for providing physical assistance and operability to follow the user's intuitive operation. Aiming to mitigate the falling risk of elderly people with reduced mobility with our pHRI approach, we propose a hierarchical algorithm to optimize an admittance control model for a walker robot. By employing dynamic trajectories such as Zero Moment Point (ZMP) and Divergent Component of Motion (DCM) with optimization, our controller provides appropriate physical interaction to improve the gait stability while considering intrinsic body dynamics. In the current implementation, based on a model predictive control (MPC) framework, we formulate the optimization problems in the form of quadratic programming (QP), making the optimization suitable for real-time interaction. Through mechanical assessments with passive walking models of compass gait, we demonstrate the feasibility of our proposed optimization framework in stabilizing the limit cycle gait with minimized assistance.

Contains fulltext : 221879.pdf (Publisher’s version ) (Open Access)

In this work, we propose a new estimation method for second-order kinematics for floating-base robots, based on highly redundant distributed inertial feedback. The linear acceleration of each robot link is measured at multiple points using a multimodal, self-configuring and self-calibrating artificial skin. The proposed algorithm is two-fold: i) the skin acceleration data is fused at the link level for state dimensionality reduction; ii) the estimated values are then fused limb-wise with data from the joint encoders and the main inertial measurement unit (IMU), using a Sigma-point Kalman filter. In this manner, it is possible to estimate the joint velocities and accelerations while avoiding the lag and noise amplification phenomena associated with conventional numerical derivation approaches. Experiments performed on the right arm and torso of a REEM-C humanoid robot, demonstrate the consistency of the proposed estimation method.

In a physical Human-Robot Interaction for industrial scenarios is paramount to guarantee the safety of the user while keeping the robot's performance. Hierarchical task approaches are not sufficient since they tend to sacrifice the low priority tasks in order to guarantee the consistency of the main task. To handle this problem, we enhance the standard hierarchical fusion by introducing a novel interactive task-reconfiguring approach (TACTO-Selector) that uses the information of the tactile interaction to adapt the dimension of the tasks, therefore guaranteeing the execution of the safety task while performing the other task as good as possible. In this work, we hierarchically combine a 6 DOF Position-Based Visual Servoing (PBVS) task with a reactive skin control. This approach was evaluated on a 6 DOF industrial robot showing an improvement of 36.37% on average in tracking error reduction compared with a standard approach.

In this paper, we propose a real-time human-robot interface (HRI) system, where Electrooculography (EOG) and Electromyography (EMG) signals were decoded to perform reach-to-grasp movements. For that, five different eye movements (up, down, left, right and rest) were classified in real-time and translated into commands to steer an industrial robot (UR-10) to one of the four approximate target directions. Thereafter, EMG signals were decoded to perform the grasping task using an attached gripper to the UR-10 robot arm. The proposed system was tested offline on three different healthy subjects, and mean validation accuracy of 93.62% and 99.50% were obtained across the three subjects for EOG and EMG decoding, respectively. Furthermore, the system was successfully tested in real-time with one subject, and mean online accuracy of 91.66% and 100% were achieved for EOG and EMG decoding, respectively. Our results obtained by combining real-time decoding of EOG and EMG signals for robot control show overall the potential of this approach to develop powerful and less complex HRI systems. Overall, this work provides a proof-of-concept for successful real-time control of robot arms using EMG and EOG signals, paving the way for the development of more dexterous and human-controlled assistive devices.

Multiple approaches to the estimation of high-order motion derivatives for innovative control applications now rely on the data collected by redundant arrays of inertial sensors mounted on robots, with promising results. However, most of these works suffer scalability issues induced by the considerable amount of data generated by such large-scale distributed sensor systems. In this article, we propose a new adaptive sensor-selection algorithm, for distributed inertial measurements. Our approach consists in using the data of a subset of sensors, selected among a larger collection of inertial sensing elements covering a rigid robot link. The sensor selection process is formulated as an optimization problem, and solved using a projected gradient heuristics. The proposed method can run online on a robot and be used to recalculate the selected sensor arrangement on the fly when physical interaction or potential sensor failure is detected. The tests performed on a simulated UR5 industrial manipulator covered with a multimodal artificial skin, demonstrate the consistency and performance of the proposed sensor-selection algorithm.

To act effectively in its environment, a cognitive robot needs to understand the causal dependencies of all intermediate actions leading up to its goal. For example, the system has to infer that it is instrumental to open a cupboard door before trying to grasp an object inside the cupboard. In this paper, we introduce a novel learning method for extracting instrumental dependencies by following the scientific approach of observations, generation of causal hypotheses, and testing through experiments. Our method uses a virtual reality dataset containing observations from human activities to generate hypotheses about causal dependencies between actions. It detects pairs of actions with a high temporal co-occurrence and verifies if one action is instrumental in executing the other action through mental simulation in a virtual reality environment which represents the system's mental model. Our system is able to extract all present instrumental action dependencies while significantly reducing the search space for mental simulation, resulting in a 6-fold reduction in computational time.

This survey presents the most relevant neural network models of autism spectrum disorder and schizophrenia, from the first connectionist models to recent deep neural network architectures. We analyzed and compared the most representative symptoms with its neural model counterpart, detailing the alteration introduced in the network that generates each of the symptoms, and identifying their strengths and weaknesses. We additionally cross-compared Bayesian and free-energy approaches, as they are widely applied to model psychiatric disorders and share basic mechanisms with neural networks. Models of schizophrenia mainly focused on hallucinations and delusional thoughts using neural dysconnections or inhibitory imbalance as the predominating alteration. Models of autism rather focused on perceptual difficulties, mainly excessive attention to environment details, implemented as excessive inhibitory connections or increased sensory precision. We found an excessively tight view of the psychopathologies around one specific and simplified effect, usually constrained to the technical idiosyncrasy of the used network architecture. Recent theories and evidence on sensorimotor integration and body perception combined with modern neural network architectures could offer a broader and novel spectrum to approach these psychopathologies. This review emphasizes the power of artificial neural networks for modeling some symptoms of neurological disorders but also calls for further developing of these techniques in the field of computational psychiatry.

Humanoid robots are expected to interact in human environments, where physical interactions are unavoidable. Therefore, whole-body control methods that include multi-contact interactions are required. The new emerging technologies in touch sensing are fundamental to acquire online and rich information about these physical interactions with the environment. These technologies lead to the design of novel control systems that can profit from the tactile sensor information in an efficient form, thus producing reactive and compliant robots capable of interacting with their environment. In this paper, we present a novel control framework to integrate the multi-modal tactile information of a robot skin with different control strategies, producing dynamic behaviours suitable for Human-Robot Interactions (HRI). The control framework was experimentally evaluated on a full-size humanoid robot covered with more than 1260 skin cells distributed in the whole robot body. The results show that multi-modal tactile information can be fused hierarchically with multiple control strategies, producing active compliance in a position-controlled stiff humanoid robot.

This work presents a new contact-based 3D path planning approach for manipulators using robot skin. We make use of the Stochastic Functional Gradient Path Planner, extending it to the 3D case, and assess its usefulness in combination with multi-modal robot skin. Our proposed algorithm is verified on a 6 DOF robot arm that has been covered with multi-modal robot skin. The experimental platform is combined with a skin based compliant controller, making the robot inherently reactive. We implement different state-of-the-art planners within our contact-based robot system to compare their performance under the same conditions. In this way, all the planners use the same skin compliant control during evaluation. Furthermore, we extend the stochastic planner with tactile-based explorative behavior to improve its performance, especially for unknown environments. We show that CHiMP is able to outperform state of the art algorithms when working with skin-based sparse contact data.

Studies have demonstrated the potential of using error-related potentials (ErrPs), online decoded from the electroencephalogram (EEG) of a human observer, for robot skill learning and mediation of co-adaptation in collaborative human-robot interaction (HRI). While these studies provided proof-of-concept of this approach as a highly promising avenue in the field of HRI, a systematic understanding of the dyadic interacting system (human and machine) remained unexplored. This research aims to address this gap by proposing a computational model of the human counterpart and simulating the integrated dyadic system. The model can be employed for the systematic study of both human behavioral and technical factors influencing co-adaptation as exemplarily demonstrated in this paper for hypothetical variations of ErrP -decoder performance. The obtained findings have practical implications for future steps along this line of research, for instance to what extent and how improvements of ErrP -decoder performance can benefit co-adaptation in ErrP -based HRI. The proposed computational model enables the prediction of human behavior in the context of ErrP -based HRI. As such it allows the simulation of future empirical studies prior to their conductance and thereby providing a means for accelerating progress along this line of research in a resource-saving manner.

This article presents a holistic approach to the engineering of an artificial robot skin for robots. An example of a multimodal skin cell is given, one that supports multiple human-like sensing modalities, and support for skin cell network is also provided; this is essential to form large-area skin patches in order to cover the surfaces of robots. The essential elements of efficiently handling a large amount of tactile data are explained. A general control framework, which supports robots commanded in position, velocity, and torque, is provided and validated. Several applications of this robot skin will be presented, demonstrating the effectiveness and efficiency of our artificial robot skin to support a wide number of robotic platforms as well as its ease of use across different domains.

In this paper, we propose a control method, providing mobile robots with whole-body compliance capabilities, in response to multicontact physical interactions with their environment. The external forces applied to the robot, as well as their localization on its kinematic tree, are measured using a multimodal, self-configuring, and self-calibrating artificial skin. We formulate a compliance control law in Cartesian space, as a set of quadratic optimization problems, solved in parallel for each limb involved in the interaction process. This specific formulation makes it possible to determine the torque commands required to generate the desired reactive behaviors, while taking the robot kinematic and dynamic constraints into account. When a given limb fails to produce the desired compliant behavior, the generalized force residual at the considered contact points is propagated to a parent limb in order to be adequately compensated. Hence, the robot's compliance range can be extended in a both robust and easily adjustable manner. The experiments performed on a dual-arm velocity-controlled mobile manipulator show that our methodology is robust to nullspace interactions and robot physical constraints.

This paper presents novel event-driven forward kinematics algorithms for open kinematic chains with O(log n) complexity. This event-driven algorithm can efficiently update forward kinematics only when new sensory data comes. This will also contribute to localization of computational resources at sensitive joints to the position of the endpoint (e.g. a fingertip), like a root joint. We constructed 3 event-driven FK algorithms. We proved that the algorithms have the complexity of O(logn) for updating 1 joint angle, and O(logn) for obtaining a homogeneous transformation matrix between links. We compared the 3 algorithms with a conventional forward kinematics algorithm in the viewpoint of complexity, computation time, time-variance and algebraic structures. The results showed that the computation time is well adequate for real-time computation. Computation time is less than 2 us per 1 query, for 40,000 kinematic chains.

Artificial robotic skins are continuously in contact with their environment, and therefore highly rely on proper connections in their skin cells' network. With a static network protocol approach, the affected skin area is unusable after a connection failure. Therefore, we developed a dynamic network protocol for large-scale artificial robotic skins, which re-routes the network upon connection failures to keep the whole skin in operation. Furthermore, the protocol balances the load for driving larger skins without packet loss. For verification, we validated the protocol on a large artificial robot skin we have developed and analyzed its performance with a skin network consisting of up to 204 cells. The failure recovery of the protocol converges in at most 50ms. We showed that the balancing method achieves a packet loss reduction of over 30% compared to the previously used protocol.

The realization of a kinesthetic robot behavior using robot skin requires a reactive skin torque controller, which fuses skin information and robot information to an appropriate skin joint torque in real-time. This fusion of information in real-time is challenging when deploying large scale skin. In this paper, we present a system which efficiently computes the torque of distributed skin cells locally at the point of contacts, completely removing this complex computations from the real-time loop. We demonstrate the feasibility of realizing the skin joint torque computations on the local micro-controllers of the skin cells. Conducting experiments with a real robot, we compare the accuracy of the distributed skin joint torque computation with the computation on the control PC. We also show that the novel distributed approach completely eliminates the computational delay of computing skin joint torques in the robot's real-time control loop. As a result, this approach removes any limits for the maximum number of skin cells in control.

The predictive functions that permit humans to infer their body state by sensorimotor integration are critical to perform safe interaction in complex environments. These functions are adaptive and robust to non-linear actuators and noisy sensory information. This paper introduces a computational perceptual model based on predictive processing that enables any multisensory robot to learn, infer and update its body configuration when using arbitrary sensors with Gaussian additive noise. The proposed method integrates different sources of information (tactile, visual and proprioceptive) to drive the robot belief to its current body configuration. The motivation is to provide robots with the embodied perception needed for self-calibration and safe physical human-robot interaction. We formulate body learning as obtaining the forward model that encodes the sensor values depending on the body variables, and we solve it by Gaussian process regression. We model body estimation as minimizing the discrepancy between the robot body configuration belief and the observed posterior. We minimize the variational free energy using the sensory prediction errors (sensed vs expected). In order to evaluate the model we test it on a real multi-sensory robotic arm. We show how different sensor modalities contributions, included as additive errors, improve the refinement of the body estimation and how the system adapts itself to provide the most plausible solution even when injecting strong sensory visuo-tactile perturbations. We further analyse the reliability of the model when different sensor modalities are disabled. This provides grounded evidence about the correctness of the perceptual model and shows how the robot estimates and adjusts its body configuration just by means of sensory information.

The automation of production lines in industrial scenarios implies solving different problems, such as the flexibility to deploy robotic solutions to different production lines, usability to allow non-robotics expert users to teach robots different tasks, and safety to enable operators to physically interact with robots without the need of fences. In this paper, we present a system that integrates three novel technologies to address the above mentioned problems. We use an autocalibrated multi-modal robot skin , a general robot control framework to generate dynamic behaviors fusing multiple sensor signals, and an intuitive and fast teaching by demonstration method based on semantic reasoning. We validate the proposed technologies with a wheeled humanoid robot in an industrial set-up. The benefits of our system are the transferability of the learned tasks to different robots, the reusability of the models when new objects are introduced in the production line, the capability of detecting and recovering from errors, and the reliable detection of collisions and pre-collisions to provide a fast reactive robot that improves the physical human-robot interaction.

In this paper, we propose a set of novel tactile descriptors to enable robotic systems to extract robust tactile information during tactile object explorations, regardless of the number of the tactile sensors, sensing technologies, type of exploratory movements, and duration of the objects' surface exploration. The performance and robustness of the tactile descriptors are verified by testing on four different sensing technologies (dynamic pressure sensors, accelerometers, capacitive sensors, and impedance electrode arrays) with two robotic platforms (one anthropomorphic hand and one humanoid), and with a large set of objects and materials. Using our proposed tactile descriptors, the Shadow Hand, which has multimodal robotic skin on its fingertips, successfully classified 120 materials (100% accuracy) and 30 in-hand objects (98% accuracy) with regular and irregular textural structure by executing human-like active exploratory movements on their surface. The robustness of the proposed descriptors was assessed further during the large object discrimination with a humanoid. With a large sensing area on its upper body, the humanoid classified 120 large objects with multiple weights and various textures while the objects slid between its sensitive hands, arms, and chest. The achieved 90% recognition rate shows that the proposed tactile descriptors provided robust tactile information from the large number of tactile signals for identifying large objects via their surface texture regardless of their weight.

Indirect force control (IFC) architectures are a common approach for dealing with unknown environments. What all IFC schemes have in common is that the relation between the set point and the actual configuration of the robot is determined by a mechanical relationship (e.g., a mass-spring-damper system). In this paper, we propose a set-point generation method for IFC schemes, enabling intuitive specification of mixed force and positioning tasks on joint and Cartesian levels. In addition, the formulation of equality and inequality tasks is supported and a passivity-based stability proof is formulated using the concept of virtual energy storage. The resulting task programming interface is demonstrated on a 7-degree-of-freedom robot, running a joint space impedance controller. One sample task demonstrates the application of the developed approach and highlights the basic features.

In this work we present a novel efficient event-driven reactive skin controller for large scale robot skin. The novel event-driven controller derives from a standard Jacobian torque controller and fully takes advantage of our multi-modal event-driven robot skin. Event-driven systems only sample, transmit and process information when the novelty of the information is guaranteed. This increases their efficiency in comparison to synchronous systems. We also use the new event-driven controller formulation to design a new synchronous reactive skin controller. We compare both controllers in a comprehensive performance evaluation with our robot TOMM. TOMM has two UR5 robot arms, each covered with 253 multi-modal skin cells. Each skin cell samples 4 different modalities and supports data mode and event mode. The results show that the event-driven reactive skin controller always outperforms the synchronous reference controller while both controllers show exactly the same response. When the robot is not moving then the event-driven controller reduces the CPU usage by 78% in comparison to the synchronous reference controller. When the robot is responding to contacts then the CPU usage reduces by 66%.

In this paper, we present the mechatronic design of our Tactile Omnidirectional Robot Manipulator (TOMM), which is a dual arm wheeled humanoid robot with 6DoF on each arm, 4 omnidirectional wheels and 2 switchable end-effectors (1 DoF grippers and 12 DoF Hands). The main feature of TOMM is its arms and hands which are covered with robot skin. We exploit the multi-modal tactile information of our robot skin to provide a rich tactile interaction system for robots. In particular, for the robot TOMM, we provide a general control framework, capable of modifying the dynamic behavior of the entire robot, e.g., producing compliance in a non-compliant system. We present the hardware, software and middleware components of the robot and provide a compendium of the base technologies deployed in it. Furthermore, we show some applications and results that we have obtained using this robot.

Skin technology enabled a powerful way to sense the environment in robotic systems. It allows simplifying the formulation of safety tasks such as collision avoidance between the robot, the environment and surrounding objects. In this paper, a hierarchy policy based on tactile feedback is proposed to let a robot interact with its environment while performing a set of tasks. Such policy lets the safety tasks as collision avoidance and physical interaction, be reduced to simple potential field rules fed directly with tactile feedback which keeps computation demand low. In this context, the concept of “Intentional Contact” is introduced to escape from classic undesired equilibrium points produced by local minima in the potential fields. Allowed contact with the environment empowers a robot to modify its surroundings in order to fulfil the main task. Such contact is permitted as long as the generated force remains under a specific limit, otherwise, a reactive action is taken to reduce it. This new concept is validated in simulation and on a real robot.

The concept of hybrid zero dynamics is a promising approach for designing exponentially stabilizing controllers for dynamic walking with some degrees of underactuation. By this approach a feedback controller is designed such that a stable periodic orbit, within an invariant submanifold for the hybrid closed-loop system is created. This is usually achieved through an exponentially fast dynamics transverse to the zero dynamics manifold and the stability properties of such periodic orbit is then transferred to the full-order dynamic system. In this paper a passivity-based controller for a planar biped with one degree of underactuation is designed. By this approach we aim to preserve the natural dynamics of the system in the transverse dynamics (i.e. the dynamics transverse to the zero dynamics manifold) in contrast to the common input-output linearization method which cancels these dynamics. A Lyapunov stability analysis of the full-order system based on the conditional stability theorem is presented. By this analysis, the asymptotic stability of the periodic orbit in lower dimensional state space is extended to the full dimensional space. The results of the analysis are verified by simulation on a seven-link biped robot walking with zero ankle torque in sagittal plane.

This paper presents a new algorithm for forward kinematics, called Asynchronous Forward Kinematics (AFK). The algorithm has the complexity of O(log n) for updating one joint angle, and O(logn) for obtaining a homogeneous transformation matrix between links. AFK enables computation for efficient forward kinematics under asynchronous sensory data. Moreover, AFK peovides localise computational resources at sensitive joints to the position of the endpoint (e.g. a fingertip), like a root joint. We provide comparative results including computation time, evaluating AFK against the conventional forward kinematics (CFK). The results showed that the computation time is well adequate for real-time computation. Computation time for 100 links takes less than 20 us for 1 query. Moreover, computation time with over 50000 links takes less than 35 us for 1 query.

Capturing realistic human behaviors is essential to learn human models that can later be transferred to robots. Recent improvements in virtual reality (VR) head-mounted displays provide a viable way to collect natural examples of human behavior without the difficulties often associated with capturing performances in a physical environment. We present a realistic, cluttered, VR environment for experimentation with household tasks paired with a semantic extraction and reasoning system able to utilize data collected in real-time and apply ontology-based reasoning to learn and classify activities performed in VR. The system performs continuous segmentation of the motions of users' hands and simultaneously classifies known actions while learning new ones on demand. The system then constructs a graph of all related activities in the environment through its observations, extracting the task space utilized by observed users during their performance. The action recognition and learning system was able to maintain a high degree of accuracy of around 92% while dealing with a more complex and realistic environment compared to earlier work in both physical and virtual spaces.

Neuroscience studies have shown that incorporating gaze view with third view perspective has a great influence to correctly infer human behaviors. Given the importance of both first and third person observations for the recognition of human behaviors, we propose a method that incorporates these observations in a technical system to enhance the recognition of human behaviors, thus improving beyond third person observations in a more robust human activity recognition system. First, we present the extension of our proposed semantic reasoning method by including gaze data and external observations as inputs to segment and infer human behaviors in complex real-world scenarios. Then, from the obtained results we demonstrate that the combination of gaze and external input sources greatly enhance the recognition of human behaviors. Our findings have been applied to a humanoid robot to online segment and recognize the observed human activities with better accuracy when using both input sources; for example, the activity recognition increases from 77% to 82% in our proposed pancake-making dataset. To provide completeness of our system, we have evaluated our approach with another dataset with a similar setup as the one proposed in this work, that is, the CMU-MMAC dataset. In this case, we improved the recognition of the activities for the egg scrambling scenario from 54% to 86% by combining the external views with the gaze information, thus showing the benefit of incorporating gaze information to infer human behaviors across different datasets.

This paper proposes an online tactile transfer learning strategy for discriminating objects through the surface texture properties via a robotic hand and an artificial robotic skin. The proposed method has the ability to autonomously select and exploit the previously learned multiple texture models while discriminating new textures with a very few available training samples or even one. The experimental results show that employing the proposed method and 10 prior texture models, the robotic hand could discriminate 12 objects via their surface textures with 97% and 100% recognition accuracy with only one and ten training samples respectively. Moreover, the experimental outcomes illustrate that our proposed algorithm is robust against of any negative tactile knowledge transfer.

The development of breakthrough technologies helps the deployment of robotic systems in the industry. The implementation and integration of such technologies will improve productivity, flexibility and competitiveness, in diverse industrial settings specially for small and medium enterprises. In this paper we present a framework that integrates three novel technologies, namely safe robot arms with multi-modal and auto-calibrated sensing skin, a robot control framework to generate dynamic behaviors fusing multiple sensor signals, and an intuitive and fast teaching by demonstration method that segments and recognizes the robot activities on-line based on re-usable semantic descriptions. In order to validate our framework, these technologies are integrated in a industrial setting to sort and pack fruits. We demonstrate that our presented framework enables a standard industrial robotic system to be flexible, modular and adaptable to different production requirements.

In this paper we describe how we realized event-based signaling for large scale artificial robotic skin. We developed a new algorithm for the event generation on multi-modal skin cells. The skin cells have two modes, the conventional data sampling mode and the event mode. A comprehensive performance evaluation and comparison of these two modes is presented. We perform different experiments on our robot TOMM which has two UR5 robot arms, each covered with 260 multi-modal skin cells. Each skin cell samples 9 signals of 4 different modalities. Finally we derive models for extrapolating CPU usage and network traffic for larger numbers of skin cells and higher sample rates. The results show that the event-based system has superior performance and its performance edge increases with larger numbers of skin cells and higher sample rates. Experimental validation on our real robot system shows that in reactive control the event-based system reduces in comparison to the conventional system the packet rate by 48.2% and the CPU usage by 17.79%. We extrapolate the worst case for 5000 cells and show that the event-based system can at least reduce the packet rate by 21.2% and the CPU usage by 17.46%.

In this paper we propose event-based signaling for large-scale artificial robotic skin to reduce bandwidth requirements on data transmission and processing power. We use the send-on-delta principle to trigger the event generation only when tactile sensors are stimulated and transduce novel information. To compare the standard non-event based method with the proposed event-based method we present a comprehensive analysis of large-scale artificial skin systems for different test applications. For this purpose we collect data of 260 CellulARSkin cells on an UR-5 arm and calculate the events off-line. We determine the optimal packet size for event-based signaling and we show that the event-based system reduces the data rate with respect to the non-event based system for an unstimulated skin cell network to 16.45% and for a heavily stimulated skin cell network to 47.69%. The obtained results show that the event-based system reduces the data redundancy and the required transmission rates without loosing information.

We propose a fast method for salient region detection which aims at providing a computationally efficient method for online image processing. It is scalable and can be adjusted on the run to adapt to different computational requirements, which makes it a perfect candidate for time crucial applications. In our approach, we apply a template sampling over the image and compare these templates with each other by calculating a dissimilarity score. Templates with a low overall response are therefore likely to be part of a salient region in the image. This conceptually easy method is simple to implement and still outperforms state-of-the-art salient region detection systems (Our model's AUC(ROC) Score 0.794-AIM 0.772).

In this paper, we present an biologically-motivated object recognition system for robots and vision tasks in general. Our approach is based on a hierarchical model of the visual cortex for feature extraction and rapid scene categorization. We modify this static model to be usable in time-crucial real-world scenarios by applying methods for optimization from signal detection theory, information theory, signal processing and linear algebra. Our system is more robust to clutter and supports object localization by approaching the binding problem in contrast to previous models. We show that our model outperforms the preceding model and that by our modifications we created a robust and fast system which integrates the capabilities of biological-inspired object recognition in a technical application.

In this work, we reformulate our previously developed strategy for operating unknown constrained mechanisms, to fit our new task specification framework for indirect force controlled robots, which we presented recently. The main improvement is a significant reduction of erroneous forces, which is achieved by breaking the manipulation task down into a hierarchical set of subtasks instead of having interfering tasks. The improvement is evaluated by conducting a series of manipulation experiments using the old and new approach and comparing the average erroneous forces.

In our previous work we derived a task specification approach for indirect force controlled robots to assign force and positioning tasks in joint and Cartesian space and execute them simultaneously in a hierarchical way. The virtual set points for an underlying joint space indirect force controller have been computed according to the specified tasks, supporting reactive control by generating virtual velocity commands.

In this paper, we present a new approach to spatially self-organize a modular artificial skin in 3D space. We were motivated by the demand to efficiently and automatically acquire the position and orientation of a steadily growing number of artificial skin sensor elements. Here, we combine our 3D surface reconstruction algorithm for individual patches of artificial skin, with a common active visual marker approach. Light emitting diodes, built into every element of our modular artificial skin, enable us to turn each reconstructed patch of skin into an active 6 DoF visual marker. With the help of a calibrated monocular camera, we can then estimate the homogeneous transformations between multiple, at least partially visible skin patches e.g. when distributed on the body of a robot. Our approach allows to quickly combine distributed tactile and visual coordinate systems into one homogeneous rigid body representation. We demonstrate the robustness of our approach by calibrating several patches mounted on a robot arm using only a standard web-cam.

Automatically segmenting and recognizing human activities from observations typically requires a very complex and sophisticated perception algorithm. Such systems would be unlikely implemented on-line into a physical system, such as a robot, due to the pre-processing step(s) that those vision systems usually demand. In this work, we present and demonstrate that with an appropriate semantic representation of the activity, and without such complex perception systems, it is sufficient to infer human activities from videos. First, we will present a method to extract the semantic rules based on three simple hand motions, i.e. move, not move and tool use. Additionally, the information of the object properties either ObjectActedOn or ObjectInHand are used. Such properties encapsulate the information of the current context. The above data is used to train a decision tree to obtain the semantic rules employed by a reasoning engine. This means, we extract lower-level information from videos and we reason about the intended human behaviors (high-level). The advantage of the abstract representation is that it allows to obtain more generic models out of human behaviors, even when the information is obtained from different scenarios. The results show that our system correctly segments and recognizes human behaviors with an accuracy of 85%. Another important aspect of our system is its scalability and adaptability toward new activities, which can be learned on-demand. Our system has been fully implemented on a humanoid robot, the iCub to experimentally validate the performance and the robustness of our system during on-line execution of the robot.

The main contribution of this paper is the general formulation of force and positioning tasks on joint and Cartesian level for indirect force controlled robots and combining them in a strict hierarchical way. As a secondary contribution, we provide a simple and intuitive programming paradigm, using the developed formulation. By building on the well-established indirect force control scheme, which is often already provided for commercial robots, we provide application programmers with a useful tool for specifying tasks, involving positioning and force components. Different physical interaction tasks have been implemented to show the potential of the proposed method and discuss the general advantages and drawbacks.

In this paper, we present a tactile approach to grasp large and unknown objects, which can not be easily manipulated with a single end-effector or two-handed grasps, with the whole upper body of a humanoid robot. Instead of conventional joint level force sensing, we equip the robot with various patches of HEX-o-SKIN - a self-organizing, multi-modal cellular artificial skin. Low-level controllers, one allocated to each sensor cell, utilize a self-explored inverted jacobian-like sensory-motor map to directly transfer tactile stimulation into reactive arm motions, altering basic grasping trajectories to the need of the current object. A high-level state machine guides those low-level controllers during the different states of the grasping action. Desired contact points, and key poses for the trajectory generation, are taught through forceless tactile stimulation. First experiments on a position controlled robot, an HRP-2 humanoid, demonstrate the feasibility of our approach. Our paper contributes to the first realization of a self-organizing tactile sensor-behavior mapping on a full-sized humanoid robot, which enables: 1) a new general approach for grasping unknown objects with the whole-body; and 2) a novel way of teaching behaviors using pre-contact tactile sensing.

In the human brain, rewards are encoded in a flexible and adaptive way after each novel stimulus. Neurons of the orbitofrontal cortex are the key reward structure of the brain. Neurobiological studies show that the anterior cingulate cortex of the brain is primarily responsible for avoiding repeated mistakes. According to vigilance threshold, which denotes the tolerance to risks, we can differentiate between a learning mechanism that takes risks and one that averts risks. The tolerance to risk plays an important role in such a learning mechanism. Results have shown the differences in learning capacity between risk-taking and risk-avert behaviors. These neurological properties provide promising inspirations for robot learning based on rewards. In this paper, we propose a learning mechanism that is able to learn from negative and positive feedback with reward coding adaptively. It is composed of two phases: evaluation and decision making. In the evaluation phase, we use a Kohonen self-organizing map technique to represent success and failure. Decision making is based on an early warning mechanism that enables avoiding repeating past mistakes. The behavior to risk is modulated in order to gain experiences for success and for failure. Success map is learned with adaptive reward that qualifies the learned task in order to optimize the efficiency. Our approach is presented with an implementation on the NAO humanoid robot, controlled by a bioinspired neural controller based on a central pattern generator. The learning system adapts the oscillation frequency and the motor neuron gain in pitch and roll in order to walk on flat and sloped terrain, and to switch between them.

In this paper, we present a twofold, open-loop method to explore, model and calibrate articulated robots equipped with artificial skin. We do so, using a 3-axis accelerometer per artificial sensor skin unit (SU) and special excitation pattern on every actuated degree of freedom (DoF) of the robotic joints. The first algorithm extracts the kinematic dependencies in between segments, equipped with artificial skin units, and joints, featuring one or multiple rotatory DoFs. A second algorithm uses this structural knowledge to automatically build and estimate kinematic models in between a static reference and an end effector segment. We show experimental results for the structural exploration with a KUKA light weight robotic arm equipped with our own SU prototypes. Additional simulation results, supporting our approach on estimating the kinematic parameters of the robot, are also presented.

In this paper we propose a set-point-generator (SPG) for indirect-force-controlled (IFC) manipulators, interacting with mechanisms, which impose narrow bilateral kinematic constraints on their end effector, like doors, cabinets and drawers. These mechanisms could also have dynamic properties, due to their inertia, friction and gravity, which demands to consider applied forces when choosing a set-point for the IFC. Neither the type of the constraint (linear, circular), nor required manipulation forces are assumed to be known. The proposed SPG consists of two parts: i) estimating the single direction of possible motion based on filtering of the measured end effector velocities; and ii) choosing an appropriate set-point for the underlying IFC, resulting in an effective operation of the mechanism. A major aspect of our approach is to explore the kinematic constraints with the manipulators desired set-point, avoiding direct force control, as the required interaction force is unknown and hence there is no definite reference force. The presented approach is a generalization of our previous work on constrained manipulation of unknown mechanisms and extends the applicability to a wider class of manipulators by considering joint-level IFC and taking into account the applied forces in yielding a robust and effective controller. The approach is evaluated in various experiments on a manipulator, providing joint space compliance.

In this paper, we present a new approach to reconstruct the 3D surface of robotic body parts equipped with artificial skins. We do so by fusing static knowledge on the shape, size and tessellation capabilities of the uniform cell our skin consists of, together with dynamic knowledge on its neighbors and measurements from its orientation sensor - a 3-axis accelerometer. Our approach makes it possible to reconstruct the 3D surface of robotic body parts equipped with a patch of skin in a very short time, providing the location and orientation of every unit in a patch relative to an automatically chosen origin on the patch, utilizing no external sensors and only robot independent information. We show experimental results on the 3D reconstruction of different skin patches.

The belief that artificial agents are useful interaction partners in cognitive therapies of social disorders such as autism fuels an increasing number of research projects involving the developments of robots and computer avatars. Yet, for an appropriate use of these new tools, it is necessary to understand how perception of and interaction with artificial agents differ from natural agents, both in normally developed adults and in patients with disorders of social cognition. Here we investigated the neural bases of social interactions with a human or with a humanoid robot using fMRI. During interaction, participants were playing a computerized version of the game chifumi (stone-paper-scissors), while believing they were interacting ‘live’ either with a fellow human (Intentional agent, Int), a humanoid robot endowed with an artificial intelligence (Artificial agent, Art), or a random number generator running on a laptop (Rnd). The belief was built both with an extensive briefing before the actual experiment, including a live interaction with the human and humanoid robot opponents, and with videos of the opponent, presented as live but actually recorded prior to the experiment, that preceded and followed each series of 5 games played against one opponent in a block. Results indicate that the brain network found when interacting with an active opponent (Art & Int vs Rnd) was more activated when interacting with the human than with the robot agent, implying that interacting with a human is more engaging than interacting with an artificial agent. Areas involved in social interactions in the posterior temporal sulcus were activated when controls, but not high-functioning autistic patients patients, interacted with a fellow human.

This paper introduces Gertie the Robotic Desk Lamp, a novel research platform that has five degrees of freedom, and is equipped with a camera and microphone in its lamp shade. These features mean that Gertie is a flexible and low-cost resource for conducting research into cognitive products and human-robot interaction. It will be available as an open hardware on http://www.opengertie.org/. Gertie was designed from first principles, and assembled using off the shelf electronic components and parts fabricated using a 3D printer. In this paper, the design of Gertie is presented, and its application as a research platform is described. Gertie has already been used to investigate a problem of simple object tracking, building on computer vision algorithms. Furthermore, it has also been used to investigate and replicate emotional body language. By imitating human body language Gertie is capable of expressing four of the basic Ekman emotions: 1) joy; 2) sadness; 3) surprise; and 4) fear. This work was validated using an online study, which investigates how well the emotions expressed by Gertie are recognized by human audiences. In total 84 participants were shown one video for each of the four emotions and they were asked to choose from a list of seven emotions, which they thought was displayed by Gertie. While joy and sadness were recognized very reliably with 81% and 88% of all people giving the correct answer, fear and surprise were more commonly misinterpreted as surprise and disgust. However, all emotions were recognized above the chance level percentage of 14%.

A retro-projected face display system has great advantages in being able to present realistic 3D appearances to users and to easily switch the appearance of the humanoid robot heads animated on the display. Therefore, it is useful to evaluate how effectively users can perceive various information from such devices and what type of animation is suitable for human-robot interaction - in particular, face-to-face communication with robots. In this paper, we examine how facial texture images affect people's ability to identify the gender of faces displayed on a retro-projected face screen system known as Mask-bot. In an evaluation study, we use a female face screen as the 3D output surface, and display various face images morphed between male and female. Subjects are asked to rate the gender of each projected face. We found that even though the output 3D mask screen has a female shape, gender identification is strongly determined by texture images, especially in the case of high-quality images.

Here we describe a parameter-driven solution for generating novel yet similar movements from a sparse example set obtained through observation. In our experiments, a humanoid learns to represent movement trajectories demonstrated by a person with intuitive parameters describing the start and end points of different motion trajectory segments. These segments are automatically produced based on changes in curvature. After rebinning to equate similar segments across the samples, we use a linear approximation framework to build a representation based on relevant task features (segment start and end points) where radial basis functions(RBFs) are used to approximate the unknown non-linear characteristics describing a trajectory. The solution is accomplished on-line and requires no interaction. With this approach a humanoid can learn from only a few examples, and quickly produce new movements.

In this paper, we present a new generation of active tactile modules (i.e., HEX-O-SKIN), which are developed in order to approach multimodal whole-body-touch sensation for humanoid robots. To better perform like humans, humanoid robots need the variety of different sensory modalities in order to interact with their environment. This calls for certain robustness and fault tolerance as well as an intelligent solution to connect the different sensory modalities to the robot. Each HEX-O-SKIN is a small hexagonal printed circuit board equipped with multiple discrete sensors for temperature, acceleration, and proximity. With these sensors, we emulate the human sense of temperature, vibration, and light touch. Off-the-shelf sensors were utilized to speed up our development cycle; however, in general, we can easily extend our design with new discrete sensors, thereby making it flexible for further exploration. A local controller on each HEX-O-SKIN preprocesses the sensor signals and actively routes data through a network of modules toward the closest PC connection. Local processing decreases the necessary network and high-level processing bandwidth, while a local analog-to-digital conversion and digital-data transfers are less sensitive to electromagnetic interference. With an active data-routing scheme, it is also possible to reroute the data around broken connections-yielding robustness throughout the global structure while minimizing wirings. To support our approach, multiple HEX-O-SKIN are embedded into a rapid-prototyped elastomer skin material and redundantly connected to neighboring modules by just four ports. The wiring complexity is shifted to each HEX-O-SKIN such that a power and data connection between two modules is reduced to four noncrossing wires. Thus, only a very simple robot-specific base frame is needed to support and wire the HEX-O-SKIN to a robot. The potential of our multimodal sensor modules is demonstrated experimentally on a robot platform.

This work aims at the development of a versatile control strategy for operating unknown mechanically constrained devices such as drawers or doors. Few assumptions on the device's shape as well as the utilized hardware are required. Our approach is based on an on-line estimation of the constraint manifold which serves as a reference input for an admittance-type controller providing the compliance required. The direction estimation is obtained from the velocity signal in task space. An on-line adaptation of the admittance controller according to the estimated moving direction reduces contact forces. The functionality of the control strategy is demonstrated on a mobile manipulator in a kitchen environment.

When a robot wants to manipulate an object, it needs to know what action to execute to obtain the desired result. In most of the cases, the actions that can be applied to an object consist of exerting forces to it. If a robot is able to predict what will happen to an object when some force is applied to it, then it's possible to build a controller that solves the inverse problem of what force needs to be applied in order to get a desired result. To accomplish this, the first task is to build an object model and second to get the right parameters for it. The goals of this paper are 1) to demonstrate the use of an object model to predict outcomes of actions, and 2) to adapt this model to an specific object instance for a specific robot.

Actuation of robotic systems is still an open question and represents a big challenge. Demanding performances including high power to mass ratio, capability of producing high power at low speed within a small-occupied volume are some of the key issues that required careful consideration. These criteria aimed to increase autonomy of humanoid robots. In this paper, a novel hydrostatic transmission actuator is proposed. The proposed actuator is controlled by displacement and has capacities for energy storage. This leads to an optimal solution in terms of power consumption. First, the proposed hydrostatic actuation principle is explained. A simplified hydraulic scheme to illustrate the energy storage capability is then provided. A mathematical model of the proposed solution is detailed showing our ability to access to the payload “jerk”. The built prototype is presented and its properties are outlined. Finally, a prototype of the actuator and the preliminary results of the actuator performance are presented, demonstrating the novelty of our solution.

Neurobiology studies showed that the role of the anterior cingulate cortex of the brain is primarily responsible for avoiding repeated mistakes. According to vigilance threshold, which denotes the tolerance to risks, we can differentiate between a learning mechanism that takes risks, and one that averts risks. The tolerance to risk plays an important role in such learning mechanism. Results have shown the differences in learning capacity between risk-taking and risk avert behaviors. In this paper, we propose a learning mechanism that is able to learn from negative and positive feedback. It is composed of two phases, evaluation and decision-making phase. In the evaluation phase, we use a Kohonen Self Organizing Map technique to represent success and failure. Decision-making is based on an early warning mechanism that enables to avoid repeating past mistakes. Our approach is presented with an implementation on a simulated planar biped robot, controlled by a reflexive low-level neural controller. The learning system adapts the dynamics and range of a hip sensor neuron of the controller in order for the robot to walk on flat or sloped terrain. Results show that success and failure maps can learn better with a threshold that is more tolerant to risk. This gives rise to robustness to the controller even in the presence of slope variations.

In this video presentation we introduce a 50 degrees of freedom humanoid robot, CB -ComputationalBrain[1]. CB is a humanoid robot created for exploring the underlying processing of the human brain while dealing with the real world. We place our investigations within real world contexts, as humans do. In so doing, we focus on utilising a system that is closer to humans - in sensing, kinematics configuration and performance. We present a full-body compliance controller that was developed for the motion control of our humanoid robot [2]. Our initial experimentation on our system includes: 1) full-body compliant control - physical interactions/balancing/motion control; 2) the integrated visual ocular-motor responses; 3) perception and control - reaching, foveation, and active object recognition; 4) our studies of Central Pattern Generator for walking.

In this paper, we propose a simple framework for learning and synthesis of fast and complex motor tasks. Where a passivity-based task-space controller acts not only as a full-body force control module, but also as an important module to generate phasic joint patterns. The generated joint patterns are encoded into the parameters of phase oscillators and form the synergy of the task. Then, similar and/or faster motions are synthesized by superposing the task space controller output and the oscillator output with the modified oscillator amplitudes and/or frequencies. We present some examples of whole-body motion synthesis on a human-sized biped humanoid robot including squatting, dancing and stepping while bipedal balancing. The simulation and experimental videos are supplemented.

We propose using the kernel dimension reduction (KDR) to extract a low-dimensional feature space for humanoid locomotion tasks. Although humanoids have many degrees of freedom, task relevant feature spaces can be much smaller than the number of dimension of the original state space. We consider an application of the proposed approach to improve the locomotive performance of humanoid robots using an extracted low-dimensional state space. To improve the locomotive performance, we use a reinforcement learning (RL) framework. While RL is a useful non-linear optimizer, it is usually difficult to apply RL to real robotic systems - due to the large number of iterations required to acquire suitable policies. In this study, we use the extracted low-dimensional feature space for RL so that the learning system can improve task performance quickly. The kernel dimension reduction method allows us to extract the feature space even if the task relevant mapping is non-linear. This is an essential property to improve humanoid locomotive performance since stepping or walking dynamics involves highly nonlinear dynamics. We show that we can improve stepping and walking policies by using a RL method on an extracted feature space by using KDR.

Biological systems seem to have a simpler but more robust locomotion strategy than that of the existing biped walking controllers for humanoid robots. We show that a humanoid robot can step and walk using simple sinusoidal desired joint trajectories with their phase adjusted by a coupled oscillator model. We use the center-of-pressure location and velocity to detect the phase of the lateral robot dynamics. This phase information is used to modulate the desired joint trajectories. We do not explicitly use dynamical parameters of the humanoid robot. We hypothesize that a similar mechanism may exist in biological systems. We applied the proposed biologically inspired control strategy to our newly developed human-sized humanoid robot computational brain (CB) and a small size humanoid robot, enabling them to generate successful stepping and walking patterns.

This paper proposes a simple passivity-based disturbance rejection scheme for force-controllable biped humanoids. The disturbance rejection by force control is useful not only for self-balance, but also for stable and safety physical interaction between human and humanoid robots. The core technique is passivity-based contact force control with gravity-compensation. This makes it easy to control the contact forces in a satisfactory dynamic range without canceling all non-linear terms. The disturbance rejection is located at the higher layer above the contact force controller. It is composed of three sub-controllers; 1) a balancing controller; 2) a stepping controller; and 3) the trigger. Numerical simulations and experiments evaluate the effectiveness of the proposed controller. Although the method is incomplete in the sense that the self-collision between the limbs is ignored, a preliminary experimental result on a real humanoid platform demonstrates that the proposed method can actually make the robot recover the balance under large unknown external perturbations.

This paper presents a novel approach to acquire dynamic whole-body movements on humanoid robots focused on learning a control policy for the center of mass. A policy-gradient method is used to acquire a CoM movement as a control policy for achieving a desired dynamic task. A CoM-Jacobian-based redundancy resolution is then used to compute angular velocities for all joints in order to achieve a whole-body movement consistent with the CoM movement acquired through learning. To demonstrate the effectiveness of our method, we apply it in simulation to the learning of a strong punching movement on the Fujitsu humanoid robot, Hoap-2.

We propose a framework for skill synthesis for robots that exploits the human capacity to learn novel control tasks. The conceptual idea is to incorporate the target robotic platform into the experimenter's body schema so that it can be controlled effortlessly as if the robot were a part of the body. Once this stage is achieved, the dexterity on a task exhibited with the new external limb -the robot- can be used for designing controllers for the task under consideration. This article exemplifies the proposed framework by showing the derivation of an effective open-loop controller that can manipulate two balls with the fingers of a 16-DOF robotic hand.

We propose to improve the locomotive performance of humanoid robots by using approximated biped stepping and walking dynamics with reinforcement learning (RL). Although RL is a useful non-linear optimizer, it is usually difficult to apply RL to real robotic systems - due to the large number of iterations required to acquire suitable policies. In this study, we first approximated the dynamics by using data from a real robot, and then applied the estimated dynamics in RL in order to improve stepping and walking policies. Gaussian processes were used to approximate the dynamics. By using Gaussian processes, we could estimate a probability distribution of a target function with a given covariance function. Thus, RL can take the uncertainty of the approximated dynamics into account throughout the learning process. We show that we can improve stepping and walking policies by using a RL method with the approximated models both in simulated and real environments. Experimental validation on a real humanoid robot of the proposed

A cognitive robot system has to acquire and efficiently store vast knowledge about the world it operates in. To cope with every day tasks, a robot needs to learn, classify and recognize a manifold of different objects. Our work focuses on an object representation scheme that allows storing perceived objects in a compact way. This will enable the system to store extensive information about the world and will ease complex recognition tasks. The human visual system deploys several mechanisms to reduce the amount of information. Our goal is to develop an artificial system that mimics these mechanisms to create representations that can be used in cognitive tasks. In particular, in this paper we will present an approach that exploits similarities among different views of objects. The proposed representation scheme allows for reduction of storage required for the representation of objects and preserves the information about the similarity among objects. This is achieved by selecting 'important views' of objects, depending on their stability. Furthermore, by extending the same approach to multiple objects, we are able to exploit similarities between objects to find a common representation and to further reduce the storage requirements.

Interaction between humans involves a plethora of sensory information, both in the form of explicit communication as well as more subtle unconsciously perceived signals. In order to enable natural human-robot interaction, robots will have to acquire the skills to detect and meaningfully integrate information from multiple modalities. In this article, we focus on sound localization in the context of a multi-sensory humanoid robot that combines audio and video information to yield natural and intuitive responses to human behavior, such as directed eye-head movements towards natural stimuli. We highlight four common sound source localization algorithms and compare their performance and advantages for real-time interaction. We also briefly introduce an integrated distributed control framework called DVC, where additional modalities such as speech recognition, visual tracking, or object recognition can easily be integrated. We further describe the way the sound localization module has been integrated in our humanoid robot, CB.

This paper proposes an effective framework of human-humanoid robot physical interaction. Its key component is a new control technique for full-body balancing in the presence of external forces, which is presented and then validated empirically. We have adopted an integrated system approach to develop humanoid robots. Herein, we describe the importance of replicating human-like capabilities and responses during human-robot interaction in this context. Our balancing controller provides gravity compensation, making the robot passive and thereby facilitating safe physical interactions. The method operates by setting an appropriate ground reaction force and transforming these forces into full-body joint torques. It handles an arbitrary number of force interaction points on the robot. It does not require force measurement at interested contact points. It requires neither inverse kinematics nor inverse dynamics. It can adapt to uneven ground surfaces. It operates as a force control process, and can therefore, accommodate simultaneous control processes using force-, velocity-, or position-based control. Forces are distributed over supporting contact points in an optimal manner. Joint redundancy is resolved by damping injection in the context of passivity. We present various force interaction experiments using our full-sized bipedal humanoid platform, including compliant balance, even when affected by unknown external forces, which demonstrates the effectiveness of the method.

We show that a humanoid robot can step and walk using simple sinusoidal desired joint trajectories with their phase adjusted by a coupled oscillator model. We use the center of pressure location and velocity to detect the phase of the lateral robot dynamics. This phase information is used to modulate the desired joint trajectories. We applied the proposed control approach to our newly developed human sized humanoid robot and a small size humanoid robot developed by Sony, enabling them to generate successful stepping and walking patterns

In this paper we discuss active humanoid vision systems that realize foveation using two rigidly connected cameras in each eye. We present an exhaustive analysis of the relationship between the positions of the observed point in the foveal and peripheral view with respect to the intrinsic and extrinsic parameters of both cameras and 3-D point position. Based on these results we propose a control scheme that can be used to maintain the view of the observed object in the foveal image using information from the peripheral view. Experimental results showing the effectiveness of the proposed foveation control are also provided

This paper presents a case study of learning to select behavioral primitives and generate subgoals from observation and practice. Our approach uses local features to generalize across tasks and global features to learn from practice. We demonstrate this approach applied to the marble maze task. Our robot uses local features to initially learn primitive selection and subgoal generation policies from observing a teacher maneuver a marble through a maze. The robot then uses this information as it tries to traverse another maze, and refines the information during learning from practice

This paper proposes a passivity-based hierarchical full-body motion controller for force-controllable multi-DOF humanoid robots. The task-space forces are treated in a uniform manner for a variety of position/force tracking and force/moment compensation. The contact force closure is optimally solved and transformed directly into the joint torques in real-time without any joint trajectory planning. With this framework, we introduce gravity compensation at the lowest layer of the controller that makes the closed-loop system passive with respect to additional inputs as well as external forces. Furthermore, we propose two upper-layers: one layer controls the ground reaction forces, which enables the robot keep the dynamic balance. The other layer is the another passification control, which constructs an invariant manifold that prevents the robot from falling during walking. Four realistic dynamic simulations: balanced squatting, reaching, externally driven, or speed-controlled walking with disturbances demonstrate the effectiveness of the proposed methods

Recently, there has been a growing interest in biologically inspired biped locomotion control with Central Pattern Generator (CPG). However, few experimental attempts on real hardware 3D humanoid robots have yet been made. Our goal in this paper is to present our achievement of 3D biped locomotion using a neural oscillator applied to a humanoid robot, QRIO. We employ reduced number of neural oscillators as the CPG model, along with a task space Cartesian coordinate system and utilizing entrainment property to establish stable walking gait. We verify robustness against lateral perturbation, through numerical simulation of stepping motion in place along the lateral plane. We then implemented it on the QRIO. It could successfully cope with unknown 3mm bump by autonomously adjusting its stepping period. Sagittal motion produced by a neural oscillator is introduced, and then overlapped with the lateral motion generator in realizing 3D biped locomotion on a QRIO humanoid robot.

We propose a model-based reinforcement learning algorithm for biped walking in which the robot learns to appropriately modulate an observed walking pattern. Via-points are detected from the observed walking trajectories using the minimum jerk criterion. The learning algorithm modulates the via-points as control actions to improve walking trajectories. This decision is based on a learned model of the Poincaré map of the periodic walking pattern. The model maps from a state in the single support phase and the control actions to a state in the next single support phase. We applied this approach to both a simulated robot model and an actual biped robot. We show that successful walking policies are acquired.

Humanoid research has made remarkable progress during the past 10 years. However, currently most humanoids use the target ZMP (zero moment point) control algorithm for bipedal locomotion, which requires precise modeling and actuation with high control gains. On the contrary, humans do not rely on such precise modeling and actuation. Our aim is to examine biologically related algorithms for bipedal locomotion that resemble human-like locomotion. This paper describes an empirical study of a neural oscillator for the control of biped locomotion. We propose a new neural oscillator arrangement applied to a compass-like biped robot. Dynamic simulations and experiments with a real biped robot were carried out and the controller performs steady walking for over 50 steps. Gait variations resulting in energy efficiency was made possible through the adjustment of only a single neural activity parameter. Aspects of adaptability and robustness of our approach are shown by allowing the robot to walk over terrains with varying surfaces with different frictional properties. Initial results suggesting optimal amplitude for dealing with perturbation are also presented.

We propose a model-based reinforcement learning algorithm for biped walking in which the robot learns to appropriately place the swing leg. This decision is based on a learned model of the Poincare map of the periodic walking pattern. The model maps from a state at the middle of a step and foot placement to a state at next middle of a step. We also modify the desired walking cycle frequency based on online measurements. We present simulation results, and are currently implementing this approach on an actual biped robot.

We propose a framework for learning biped locomotion using dynamical movement primitives based on nonlinear oscillators. In our previous work, we suggested dynamical movement primitives as a central pattern generator (CPG) to learn biped locomotion from demonstration. We introduced an adaptation algorithm for the frequency of the oscillators based on phase resetting at the instance of heel strike and entrainment between the phase oscillator and mechanical system using feedback from the environment. In this paper, we empirically explore the role of phase resetting in the proposed algorithm for robust biped locomotion. We demonstrate that phase resetting contributes to robustness against external perturbations and environmental changes by numerical simulations and experiments with a physical biped robot.

Object recognition requires a robot to perform a number of nontrivial tasks such as finding objects of interest, directing its eyes towards the objects, pursuing them, and identifying the objects once they appear in the robot's central vision. We have recently developed a recognition system on a humanoid robot, which makes use of foveated vision to accomplish these tasks (A Ude, et al., 2003). In this paper we present several substantial improvements to this system. We present a biologically motivated object representation scheme based on Gabor kernel functions and show how to employ support vector machines to identify known objects in foveal images based on this representation. A mechanism for visual search is integrated into the system to find objects of interest in peripheral images. The framework also includes a control scheme for eye movements, which are directed using the results of attentive processing in peripheral images.

Smooth control using an active vision head's verge-axis joint is performed through continuous state and action reinforcement learning. The system learns to perform visual servoing based on rewards given relative to tracking performance. The learned controller compensates for the velocity of the target and performs lag-free pursuit of a swinging target. By comparing controllers exposed to different environments we show that the controller is predicting the motion of the target by forming an implicit model of the target's motion. Experimental results are presented that demonstrate the advantages and disadvantages of implicit modelling.

This paper focuses on learning to select behavioral primitives and generate sub-goals from practicing a task. We present a novel algorithm that combines Q-learning and a locally weighted learning method to improve primitive selection and sub-goal generation. We demonstrate this approach applied to the tilt maze task. Our robot initially learns to perform this task using learning from observation, and then learns from practice.

An essential problem of imitation is that of determining "what to imitate", i.e. to determine which of the many features of the demonstration are relevant to the task and which should be reproduced. The strategy followed by the imitator can be modeled as a hierarchical optimization system, which minimizes the discrepancy between two multi-dimensional datasets. We consider imitation of a manipulation task. To classify across manipulation strategies, we apply a probabilistic analysis to data in Cartesian and joint spaces. We determine a general metric that optimizes the policy of task reproduction, following strategy determination. The model successfully discovers strategies in six different manipulation tasks and controls task reproduction by a full body humanoid robot.

In this paper we present a humanoid system that can integrate information provided by its foveal and peripheral cameras. We use peripheral vision to detect and pursue objects of interest based on simple shape and color models. A detection event triggers the robot to direct its eyes towards the object, thus making a more detailed analysis of the observed objects in higher resolution foveal images feasible. The recognition is based on principal component analysis and is performed while the robot actively pursues the detected object. The classification results are inferred using information from a video stream rather than just a single image. Once the desired object is recognized, the robot reaches for it while ignoring other objects.

This paper describes humanoid robot learning from observation and game playing using information provided by a real-time PC-based vision system. To cope with extremely fast motions that arise in the environment, a visual system capable of perceiving the motion of several objects at 60 fields per second was developed. We have designed a suitable error recovery scheme for our vision system to ensure successful game playing over longer periods of time. To increase the learning rate of the robot it is given domain knowledge in the form of primitives. The robot learns how to perform primitives from data collected while observing a human. The robot control system and primitive use strategy are also explained.

This paper presents the final stage of development of the humanoid system, ETL-Humanoid. It is full-scale humanoid system with 46 degrees of freedom, with the height and weight of an average Japanese person. It was designed as an experimental platform, to explore the general principle of controls of complex embodied systems. The complete system will be presented; the mechanical configuration of the system and the low-level network-based control system will also be presented. The final system possesses properties of compactness, modularity and is light in weight. The mechanical system is high in performance, is backdrivable and compliant, allowing the possibility of a wide range of motions and capabilities. The general capability of being able to support itself is demonstrated. A "Chin Up" experiment showing the physical strength of our system is presented. The system is able to support its own body weight while rising up to a supporting bar. Aside from its physical strength, the system is also capable of performing higher-level perceptions and actions. These capabilities will be briefly presented.

Human interaction involves a number of factors. One key and noticeable factor is the mass perceptual problem. Humans are equipped with a large number of receptors, equipped for seeing, hearing and touching, to name just a few. These stimuli bombard us continuously, often not on a singular basis. Typically multiple stimuli are activated at once, and in responding to these stimuli, variations of responses are exhibited. The current aim of our project is to provide an architecture, that will enable a humanoid robot to yield meaningful responses to complex and continuous interactions, similar to that of humans. We present our humanoid, a system which is able to simultaneously detect the spatial orientation of a sound source, and is also able to detect and mimic the motion of the upper body of a person. The motion produced by our system is human like-ballistic motion. The focus of the paper is on how we have come about the integration of these components. A continuous interactive experiment is presented in demonstrating our initial effort. The demonstration is in the context of our humanoid interacting with a person. Through the use of spatial hearing and multiple visual cues, the system is able to track a person, while mimicking the persons upper body motion. The system has shown to be robust and tolerable to failure, in performing experiments for a long duration of time.

Presents the hardware of the upper body of the ETL-Humanoid system. It has 24 degrees of freedom with high performance actuators, which gives the system high mobility, such as is required for pushups, lifting itself or another person. The mechanical design is able to achieve human proportions, with smooth outer shape, and is light weight at a very high level, considering the high motor performance embedded inside. Custom compact electronics for AC servo control and fast embedded network for control and measurement was also developed. The system is still evolving, being used as a working testbed for versatile human interaction experiments.

We describe a mobile robot system that performs goal-oriented visual navigation using a behaviour-based architecture. The system was constructed as a result of our previous research (Zelinsky et al. (1995)) into behaviour-based mobile robot systems. Since the system is implemented using real-time vision it is able to navigate in dynamic and unknown environments. The most important feature of this navigation system is that the mobile robot can learn to resolve conflicts between its own internal behaviours, such as moving to a goal while avoiding obstacles. Our robot is able to efficiently avoid obstacles while trying to reach a goal. Experimental results of an implementation on a Yamabico mobile robot are presented.

In this paper we propose a new paradigm for teleoperating a mobile robot. Our teleoperation paradigm is made up of five major components: "self-preservation"; "instructive feedback"; "qualitative instructions"; "qualitative explanations"; and a "user interface". Our aim is to provide a mobile robot with autonomy while being teleoperated. Our approach can be used to combat the continuous closed-loop problem. A qualitative approach has been taken in the design of each of the components. The usefulness of our approach is demonstrated with an implementation of a user interface to teleoperate an autonomous mobile robot equipped with a vision based navigation system. In this paper we describe the implementation of our teleoperation system.

We present an approach for using vision as the primary source of sensing to guide a mobile robot in an unknown environment. We define a set of primitive visual behaviours for navigating a mobile robot in real-time. By combining such behaviours with a purposive map, our mobile robot exhibits a goal seeking behaviour. We present a fast segmentation technique for vision processing. This processing technique is used by different behaviours to produce an overall competent behaviour in our Yamabico robot. Experimental results show that our robot can navigate competently in dynamic indoor environments.