Deep Reinforcement Learning (DRL) has made considerable advances in simulated and physical robot control tasks, especially when problems admit a fully observed Markov Decision Process (MDP) formulation. When observations only partially capture the underlying state, the problem becomes a Partially Observable MDP (POMDP), and performance rankings between algorithms can change. We empirically compare Proximal Policy Optimization (PPO), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Soft Actor-Critic (SAC) on representative POMDP variants of continuous-control benchmarks. Contrary to widely reported MDP results where TD3 and SAC typically outperform PPO, we observe an inversion: PPO attains higher robustness under partial observability. We attribute this to the stabilizing effect of multi-step bootstrapping. Furthermore, incorporating multi-step targets into TD3 (MTD3) and SAC (MSAC) improves their robustness. These findings provide practical guidance for selecting and adapting DRL algorithms in partially observable settings without requiring new theoretical machinery.

A promising characteristic of Deep Reinforcement Learning (DRL) is its capability to learn optimal policy in an end-to-end manner without relying on feature engineering. However, most approaches assume a fully observable state space, i.e. fully observable Markov Decision Processes (MDPs). In real-world robotics, this assumption is unpractical, because of issues such as sensor sensitivity limitations and sensor noise, and the lack of knowledge about whether the observation design is complete or not. These scenarios lead to Partially Observable MDPs (POMDPs). In this paper, we propose Long-Short-Term-Memory-based Twin Delayed Deep Deterministic Policy Gradient (LSTM-TD3) by introducing a memory component to TD3, and compare its performance with other DRL algorithms in both MDPs and POMDPs. Our results demonstrate the significant advantages of the memory component in addressing POMDPs, including the ability to handle missing and noisy observation data.

Physical agents that can autonomously generate engaging, life-like behavior will lead to more responsive and user-friendly robots and other autonomous systems. Although many advances have been made for one-to-one interactions in well-controlled settings, physical agents should be capable of interacting with humans in natural settings, including group interaction. To generate engaging behaviors, the autonomous system must first be able to estimate its human partners’ engagement level. In this article, we propose an approach for estimating engagement during group interaction by simultaneously taking into account active and passive interaction, and use the measure as the reward signal within a reinforcement learning framework to learn engaging interactive behaviors. The proposed approach is implemented in an interactive sculptural system in a museum setting. We compare the learning system to a baseline using pre-scripted interactive behaviors. Analysis based on sensory data and survey data shows that adaptable behaviors within an expert-designed action space can achieve higher engagement and likeability.

Multi-step (also called n-step) methods in Reinforcement Learning (RL) have been shown to be more efficient than the 1-step method due to faster propagation of the reward signal, both theoretically and empirically, in tasks exploiting tabular representation of the value-function. Recently, research in Deep Reinforcement Learning (DRL) also shows that multi-step methods improve learning speed and final performance in applications where the value-function and policy are represented with deep neural networks. However, there is a lack of understanding about what is actually contributing to the boost of performance. In this work, we analyze the effect of multi-step methods on alleviating the overestimation problem in DRL, where multi-step experiences are sampled from a replay buffer. Specifically building on top of Deep Deterministic Policy Gradient (DDPG), we propose Multi- step DDPG (MDDPG), where different step sizes are manually set, and a variant called Mixed Multi-step DDPG (MMDDPG) where an average over different multi-step backups is used as an update target for the Q-value function. Empirically, we show that both MDDPG and MMDDPG are significantly less affected by the overestimation problem than DDPG with 1-step backup, which consequently results in better final performance and learning speed. We also discuss the advantages and disadvantages of different ways to do multi-step expansion in order to reduce approximation error, and expose the tradeoff between overestimation and underestimation that underlies offline multi-step methods. Finally, we compare the computational resource needs of MDDPG and MMDDPG with those of Twin Delayed Deep Deterministic Policy Gradient (TD3), a state-of-the-art algorithm proposed to address overestimation in actor-critic methods, since they show comparable final performance and learning speed.

One of the main challenges of bipedal gait is to avoid falling due to unknown disturbances. Compensating for these disturbances in bipeds is often achieved by leaning or stepping. In this work, the Spherical Foot Placement Estimator (SFPE) is introduced, which uses the biped's current kinematics and dynamics to predict if a step is needed, and if so where to step, to restore balance in 3D. An example of a controller using the SFPE is shown, which augments an existing optimal controller with both leaning and stepping: SFPE-based feedback is used to generate a desired momentum for momentum-based leaning while the SFPE point is used as a control reference for stepping. The new estimator outperforms existing balance criteria by providing both recovery step location prediction and momentum objectives with smooth dynamics.

To enable long term, engaging social human-machine interaction, robots and other autonomous systems must be able to move beyond purely reactive interaction control strategies, and engage in shared-initiative interaction. In this paper, we describe an implementation of an interactive art sculpture which generates interactive behaviours using curiosity-based learning. Using its own internal motivation formulated as a curiosity drive, the system initiates interaction with and responds to human visitors, generating continuously evolving interactive behaviours. The proposed system was tested in a user study with a prototype interactive sculpture installation.

The ability to engage human observers is a key requirement for both social robots and the arts. In this paper, we propose an approach for adapting the Intelligent Adaptive Curiosity learning algorithm to distributed interactive sculptural systems. This Curiosity-Based Learning Algorithm (CBLA) allows the system to learn about its own mechanisms and its surroundings through self-experimentation and interaction. A novel formulation using multiple agents as learning subsets of the system that communicate through shared input variables enables us to scale to a much larger system with diverse types of sensors and actuators. Experiments on a prototype interactive sculpture demonstrate the exploratory patterns of the CBLA and collective learning behaviours through the integration of multiple learning agents.

This paper investigates the design and implementation of a finger-like robotic structure capable of reproducing human hand gestural movements performed by a multi-fingered, hand-like structure. In this work, we present a pneumatic circuit and a closed-loop controller for a finger-like soft pneumatic actuator. Experimental results demonstrate the performance of the pneumatic and control systems of the soft pneumatic actuator, and its ability to track human movement trajectories with affective content.

Soft pneumatic artificial muscles (SPAMs) are a type of pneumatic actuator that provide customizable motion trajectories in three dimensional space without the need for rigid links or a transmission mechanism. This paper presents a novel design for producing SPAMs, named wrapped SPAMs (WSPAMs). Unlike previous SPAM designs, the production process of WSPAM is highly repeatable, while the motion trajectory can be easily modified. A model for predicting the steady-state angular displacement of a WSPAM actuator based on its geometrical parameters and the elasticity of the materials used in its production is presented and experimentally validated.

For an engaging human-machine interaction, machines need to be equipped with affective communication abilities. Such abilities enable interactive machines to recognize the affective expressions of their users, and respond appropriately through different modalities including movement. This paper focuses on bodily expressions of affect, and presents a new computational model for affective movement recognition, robust to kinematic, interpersonal, and stochastic variations in affective movements. The proposed approach derives a stochastic model of the affective movement dynamics using hidden Markov models (HMMs). The resulting HMMs are then used to derive a Fisher score representation of the movements, which is subsequently used to optimize affective movement recognition using support vector machine classification. In addition, this paper presents an approach to obtain a minimal discriminative representation of the movements using supervised principal component analysis (SPCA) that is based on Hilbert-Schmidt independence criterion in the Fisher score space. The dimensions of the resulting SPCA subspace consist of intrinsic movement features salient to affective movement recognition. These salient features enable a low-dimensional encoding of observed movements during a human-machine interaction, which can be used to recognize and analyze human affect that is displayed through movement. The efficacy of the proposed approach in recognizing affective movements and identifying a minimal discriminative movement representation is demonstrated using two challenging affective movement datasets.

The Laban Effort and Shape components provide a systematic tool for a compact and informative description of the dynamic qualities of movements. To enable the application of Laban notation in computational movement analysis, measurable physical correlates of Effort and Shape components need to be identified. Such physical correlates enable quantification of Effort and Shape components, which in turn facilitates computational analysis of affective movements. In this work, two existing approaches to quantification of Laban Effort components (Weight, Time, Space, and Flow) based on measurable movement features (position, velocity, acceleration, and jerk) are adapted for hand and arm movements, and a new approach for quantifying Shape Directional based on the average trajectory curvature is proposed. The results show a high correlation between Laban annotations provided by a certified movement analyst (CMA) and the quantified Effort Weight (81%), Time (77%) and Shape Directional (93%) for an affective hand and arm movement dataset.

Aurora is a large-scale kinetic art installation that reacts to human presence directly, with sensors triggering outputs, and indirectly, by modifying output behaviour rules. This paper describes a novel method for estimating visitors' fields of interest, their attention to specific parts of the installation, with a future goal of using this measure as a fitness function for output behaviour modification based on genetic algorithms. Due to constraints in Aurora, distributed overhead distance sensors were used as the sensory inputs. A low resolution height graph of the space below the installation is created, and the active sensors are clustered into groups. The height graph and sensor groups are used to produce a probability map of possible visitor locations. Based on these, particle filters are created to estimate the visitors' state, and by extension their fields of interest. Using this overall strategy for tracking and interest prediction, an average prediction accuracy of 92% is found when compared to a set of simulated people moving within a simulated space.

Body movements communicate affective expressions and, in recent years, computational models have been developed to recognize affective expressions from body movements or to generate movements for virtual agents or robots which convey affective expressions. This survey summarizes the state of the art on automatic recognition and generation of such movements. For both automatic recognition and generation, important aspects such as the movements analyzed, the affective state representation used, and the use of notation systems is discussed. The survey concludes with an outline of open problems and directions for future work.

This paper presents a study on human performance in recognizing affective expressions conveyed through movements of hand-like structures. One movement sequence, closing and opening the hand, was performed by a demonstrator in 3 sets of 5 repeated trials, each set intending to convey a different affective expression. Three different expressions, sadness, happiness and anger, were considered. Expressive movement animations were replicated with a human-like hand model, a stick hand model and with a model resembling a palm frond structure. The structures tested have identical kinematics but different physical appearance. The ability of a human to correctly identify the intended expressive movements performed on these different structures was tested with 66 users viewing videos of the animated structures and reporting via an online questionnaire. Results show that anger is reliably perceived by observers from animated movements on different structures, while the other emotions are easily misperceived. The physical appearance of the structure has some impact on perception performance, but was not found to be statistically significant in this study. Furthermore, analyzing the participants' responses in the context of the valence-arousal model of emotion shows that the subjects were able to recognize the arousal component of the affective hand movements across all structures.

In recent years, the Preisach hysteresis model (1935) has emerged as the model of choice for the behaviour of many smart materials, such as shape memory alloys (SMA). This research treats the identification of Preisach models for a differential SMA actuator. The traditional identification technique is applied, and several models are derived for the actuator. It is seen that the classical Preisach model is able to model its behaviour. However, the results raise questions concerning the robustness of the traditional identification technique, as well as the conservative nature of current passivity-based control results for the Preisach model.

The motivation for this research arises from interest in actuators made from "smart materials" such as shape memory alloys and piezoceramics. One common characteristic among these new materials is that they exhibit complex hysteresis nonlinearities. The input-output nature of the Preisach model of hysteresis results in a simple model for which identification strategies are known, and implementation is straightforward both in simulation and real-time control. This paper examines the passivity of the Preisach model, and the stability of velocity feedback control for hysteretic actuators.

This paper deals with the L/sub 2/ stability of a shape memory alloy actuated position control system, based on a single wire under constant axial load. Presuming a controller has certain specific dissipativity characteristics, it is shown that system stability can be guaranteed for a large class of controllers. In particular, the results can be used to show stability under approximated PID, PI and PD control. The result is used to find a stabilizing PI controller, and the corresponding experimental data is shown.

Shape memory alloy (SMA) based actuators have a number of attributes which make them useful for robotic applications. Unfortunately their response is relatively slow being limited by the speed with which the SMA can be heated and cooled. This paper describes two novel developments of the basic SMA actuator which improve dynamic performance. To allow rapid heating of the shape memory alloy wire without the danger of over heating an infrared temperature sensor is introduced. Cooling is improved with the aid of a mobile heat sink. These developments are described and preliminary results are presented.