Policy optimization methods benefit from a simple and tractable policy parametrization, usually the Gaussian for continuous action spaces. In this paper, we consider a broader policy family that remains tractable: the $q$-exponential family. This family of policies is flexible, allowing the specification of both heavy-tailed policies ($q>1$) and light-tailed policies ($q<1$). This paper examines the interplay between $q$-exponential policies for several actor-critic algorithms conducted on both online and offline problems. We find that heavy-tailed policies are more effective in general and can consistently improve on Gaussian. In particular, we find the Student's t-distribution to be more stable than the Gaussian across settings and that a heavy-tailed $q$-Gaussian for Tsallis Advantage Weighted Actor-Critic consistently performs well in offline benchmark problems. In summary, we find that the Student's t policy a strong candidate for drop-in replacement to the Gaussian. Our code is available at \url{https://github.com/lingweizhu/qexp}.

Sparse continuous policies are distributions that can choose some actions at random yet keep strictly zero probability for the other actions, which are radically different from the Gaussian. They have important real-world implications, e.g. in modeling safety-critical tasks like medicine. The combination of offline reinforcement learning and sparse policies provides a novel paradigm that enables learning completely from logged datasets a safety-aware sparse policy. However, sparse policies can cause difficulty with the existing offline algorithms which require evaluating actions that fall outside of the current support. In this paper, we propose the first offline policy optimization algorithm that tackles this challenge: Fat-to-Thin Policy Optimization (FtTPO). Specifically, we maintain a fat (heavy-tailed) proposal policy that effectively learns from the dataset and injects knowledge to a thin (sparse) policy, which is responsible for interacting with the environment. We instantiate FtTPO with the general $q$-Gaussian family that encompasses both heavy-tailed and sparse policies and verify that it performs favorably in a safety-critical treatment simulation and the standard MuJoCo suite. Our code is available at https://github.com/lingweizhu/fat2thin.

Many individuals—especially those with autism spectrum disorder (ASD), alexithymia, or other neurodivergent profiles—face challenges in recognizing, expressing, or interpreting emotions. To support more inclusive and personalized emotion technologies, we present a real-time multimodal emotion estimation system that combines neurophysiological EEG, ECG, blood volume pulse (BVP), and galvanic skin response (GSR/EDA) and behavioral modalities (facial expressions, and speech) in a unified arousal-valence 2D interface to track moment-to-moment emotional states. This architecture enables interpretable, user-specific analysis and supports applications in emotion education, neuroadaptive feedback, and interaction support for neurodiverse users. Two demonstration scenarios illustrate its application: (1) passive media viewing (2D or VR videos) reveals cortical and autonomic responses to affective content, and (2) semi-scripted conversations with a facilitator or virtual agent capture real-time facial and vocal expressions. These tasks enable controlled and naturalistic emotion monitoring, making the system well-suited for personalized feedback and neurodiversity-informed interaction design.

Homeostasis is a self-regulatory process, wherein an organism maintains a specific internal physiological state. Homeostatic reinforcement learning (RL) is a framework recently proposed in computational neuroscience to explain animal behavior. Homeostatic RL organizes the behaviors of autonomous embodied agents according to the demands of the internal dynamics of their bodies, coupled with the external environment. Thus, it provides a basis for real-world autonomous agents, such as robots, to continually acquire and learn integrated behaviors for survival. However, prior studies have generally explored problems pertaining to limited size, as the agent must handle observations of such coupled dynamics. To overcome this restriction, we developed an advanced method to realize scaled-up homeostatic RL using deep RL. Furthermore, several rewards for homeostasis have been proposed in the literature. We identified that the reward definition that uses the difference in drive function yields the best results. We created two benchmark environments for homeostasis and performed a behavioral analysis. The analysis showed that the trained agents in each environment changed their behavior based on their internal physiological states. Finally, we extended our method to address vision using deep convolutional neural networks. The analysis of a trained agent revealed that it has visual saliency rooted in the survival environment and internal representations resulting from multimodal input.

Learning to interact with the environment not only empowers the agent with manipulation capability but also generates information to facilitate building of action understanding and imitation capabilities. This seems to be a strategy adopted by biological systems, in particular primates, as evidenced by the existence of mirror neurons that seem to be involved in multi-modal action understanding. How to benefit from the interaction experience of the robots to enable understanding actions and goals of other agents is still a challenging question. In this study, we propose a novel method, deep modality blending networks (DMBN), that creates a common latent space from multi-modal experience of a robot by blending multi-modal signals with a stochastic weighting mechanism. We show for the first time that deep learning, when combined with a novel modality blending scheme, can facilitate action recognition and produce structures to sustain anatomical and effect-based imitation capabilities. Our proposed system, which is based on conditional neural processes, can be conditioned on any desired sensory/motor value at any time step, and can generate a complete multi-modal trajectory consistent with the desired conditioning in one-shot by querying the network for all the sampled time points in parallel avoiding the accumulation of prediction errors. Based on simulation experiments with an arm-gripper robot and an RGB camera, we showed that DMBN could make accurate predictions about any missing modality (camera or joint angles) given the available ones outperforming recent multimodal variational autoencoder models in terms of long-horizon high-dimensional trajectory predictions. We further showed that given desired images from different perspectives, i.e. images generated by the observation of other robots placed on different sides of the table, our system could generate image and joint angle sequences that correspond to either anatomical or effect-based imitation behavior. To achieve this mirror-like behavior, our system does not perform a pixel-based template matching but rather benefits from and relies on the common latent space constructed by using both joint and image modalities, as shown by additional experiments. Moreover, we showed that mirror learning (in our system) does not only depend on visual experience and cannot be achieved without proprioceptive experience. Our experiments showed that out of ten training scenarios with different initial configurations, the proposed DMBN model could achieve mirror learning in all of the cases where the model that only uses visual information failed in half of them. Overall, the proposed DMBN architecture not only serves as a computational model for sustaining mirror neuron-like capabilities, but also stands as a powerful machine learning architecture for high-dimensional multi-modal temporal data with robust retrieval capabilities operating with partial information in one or multiple modalities.

Understanding information processing in the brain-and creating general-purpose artificial intelligence-are long-standing aspirations of scientists and engineers worldwide. The distinctive features of human intelligence are high-level cognition and control in various interactions with the world including the self, which are not defined in advance and are vary over time. The challenge of building human-like intelligent machines, as well as progress in brain science and behavioural analyses, robotics, and their associated theoretical formalisations, speaks to the importance of the world-model learning and inference. In this article, after briefly surveying the history and challenges of internal model learning and probabilistic learning, we introduce the free energy principle, which provides a useful framework within which to consider neuronal computation and probabilistic world models. Next, we showcase examples of human behaviour and cognition explained under that principle. We then describe symbol emergence in the context of probabilistic modelling, as a topic at the frontiers of cognitive robotics. Lastly, we review recent progress in creating human-like intelligence by using novel probabilistic programming languages. The striking consensus that emerges from these studies is that probabilistic descriptions of learning and inference are powerful and effective ways to create human-like artificial intelligent machines and to understand intelligence in the context of how humans interact with their world.

Recurrent neural networks that can capture temporal characteristics on multiple timescales are a key architecture in machine learning solutions as well as in neurocognitive models. A crucial open question is how these architectures can adopt both multi-term dependencies and systematic fluctuations from the data or from sensory input, similar to the adaptation and abstraction capabilities of the human brain. In this paper, we propose an extension of the classic Continuous Time Recurrent Neural Network (CTRNN) by allowing it to learn to gate its timescale characteristic during activation and thus dynamically change the timescales in processing sequences. This mechanism is simple but bio-plausible as it is motivated by the modulation of oscillation modes between neural populations. We test how the novel Gating Adaptive CTRNNs can solve difficult synthetic sequence prediction problems and explore the development of the timescale characteristics as well as the interplay of multiple timescales. As a particularly interesting finding, we report that timescale distributions emerge, which simultaneously capture systematic patterns as well as spontaneous fluctuations. Our extended architecture is interesting for cognitive models that aim to investigate the development of specific timescale characteristic under temporally complex perception and action, and vice versa.

This survey presents the most relevant neural network models of autism spectrum disorder and schizophrenia, from the first connectionist models to recent deep neural network architectures. We analyzed and compared the most representative symptoms with its neural model counterpart, detailing the alteration introduced in the network that generates each of the symptoms, and identifying their strengths and weaknesses. We additionally cross-compared Bayesian and free-energy approaches, as they are widely applied to model psychiatric disorders and share basic mechanisms with neural networks. Models of schizophrenia mainly focused on hallucinations and delusional thoughts using neural dysconnections or inhibitory imbalance as the predominating alteration. Models of autism rather focused on perceptual difficulties, mainly excessive attention to environment details, implemented as excessive inhibitory connections or increased sensory precision. We found an excessively tight view of the psychopathologies around one specific and simplified effect, usually constrained to the technical idiosyncrasy of the used network architecture. Recent theories and evidence on sensorimotor integration and body perception combined with modern neural network architectures could offer a broader and novel spectrum to approach these psychopathologies. This review emphasizes the power of artificial neural networks for modeling some symptoms of neurological disorders but also calls for further developing of these techniques in the field of computational psychiatry.

Interactive reinforcement learning (IRL) has become an important apprenticeship approach to speed up convergence in classic reinforcement learning (RL) problems. In this regard, a variant of IRL is policy shaping which uses a parent-like trainer to propose the next action to be performed and by doing so reduces the search space by advice. On some occasions, the trainer may be another artificial agent which in turn was trained using RL methods to afterward becoming an advisor for other learner-agents. In this work, we analyse internal representations and characteristics of artificial agents to determine which agent may outperform others to become a better trainer-agent. Using a polymath agent, as compared to a specialist agent, an advisor leads to a larger reward and faster convergence of the reward signal and also to a more stable behaviour in terms of the state visit frequency of the learner-agents. Moreover, we analyse system interaction parameters in order to determine how influential they are in the apprenticeship process, where the consistency of feedback is much more relevant when dealing with different learner obedience parameters.

Humans use various types of modalities to express own internal states. If a robot interacting with humans can pay attention to limited signals, it should select more informative ones to estimate the partners' states. We propose an active perception method that controls the robot's attention based on an energy minimization criterion. An energy-based model, which has learned to estimate the latent state from sensory signals, calculates energy values corresponding to occurrence probabilities of the signals; The lower the energy is, the higher the likelihood of them. Our method therefore selects the modality that provides the lowest expectation energy among available ones to exploit more frequent experiences. We employed a multimodal deep belief network to represent relationships between humans' states and expressions. Our method demonstrated better performance for the modality selection than other methods in a task of emotion estimation. We discuss the potential of our method to advance human-robot interaction.

Collaborative robots are quickly gaining momentum in real-world settings. This has motivated many new research questions in human-robot collaboration. In this paper, we address the questions of whether and when a robot should take initiative during joint human-robot task execution. We develop a system capable of autonomously tracking and performing table-top object manipulation tasks with humans and we implement three different initiative models to trigger robot actions. Human-initiated help gives control of robot action timing to the user; robot-initiated reactive help triggers robot assistance when it detects that the user needs help; and robot-initiated proactive help makes the robot help whenever it can. We performed a user study (N=18) to compare these trigger mechanisms in terms of task performance, usage characteristics, and subjective preference. We found that people collaborate best with a proactive robot, yielding better team fluency and high subjective ratings. However, they prefer having control of when the robot should help, rather than working with a reactive robot that only helps when it is needed.

It is generally accepted that a robot should exhibit a contingent behavior, adaptable to the needs of each individual user, to achieve a more natural and pleasant interaction. In this paper we have evaluated whether this general rule applies also when the robot plays a leading role and needs to motivate the human partner to keep a certain pace, as during training or teaching. Also among humans, in schools or factories, structured interaction is often guided by a predefined rhythm, which facilitates the coordination of the partners involved and is thought to maximize their efficiency. On the other hand, a pre-established timing forces all participants to adjust their natural speed to the external, sometimes not appropriate, timing requirement. Where does the optimal trade-off between these two paradigms lie? We have addressed this question in a dictation scenario where the humanoid robot iCub plays the role of a teacher and dictates brief English or Italian sentences to the participants. In particular we compare a condition in which the dictation is performed at a fixed timing with a condition in which iCub monitors subjects' gaze to adjust its dictation speed. The results are discussed both in terms of participants' subjective evaluation and their objective performance, by highlighting the advantages and drawbacks of the choice of contingent robot behavior.

A fundamental challenge for robotics is to transfer the human natural social skills to the interaction with a robot. At the same time, neuroscience and psychology are still investigating the mechanisms behind the development of human-human interaction. HRI becomes therefore an ideal contact point for these different disciplines, as the robot can join these two research streams by serving different roles. From a robotics perspective, the study of interaction is used to implement cognitive architectures and develop cognitive models, which can then be tested in real world environments. From a neuroscientific perspective, robots could represent an ideal stimulus to establish an interaction with human partners in a controlled manner and make it possible studying quantitatively the behavioral and neural underpinnings of both cognitive and physical interaction. Ideally, the integration of these two approaches could lead to a positive loop: the implementation of new cognitive architectures may raise new interesting questions for neuroscientists, and the behavioral and neuroscientific results of the human-robot interaction studies could validate or give new inputs for robotics engineers. However, the integration of two different disciplines is always difficult, as often even similar goals are masked by difference in language or methodologies across fields. The aim of this workshop will be to provide a venue for researchers of different disciplines to discuss and present the possible point of contacts, to address the issues and highlight the advantages of bridging the two disciplines in the context of the study of interaction.

It has been suggested that perceptual immaturity in early infancy enhances learning for various cognitive functions. This paper demonstrates the role of visual development triggered by self-organization in a learner's visual space in a case of the mirror neuron system (MNS). A robot learns a function of the MNS by associating self-induced motor commands with observed motions while the observed motions are gradually self-organized in the visual space. A temporal convergence of the self-organization triggers visual development, which improves spatiotemporal blur filters for the robot's vision and thus further advances self-organization in the visual space. Experimental results show that the self-triggered development enables the robot to adaptively change the speed of the development (i.e., slower in the early stage and faster in the later stage) and thus to acquire clearer correspondence between self and other (i.e., the MNS).

Human interaction is highly intuitive: we infer reactions of our opponents mainly from what we have learned in years of experience and often assume that other people have the same knowledge about certain situations, abilities, and expectations as we do. In human-robot interaction (HRI) we cannot take for granted that this is equally true since HRI is asymmetrical. In other words, robots have different abilities, knowledge, and expectations than humans. They need to react appropriately to human expectations and behaviour. With this respect, scientific advances have been made to date for applications in entertainment and service robotics that largely depend on intuitive interaction. However, HRI today is often still unnatural, slow, and unsatisfactory for the human interlocutor. Both the sensorimotor interaction with environment and interlocutor, and the social aspects of the interaction still need to be researched and improved. Therefore, this full-day workshop aims to bring together researchers from different scientific fields to discuss these crosscutting issues and to exchange views on what are the preconditions and principles of intuitive interaction.

This paper presents an architecture extending bottom-up visual attention for dynamic scene analysis. In dynamic scenes, particularly when learning actions from demonstrations, robots have to stably focus on the relevant movement by disregarding surrounding noises, but still maintain sensitivity to a new relevant movement, which might occur in the surroundings. In order to meet the contradictory requirements of stability and sensitivity for attention, this paper introduces biologically-inspired mechanisms for retinal filtering and stochastic attention selection. The former reduces the complexity of peripheral signals by filtering an input image. It results in enhancing bottom-up saliency in the fovea as well as in detecting only prominent signals from the periphery. The latter allows robots to shift attention to a less but still salient location in the periphery, which is likely relevant to the demonstrated action. Integrating these mechanisms with computation for bottom-up saliency enables robots to extract important action sequences from task demonstrations. Experiments with a simulated and a natural scene show better performance of the proposed model than comparative models.

Abstract We propose a novel method for smoothing partition of unity (PU) implicit surfaces consisting of sets of non‐conforming linear functions with spherical supports. We derive new discrete differential operators and Laplacian smoothing using a spherical covering of PU as a grid‐like data structure. These new differential operators are applied to the smoothing of PU implicit surfaces. First, Laplacian smoothing is performed for the vector field defined by the gradient of the PU implicit surface, which is then updated to reflect the smoothing of the gradient field. This process achieves a method for noise robust surface reconstruction from scattered points.

How to teach actions to a robot as well as how a robot learns actions is an important issue to be discussed in designing robot learning systems. Inspired by human parent-infant interaction, we hypothesize that a robot equipped with infant-like abilities can take advantage of parental proper teaching. Parents are known to significantly alter their infant-directed actions versus adult-directed ones, e.g. make more pauses between movements, which is assumed to aid the infants' understanding of the actions. As a first step, we analyzed parental actions using a primal attention model. The model based on visual saliency can detect likely important locations in a scene without employing any knowledge about the actions or the environment. Our statistical analysis revealed that the model was able to extract meaningful structures of the actions, e.g. the initial and final state of the actions and the significant state changes in them, which were highlighted by parental action modifications. We further discuss the issue of designing an infant-like robot that can induce parent-like teaching, and present a human-robot interaction experiment evaluating our robot simulation equipped with the saliency model.

The skeletal structures of solid objects play an important role in medical and industrial applications. Given a volumetrically sampled solid object, our method extracts a nice-looking skeletal structure represented as a polygon mesh. The purpose is to achieve a noise-robust extraction of the skeletal mesh from a real-world object obtained using a scanning technology such as the CT scan method. We first approximate the input through a set of spherically supported polynomials that provide an adaptively smoothed intensity field, and then perform a polygonization process to find the extremal sheet of the field, which is regarded as a skeletal sheet in this research. In our polygonization, a subset of the weighted Delaunay tetrahedrization defined by a set of spherical supports is used as an adaptively sampled grid. The derivatives for detecting extremality are analytically evaluated at the tetrahedron vertices. We also demonstrate the effectiveness of our method by extracting skeletal meshes from noisy CT images.

We suggest that people's responses to a robot of which attention starts to be distracted show whether they accept the robot as an intentional communication partner or not. Human-robot interaction (HRI) as well as human-human interaction (HHI) is sometimes interrupted by disturbing factors. However, in HHI people continue to communicate with a partner because they presuppose that the partner may shift his/her interactive orientation based on his/her internal state. We designed a communication robot equipped with a mechanism of saliency-based visual attention and evaluated it in an observational experiment of HRI. Our sociological analysis of people's responses to our robot showed that it was accepted as a proactive communication agent. When the robot shifted its attention to an irrelative target, the human partners, for example, followed the line of the robot's gaze and tried to regain its attention by exaggerating their actions and increasing their communication channels as they would do toward a human partner. Based on these results, we conclude that disturbance can be an encouraging factor for human activity in HRI. The results are discussed from both a sociological and an engineering point of view.

To realize natural human-robot interactions and investigate the developmental mechanism of human communication, an effective approach is to construct models by which a robot imitates cognitive functions of humans. Focusing on the knowledge that humans utilize motion information of others’ action, this paper presents a learning model that enables a robot to acquire the ability to establish joint attention with a human by utilizing both static and motion information. As the motion information, the robot uses the optical flow detected when observing a human who is shifting his/her gaze from looking at the robot to looking at another object. As the static information, it extracts the edge image of the human face when he/she is gazing at the object. The static and motion information have complementary characteristics. The former gives the exact direction of gaze, even though it is difficult to interpret. On the other hand, the latter provides a rough but easily understandable relationship between the direction of gaze shift and motor output to follow the gaze. The learning model utilizing both static and motion information acquired from observing a human’s gaze shift enables the robot to efficiently acquire joint attention ability and to naturally interact with the human. Experimental results show that the motion information accelerates the learning of joint attention while the static information improves the task performance. The results are discussed in terms of analogy with cognitive development in human infants.

A human-like intelligent robot is expected to have the capability to develop its cognitive functions through experience without a priori knowledge or explicit teaching. In addition, the realization of this kind of robot leads us to understand the developmental mechanisms of human beings. This paper proposes a bootstrap learning model by which a robot acquires the ability of joint attention without a caregiver's evaluation or a controlled environment based on the robot's embedded mechanisms: visual attention and learning with self-evaluation. Through learning based on the proposed model, the robot finds a correlation in sensorimotor coordination when joint attention succeeds and consequently acquires the ability of joint attention by accumulating the appropriate correlation and losing the uncorrelated coordination as statistical outliers. The experimental results show the validity of the proposed model.

This paper presents a constructive model by which a robot acquires the ability of joint attention with a human caregiver based on its embedded mechanisms of visual attention and learning with self-evaluation. The former is to look at a salient object in the robot's view, and the latter is to learn sensorimotor co-ordination when visual attention has succeeded. Since the success of visual attention does not always correspond to the success of joint attention, the robot has incorrect learning data for joint attention as well as correct data. However, the robot is expected statistically to lose incorrect data as outliers since such data do not have any correlation in the sensorimotor co-ordination while correct data have a correlation. The robot consequently acquires the ability of joint attention by finding the correlation in the sensorimotor co-ordination even if multiple objects are placed at random positions in an environment and a human caregiver does not provide any task evaluation to the robot. The experimental results show that the proposed model makes the robot reproduce the developmental process of infants' joint attention. Therefore, the proposed model could be one of the models to explain how infants develop the ability of joint attention.

This paper proposes a developmental learning model for joint attention between a robot and a human caregiver. The proposed model has abilities to accelerate the learning and improve the final task performance owing to two kinds of developments: a robot's development and a caregiver's one. The robot's development means that the sensing and actuating capabilities of the robot change from immaturity to maturity. On the other hand, the caregiver's development is defined as that the caregiver changes the task from easy situation to difficult one. The proposed model causes these developments according to the learning progress of the robot. The experimental results show what kinds of effects the developments bring to the learning.