Goal-conditioned hierarchical reinforcement learning has demonstrated effectiveness in addressing complicated decision-making tasks by providing ''temporal extraction'', which decomposes tasks into smaller and more manageable ''subgoals''. This enables agents to plan over a longer time scale. However, achieving optimal exploration and exploitation still remains a challenge, especially for long-horizon or sparse-reward scenarios. In this paper, we introduce Active exploraion and hierarchical Self-Imitation (ASI), an effective scheme to enhance exploration and exploitation based on subgoal representation learning. The key point of ASI is to utilize temporal adjacency information in the representation space. We construct and dynamically update an adjacency graph that captures the relationships between subgoals. Based on the adjacency information provided by the graph, we design two mechanisms: active ``frontier-reaching'' exploration that faster expands the explored area by targeting boundary regions, and hierarchical self-imitation learning that leverages historical experience to facilitate both frontier reaching and policy training. Experimental results show that our method accelerates exploration and outperforms existing baselines in challenging long-horizon continuous control tasks.

Many real-world applications involve a team of agents who must coordinate each other's policies in real-time to achieve a shared goal. Previous studies mainly focus on decentralized control to maximize common rewards, with little consideration of coordination between control policies, which is critical in dynamic and complicated environments. Viewing this issue, we propose a novel adaptive graph coordination strategy that factorizes the joint policy into an adaptive graph generator and a graph-based coordinated policy. We employ a difference aware module to control when to generate graphs and an encoder-decoder module to acquire the underlying decision graph structure. Moreover, we introduce DAGness- and DAG depth constrained optimization to adjust the graph structure and strike a balance between efficiency and performance. We also present a graph-based coordinated policy to make asynchronous decisions based on the inter-agent coordination dependencies implied in the generated graph. Empirical evaluations on some cooperative multi-agent environments demonstrate the superiority of the proposed method, with faster convergence and more efficient coordinated policies.

Offline multi-agent reinforcement learning (MARL) with pre-training paradigm, which uses a large quantity of trajectories for offline pre-training and online deployment, has become fashionable lately. While performing well on various tasks, conventional pre-trained decision-making models based on imitation learning typically require many expert trajectories or demonstrations, which limits the development of pre-trained policies in multi-agent case. To address this problem, we propose a new setting, where a multi-agent policy is pre-trained offline using suboptimal (non-expert) data and then tested online with the expectation of high rewards. In this practical setting inspired by contrastive learning , we propose YANHUI, a simple yet effective framework utilizing a well-designed reward contrast function for multi-agent policy representation learning from a dataset including various reward-level data instead of just expert trajectories. Furthermore, we enrich the multi-agent policy pre-training with mixture-of-experts to dynamically represent it. With the same quantity of offline StarCraft Multi-Agent Challenge datasets, YANHUI achieves significant improvements over offline MARL baselines. In particular, our method surprisingly competes in performance with earlier state-of-the-art approaches, even with 10% of the expert data used by other baselines and the rest replaced by poor data.

Causal dynamics models (CDMs) have demonstrated significant potential in addressing various challenges in reinforcement learning. To learn CDMs, recent studies have performed causal discovery to capture the causal dependencies among environmental variables. However, the learning of CDMs is still confined to small-scale environments due to computational complexity and sample efficiency constraints. This paper aims to extend CDMs to large-scale object-oriented environments, which consist of a multitude of objects classified into different categories. We introduce the Object-Oriented CDM (OOCDM) that shares causalities and parameters among objects belonging to the same class. Furthermore, we propose a learning method for OOCDM that enables it to adapt to a varying number of objects. Experiments on large-scale tasks indicate that OOCDM outperforms existing CDMs in terms of causal discovery, prediction accuracy, generalization, and computational efficiency.

Hearthstone is a widely played collectible card game that challenges players to strategize using cards with various effects described in natural language. While human players can easily comprehend card descriptions and make informed decisions, artificial agents struggle to understand the game’s inherent rules and are unable to generalize their policies through natural language. To address this issue, we propose Cardsformer, a method capable of acquiring linguistic knowledge and learning a generalizable policy in Hearthstone. Cardsformer consists of a Prediction Model trained with offline trajectories to predict state transitions based on card descriptions and a Policy Model capable of generalizing its policy on unseen cards. To our knowledge, this is the first work to consider language knowledge in a card game. Experiments show that our approach significantly improves data efficiency and outperforms the state-of-the-art in Hearthstone even when there are untrained cards in the deck, inspiring a new perspective of tackling problems as such with knowledge representation from large language models. As the game constantly releases new cards along with new descriptions and new effects, the challenge in Hearthstone remains. To encourage further research, we make our code publicly available and publish PyStone, the code base of Hearthstone on which we conducted our experiments, as an open benchmark.

Offline reinforcement learning (RL) circumvents costly interactions with the environment by utilising historical trajectories. Incorporating a world model into this method could substantially enhance the transfer performance of various tasks without expensive calculations from scratch. However, due to the complexity arising from different types of generalisation, previous works have focused almost exclusively on single-environment tasks. In this study, we introduce a multi-environment offline RL setting to investigate whether a generalised world model can be learned from large, diverse datasets and serve as a good surrogate for policy learning in different tasks. Inspired by the success of multi-task prompt methods, we propose the Task-prompt Generalised World Model (TGW) framework, which demonstrates notable performance in this setting. TGW comprises three modules: a task-state prompter, a generalised dynamics module, and a reward module. We implement the generalised dynamics module as a transformer-based recurrent state-space model and employ prompts to provide task-specific instructions, enabling TGW to address the internal stochasticity of the generalised world model. On the MuJoCo control benchmarks, TGW significantly outperforms previous offline RL algorithms in multi-environment setting.

Generating explanations for reinforcement learning (RL) is challenging as actions may produce long-term effects on the future. In this paper, we develop a novel framework for explainable RL by learning a causal world model without prior knowledge of the causal structure of the environment. The model captures the influence of actions, allowing us to interpret the long-term effects of actions through causal chains, which present how actions influence environmental variables and finally lead to rewards. Different from most explanatory models which suffer from low accuracy, our model remains accurate while improving explainability, making it applicable in model-based learning. As a result, we demonstrate that our causal model can serve as the bridge between explainability and learning.

Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) is a promising paradigm to address the exploration-exploitation dilemma in reinforcement learning. It decomposes the source task into sub goal conditional subtasks and conducts exploration and exploitation in the subgoal space. The effectiveness of GCHRL heavily relies on sub goal representation functions and sub goal selection strategy. However, existing works often overlook the temporal coherence in GCHRL when learning latent sub goal representations and lack an efficient sub goal selection strategy that balances exploration and exploitation. This paper proposes HIerarchical reinforcement learning via dynamically building Latent Landmark graphs (HILL) to overcome these limitations. HILL learns latent subgoal representations that satisfy temporal coherence using a contrastive representation learning objective. Based on these representations, HILL dynamically builds latent landmark graphs and employs a novelty measure on nodes and a utility measure on edges. Finally, HILL develops a subgoal selection strategy that balances exploration and exploitation by jointly considering both measures. Experimental results demonstrate that HILL outperforms state-of-the-art baselines on continuous control tasks with sparse rewards in sample efficiency and asymptotic performance. Our code is available at https://github.com/papercode2022/HILL.

Acquiring dynamics is critical for robot learning and is fundamental to planning and control. This paper concerns two fundamental questions: How can we learn a model that covers massive, diverse robot dynamics? Can we construct a model that lifts the data-collection pain and domain expertise required for building specific robot models? We learn the dynamics involved in a dataset containing a large number of serial articulated robots and propose a new concept, “Gen2Real”, to transfer simulated, generalized models to physical, specific robots. We generate a large-scale dataset by randomizing dynamics parameters, topology configurations, and model dimensions, which, in sequence, correspond to different properties, connections, and numbers of robot links. A structure modified from the generative pre-trained transformer is applied to approximate the dynamics of massive heterogeneous robots. In Gen2Real, we transfer the pre-trained model to a target robot using distillation, for the sake of real-time computation. The results demonstrate the superiority of the proposed method in terms of its accuracy in learning a tremendous amount of robot dynamics and its generality to transfer to different robots.

This article presents a novel structure of spiking neural networks (SNNs) to simulate the joint function of multiple brain regions in handling precision physical interactions. This task desires efficient movement planning while considering contact prediction and fast radial compensation. Contact prediction demands the cognitive memory of the interaction model, and we novelly propose a double recurrent network to imitate the hippocampus, addressing the spatiotemporal property of the distribution. Radial contact response needs rich spatial information, and we use a cerebellum-inspired module to achieve temporally dynamic prediction. We also use a block-based feedforward network to plan movements, behaving like the prefrontal cortex. These modules are integrated to realize the joint cognitive function of multiple brain regions in prediction, controlling, and planning. We present an appropriate controller and planner to generate teaching signals and provide a feasible network initialization for reinforcement learning, which modifies synapses in accordance with reality. The experimental results demonstrate the validity of the proposed method.

Kinematics and instantaneous kinematics are fundamental in many robotic tasks, such as positioning and collision avoidance. Existing learning methods mainly concern a single robot, and small-scale networks are sufficient for considerable approximation accuracy. A question is: Can we learn a kinematics model that can generalize to various robots rather than a single robot? This paper studies the kinematics learning of massive heterogeneous serial robots and the transfer of these general models to reality. We generate a dataset by randomizing dimensions, configurations, and link lengths and employ a network based on the generative pre-trained transformer to learn general kinematics mappings. We directly transfer our models for accuracy and use distillation-based transfer for computational efficiency. The results validate that our method can accurately approximate the kinematics of thousands of robot models and demonstrates generality in transfer.

Although multi-agent reinforcement learning (MARL) has demonstrated remarkable progress in tackling sophisticated cooperative tasks, the assumption that agents take simultaneous actions still limits the applicability of MARL for many real-world problems. In this work, we relax the assumption by proposing the framework of the bi-level Markov game (BMG). BMG breaks the simultaneity by assigning two players with a leader-follower relationship in which the leader considers the policy of the follower who is taking the best response based on the leader's actions. We propose two provably convergent algorithms to solve BMG: BMG-1 and BMG-2. The former uses the standard Q-learning, while the latter relieves solving the local Stackelberg equilibrium in BMG-1 with the further two-step transition to estimate the state value. For both methods, we consider temporal difference learning techniques with both tabular and neural network representations. To verify the effectiveness of our BMG framework, we test on a series of games, including Seeker, Cooperative Navigation, and Football, that are challenging to existing MARL solvers find challenging to solve: Seeker, Cooperative Navigation, and Football. Experimental results show that our BMG methods achieve competitive advantages in terms of better performance and lower variance.

In a small operational space, e.g., mesoscale or microscale, we need to control movements carefully because of fragile objects. This article proposes a novel structure based on spiking neural networks to imitate the joint function of multiple brain regions in visual guiding in the small operational space and offers two channels to achieve collision-free movements. For the state sensation, we simulate the primary visual cortex to directly extract features from multiple input images and the high-level visual cortex to obtain the object distance, which is indirectly measurable, in the Cartesian coordinates. Our approach emulates the prefrontal cortex from two aspects: multiple liquid state machines to predict distances of the next several steps based on the preceding trajectory and a block-based excitation-inhibition feedforward network to plan movements considering the target and prediction. Responding to "too close" states needs rich temporal information, and we leverage a cerebellar network for the subconscious reaction. From the viewpoint of the inner pathway, they also form two channels. One channel starts from state extraction to attraction movement planning, both in the camera coordinates, behaving visual-servo control. The other is the collision-avoidance channel, which calculates distances, predicts trajectories, and reacts to the repulsion, all in the Cartesian coordinates. We provide appropriate supervised signals for coarse training and apply reinforcement learning to modify synapses in accordance with reality. Simulation and experiment results validate the proposed method.

This article uses a multiple compliant degree-of-freedoms (DOFs) mechanism (i.e., a spring) to facilitate compliant insertion in precision assembly and proposes an efficient insertion strategy accordingly. The addition of a spring increases the insertion compliance while resulting in the object being not directly controllable. The proposed strategy handles both vertical insertion and inclined insertion with an unknown posture according to force feedback. We investigate horizontal compliance when the spring is compressed or stretched and introduce the withdrawal process for exceptional situations by taking advantage of the insertion compliance. The inclined insertion is a process of inserting while estimating the hole posture and a radial compensation strategy is presented while not affecting the axial length of the spring. Efficient insertion planning is discussed in the presence of uncertainty caused by the spring for both insertion types. Experiments are carried out to demonstrate the validation of the proposed method.

This article investigates simultaneous precision assembly of multiple objects, which plays a significant role in forming combined and complicated structures. Taking circular assembly as an example, we present a general alignment and insertion strategy, considering the difficulties of joint alignment as well as complicated interaction. In alignment, this article uses movable microscopic cameras to realize a large view area required by the spatial alignment and an optimization approach to achieve the even display of multiple objects. In insertion, we propose a wriggling insertion method to distinguish interaction forces, iteratively counting according to the relative movements of adjacent objects, and also an algorithm to plan the insertion: the compensational motion regarding the mutual effects of all objects and the toward-center motion based on radial forces. Experiments are carried out to demonstrate the validation of the proposed method.

Simultaneously inserting multiple objects is an essential topic in precision assembly to compose complicated shapes. This task involves the acquisition difficulty of unobservable interaction states, which makes it hard to plan the insertion. To solve it, this article investigates the circular assembly of multiple objects and proposes a strategy to control the simultaneous insertion in belief space. We first present the insertion state transition and observation models, in which the stochastic parts are modeled as Gaussian noise, and then estimate the belief state using an extended Kalman filter. An optimization approach is discussed, for the compensational movement planning, to decrease the estimated radial interaction forces and the toward-center movement is thus determined, considering the optimized compensational movement and the belief state. Experiments are carried out to demonstrate the validation of the proposed method.

The problem of simultaneous precision assembly of multiple objects is quite practical one to form compact physical structures and functionalities in mechatronics and advanced robotics. The core research aspects facing the problem are the contact status perception between each two object and the motion planning of each separate object. These two aspects mutually affect each other and cannot be discussed separately. In this paper, we first strategically discuss the possible approaches to solve the simultaneous assembly problem and analyze their advantages and drawbacks. Then, a probabilistic control method is developed based on the incomplete perceived information of the assembly process, which can achieve the highest assembly efficiency from the strategic perspective. Specifically, by fully utilizing the mechanical properties of materials in micrometer scale, the interaction between objects is first characterized as stochastic state-transition process. Second, adopting the simultaneous feeding strategy instead of serial feeding, the current contact status between each two object is determined based on the state-transition equation as a probability distribution along a hyperline. Finally, the motion planning technique is designed taking all possible radial forces on every contact surface into consideration. The experimental results demonstrate the effectiveness of the proposed method.

This article proposes an efficient insertion strategy for inclined precision assembly. According to the characteristics of the inclined insertion, the radial force between objects is separated into three parts: the force due to insertion, the force retained from previous compensation, and the force caused by estimation error of object inclination. We employ Gaussian distribution to model the probability of the radial force in insertion and predict the future contact based on the current force and the compensation movement. Object inclination is online estimated by peeling off the forces not related with inclination and investigating the relationship between inclination difference and radial force error. The insertion of the lower object is originated from the parameter iteratively updated by past performance, the current assessment depicting the current state, the confidence on future execution, and the coordinated motion of the upper object is then planned based on the inclined insertion, the estimated inclination, and the force to be compensated. Experiments are carried out to demonstrate the validation of the proposed method.

This article investigates the coordinated motion planning of independent manipulators in the domain of precision manipulation, where forces are considered in the level of tens or hundreds of milliNewtons. Independent manipulating is crucial in randomly combining robot arms to temporarily fulfill certain tasks. We study the cases where a part of task information is known to each manipulator and yet no communication exists between them. In the coordinated structure, the leading manipulator plans its movement according to the desired state and its evaluation of the object force. The follower needs to compensate for the offset errors, estimate the intention of the leader, evaluate its confidence on the estimation, and plan its movement accordingly. The handling of special states, i.e., zero force feedback, is also discussed. Experiments are carried out to demonstrate the validation of the proposed method.

The inherent hysteresis property of piezoelectric actuator (PEA) brings challenges to its modeling and control. This paper proposes a model learning method that is suitable for both forward and inverse PEA models. The hysteresis property is learned based on least squares support vector machines (LS-SVMs). A larger dataset is used for training LS-SVM to guarantee a good generalization performance. Support vectors pruning is utilized to reduce the model complexity. The rate-dependent property of PEA is identified as a linear dynamic submodel. Moreover, a pointing control system with two dualPEA-axis steering mirrors is developed, which can regulate the 4-degree-of-freedom pose of a laser beam. The coordinated control of four PEAs is realized based on the Jacobian matrix. The learned inverse PEA models are used for the feedforward compensation of each PEA's nonlinearity. A series of experiments were conducted to evaluate the proposed method's effectiveness.

This paper proposes an efficient strategy for the insertion of the partially flexible object in precision assembly. The partially flexible object refers to the component with unevenly distributed flexibilities: coupling rigidity and flexibility. This paper focuses on the insertion of one class of partially flexible objects: rigid shapes connected by a compliant mechanism. We first analyze the horizontal compliance of the compliant mechanism and build a model to relate its state and force. The insertion is separated into two stages according to the insertion type. The first stage is a compliant insertion and we estimate the insertion direction based on the built model, horizontally compensate resorting to the horizontal compliance and the updated direction, and efficiently plan the vertical insertion in an aggressive strategy regarding the uncertainties caused by the compliant mechanism and predicting the future insertion. The second stage is a hybrid insertion with both rigid and compliant gripping and its features include that the object states are not precisely measurable and the motion of a part of the object is not directly controllable. To solve it, we qualitatively and quantitatively analyze all possible configurations and, taking advantage of the insertion property, conclude one insertion posture based on which a control strategy is proposed. Additionally, a conservative insertion strategy is planned resorting to the past execution performance and the current state evaluation. Experiments are carried out to demonstrate the validation of the proposed method.

This paper proposes an efficient strategy to simultaneously insert multiple objects, which are parallel connected by passive compliant mechanisms, in precision assembly. The distinctions of this task include: each object is held compliantly; multiple objects are parallel connected to a manipulator; not all the peg-in-hole has the same insertion condition; and high accuracy is required for each insertion. This configuration can provide sufficient compliance and improve insertion efficiency for massive precision assembly. We model the relationship between the state and force of a single compliant mechanism, and analyze the horizontal compliance of parallel mechanisms. Based on the model, with a fitting and optimization method the states of all but one compliant mechanisms are acquired from microscopic views and the remaining states are optimized with resultant forces provided by a force sensor. To efficiently plan the parallel insertion, we propose a strategy to horizontally compensate according to the resultant force and the horizontal compliance, and to vertically insert based on the insertion ratio expectation, the horizontal offsets of each individual insertion, and the horizontal force. Experiments are carried out to demonstrate the validation of the proposed method.

This paper proposes an efficient and accurate method for collision detection between convex prisms in precision alignment and a detach controller based on the contact location for the collision occurrence. We project the objects onto an appropriate plane and detect the collision status considering the relationship between their planar contours. Efficient methods are presented for the overlap checking of several elementary contours, and the way to obtain the vertices of the projected shapes is also introduced. The detection is then accelerated by classifying into the corresponding case based on the relative configuration and using the efficient planar checking to replace the cubic calculation. A detach controller is presented to immediately separate the objects according to the contact location once collision occurs. The computational efficiency and comparison are demonstrated in simulations, and experiments are carried out to validate the detection and the controller.

This paper proposes two methods for collision detection between cylindrical components when mutual blocking occurs in the view of cameras. In reconstruction approach, 3D models are built according to key features captured by cameras and constraint optimization is employed to rapidly find possible intersections. To satisfy the computational efficiency required by real time operation, we present another way, projection method, to convert two planar views to contours on a projection plane and to detect high dimension collisions by studying the projection's relationships in low dimension. Nine cases are totally categorized and eleven parameters are constructed for detection on the basis of relative postures and positions. Simulations and experiments are carried out to demonstrate their validity.

The microscope has characteristics of a planar vision with small view field and small view depth. For micro operation systems with multiple manipulators, the handling of irregular objects may lead to a nonorthogonal microscopic system, which needs to focus on clear viewing interested features, and it may also hardly locate the exact position and posture of the robot arms. In view of these, this paper proposes an active calibration method to compute image Jacobian matrix, which maps from the relative motion of the manipulators to the image coordination changes in the microscopes. We also investigate the applications in micro operator positioning, tracking for distributed systems, and movement optimization in micro-assembly. Experiments are carried out on a micro-assembly platform equipped with three microscopes and six robot arms, and the results validate the effectiveness of the proposed method.

Difficulties arise in the micro-assembly of many irregular objects and in the insertion with contact between components of soft materials. To handle these problems, we design a micro-operational platform with multiple manipulators to facilitate a sequence of assembly. Six robot arms and three microscopes are incorporated, together with macro and micro motion systems. We also propose a hybrid control strategy to achieve high precision and protect objects. This hybrid scheme includes vision based positioning controllers for alignment, which employ incremental PI controllers and image Jacobian matrix, force based controllers for insertion, and a decision mechanism determining the assembly state. Experiments demonstrate the effectiveness of the proposed platform and control methods.

We present full state feedback controllers for standing and walking balance of humanoid robot. The robot is simulated as a two-joint inverted pendulum for standing and a five-link model for walking, and is disturbed by a horizontal push with given size and location in the sagittal plane. We optimize the parametric controllers for different push sizes, locations, and directions. For standing balance, both impulsive and constant pushes are applied to simulate the hip strategy; for bipedal walking, instantaneous pushes are used as perturbations. The performance of optimized controllers are shown in handling different pushes for standing and walking balance.

This paper develops full state feedback parametric controllers for perturbed walking of humanoid robot in response to external perturbations. We simulate models in the sagittal and lateral plane and in 3-dimensions, use a horizontal push of a given size, direction, location, and time as a perturbation, and optimize parametric controllers for different push sizes, directions, locations, and times. During a simulated perturbation experiment, the appropriate controller is selected based on the detected push information. The performance of optimized controllers are shown in handling different instantaneous pushes.

This paper develops full-state parametric controllers for standing balance of humanoid robots in response to impulsive and constant pushes. We also explore a hypothesis that postural feedback gains in standing balance should change with perturbation size. From an engineering point of view this is known as gain scheduling. We use an optimization approach to see if feedback gains should scale with the perturbation for a simulated robot. We simulate models in the sagittal and lateral plane and in 3-dimensions, use a horizontal push of a given size, direction and location as a perturbation, and optimize parametric controllers for different push sizes, directions and locations. During a simulated perturbation experiment, the appropriate controller is continuously selected based on the current push. For an impulse, the simulated robot recovers back to the initial state; for a constant push, the robot moves to an equilibrium position which leans into the push and has zero joint torques. We show the performance of optimized parametric controllers in response to different external pushes.