It is of significance for an agent to autonomously explore the environment and learn a widely applicable and general-purpose goal-conditioned policy that can achieve diverse goals including images and text descriptions. Considering such perceptually-specific goals, one natural approach is to reward the agent with a prior non-parametric distance over the embedding spaces of states and goals. However, this may be infeasible in some situations, either because it is unclear how to choose suitable measurement, or because embedding (heterogeneous) goals and states is non-trivial. The key insight of this work is that we introduce a latent-conditioned policy to provide goals and intrinsic rewards for learning the goal-conditioned policy. As opposed to directly scoring current states with regards to goals, we obtain rewards by scoring current states with associated latent variables. We theoretically characterize the connection between our unsupervised objective and the multi-goal setting, and empirically demonstrate the effectiveness of our proposed method which substantially outperforms prior techniques in a variety of tasks.

By maximizing an information theoretic objective, a few recent methods empower the agent to explore the environment and learn skills without extrinsic reward. However, when considering using multiple consecutive skills to complete a specific task, the transition from one to another cannot guarantee the success of the process due to the evident gap between skills. In this paper, we propose a novel unsupervised reinforcement learning approach to learn transitional skills in addition to pursuing diverse primitive skills. By introducing an extra latent variable for exploring the dependence between skills, our method discovers both primitive and transitional skills by optimizing a novel information theoretic objective. Considering various robotic tasks, our results demonstrate the effectiveness on learning both diverse primitive skills and transitional skills, and further exhibit the superiority of our method in smooth transition of skills over the baselines. Videos of transitional skills can be found on the project website: https://sites.google.com/view/udts-skill.

Unsupervised reinforcement learning aims to acquire skills without prior goal representations, where an agent automatically explores an open-ended environment to represent goals and learn the goal-conditioned policy. However, this procedure is often time-consuming, limiting the rollout in some potentially expensive target environments. The intuitive approach of training in another interaction-rich environment disrupts the reproducibility of trained skills in the target environment due to the dynamics shifts and thus inhibits direct transferring. Assuming free access to a source environment, we propose an unsupervised domain adaptation method to identify and acquire skills across dynamics. Particularly, we introduce a KL regularized objective to encourage emergence of skills, rewarding the agent for both discovering skills and aligning its behaviors respecting dynamics shifts. This suggests that both dynamics (source and target) shape the reward to facilitate the learning of adaptive skills. We also conduct empirical experiments to demonstrate that our method can effectively learn skills that can be smoothly deployed in target.

Experience replay plays a key role in the success of deep reinforcement learning (DRL), where the experience collection is defined as the process of placing experience tuples into a replay buffer. Although this deterministic experience collection works well for the posterior replay, there is still a potential to improve the efficiency by using a preferential way. In this paper, we propose a preferential experience collection (PEC) method for experience replay in model-free DRL, where the reward is composed of extrinsic and intrinsic components. Since designing a good extrinsic reward is difficult especially in a complicated environment, this paper considers to establish a novel self-supervised intrinsic reward (IR) for model-free DRL. By looking into the frequency domain, the intrinsic reward is designed such that the agent can understand the behavior itself to some extent in addition to completing the task. Extensive experimental results demonstrate that our method can improve the efficiency and stability of the model-free DRL.

Recently, diverse primitive skills have been learned by adopting the entropy as intrinsic reward, which further shows that new practical skills can be produced by combining a variety of primitive skills. This is essentially skill transfer, very useful for learning high-level skills but quite challenging due to the low efficiency of transferring primitive skills. In this paper, we propose a novel efficient skill transfer method, where we learn independent skills and only independent components of skills are transferred instead of the whole set of skills. More concretely, independent components of skills are obtained through independent component analysis (ICA), which always have a smaller amount (or lower dimension) compared with their mixtures. With a lower dimension, independent skill transfer (IST) exhibits a higher efficiency on learning a given task. Extensive experiments including three robotic tasks demonstrate the effectiveness and high efficiency of our proposed IST method in comparison to direct primitive-skill transfer and conventional reinforcement learning.

Multivariate time series prediction has fundamental importance to various practical domains. Many useful techniques have been proposed in literature for improving the accuracy and efficiency of prediction. However, due to the complicated dependency between time stamps and the interactive dependency between multiple time series, it is very difficult to pursue a high accuracy and alleviate the prediction error. In this paper, we propose a residual compensation scheme for multivariate time series prediction, where the prediction error is modeled by a PID-based residual network (PID-R) to cover various linear, cumulative and differential effects. To evaluate the effectiveness of the residual compensation, a hybrid structure integrating the Seq2Seq model with the classic vector auto-regression (VAR) is built as an initial predictor. The proposed final predictor with residual compensation (FPRC) incorporates the initial predictor and PID-R. Extensive experiments show that the FPRC achieves superior accuracy in comparison to state-of-the-art methods and ablation analysis demonstrates the effectiveness of residual compensation.

Time series prediction plays a key role in wide applications and has been investigated for a couple of decades. Nevertheless, most of the prior works fail to identify the most effective frequency components of time series before passing through the prediction, which induces the drop of the performance. In this paper, we propose a novel predictor which integrates the sequence to sequence (seq2seq) model based on long short-term memory units (LSTM) with interpretable data reconstruction, where the learned hidden state is taken as a bridge. The reconstructor can effectively regularize data based on the learned frequency components and extract the effective components of time series. Moreover, we present an alternative training mechanism and a dedicated loss function to guarantee the success of prediction. Experimental results on extensive real-world datasets show a prominent superiority in comparison to state-of-the-art methods.