Skill learning from language instructions is a critical challenge in developing intelligent agents that can generalize across diverse tasks and follow complex human instructions. Hierarchical methods address this by decomposing the learning problem into multiple levels, where the high-level and low-level policies are mediated through a latent plan space. Effective modeling and learning of this latent plan space are key to enabling robust and interpretable skill learning. In this paper, we introduce LADS, a hierarchical approach that learns language-conditioned discrete latent plans through semantic skill abstractions. Our method decouples the learning of the latent plan space from the language-conditioned high-level policy to improve training stability. First, we incorporate a trajectory encoder to learn a discrete latent space with the low-level policy, regularized by language instructions. Next, we model the high-level policy as a categorical distribution over these discrete latent plans to capture the multi-modality of the dataset. Through experiments in simulated control environments, we demonstrate that LADS outperforms state-of-the-art methods in both skill learning and compositional generalization.

In this paper, we study the problem of video-conditioned policy learning. While previous works mostly focus on learning policies that perform a single skill specified by the given video, we take a step further and aim to learn a policy that can perform multiple skills according to the given video, and generalize to unseen videos by recombining these skills. To solve this problem, we propose our algorithm, Watch-Less-Do-More, an information bottleneck-based imitation learning framework for implicit skill discovery and video-conditioned policy learning. In our method, an information bottleneck objective is employed to control the information contained in the video representation, ensuring that it only encodes information relevant to the current skill (Watch-Less). By discovering potential skills from training videos, the learned policy is able to recombine them and generalize to unseen videos to achieve compositional generalization (Do-More). To evaluate our method, we perform extensive experiments in various environments and show that our algorithm substantially outperforms baselines (up to 2x) in terms of compositional generalization ability.

The in-context learning ability of Transformer models has brought new possibilities to visual navigation. In this paper, we focus on a novel video navigation setting, where an in-context navigation policy needs to be learned purely from videos in an offline manner, without access to the actual environment. For this setting, we propose Navigate Only Look Once (NOLO), a method for learning a navigation policy that possesses the in-context ability and adapts to new scenes by taking corresponding context videos as input without finetuning or re-training. To enable learning from videos, we first propose a pseudo action labeling procedure using optical flow to recover the action label from egocentric videos. Then, offline reinforcement learning is applied to learn the navigation policy. Through extensive experiments on different scenes both in simulation and the real world, we show that our algorithm outperforms baselines by a large margin, which demonstrates the effectiveness of the learned policy in in-context learning. For videos and more information, visit our project page.

Reinforcement learning with natural language action space often suffers from the curse of dimensionality due to the combinatorial nature of the natural language.Previous research leverages pretrained language models to capture action semantics and reduce the size of the action space.However, since pretrained models are typically trained on general corpora, there can be an unpredictable mismatch between the priors encoded in pretrained models and the characteristics of the specific RL environment.To address this issue, we propose Mutual-Information Regularized Policy Optimization, MIPO.MIPO enables implicit and dynamic reduction of the action space.Starting from the prior provided by the pretrained language model, our method dynamically adjusts the prior during the learning process based on the guidance of mutual information regularization.Theoretically, we demonstrate that this policy optimization process leads to the monotonic improvement on the mutual-information regularized RL objective.Empirically, we conduct experiments in various environments and demonstrate the effectiveness of MIPO.

Despite the success of cooperative multi-agent reinforcement learning algorithms, most of them focus on a single team composition, which prevents them from being used in more realistic scenarios where dynamic team composition is possible. While some studies attempt to solve this problem via multi-task learning in a fixed set of team compositions, there is still a risk of overfitting to the training set, which may lead to catastrophic performance when facing dramatically varying team compositions during execution. To address this problem, we propose to use mutual information (MI) as an augmented reward to prevent individual policies from relying too much on team-related information and encourage agents to learn policies that are robust in different team compositions. Optimizing this MI-augmented objective in an off-policy manner can be intractable due to the existence of dynamic marginal distribution. To alleviate this problem, we first propose a multi-agent policy iteration algorithm with a fixed marginal distribution and prove its convergence and optimality. Then, we propose to employ the Blahut–Arimoto algorithm and an imaginary team composition distribution for optimization with approximate marginal distribution as the practical implementation. Empirically, our method demonstrates strong zero-shot generalization to dynamic team compositions in complex cooperative tasks.

The automatic ICD coding task is to assign the most relevant International Classification of Disease diagnosis codes (ICD codes) to the patient’s clinical text, which is defined as a multi-label text classification problem. In practical applications, the clinical text is lengthy and the distribution of ICD codes is skewed with a long tail, making it difficult to predict the labels for rare disease. To tackle the problem, we propose a new deep learning framework which combines diseases correlation and multi-attention mechanism into Bidirectional Recurrent Neural Network. In particular, we apply the Graph Convolutional Network to capture the disease relevance between ICD codes, which can be highly valuable for those diseases with limited data. In addition, we leverage multi-head attention and label-wise attention to encode clinical text, thus our model can learn label-specific representations and consider disease correlation in the same time. Extensive experiments demonstrate that our proposed model is effective. On the MIMIC-III-full dataset, our model outperformed the state-of-the-art model in 5 out of 7 evaluation metrics. On the MIMIC-III-50 dataset, our model outperformed the state-of-the-art model in 4 out of 7 evaluation metrics.