Context-based offline meta-reinforcement learning (meta-RL) is a paradigm that integrates meta-learning with offline reinforcement learning. It learns a strategy to extract task-specific contexts from trajectories of meta-training tasks and leverages this strategy for adapting to unseen target tasks. However, existing methods struggle to generate generalizable contexts for adaptations due to context shift, which arises from the context-based policy overfitting to offline data. We argue that leveraging the internal relationships among tasks, rather than treating each task in isolation, is crucial for mitigating the impact of context shift. Hence, we propose a framework called cross-task contexts for improving generalization in meta-RL (CTMRL). Specifically, we design a context quantization variational auto-encoder (CQ-VAE), which clusters task-specific contexts of meta-training tasks into discrete codes based on the internal relationships among tasks. Cross-task contexts are constructed with these codes, reflecting shared information across similar tasks. These cross-task contexts not only serve as high-level structures to capture similarity across tasks but also provide a foundation for hard contrastive learning that enhances the distinguishability of similar yet distinct tasks, thereby improving the generalization of contexts and facilitating adaptation to unseen target tasks. The evaluation in meta-environments confirms the performance advantage of CTMRL over existing methods.

A major challenge in Reinforcement Learning (RL) is the difficulty of learning an optimal policy from sparse rewards. Prior works enhance online RL with conventional Imitation Learning (IL) via a handcrafted auxiliary objective, at the cost of restricting the RL policy to be sub-optimal when the offline data is generated by a non-expert policy. Instead, to better leverage valuable information in offline data, we develop Generalized Imitation Learning from Demonstration (GILD), which meta-learns an objective that distills knowledge from offline data and instills intrinsic motivation towards the optimal policy. Distinct from prior works that are exclusive to a specific RL algorithm, GILD is a flexible module intended for diverse vanilla off-policy RL algorithms. In addition, GILD introduces no domain-specific hyperparameter and minimal increase in computational cost. In four challenging MuJoCo tasks with sparse rewards, we show that three RL algorithms enhanced with GILD significantly outperform state-of-the-art methods.

Intrinsic motivation lies at the heart of the exploration of reinforcement learning, which is primarily driven by the agent's inherent satisfaction rather than external feedback from the environment. However, in recent more challenging procedurally-generated environments with high stochasticity and uninformative extrinsic rewards, we identify two significant issues of applying intrinsic motivation. (1) State representation collapse: In existing methods, the learned representations within intrinsic motivation have a high probability to neglect the distinction among different states and be distracted by the task-irrelevant information brought by the stochasticity. (2) Insufficient interrelation among dynamics: Unsuccessful guidance provided by the uninformative extrinsic reward makes the dynamics learning in intrinsic motivation less effective. In light of the above observations, a novel Behavioral metric with Cyclic Dynamics (BCD) is proposed, which considers both cumulative and immediate effects and facilitates the abstraction and exploration of the agent. For the behavioral metric, the successor feature is utilized to reveal the expected future rewards and alleviate the heavy reliance of previous methods on extrinsic rewards. Moreover, the latent variable and vector quantization techniques are employed to enable an accurate measurement of the transition function in a discrete and interpretable manner. In addition, cyclic dynamics is established to capture the interrelations between state and action, thereby providing a thorough awareness of environmental dynamics. Extensive experiments conducted on procedurally-generated environments demonstrate the state-of-the-art performance of our proposed BCD.

Partial index tracking (PIT) is a popular passive investment strategy aiming at replicating the performance of a market index (e.g., S&P 500). Existing PIT methods typically treat it as a regression problem and divide it into two tasks: (i) asset selection (determining which assets to choose from the index constituents) and (ii) asset allocation (deciding how to allocate capital among the selected assets). However, these methods either optimize these two tasks jointly, which has been proven to be NP-hard and inefficient when tracking large-scale constituent indices (e.g., Russell 2000), or attempt an independent optimization, lacking a connection to ensure collaborative optimization. In this paper, we present a hierarchical model for partial index tracking (HIT), which formulates PIT as a hierarchical Markov decision process (MDP) and is optimized via hierarchical reinforcement learning (HRL). HIT consists of (1) a high-level policy learns to select assets from constituents to handle task (i) and (2) a low-level policy learns to allocate capital weights among the selected assets to handle task (ii). We further propose a novel cost-sensitive reward function that serves as a connection to collaboratively optimize the two policies, aiming to replicate the index closely while considering transaction cost. Compared with existing jointly optimized approaches, our model simplifies the problem by learning separate policies for the two tasks, and the reward function serves as a connection to ensure collaborative optimization between them, avoiding challenges faced by joint optimization methods in existing literature. Remarkable performance across 6 benchmarks, ranging from small to large-scale constituents demonstrate the superiority of HIT. Moreover, the experiments conducted on a real-world market dataset spanning over 10 years show its effectiveness and practicality.

Deep reinforcement learning (RL) has emerged as a promising approach for portfolio management due to its ability to make sequential decisions. However, applying RL techniques to this domain is still challenging due to the non-stationary nature of financial markets. Existing RL-based solutions fail to consider the intrinsic causes behind this non-stationary, which primarily stem from the involvement of diverse traders with distinct investment horizons and their varied investment strategies. In this paper, we tackle the non-stationary problem by examining its intrinsic causes and propose cross-insight trader, a novel two-step RL-based approach that integrates multiple trading policies with different investment horizons to adapt to the changing market conditions. In the first step, we learn multiple horizon-specific policies by providing each policy with tailored information specific to its investment horizon. This allows each policy to recognize dynamic patterns within its respective horizon and make insightful pre-decisions. In the second step, we learn a cross-insight policy to make the final trade decision by considering the investment pre-decisions made by multiple horizon-specific policies in the first step. To enable effective learning of two types of policies, our approach employs a centralized critic to evaluate the actions performed by both horizon-specific and cross-insight policies. By incorporating multiple insights from different investment horizons into the decision-making process, our approach enhances its adaptability to changing market conditions. Experimental results conducted on three stock markets demonstrate the superiority of our framework.

The integration of artificial intelligence has become a hot topic in the field of education. However, current studies primarily focus on personalization for learners, with the aim of accurately modeling learners’ knowledge level based on their learning history and providing better personalized services, while overlooking the needs of educators. In contrast to the focus on personalization of learners, educators place greater emphasis on accurately assessing collective performance and relative differences within a group, which serves as a qualitative measure of the teaching quality. In this study, we investigate collective knowledge tracing (CeKT), a method designed to estimate the average knowledge level of all students in a course based on a sequence of exercises. To achieve this objective, we propose a graph-based solution capable of estimating the average knowledge level solely from the exercise sequence as well as capturing the intrinsic structure of the exercise sequence (i.e., the sequential order of exercises and the repetition of exercises). Through experimental validation, we affirm that our approach enables a precise estimation of the average knowledge mastery of all students given an exercise sequence and also holds distinct value in three applications within the education domain.

Stock prediction is an attractive topic in fintech. However, traditional solutions for stock prediction have two drawbacks: (1) Some focus on the temporal patterns of stocks and model each stock as an independent individual but neglect their relations. Some models consider the relations among stocks, but work in a two-step format (i.e., capturing the temporal patterns first and then considering the relation dependency), which makes them complex and inefficient; (2) They model the stock prediction as a regression (predicting stock price) or classification task (predicting stock trend), which cannot optimize the target of investment, i.e., selecting the best stocks from the exchange market with the highest expected revenue in the future. To fully utilize the relations among stocks and achieve the highest revenue, a relation-temporal graph convolutional network (RT-GCN) is proposed. We first model the relations among stocks and their daily features into a relation-temporal graph. Then, we apply RT-GCN and three relation-aware strategies to realize the relation-temporal feature extraction for each stock. Finally, the features are fed for score calculation in a learning-to-rank way, and the stock with the highest score represents the highest investment revenue in the future. Extensive experiments demonstrate the effectiveness and efficiency of our method.

A protein complex is a group of associated polypeptide chains which plays essential roles in the biological process. Given a graph representing protein-protein interactions (PPI) network, it is critical but non-trivial to detect protein complexes, the subsets of proteins that are tightly coupled, from it. Network embedding is a technique to learn low-dimensional representations of vertices in networks. It has been proved quite useful for community detection in social networks in recent years. However, unlike social networks, PPI network does not contain rich metadata, so that existing network embedding methods cannot fully capture the network structure of PPI to improve the effect of protein complexes detection significantly. We propose a semi-supervised network embedding model by adopting graph convolutional networks to detect densely connected subgraphs effectively. We compare the performance of our model with state-of-the-art approaches on three popular PPI networks with various data sizes and densities. The experimental results show that our approach significantly outperforms other approaches on all three PPI networks.

With the development of modern technology, more and more research papers have been published and shared in various digital databases. However, it is time-consuming for researchers to find suitable scholars who study the same research field with them. To address this issue, we focus on proposing a keyword-based scholar recommendation framework, which can help users to advance their research by recommending scholars that align with the users interests. We first utilize keywords that are extracted from abstract to construct a word-word co-occurrence graph in each query. Based on the graph, we propose an approach to find core nodes to solve cold start problem by treating these core nodes as the users interests. We then combine these core nodes and authors to build a bipartite graph, and adopted the PersonalRank algorithm to rank authors based on the bipartite graph. Finally, we design a recommendation evaluation criterion by comparing our recommendation lists with the results lists of Microsoft Academic search. Experimental results show that our recommendation framework can effectively and efficiently generate scholar recommendation in the situation that lack of the citations times, user behavior and other important indicators.