As a data-driven learning approach, model-based offline reinforcement learning (MORL) aims to learn a policy by exploiting a dynamics model derived from an existing dataset. Applying conservative quantification to the dynamics model, most existing works on MORL generate trajectories that approximate the real data distribution to facilitate policy learning. However, these methods typically overlook the influence of historical information on environmental dynamics, thus generating unreliable trajectories that fail to align with the true data distribution. In this paper, we propose a new MORL algorithm called Reliability-guaranteed and Reward-seeking Transformer (RT). RT can avoid generating unreliable trajectories through the calculation of cumulative reliability of the trajectories, which is a weighted variational distance between the generated trajectory distribution and the true data distribution. Moreover, by sampling candidate actions with high rewards, RT can efficiently generate high-reward trajectories from the existing offline data, thereby further facilitating policy learning. We theoretically prove the performance guarantees of RT in policy learning, and empirically demonstrate its effectiveness against state-of-the-art model-based methods on several offline benchmark tasks and a large-scale industrial dataset from an on-demand food delivery platform.

Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large joint state-action space and coupled behaviors of multiple agents, which bring extra complexity to offline policy optimization. In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic, leading to uncoordinated behaviors and out-of-distribution (OOD) joint actions. To address these issues, we propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO). InSPO sequentially updates each agent's policy in an in-sample manner, which not only avoids selecting OOD joint actions but also carefully considers teammates' updated policies to enhance coordination. Additionally, by thoroughly exploring low-probability actions in the behavior policy, InSPO can well address the issue of premature convergence to sub-optimal solutions. Theoretically, we prove InSPO guarantees monotonic policy improvement and converges to quantal response equilibrium (QRE). Experimental results demonstrate the effectiveness of our method compared to current state-of-the-art offline MARL methods.

The recent past has witnessed a notable surge in on-demand food delivery (OFD) services, offering delivery fulfillment within dozens of minutes after an order is placed. In OFD, pooling multiple orders for simultaneous delivery in real-time order assignment is a pivotal efficiency source, which may in turn extend delivery time. Constructing high-quality order pooling to harmonize platform efficiency with the experiences of consumers and couriers, is crucial to OFD platforms. However, the complexity and real-time nature of order assignment, making extensive calculations impractical, significantly limit the potential for order consolidation. Moreover, offline environment is frequently riddled with unknown factors, posing challenges for the platform's perceptibility and pooling decisions.

As one representative of last-mile logistics in intelligent transportation systems, the on-demand food delivery (OFD) service has gained rapid market growth but also faces multiple challenges. One of the critical issues is the order dispatching problem (ODP) with an NP-hard nature, which refers to dispatching a large number of orders to riders reasonably in real time with very limited decision time. To address the ODP, this paper proposes an optimization algorithm based on graph neural networks (GNN) by combining the advantages of machine learning (ML) techniques and operational research (OR) methods: 1) The ML component learns to reduce the solution space by filtering out inappropriate riders for each order, handling the large-scale complexity of ODP. Specifically, we present a rider modeling approach by using GNN to better characterize rider information; besides, two attention mechanisms are designed to adaptively learn the matching relationship between riders and orders. 2) The OR component ensures the solution quality with a greedy and regret value-based dispatching heuristic. Extensive experiments are conducted on real-world datasets to evaluate the performance of the proposed method by comparing it with other existing models and algorithms. The results show that the design of our ML model is effective in yielding better prediction results, and the proposed GNN-based optimization algorithm can effectively and efficiently solve the ODP by improving delivery efficiency and customer satisfaction.

On-demand food delivery (OFD) service has gained fast-growing popularity all around the world. Order dispatching is instrumental to large-scale OFD platforms, such as Meituan, which continuously match food order requests to couriers at a scale of tens of millions each day to satisfy the needs of consumers, couriers, and merchants. However, due to high dynamism and inevitable uncertainties in the real-world environment, it is not an easy task to achieve long-term global objective optimization through continuous isolated optimization decisions at each dispatch moment. Our work proposes the concept of "courier occupancy" (CO) to precisely quantify the impact of order assignment on the courier's delivery efficiency, realizing a decomposition of long-term and macro goals into various dispatch moments and micro decision-making dimensions. Then in the prediction phase, an improved and universally applicable distribution estimation method is designed to quantify CO which is a stochastic variable and contains future information, combining Monte Carlo dropout and knowledge distillation. In the optimization phase, we use CO to model the objective function at each dispatch moment to introduce future information and extend dispatch decisions from merely who to assign the order to both when and who to assign it, significantly enhancing the long-term optimization capability of dispatching decisions and avoiding local greed. We conduct extensive offline simulations based on real dispatching data as well as online AB tests through Meituan's platform. Results show that our method consistently improves the couriers' delivery efficiency and consumers' satisfaction.

Recently, deep learning based methods, especially graph neural network (GNN), have made impressive progress on rating prediction problem in recommender systems. However, the performance of existing methods drops quickly in the cold start scenario. More importantly, such methods are unable to learn the preference embedding of a strict cold start user/item since there is no interaction for this user/item. In this work, we develop a novel framework Attribute Graph Neural Networks (AGNN) by exploiting the attribute graph rather than the commonly used interaction graph. AGNN can produce the preference embedding for a strict cold user/item by learning on the distribution of attributes with an extended variational auto-encoder (eVAE) structure. It also contains a new graph neural network variant (gated-GNN) to effectively aggregate various attributes of different dimensions in a neighborhood. Empirical results demonstrate that AGNN achieves the new state-of-the-art performance.

Reinforcement learning based recommender systems (RL-based RS) aim at learning a good policy from a batch of collected data, by casting recommendations to multi-step decision-making tasks. However, current RL-based RS research commonly has a large reality gap. In this paper, we introduce the first open-source real-world dataset, RL4RS, hoping to replace the artificial datasets and semi-simulated RS datasets previous studies used due to the resource limitation of the RL-based RS domain. Unlike academic RL research, RL-based RS suffers from the difficulties of being well-validated before deployment. We attempt to propose a new systematic evaluation framework, including evaluation of environment simulation, evaluation on environments, and counterfactual policy evaluation. In summary, the RL4RS (Reinforcement Learning for Recommender Systems), a new resource with special concerns on the reality gaps, contains two real-world datasets, data understanding tools, tuned simulation environments, related advanced RL baselines, batch RL baselines, and counterfactual policy evaluation algorithms. The RL4RS suite can be found at https://github.com/fuxiAIlab/RL4RS.

On-demand food delivery (OFD) service is a representative scenario of last-mile logistics. It has gained a fast-growing market but also encounters many challenges, e.g., high dynamism, large-scale complexity, and immediacy requirement. To solve the OFD problem, a predictive-reactive optimization framework with feedback-based knowledge distillation is presented by organically combining deep learning technology and operational research method. In the prediction phase, a deep learning model is designed to predict future information which can reflect the delivery efficiency of dispatching results. To improve model performance, a feedback-based knowledge distillation is proposed which balances the diversity and effectiveness of the ensembled models by adaptively controlling learning weights. In the optimization phase, to avoid myopic decisions and obtain high-quality solutions for long-term objectives, a greedy heuristic with a multi-stage decision-making strategy is designed by employing the predicted future information to assist in making decisions. Extensive experiments are conducted on real-world datasets to test the performance of the proposed model and heuristic. Besides, the simulation results illustrate the superiority of the proposed framework for solving the OFD problem in both delivery efficiency and customer satisfaction.

One key property in recommender systems is the long-tail distribution in user-item interactions where most items only have few user feedback. Improving the recommendation of tail items can promote novelty and bring positive effects to both users and providers, and thus is a desirable property of recommender systems. Current novel recommendation methods over-emphasize the importance of tail items without differentiating the degree of users’ intent on popularity and often incur a sharp decline of accuracy. Moreover, none of existing studies has ever taken the extreme case of tail items, i.e., cold-start items without any interaction, into consideration. In this work, we first disclose the mechanism that drives a user's interaction towards popular or niche items by disentangling her intent into conformity influence (popularity) and personal interests (preference). We then present a unified end-to-end framework to simultaneously optimize accuracy and novelty targets based on the disentangled intent of popularity and that of preference. We further develop a new paradigm for novel recommendation of cold-start items which exploits the self-supervised learning technique to model the correlation between collaborative features and content features. We conduct extensive experiments on three real-world datasets. The results demonstrate that our proposed model yields significant improvements over the state-of-the-art baselines in terms of the trade-off between accuracy and novelty.

Recent years have witnessed the great accuracy performance of graph-based Collaborative Filtering (CF) models for recommender systems. By taking the user-item interaction behavior as a graph, these graph-based CF models borrow the success of Graph Neural Networks (GNN), and iteratively perform neighborhood aggregation to propagate the collaborative signals. While conventional CF models are known for facing the challenges of the popularity bias that favors popular items, one may wonder "Whether the existing graph-based CF models alleviate or exacerbate the popularity bias of recommender systems?" To answer this question, we first investigate the two-fold performances w.r.t. accuracy and novelty for existing graph-based CF methods. The empirical results show that symmetric neighborhood aggregation adopted by most existing graph-based CF models exacerbates the popularity bias and this phenomenon becomes more serious as the depth of graph propagation increases. Further, we theoretically analyze the cause of popularity bias for graph-based CF. Then, we propose a simple yet effective plugin, namely r-AdjNorm, to achieve an accuracy-novelty trade-off by controlling the normalization strength in the neighborhood aggregation process. Meanwhile, r-AdjNorm can be smoothly applied to the existing graph-based CF backbones without additional computation. Finally, experimental results on three benchmark datasets show that our proposed method can improve novelty without sacrificing accuracy under various graph-based CF backbones.

Rating prediction is a classic problem underlying recommender systems. It is traditionally tackled with matrix factorization. Recently, deep learning based methods, especially graph neural networks, have made impressive progress on this problem. Despite their effectiveness, existing methods focus on modeling the user-item interaction graph. The inherent drawback of such methods is that their performance is bound to the density of the interactions, which is however usually of high sparsity. More importantly, for a strict cold start user/item that neither appears in the training data nor has any interactions in the test stage, such methods are unable to learn the preference embedding of the user/item since there is no link to this user/item in the graph. In this work, we develop a novel frameworkAttribute Graph Neural Networks(AGNN) by exploiting the attribute graph rather than the commonly used interaction graph. This leads to the capability of learning embeddings for the strict cold start users/items. Our AGNN can produce the preference embedding for a strict cold user/item by learning on the distribution of attributes with an extended variational auto-encoder (eVAE) structure. Moreover, we propose a new graph neural network variant, i.e., gated-GNN, to effectively aggregate various attributes of different modalities in a neighborhood. Empirical results on three real-world datasets demonstrate that our model yields significant improvements for strict cold start recommendations and outperforms or matches the state-of-the-art performance in the warm start scenario.

Recommender systems are playing a vital role in online platforms due to the ability of incorporating users' personal tastes. Beyond accuracy, diversity has been recognized as a key factor to broaden users' horizons as well as to promote enterprises' sales. However, the trade-off between accuracy and diversity remains to be a big challenge. More importantly, none of existing methods has explored the domain and user biases toward diversity.

Point-of-Interest (POI) recommendation has been a trending research topic as it generates personalized suggestions on facilities for users from a large number of candidate venues. Since users' check-in records can be viewed as a long sequence, methods based on recurrent neural networks (RNNs) have recently shown promising applicability for this task. However, existing RNN-based methods either neglect users' long-term preferences or overlook the geographical relations among recently visited POIs when modeling users' short-term preferences, thus making the recommendation results unreliable. To address the above limitations, we propose a novel method named Long- and Short-Term Preference Modeling (LSTPM) for next-POI recommendation. In particular, the proposed model consists of a nonlocal network for long-term preference modeling and a geo-dilated RNN for short-term preference learning. Extensive experiments on two real-world datasets demonstrate that our model yields significant improvements over the state-of-the-art methods.

Abstract Recent years have witnessed a growing trend of utilizing reviews to improve the performance and interpretability of recommender systems. Almost all existing methods learn the latent representations from the user’s and the item’s historical reviews and then combine these two representations for rating prediction. The fatal limitation in these methods is that they are unable to utilize the most predictive review of the target user for the target item since such a review is not available at test time. In this paper, we propose a novel recommendation model, called AGTR, which cangenerate the unseen target review with adversarial training for rating prediction. To this end, we develop a unified framework to combinethe rating tailored generative adversarial netsfor synthetic review generation andthe neural latent factor moduleusing the generated target review along with historical reviews for rating prediction. Extensive experiments on four real-world datasets demonstrate that our model achieves the state-of-the-art performance in both rating prediction and review generation tasks.

The development of the cutting-edge technologies in cogeneration and trigeneration has led to a rapid transition toward integrated energy systems and the mushrooming of energy hubs, calling for effective energy management schemes. This paper proposes a distributed algorithm for autonomous energy management (AEM) of a cluster of residential energy hubs. Given the interactive behaviors of energy purchasing at the supply side, we treat each hub as a self-interested agent, and formulate the AEM problem of these hubs as a monotone generalized Nash game (MON-GNG). On one hand, there are global coupling constraints representing the supply limits of the input energy systems, which are imposed by the limited capacities of electrical feeders and natural gas pipelines, thus making it a GNG. On the other hand, the cost function of each hub is merely convex in its actions considering the input-to-output energy transformation inside and the impacts of the energy storage devices, which leads to an MON game. The existence of the generalized Nash equilibria (GNEs) of this MON-GNG can be theoretically guaranteed. Then, by reformulating the MON-GNG as a variational inequality problem with special decomposition structure, an efficient and single-loop distributed algorithm is then proposed for computing a GNE with clear economic interpretation based on an improved Tikhonov regularization technique. Principles of parameter selection that will guarantee convergence are suggested. Numeric simulations validate the convergence performance and effectiveness of the proposed algorithm.