Advantage-Conditioned Flow Policy for Offline Reinforcement Learning in Recommendation
Abstract
Offline reinforcement learning (RL) is a useful approach for recommender systems because it can optimize long-term user feedback from logged interaction data without online exploration. A key challenge is the multi-modal nature of user preferences: a user may like several unrelated item types, so a unimodal policy (for example, a Gaussian) tends to average across modes and generate actions that do not match any interest. Recent diffusion-based policies can model complex preference distributions, but they often require many denoising steps. We propose PerfRec (Preference-aware Flow for Recommendation), a flow-matching offline RL framework that learns an expressive behavioral policy and distills it into an efficient one-step policy. PerfRec (i) trains a conditional flow model to clone the logged action distribution, (ii) trains twin Q-networks using next actions sampled from the learned flow policy, and (iii) trains an advantage-conditioned one-step policy with Q-guidance for improvement and a distillation loss that keeps the policy close to the flow policy. We use binary advantage conditioning to separate high-advantage and low-advantage regions of the flow-induced action distribution, so that at inference we can sample from the high-advantage mode with a single forward pass. Experiments on five benchmark datasets and one online simulation platform show that PerfRec improves recommendation performance over strong offline RL baselines.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Shuchang Liu 0001 | 58 | 88 Modeling User Retention through Generative Flow Networks (2024) Diffusion and Flow Recommenders 74 Exploration and Regularization of the Latent Action Space in Recommendation (2023) Offline RL for Recommendation 69 Fading to Grow: Growing Preference Ratios via Preference Fading Discrete Diffusion for Recommendation (2025) Diffusion and Flow Recommenders |
|
| 2 | Yuta Saito | 47 | 51 Off-Policy Evaluation of Ranking Policies under Diverse User Behavior (2023) Recommendation with Search and Ranking 49 Evaluating the Robustness of Off-Policy Evaluation (2021) Recommendation Systems and Evaluation 49 Asymmetric Tri-training for Debiasing Missing-Not-At-Random Explicit Feedback (2020) Popularity Bias and Debiasing in Recommendation |
|
| 3 | Ziru Liu | 47 | 69 Diff-MSR: A Diffusion Model Enhanced Paradigm for Cold-Start Multi-Scenario Recommendation (2024) Diffusion and Flow Recommenders 60 Modeling User Retention through Generative Flow Networks (2024) Multi-Behavior Recommendation 60 Multi-Task Recommendations with Reinforcement Learning (2023) Offline RL for Recommendation |
Area chair
| Area chair | Match | |
|---|---|---|
| Leyu Lin | 48 |
Area-chair candidates (6)
| Leyu Lin assigned | 48 | 15 papers | |
| Xin Xin 0003 | 47 | 10 papers | |
| Yanan Niu | 45 | 15 papers | |
| Kun Gai | 44 | 5 papers | |
| Maolin Wang 0001 | 43 | 15 papers | |
| Pengyue Jia | 42 | 10 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Zhengyi Yang 0007 | 59 | 3 | 74 Generate What You Prefer: Reshaping Sequential Recommendation via Guided Diffusion (2023) Diffusion and Flow Recommenders 74 On Efficiency-Effectiveness Trade-off of Diffusion-based Recommenders (2025) Diffusion and Flow Recommenders |
|
| 2 | Shuchang Liu 0001 assigned | 58 | 3 | 88 Modeling User Retention through Generative Flow Networks (2024) Diffusion and Flow Recommenders 74 Exploration and Regularization of the Latent Action Space in Recommendation (2023) Offline RL for Recommendation |
|
| 3 | Lantao Hu | 54 | 3 | 74 Fading to Grow: Growing Preference Ratios via Preference Fading Discrete Diffusion for Recommendation (2025) Diffusion and Flow Recommenders 70 Distinguished Quantized Guidance for Diffusion-based Sequence Recommendation (2025) Diffusion and Flow Recommenders |
|
| 4 | Xuanhua Yang | 49 | 3 | 69 Towards Personalized Bundle Creative Generation with Contrastive Non-Autoregressive Decoding (2022) Diffusion and Flow Recommenders 56 GenRec: A Preference-Oriented Generative Framework for Large-Scale Recommendation (2026) Generative Retrieval and Ranking |
|
| 7 | Chongming Gao | 47 | 3 | 60 Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation (2023) Popularity Bias and Debiasing in Recommendation 52 Reformulating Conversational Recommender Systems as Tri-Phase Offline Policy Learning (2024) Conversational Recommendation Systems |
|
| 8 | Fei Sun 0001 | 47 | 3 | 58 Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning (2021) Conversational Recommendation Systems 54 Exact-K Recommendation via Maximal Clique Optimization (2019) Graph-based User and Item Representation Learning |
|
| 9 | Yuta Saito assigned | 47 | 3 | 51 Off-Policy Evaluation of Ranking Policies under Diverse User Behavior (2023) Recommendation with Search and Ranking 49 Evaluating the Robustness of Off-Policy Evaluation (2021) Recommendation Systems and Evaluation |
|
| 10 | Ziru Liu assigned | 47 | 3 | 69 Diff-MSR: A Diffusion Model Enhanced Paradigm for Cold-Start Multi-Scenario Recommendation (2024) Diffusion and Flow Recommenders 60 Modeling User Retention through Generative Flow Networks (2024) Multi-Behavior Recommendation |
|
| 11 | Jianling Wang | 46 | 3 | 53 A Hierarchical Self-Attentive Model for Recommending User-Generated Item Lists (2019) Efficient and Long-Sequence Sequential Recommendation 49 Learning to Augment for Casual User Recommendation (2022) Contrastive and Self-Supervised Sequential Recommendation |
|
| 12 | Seongmin Park 0002 | 46 | 3 | 52 Temporal Linear Item-Item Model for Sequential Recommendation (2025) Efficient and Long-Sequence Sequential Recommendation 48 MUFFIN: Mixture of User-Adaptive Frequency Filtering for Sequential Recommendation (2025) Sequential Recommendation with User Modeling and Profiling |
|
| 13 | Yongjing Hao | 46 | 3 | 63 Meta-Optimized Joint Generative and Contrastive Learning for Sequential Recommendation (2024) Diffusion and Flow Recommenders 60 Click is not equal to purchase: multi-task reinforcement learning for multi-behavior recommendation (2023) Multi-Behavior Recommendation |
|
| 14 | Ching Law | 45 | 3 | 74 Uni-Layout: Integrating Human Feedback in Unified Layout Generation and Evaluation (2025) Diffusion and Flow Recommenders 74 MoFu: Scale-Aware Modulation and Fourier Fusion for Multi-Subject Video Generation (2026) Diffusion and Flow Recommenders |
|
| 16 | Han Li 0005 | 45 | 3 | 74 DiffusionGS: Generative Search with Query Conditioned Diffusion in Kuaishou (2026) Diffusion and Flow Recommenders 49 Stratified Expert Cloning for Retention-Aware Recommendation at Scale (2025) Long-Term Engagement and Retention Optimization |
|
| 17 | Jiancan Wu | 45 | 3 | 69 Generate What You Prefer: Reshaping Sequential Recommendation via Guided Diffusion (2023) Diffusion and Flow Recommenders 63 Fading to Grow: Growing Preference Ratios via Preference Fading Discrete Diffusion for Recommendation (2025) Diffusion and Flow Recommenders |
|
| 18 | Bo Peng 0009 | 44 | 3 | 53 M2: Mixed Models With Preferences, Popularities and Transitions for Next-Basket Recommendation (2023) Efficient and Long-Sequence Sequential Recommendation 48 eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data (2024) Efficiency and Deployment |
|
| 20 | Ed H. Chi | 44 | 3 | 48 PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations (2026) Item Tokenization and Semantic IDs 46 Top-K Off-Policy Correction for a REINFORCE Recommender System (2019) Long-Term Engagement and Retention Optimization |
|
| 21 | Mengyuan Jing | 44 | 3 | 67 Generating Difficulty-aware Negative Samples via Conditional Diffusion for Multi-modal Recommendation (2025) Diffusion and Flow Recommenders 66 Interest-Aware Graph Contrastive Learning for Recommendation With Diffusion-Based Augmentation (2026) Graph Contrastive and Self-Supervised Learning |
|
| 22 | Xiao Zhang 0034 | 44 | 3 | 50 When Search Meets Recommendation: Learning Disentangled Search Representation for Recommendation (2023) Recommendation with Search and Ranking 48 Test-Time Alignment with State Space Model for Tracking User Interest Shifts in Sequential Recommendation (2025) Sequential Recommendation with User Modeling and Profiling |
|
| 23 | Hao Liu 0078 | 44 | 3 | 49 WeaveRec: An LLM-Based Cross-Domain Sequential Recommendation Framework with Model Merging (2026) Efficiency and Deployment 48 Mitigating Distribution Shifts in Sequential Recommendation: An Invariance Perspective (2025) Contrastive and Self-Supervised Sequential Recommendation |
|
| 24 | Jieming Zhu | 43 | 3 | 48 EAGER: Two-Stream Generative Recommender with Behavior-Semantic Collaboration (2024) Item Tokenization and Semantic IDs 47 Directional Adversarial Training for Recommender Systems (2020) Robustness to Noise and Distribution Shift |
|
| 25 | Jin Chen 0008 | 43 | 3 | 48 Cooperative Retriever and Ranker in Deep Recommenders (2023) Recommendation with Search and Ranking 47 Dual Test-Time Training for Out-of-Distribution Recommender System (2025) Robustness to Noise and Distribution Shift |
|
| 26 | Bowen Zheng 0005 | 43 | 3 | 48 Enhancing Sequential Recommender with Large Language Models for Joint Video and Comment Recommendation (2025) Sequential and Session-based Recommendation 48 Reciprocal Sequential Recommendation (2023) Sequential Recommendation with User Modeling and Profiling |
|
| 27 | Qiang Luo 0004 | 43 | 3 | 56 OneLoc: Geo-Aware Generative Recommender Systems for Local Life Service (2026) Item Tokenization and Semantic IDs 56 DiffGRM: Diffusion-based Generative Recommendation Model (2026) Diffusion and Flow Recommenders |
|
| 28 | Bo Chen 0023 | 43 | 3 | 49 ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation (2024) Sequential and Session-based Recommendation 48 Full-Stack Optimized Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation (2026) Sequential and Session-based Recommendation |
|
| 30 | Jizhi Zhang | 43 | 3 | 48 Navigating Large Language Models for Recommendation: From Architecture to Learning Paradigms and Deployment (2025) Efficiency and Deployment 48 BSL: Understanding and Improving Softmax Loss for Recommendation (2024) Popularity Bias and Debiasing in Recommendation |
|
| 31 | Yongxin Ni | 43 | 3 | 46 Bridging NIP and MLM: A Unified Meta-Learning Framework for Sequential Recommendation (2026) Meta-Learning for Cold-Start Recommendation 45 Collaborative filtering with implicit feedback via learning pairwise preferences over user-groups and item-sets (2022) Negative Sampling and Loss Functions |
|
| 32 | Jiangxia Cao | 43 | 3 | 72 From Agnostic to Specific: Latent Preference Diffusion for Multi-Behavior Sequential Recommendation (2026) Diffusion and Flow Recommenders 46 CDSRNP: Cross-Domain Sequential Recommendation via Neural Process (2025) Cross-Domain Sequential Recommendation |
|
| 33 | Riwei Lai | 43 | 3 | 47 DAR: Dimension-Adaptive Recommendation with Multi-Granular Noise Control (2025) Robustness to Noise and Distribution Shift 47 Proxy-Aware Cross-Domain Sequential Recommendation (2023) Cross-Domain Sequential Recommendation |
|
| 34 | Lichan Hong | 43 | 3 | 51 Off-policy Learning in Two-stage Recommender Systems (2020) Recommendation with Search and Ranking 47 PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations (2026) Generative Recommendation and Item Tokenization |
|
| 35 | Noveen Sachdeva | 43 | 3 | 48 Explicit Modelling of the Implicit Short Term User Preferences for Music Recommendation (2018) Sequential Recommendation with User Modeling and Profiling 47 ActionPiece: Contextually Tokenizing Action Sequences for Generative Recommendation (2025) Item Tokenization and Semantic IDs |