Factored Markov Decision Processes (FMDPs) offer a promising framework for overcoming the curse of dimensionality in reinforcement learning (RL) by decomposing high-dimensional MDPs into smaller and independently evolving components. Despite their potential, existing studies on FMDPs face three key limitations: reliance on perfectly factorizable models, suboptimal sample complexity guarantees for model-based algorithms, and the absence of model-free algorithms. To address these challenges, we introduce approximate factorization, which extends FMDPs to handle imperfectly factored models. Moreover, we develop a model-based algorithm and a model-free algorithm (in the form of variance-reduced Q-learning), both achieving the first near-minimax sample complexity guarantees for FMDPs. A key novelty in the design of these two algorithms is the development of a graph-coloring-based optimal synchronous sampling strategy. Numerical simulations based on the wind farm storage control problem corroborate our theoretical findings.

Traditional reinforcement learning (RL) assumes the agents make decisions based on Markov decision processes (MDPs) with one-step transition models. In many real-world applications, such as energy management and stock investment, agents can access multi-step predictions of future states, which provide additional advantages for decision making. However, multi-step predictions are inherently high-dimensional: naively embedding these predictions into an MDP leads to an exponential blow-up in state space and the curse of dimensionality. Moreover, existing RL theory provides few tools to analyze prediction-augmented MDPs, as it typically works on one-step transition kernels and cannot accommodate multi-step predictions with errors or partial action-coverage. We address these challenges with three key innovations: First, we propose the \emph{Bayesian value function} to characterize the optimal prediction-aware policy tractably. Second, we develop a novel \emph{Bellman–Jensen Gap} analysis on the Bayesian value function, which enables characterizing the value of imperfect predictions. Third, we introduce BOLA (Bayesian Offline Learning with Online Adaptation), a two-stage model-based RL algorithm that separates offline Bayesian value learning from lightweight online adaptation to real-time predictions. We prove that BOLA remains sample-efficient even under imperfect predictions. We validate our theory and algorithm on synthetic MDPs and a real-world wind energy storage control problem.

The online ride-hailing platform has revolutionized urban transport. However, there is much room for improvement. We consider meeting the demand for an online ride-hailing at transport hubs late at night, when the public transport system stops its operations. Passengers arriving late at night face a long wait before service. We launch the ride-hailing model in the theoretic framework of queueing and introduce the arrival and the service processes. To improve the efficiency of the ride-hailing platform, as well as to maintain fairness between different types of passengers, we study three variations of carpool service policies. We then provide practical guidelines on the trade-off between efficiency and fairness to assist the online platform designers. Specifically, we derive the analytical trade-off bounds with the passenger parameters. Furthermore, we suggest that these bounds can be good performance estimators for the empirical trade-off when only limited passenger information is available. This analysis motivates us to design the optimal service rate for the entire platform. Finally, we conduct numerical studies based on field data retrieved from Didi Chuxing, highlighting the remarkable performance of our proposed method in terms of improving the quality of online ride-hailing service.