Offline Reinforcement Learning (RL) aims to learn policies from pre-collected datasets without any additional interaction. In order to perform well and robustly in dynamic environments with noise or disturbances, the learned value function and derived policy should generalize well within and near the dataset distribution, rather than ‘over-fitting’ to training samples. To meet this requirement, we propose a new approach called Conservative In-Distribution Q-learning (CIDQL) that takes a step towards in-distribution offline RL. CIDQL is designed to learn in-distribution with respect to the dataset, using a perturbation-based interpolation technique and a quantile method for value regularization. It prohibits bootstrapping during value iteration, ensuring stable Q-value learning that is separated from policy improvement. The approach has theoretical guarantees for both Q-value underestimation and non-underestimation, and outperforms most SOTA algorithms on D4RL gym-MuJoCo benchmarks.

Offline reinforcement learning faces a significant challenge of value over-estimation due to the distributional drift between the dataset and the current learned policy, leading to learning failure in practice. The common approach is to incorporate a penalty term to reward or value estimation in the Bellman iterations. Meanwhile, to avoid extrapolation on out-of-distribution (OOD) states and actions, existing methods focus on conservative Q-function estimation. In this paper, we propose Conservative State Value Estimation (CSVE), a new approach that learns conservative V-function via directly imposing penalty on OOD states. Compared to prior work, CSVE allows more effective state value estimation with conservative guarantees and further better policy optimization. Further, we apply CSVE and develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states around the dataset, and the actor applies advantage weighted updates extended with state exploration to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods and is strongly competitive among recent SOTA methods.

This paper investigates a critical resource allocation problem in the first party cloud: scheduling containers to machines. There are tens of services, and each service runs a set of homogeneous containers with dynamic resource usage; containers of a service are scheduled daily in a batch fashion. This problem can be naturally formulated as Stochastic Bin Packing Problem (SBPP). However, traditional SBPP research often focuses on cases of empty machines, whose objective, i.e., to minimize the number of used machines, is not well-defined for the more common reality with nonempty machines. This paper aims to close this gap. First, we define a new objective metric, Used Capacity at Confidence (UCaC), which measures the maximum used resources at a probability and is proved to be consistent for both empty and nonempty machines and reformulate the SBPP under chance constraints. Second, by modeling the container resource usage distribution in a generative approach, we reveal that UCaC can be approximated with Gaussian, which is verified by trace data of real-world applications. Third, we propose an exact solver by solving the equivalent cutting stock variant as well as two heuristics-based solvers -- UCaC best fit, bi-level heuristics. We experimentally evaluate these solvers on both synthetic datasets and real application traces, demonstrating our methodology's advantage over traditional SBPP optimal solver minimizing the number of used machines, with a low rate of resource violations.

On-demand air mobility services, often called air taxis, are on the way to revolutionize our urban/regional transportation sector by lifting transportation to the third dimension and thus possibly contribute to solving the congestion-induced transportation deadlock many metropolitan regions face today. Although existing research mainly focuses on the design of efficient vehicles and specifically battery technology, in the near future, a new question will arise: Where to locate the vertiports/landing pads for such air taxis? In this study, we propose a vertiport location selection problem. In contrast to existing studies, we allow the demand to be distributed over the whole metropolitan area, modeled as a grid, and exclude certain grid cells from becoming hubs, for example, because of safety/geographical constraints. The combination of these two contributions makes the problem intriguingly difficult to solve with standard solution techniques. We propose a novel variable neighborhood search heuristic, which is able to solve 12 × 12 grid instances within a few seconds of computation time and zero gaps in our experiments, whereas CPLEX needs up to 10 hours. We believe that our study contributes toward the scalable selection of vertiport locations for air taxis. Summary of Contribution: The increasing interest in opening the third dimension, that is, altitude, to transportation inside metropolitan regions raises new research challenges. Existing research mainly focuses on the design of efficient vehicles and control problems. In the near future, however, the actual operation of air taxis will lead to new set of operations research problems for so-called air taxi operations. Our contribution focuses on the optimization of vertiports for air taxi operations in a metropolitan region. We choose to model the problem over a grid-like demand structure, with a novel side constraint: selected grid cells are unavailable as hubs, for example, because of environmental, technical, cultural, or other reasons. This makes our model a special case in between the two traditional models: discrete location and continuous location. Our model is inherently difficult to solve for exact methods; for instance, solving a grid of 12 × 12 grid cells needs more than 10 hours with CPLEX, when modeled as a discrete location problem. We show that a straightforward application of existing neighborhood search heuristics is not suitable to solve this problem well. Therefore, we design an own variant of mixed variable neighborhood search, which consists of novel local search steps, tailored toward our grid structure. Our evaluation shows that by using our novel heuristic, almost all instances can be solved toward optimality.

Prediction+optimization is a common real-world paradigm where we have to predict problem parameters before solving the optimization problem. However, the criteria by which the prediction model is trained are often inconsistent with the goal of the downstream optimization problem. Recently, decision-focused prediction approaches, such as SPO+ and direct optimization, have been proposed to fill this gap. However, they cannot directly handle the soft constraints with the max operator required in many real-world objectives. This paper proposes a novel analytically differentiable surrogate objective framework for real-world linear and semi-definite negative quadratic programming problems with soft linear and non-negative hard constraints. This framework gives the theoretical bounds on constraints’ multipliers, and derives the closed-form solution with respect to predictive parameters and thus gradients for any variable in the problem. We evaluate our method in three applications extended with soft constraints: synthetic linear programming, portfolio optimization, and resource provisioning, demonstrating that our method outperforms traditional two-staged methods and other decision-focused approaches