ABSTRACT This paper studies a combined delivery strategy involving a private vehicle and external carriers under stochastic customer demands. The routing problem focuses on a single private vehicle, while external carriers are allowed to determine their own routes independently and are compensated with a fixed price per unit demand served. A strategy incorporating routing re‐optimization is proposed, along with a new recourse mechanism that leverages outsourcing through external carriers. To enable routing re‐optimization, a novel approximate linear programming (ALP) approach is introduced. This offers a new pathway for addressing vehicle routing problems (VRPs) under stochastic demand considerations. The ALP approach is adapted to the specific structure of routing under stochastic demands, leading to the development of a decomposition‐based ALP solution framework. This adaptation arises from changes in the decision sequence of routing and restocking at each step of the Markov decision process (MDP), which differs from previous formulations of vehicle routing under stochastic demands. Additionally, further adaptations are made to facilitate the computation of the proposed strategy by exploring the relationships among variables and constraints specific to the problem context, as well as by developing a constraint sampling procedure designed to mimic the near‐optimal heuristic policy. Our numerical results show that the proposed outsourcing‐based policy yields notable operating‐cost savings, with an average improvement of 4.06% over the traditional recourse strategy in midpoint‐depot instances. Moreover, in small instances where the optimal policy within the traditional partial re‐optimization framework can be computed, the proposed price‐directed (PD) policy still provides cost advantages over this re‐optimization scheme, demonstrating the value of our ALP‐based framework.

In restless multi-arm bandits, a central agent is tasked with optimally distributing limited resources across several bandits (arms), with each arm being a Markov decision process. In this work, we generalize the traditional restless multi-arm bandit problem with a risk-neutral objective by incorporating risk-awareness. We establish indexability conditions for the case of a risk-aware objective and provide a solution based on Whittle index. In addition, we address the learning problem when the true transition probabilities are unknown by proposing a Thompson sampling approach and show that it achieves bounded regret that scales sublinearly with the number of episodes and quadratically with the number of arms. The efficacy of our method in reducing risk exposure in restless multi-arm bandits is illustrated through a set of numerical experiments in the contexts of machine replacement and patient scheduling applications under both planning and learning setups.

We consider fair resource allocation in sequential decision-making environments modeled as weakly coupled Markov decision processes, where resource constraints couple the action spaces of $N$ sub-Markov decision processes (sub-MDPs) that would otherwise operate independently. We adopt a fairness definition using the generalized Gini function instead of the traditional utilitarian (total-sum) objective. After introducing a general but computationally prohibitive solution scheme based on linear programming, we focus on the homogeneous case where all sub-MDPs are identical. For this case, we show for the first time that the problem reduces to optimizing the utilitarian objective over the class of "permutation invariant" policies. This result is particularly useful as we can exploit efficient algorithms that optimizes the utilitarian objective such as Whittle index policies in restless bandits to solve the problem with this fairness objective. For more general settings, we introduce a count-proportion-based deep reinforcement learning approach. Finally, we validate our theoretical findings with comprehensive experiments, confirming the effectiveness of our proposed method in achieving fairness.

Yield uncertainty is an important issue in various industries such as agriculture, food, and textile where the production output relies on uncontrollable factors and fluctuating raw material quality. To systematically leverage data to deal with uncertainty cost-effectively, distributionally robust optimization combines the strengths of stochastic programming and robust optimization by optimizing the expected costs against an ambiguity set that defines possible distributions. In this work, we leverage a data-driven robust optimization framework and formulate a mixed-integer distributionally robust multi-item lot-sizing model with uncertain production yield to determine a robust production plan. To this end, we use a scenario-wise formulation that partitions the available data into scenarios that define different patterns influencing the quality of the product and production process. In addition, we apply the proposed approach to real-world data of a case study to demonstrate the effectiveness of the proposed framework in dealing with yield uncertainty. Our experimental results show that distributionally robust plans lead to more effective cost-saving strategies and decreased risk of stock-outs. Additionally, our findings suggest that the proposed model exhibits lower sensitivity to variations in production yield realizations and it is more proficient in incorporating historical data into the decision-making process. This results in a more effective response to challenges encountered within the production system under yield uncertainty. Note to Practitioners–In a production context with various sources of uncertainty for which the mathematical estimation of the uncertain parameter can be complex or hard to perform, it would be better to use the proposed robust approach. Here, any information, accurate or not, new or historical, can be integrated into the system to improve the quality of the obtained production plan, yet still robust and mitigate nervousness. For the reduction of conservatism, the proposed approaches indicate how the manufacturer risk aversion could be integrated into the decision models to respond to the strategic need for robustness for production planning under uncertainty.

We propose a general logic-based Benders decomposition (LBBD) for production planning problems with process configuration decisions. This family of problems appears in contexts where the machines are set up according to specific patterns, templates, or, in general, process configurations that allow for simultaneously producing products of different types. The problem requires determining feasible configurations for the machines and their corresponding production levels to fulfill the demand at the minimum total cost. The structure of this problem contains nonlinear constraints that link the number of units produced of each product with the used configurations and their production levels. We decompose the original problem into a master problem, where the configurations are determined, and a subproblem, where the production amounts are determined. This allows us to apply the LBBD technique to solve the problem using a standard LBBD implementation and a branch-and-check algorithm. LBBD enhancements through logic-based inequalities generated for subsets of products with common characteristics are proposed. Such inequalities represent a form of the subproblem relaxation added to the master problem during its resolution. In our computational experiments, we apply the proposed LBBD approaches to two different applications from the literature: cutting stock problems in the steel industry and a printing problem. Results show that the LBBD methods find optimal solutions much faster than the solution approaches in the literature and have a superior performance with respect to the number of instances solved to optimality and the solution quality. Summary of Contribution: In this work, we introduce a unified exact solution algorithm based on logic-based Benders decomposition to solve a class of integrated production planning problems that include process configuration decisions. We propose a general mathematical representation of the original integrated planning problem and logic-based Benders reformulations that can be applied to solve several problems within the studied class. Our implementation frameworks provide guidelines to practitioners in the field. The solution approaches in this paper together with the proposed methodological enhancements can be adapted to solve other integrated planning problems in a similar context, including the case when the original problem has a complex combinatorial and nonlinear structure.

This work investigates lot sizing with component substitution under demand uncertainty. The integration of component substitution with lot sizing in an uncertain demand context is important because the consolidation of the demand for components naturally allows risk-pooling and reduces operating costs. The considered problem is relevant not only in a production context, but also in the context of distribution planning. We propose a stochastic programming formulation for the static–dynamic type of uncertainty, in which the setup decisions are frozen but the production and consumption quantities are decided dynamically. To tackle the scalability issues commonly encountered in multistage stochastic optimization, this paper investigates the use of stochastic dual dynamic programming (SDDP). In addition, we consider various improvements of SDDP, including the use of strong cuts, the fast generation of cuts by solving the linear relaxation of the problem, and retaining the average demand scenarios. Finally, we propose two heuristics, namely, a hybrid of progressive hedging with SDDP and a heuristic version of SDDP. Computational experiments conducted on well-known instances from the literature show that the heuristic version of SDDP outperforms other methods. The proposed method can plan with up to 10 decision stages and 20 scenarios per stage, which results in 2010 scenario paths in total. Moreover, as the heuristic version of SDDP can replan to account for new information in less than a second, it is convenient in a dynamic context. Summary of Contribution: We believe our paper is suitable for the mission and scope of IJOC because we design efficient algorithms to solve an operations research problem. More precisely, we investigate the use of stochastic dual dynamic programming (SDDP) for lot sizing with component substitution under demand uncertainty. In this work, we consider the static–dynamic decision framework, and a good approximation of the expected costs in this context requires us to solve the problem with a large number of scenarios of future demand. As solving the considered problem is computationally intensive, we investigate the use of SDDP, which decomposes the problem per decision stage. We study several enhancements of SDDP, such as the use of strong cuts, the incorporation of a lower bound computed with the average demand scenario, the multicut version of SDDP, and scenario sampling with randomized quasi–Monte Carlo. Despite these improvements, the convergence of SDDP remains slow. Consequently, we propose a heuristic version of SDDP and a hybrid of progressive hedging and SDDP. We present the results of an extensive computational study performed on well-known instances from the literature. The results show that the heuristic SDDP outperforms the hybrid of progressive hedging with SDDP and state-of-the-art methods from the literature. Besides, our analysis shows that component substitution can pool the risk, and it allows maintaining the same service level with less inventory. The presented methodology can be used by practitioners to size their production lots, and subsequent researchers can build upon our results to consider uncertainty in other parameters, such as lead times, yields, and production capacities. History: Accepted by Andrea Lodi, Area Editor for Design & Analysis of Algorithms – Discrete. Funding: This work was supported by Mitacs and the Institut de Valorisation des Données (IVADO). Supplemental Material: The online supplement is available at https://doi.org/10.1287/ijoc.2022.1215 .

This paper studies the team orienteering problem, where the arrival time and service time affect the collection of profits. Such interactions result in a nonconcave profit function. This problem integrates the aspect of time scheduling into the routing decision, which can be applied in humanitarian search and rescue operations where the survival rate declines rapidly. Rescue teams are needed to help trapped people in multiple affected sites, whereas the number of people who could be saved depends as well on how long a rescue team spends at each site. Efficient allocation and scheduling of rescue teams is critical to ensure a high survival rate. To solve the problem, we formulate a mixed-integer nonconcave programming model and propose a Benders branch-and-cut algorithm, along with valid inequalities for tightening the upper bound. To solve it more effectively, we introduce a hybrid heuristic that integrates a modified coordinate search (MCS) into an iterated local search. Computational results show that valid inequalities significantly reduce the optimality gap, and the proposed exact method is capable of solving instances where the mixed-integer nonlinear programming solver SCIP fails in finding an optimal solution. In addition, the proposed MCS algorithm is highly efficient compared with other benchmark approaches, whereas the hybrid heuristic is proven to be effective in finding high-quality solutions within short computing times. We also demonstrate the performance of the heuristic with the MCS using instances with up to 100 customers. Summary of Contribution: Motivated by search and rescue (SAR) operations, we consider a generalization of the well-known team orienteering problem (TOP) to incorporate a nonlinear time-varying profit function in conjunction with routing and scheduling decisions. This paper expands the envelope of operations research and computing in several ways. To address the scalability issue of this highly complex combinatorial problem in an exact manner, we propose a Benders branch-and-cut (BBC) algorithm, which allows us to efficiently deal with the nonconcave component. This BBC algorithm is computationally enhanced through valid inequalities used to strengthen the bounds of the BBC. In addition, we propose a highly efficient hybrid heuristic that integrates a modified coordinate search into an iterated local search. It can quickly produce high-quality solutions to this complex problem. The performance of our solution algorithms is demonstrated through a series of computational experiments.

Markov Decision Processes (MDPs) are an effective model to represent decision processes in the presence of transitional uncertainty and reward tradeoffs. However, due to the difficulty in exactly specifying the transition and reward functions in MDPs, researchers have proposed uncertain MDP models and robustness objectives in solving those models. Most approaches for computing robust policies have focused on the computation of maximin policies which maximize the value in the worst case amongst all realisations of uncertainty. Given the overly conservative nature of maximin policies, recent work has proposed minimax regret as an ideal alternative to the maximin objective for robust optimization. However, existing algorithms for handling minimax regret are restricted to models with uncertainty over rewards only and they are also limited in their scalability. Therefore, we provide a general model of uncertain MDPs that considers uncertainty over both transition and reward functions. Furthermore, we also consider dependence of the uncertainty across different states and decision epochs. We also provide a mixed integer linear program formulation for minimizing regret given a set of samples of the transition and reward functions in the uncertain MDP. In addition, we provide two myopic variants of regret, namely Cumulative Expected Myopic Regret (CEMR) and One Step Regret (OSR) that can be optimized in a scalable manner. Specifically, we provide dynamic programming and policy iteration based algorithms to optimize CEMR and OSR respectively. Finally, to demonstrate the effectiveness of our approaches, we provide comparisons on two benchmark problems from literature. We observe that optimizing the myopic variants of regret, OSR and CEMR are better than directly optimizing the regret.

Bike Sharing Systems (BSSs) are widely adopted in major cities of the world due to concerns associated with extensive private vehicle usage, namely, increased carbon emissions, traffic congestion and usage of nonrenewable resources. In a BSS, base stations are strategically placed throughout a city and each station is stocked with a pre-determined number of bikes at the beginning of the day. Customers hire the bikes from one station and return them at another station. Due to unpredictable movements of customers hiring bikes, there is either congestion (more than required) or starvation (fewer than required) of bikes at base stations. Existing data has shown that congestion/starvation is a common phenomenon that leads to a large number of unsatisfied customers resulting in a significant loss in customer demand. In order to tackle this problem, we propose an optimisation formulation to reposition bikes using vehicles while also considering the routes for vehicles and future expected demand. Furthermore, we contribute two approaches that rely on decomposability in the problem (bike repositioning and vehicle routing) and aggregation of base stations to reduce the computation time significantly. Finally, we demonstrate the utility of our approach by comparing against two benchmark approaches on two real-world data sets of bike sharing systems. These approaches are evaluated using a simulation where the movements of customers are generated from real-world data sets.

Markov Decision Problems, MDPs offer an effective mechanism for planning under uncertainty. However, due to unavoidable uncertainty over models, it is difficult to obtain an exact specification of an MDP. We are interested in solving MDPs, where transition and reward functions are not exactly specified. Existing research has primarily focussed on computing infinite horizon stationary policies when optimizing robustness, regret and percentile based objectives. We focus specifically on finite horizon problems with a special emphasis on objectives that are separable over individual instantiations of model uncertainty (i.e., objectives that can be expressed as a sum over instantiations of model uncertainty): (a) First, we identify two separable objectives for uncertain MDPs: Average Value Maximization (AVM) and Confidence Probability Maximisation (CPM). (b) Second, we provide optimization based solutions to compute policies for uncertain MDPs with such objectives. In particular, we exploit the separability of AVM and CPM objectives by employing Lagrangian dual decomposition(LDD). (c) Finally, we demonstrate the utility of the LDD approach on a benchmark problem from the literature.

Vehicle sharing (ex: bike sharing, car sharing) systems, an attractive alternative of private transportation, are widely adopted in major cities around the world. In vehicle-sharing systems, base stations (ex: docking stations for bikes) are strategically placed throughout a city and each of the base stations contain a pre-determined number of vehicles at the beginning of each day. Due to the stochastic and individualistic movement of customers, there is typically either congestion (more than required) or starvation (fewer than required) of vehicles at certain base stations, which causes a significant loss in demand. We propose to dynamically redeploy idle vehicles using carriers so as to minimize lost demand or alternatively maximize revenue for the vehicle sharing company. To that end, we contribute an optimization formulation to jointly address the redeployment (of vehicles) and routing (of carriers) problems and provide two approaches that rely on decomposability and abstraction of problem domains to reduce the computation time significantly.

In this paper, we solve cooperative decentralized stochastic planning problems, where the interactions between agents (specified using transition and reward functions) are dependent on the number of agents (and not on the identity of the individual agents) involved in the interaction. A collision of robots in a narrow corridor, defender teams coordinating patrol activities to secure a target, etc. are examples of such anonymous interactions. Formally, we consider problems that are a subset of the well known Decentralized MDP (DEC-MDP) model, where the anonymity in interactions is specified within the joint reward and transition functions. In this paper, not only do we introduce a general model model called D-SPAIT to capture anonymity in interactions, but also provide optimization based optimal and local-optimal solutions for generalizable sub-categories of D-SPAIT.

The inventory routing problem (IRP) and the production routing problem (PRP) are two difficult problems arising in the planning of integrated supply chains. These problems are solved in an attempt to jointly optimize production, inventory, distribution, and routing decisions. Although several studies have proposed exact algorithms to solve the single-vehicle problems, the multivehicle aspect is often neglected because of its complexity. We introduce multivehicle PRP and IRP formulations, with and without a vehicle index, to solve the problems under both the maximum level (ML) and order-up-to level (OU) inventory replenishment policies. The vehicle index formulations are further improved using symmetry breaking constraints; the nonvehicle index formulations are strengthened by several cuts. A heuristic based on an adaptive large neighborhood search technique is also developed to determine initial solutions, and branch-and-cut algorithms are proposed to solve the different formulations. The results show that the vehicle index formulations are superior in finding optimal solutions, whereas the nonvehicle index formulations are generally better at providing good lower bounds on larger instances. IRP and PRP instances with up to 35 customers, three periods, and three vehicles can be solved to optimality within two hours for the ML policy. By using parallel computing, the algorithms could solve the instances for the same policy with up to 45 and 50 customers, three periods, and three vehicles for the IRP and PRP, respectively. For the more difficult IRP (PRP) under the OU policy, the algorithms could handle instances with up to 30 customers, three (six) periods, and three vehicles on a single core machine, and up to 45 (35) customers, three (six) periods, and three vehicles on a multicore machine.

In this paper, we seek robust policies for uncertain Markov Decision Processes (MDPs). Most robust optimization approaches for these problems have focussed on the computation of {\em maximin} policies which maximize the value corresponding to the worst realization of the uncertainty. Recent work has proposed {\em minimax} regret as a suitable alternative to the {\em maximin} objective for robust optimization. However, existing algorithms for handling {\em minimax} regret are restricted to models with uncertainty over rewards only. We provide algorithms that employ sampling to improve across multiple dimensions: (a) Handle uncertainties over both transition and reward models; (b) Dependence of model uncertainties across state, action pairs and decision epochs; (c) Scalability and quality bounds. Finally, to demonstrate the empirical effectiveness of our sampling approaches, we provide comparisons against benchmark algorithms on two domains from literature. We also provide a Sample Average Approximation (SAA) analysis to compute a posteriori error bounds.