In multi-floor fabrication facilities (fabs), lifters’ efficient transportation of resources is crucial for the overall productivity of the modern semiconductor industry. Unfortunately, most of the existing methods for controlling fab lifters have difficulty obtaining exact system models and applying traditional numerical schemes. In this paper, we propose two off-policy deep reinforcement learning algorithms that can learn to efficiently control the lifters in complex multi-floor fab environments with limited prior knowledge. The proposed algorithms exploit a novel semi-Markov decision process (semi-MDP) model and resolve several challenges that arise from the complex structures of multi-floor fabs to achieve highly efficient transportation. Extensive empirical analyses confirm that the controllers trained with our method automatically learn the intricate structure of the problem and effectively react to the real-time information flow of the multi-floor fab without referring to any domain knowledge. We empirically show that the proposed methods outperform advanced techniques including model predictive control and proximal policy optimization in terms of the computation time and transportation efficiency in various simulated fab environments.Note to Practitioners—In terms of implementation, our method is readily integrated into any fab that tracks real-time operational data, which is a common requirement in modern fabs. The method is particularly well-suited for facilities where the manufacturing processes are consistent over time. The crucial aspect of the lifter control problem is that the arrival of lots is independent of the lifter’s behavior. This makes it possible to collect data from real-world fabs and build multiple scenarios for training the reinforcement learning controllers. Furthermore, the robustness of the proposed method to small perturbations in the scenarios makes it applicable even when the frequencies of the lot arrivals change slightly. However, if there is a significant shift in lot arrival frequencies from the training data, the method may not perform as expected and may need to be extended using an advanced technique such as meta-reinforcement learning.

Although Nesterov’s accelerated gradient method (AGM) has been studied from various perspectives, it remains unclear why the most popular forms of AGMs must handle convex and strongly convex objective functions separately. To address this inconsistency, we propose a novel unified framework for Lagrangians, ordinary differential equation (ODE) models, and algorithms. As a special case, our new simple momentum algorithm, which we call the unified AGM, seamlessly bridges the gap between the two most popular forms of Nesterov’s AGM and has a superior convergence guarantee compared to existing algorithms for non-strongly convex objective functions. This property is beneficial in practice when considering ill-conditioned $\mu$-strongly convex objective functions (with small $\mu$). Furthermore, we generalize this algorithm and the corresponding ODE model to the higher-order non-Euclidean setting. Last but not least, our unified framework is used to construct the unified AGM-G ODE, a novel ODE model for minimizing the gradient norm of strongly convex functions.

Safety is one of the main challenges when applying learning-based motion controllers to practical robotic systems, especially when the dynamics of the robots and their surrounding dynamic environments are unknown. This issue is further exacerbated when the learned information is unreliable and inaccurate. In this paper, we aim to enhance the safety of learning-enabled mobile robots in dynamic environments from the perspective of distributionally robust optimization (DRO) and the unscented transform (UT). Our method infers the unknown dynamics of both the robot and the environment by adopting Gaussian process regression with an uncertainty propagation scheme based on UT to improve prediction accuracy. This leads to a novel learning-based model predictive control (MPC) method in which state information about both the robot and the environment is propagated via UT. The proposed method uses DRO to proactively limit the risk of collisions or other unsafe events in the presence of learning errors. However, the distributionally robust risk constraint is intractable because it involves a separate infinite-dimensional optimization problem. To overcome this challenge, we exploit UT with modern DRO techniques to replace the risk constraint with its simple upper bound. The performance and the utility of our method are demonstrated through simulations in autonomous driving scenarios, showing its capability to enhance safety and computational efficiency.

We propose a novel methodology that systematically analyzes ordinary differential equation (ODE) models for first-order optimization methods by converting the task of proving convergence rates into verifying the positive semidefiniteness of specific Hilbert-Schmidt integral operators. Our approach is based on the performance estimation problems (PEP) introduced by Drori and Teboulle. Unlike previous works on PEP, which rely on finite-dimensional linear algebra, we use tools from functional analysis. Using the proposed method, we establish convergence rates of various accelerated gradient flow models, some of which are new. As an immediate consequence of our framework, we show a correspondence between minimizing function values and minimizing gradient norms.

In this article, we propose a novel safety specification tool, called thedistributionally robust risk map(DR-risk map), for a mobile robot operating in a learning-enabled environment. Given the robot's position, the map aims to reliably assess the conditional value-at-risk (CVaR) of collision with obstacles whose movements are inferred by Gaussian process regression (GPR). Unfortunately, the inferred distribution is subject to errors, making it difficult to accurately evaluate the CVaR of collision. To overcome this challenge, our tool measures the risk under the worst-case distribution in a so-calledambiguity setthat characterizes allowable distribution errors. To resolve the infinite-dimensionality issue inherent in the construction of the DR-risk map, we derive a tractable semidefinite programming formulation that provides an upper bound of the risk, exploiting techniques from modern distributionally robust optimization. As a concrete application for motion planning, a distributionally robust RRT* algorithm is considered using the risk map that addresses distribution errors caused by GPR. Furthermore, a motion control method is devised using the DR-risk map in a learning-based model predictive control (MPC) formulation. In particular, a neural network approximation of the risk map is proposed to reduce the computational cost in solving the MPC problem. The performance and utility of the proposed risk map are demonstrated through simulation studies that show its ability to ensure the safety of mobile robots despite learning errors.

We propose computationally tractable accelerated first-order methods for Riemannian optimization, extending the Nesterov accelerated gradient (NAG) method. For both geodesically convex and geodesically strongly convex objective functions, our algorithms are shown to have the same iteration complexities as those for the NAG method on Euclidean spaces, under only standard assumptions. To the best of our knowledge, the proposed scheme is the first fully accelerated method for geodesically convex optimization problems. Our convergence analysis makes use of novel metric distortion lemmas as well as carefully designed potential functions. A connection with the continuous-time dynamics for modeling Riemannian acceleration in (Alimisis et al., 2020) is also identified by letting the stepsize tend to zero. We validate our theoretical results through numerical experiments.

In online learning problems, exploiting low variance plays an important role in obtaining tight performance guarantees yet is challenging because variances are often not known a priori. Recently, considerable progress has been made by Zhang et al. (2021) where they obtain a variance-adaptive regret bound for linear bandits without knowledge of the variances and a horizon-free regret bound for linear mixture Markov decision processes (MDPs). In this paper, we present novel analyses that improve their regret bounds significantly. For linear bandits, we achieve $\tilde O(\min\{d\sqrt{K}, d^{1.5}\sqrt{\sum_{k=1}^K \sigma_k^2}\} + d^2)$ where $d$ is the dimension of the features, $K$ is the time horizon, and $\sigma_k^2$ is the noise variance at time step $k$, and $\tilde O$ ignores polylogarithmic dependence, which is a factor of $d^3$ improvement. For linear mixture MDPs with the assumption of maximum cumulative reward in an episode being in $[0,1]$, we achieve a horizon-free regret bound of $\tilde O(d \sqrt{K} + d^2)$ where $d$ is the number of base models and $K$ is the number of episodes. This is a factor of $d^{3.5}$ improvement in the leading term and $d^7$ in the lower order term. Our analysis critically relies on a novel peeling-based regret analysis that leverages the elliptical potential `count' lemma.

In this article, a risk-aware motion control scheme is considered for mobile robots to avoid randomly moving obstacles when the true probability distribution of uncertainty is unknown. We propose a novel model-predictive control (MPC) method for limiting the risk of unsafety even when the true distribution of the obstacles’ movements deviates, within anambiguity set, from the empirical distribution obtained using a limited amount of sample data. By choosing the ambiguity set as a statistical ball with its radius measured by theWasserstein metric, we achieve a probabilistic guarantee of theout-of-sample risk, evaluated using new sample data generated independently of the training data. To resolve the infinite-dimensionality issue inherent in the distributionally robust MPC problem, we reformulate it as a finite-dimensional nonlinear program using modern distributionally robust optimization techniques based on the Kantorovich duality principle. To find a globally optimal solution in the case of affine dynamics and output equations, a spatial branch-and-bound algorithm is designed using McCormick relaxation. The performance of the proposed method is demonstrated and analyzed through simulation studies using nonlinear dynamic and kinematic vehicle models and a linearized quadrotor model. The simulation results indicate that, even when the sample size is small, the proposed method can successfully avoid randomly moving obstacles with a guarantee of out-of-sample risk, while its sample average approximation counterpart fails to do so.

In this paper, we propose Q-learning algorithms for continuous-time deterministic optimal control problems with Lipschitz continuous controls. A new class of Hamilton-Jacobi-Bellman (HJB) equations is derived from applying the dynamic programming principle to continuous-time Q-functions. Our method is based on a novel semi-discrete version of the HJB equation, which is proposed to design a Q-learning algorithm that uses data collected in discrete time without discretizing or approximating the system dynamics. We identify the conditions under which the Q-function estimated by this algorithm converges to the optimal Q-function. For practical implementation, we propose the Hamilton-Jacobi DQN, which extends the idea of deep Q-networks (DQN) to our continuous control setting. This approach does not require actor networks or numerical solutions to optimization problems for greedy actions since the HJB equation provides a simple characterization of optimal controls via ordinary differential equations. We empirically demonstrate the performance of our method through benchmark tasks and high-dimensional linear-quadratic problems.

In this paper, we propose an optimization-based decision-making tool for safe motion planning and control in an environment with randomly moving obstacles. The unique feature of the proposed method is that it limits the risk of unsafety by a pre-specified threshold even when the true probability distribution of the obstacles' movements deviates, within a Wasserstein ball, from an available empirical distribution. Another advantage is that it provides a probabilistic out-of-sample performance guarantee of the risk constraint. To develop a computationally tractable method for solving the distributionally robust model predictive control problem, we propose a set of reformulation procedures using (i) the Kantorovich duality principle, (ii) the extremal representation of conditional value-at-risk, and (iii) a geometric expression of the distance to the union of halfspaces. The performance and utility of this distributionally robust method are demonstrated through simulations using a 12D quadrotor model in a 3D environment.

Safety is a critical issue in learning-based robotic and autonomous systems as learned information about their environments is often unreliable and inaccurate. In this paper, we propose a risk-aware motion control tool that is robust against errors in learned distributional information about obstacles moving with unknown dynamics. The salient feature of our model predictive control (MPC) method is its capability of limiting the risk of unsafety even when the true distribution deviates from the distribution estimated by Gaussian process (GP) regression, within an ambiguity set. Unfortunately, the distributionally robust MPC problem with GP is intractable because the worst-case risk constraint involves an infinite-dimensional optimization problem over the ambiguity set. To remove the infinite-dimensionality issue, we develop a systematic reformulation approach exploiting modern distributionally robust optimization techniques. The performance and utility of our method are demonstrated through simulations using a nonlinear car-like vehicle model for autonomous driving.

We present a numerical method for computing backward reachable sets in differential games. A backward reachable set for time t is captured by the t sublevel set of the lower value function of the game, which coincides with the viscosity solution of a stationary Hamilton-Jacobi-Isaacs (HJI) equation. We solve the stationary HJI equation in a computationally efficient way that does not involve any numerical integration over time, which would otherwise be required for time-dependent HJI equations. Backward reachable sets for all time points can simultaneously be extracted from the solution. The performance of the method is demonstrated by investigating the growth of multicellular structures of non-malignant and malignant breast cells as a proof of principle.