Reinforcement learning (RL) has emerged as a key approach for training agents in complex and uncertain environments. Incorporating statistical inference in RL algorithms is essential for understanding and managing uncertainty in model performance. This paper introduces a generalized framework for time-varying batch-averaged Q-learning, termed sample-averaged Q-learning (SA-QL), which extends traditional single-sample Q-learning by aggregating samples of rewards and next states to better account for data variability and uncertainty. We leverage the functional central limit theorem (FCLT) to establish a novel framework that provides insights into the asymptotic normality of the sample-averaged algorithm under mild conditions. Additionally, we develop a random scaling method for interval estimation, enabling the construction of confidence intervals without requiring extra hyperparameters. Extensive numerical experiments across classic stochastic OpenAI Gym environments, including windy gridworld and slippery frozenlake, demonstrate how different batch scheduling strategies affect learning efficiency, coverage rates, and confidence interval widths. This work establishes a unified theoretical foundation for sample-averaged Q-learning, providing insights into effective batch scheduling and statistical inference for RL algorithms.

In the past decade, the sensor and surveillance technology have been widely used in condition monitoring. The hidden states of systems can be inferred from collected sensor data. However, the maintenance decision problem becomes more challenging when ordering decision of the spare parts must be considered jointly. In this article, we consider a multistate deteriorating system whose states are hidden but partially observable, and determine the optimal maintenance and spare parts inventory ordering policy. We use the partially observable Markov decision process to model the problem of interest and adopt the state-of-the-art heuristic search value iteration algorithm to solve the optimization problem. The proposed policy is illustrated and compared with$( {{\bm{s}},{\bm{S}}} )$inventory policy through a series of numerical examples. The numerical results indicate that our proposed policy is cost effective. Further, the model of multicomponent system is formulated, highlighting the adaptability of our framework. This research shows that considering the system component conditions and spare parts ordering jointly can result a lower operation and maintenance cost.

Cardiac conduction disease is a major cause of morbidity and mortality worldwide. There is considerable clinical significance and an emerging need of early detection of these diseases for preventive treatment success before more severe arrhythmias occur. However, developing such early screening tools is challenging due to the lack of early electrocardiograms (ECGs) before symptoms occur in patients. Mouse models are widely used in cardiac arrhythmia research. The goal of this paper is to develop deep learning models to predict cardiac conduction diseases in mice using their early ECGs. We hypothesize that mutant mice present subtle abnormalities in their early ECGs before severe arrhythmias present. These subtle patterns can be detected by deep learning though they are hard to be identified by human eyes. We propose a deep transfer learning model, DeepMiceTL, which leverages knowledge from human ECGs to learn mouse ECG patterns. We further apply the Bayesian optimization and $k$-fold cross validation methods to tune the hyperparameters of the DeepMiceTL. Our results show that DeepMiceTL achieves a promising performance (F1-score: 83.8%, accuracy: 84.8%) in predicting the occurrence of cardiac conduction diseases using early mouse ECGs. This study is among the first efforts that use state-of-the-art deep transfer learning to identify ECG patterns during the early course of cardiac conduction disease in mice. Our approach not only could help in cardiac conduction disease research in mice, but also suggest a feasibility for early clinical diagnosis of human cardiac conduction diseases and other types of cardiac arrythmias using deep transfer learning in the future.

Maintenance optimization has been extensively studied in the past decades. However, most of the existing maintenance models focus on single-component systems and are not applicable to complex systems consisting of multiple components, due to various interactions among the components. The multicomponent maintenance optimization problem, which joins the stochastic processes regarding the failures of components with the combinatorial problems regarding the grouping of maintenance activities, is challenging in both modeling and solution techniques, and has remained an open issue in the literature. In this paper, we study the multicomponent maintenance problem over a finite planning horizon and formulate the problem as a multistage stochastic integer program with decision-dependent uncertainty. There is a lack of general efficient methods to solve this type of problem. To address this challenge, we use an alternative approach to model the underlying failure process and develop a novel two-stage model without decision-dependent uncertainty. Structural properties of the two-stage problem are investigated, and a progressive-hedging-based heuristic is developed based on the structural properties. Our heuristic algorithm demonstrates a significantly improved capacity to handle large-size two-stage problems comparing to three conventional methods for stochastic integer programming, and solving the two-stage model by our heuristic in a rolling horizon provides a good approximation of the multistage problem. The heuristic is further benchmarked with a dynamic programming approach and a structural policy, which are two commonly adopted approaches in the literature. Numerical results show that our heuristic can lead to significant cost savings compared with the benchmark approaches.

Preventive maintenance has received considerable attention in industries and the literature. Conventional preventive maintenance models often assume that equipment is always available for maintenance activities. However, in many mission-critical industries, equipment may not be available for scheduled maintenance due to busy operational schedules. Forced shutdown of the equipment may incur extra costs that cannot be offset by the benefits from preventively maintaining the equipment. In this paper, we propose innovative preventive maintenance policies to address the challenges caused by equipment unavailability. Maintenance models with possible rescheduling are developed for both time-based and condition-based maintenance policies, and the objective is to minimize the long-run cost rate of all maintenance activities. The proposed policies, with consideration of equipment unavailability for prescheduled PM, are compared with the policies that ignore this unavailability. Numerical examples are provided to illustrate the proposed policies.