To date, distributional reinforcement learning (distributional RL) methods have exclusively focused on the discounted setting, where an agent aims to optimize a discounted sum of rewards over time. In this work, we extend distributional RL to the average-reward setting, where an agent aims to optimize the reward received per time step. In particular, we utilize a quantile-based approach to develop the first set of algorithms that can successfully learn and/or optimize the long-run per-step reward distribution, as well as the differential return distribution of an average-reward MDP. We derive proven-convergent tabular algorithms for both prediction and control, as well as a broader family of algorithms that have appealing scaling properties. Empirically, we find that these algorithms yield competitive and sometimes superior performance when compared to their non-distributional equivalents, while also capturing rich information about the long-run per-step reward and differential return distributions.

People with severe speech and motor impairment (SSMI) often require assistive technologies to control their environment, including robots. Current eye-gaze-controlled robotic systems, however, are limited in scope, focusing on specific tasks or requiring structured command sequences. In this work, we introduce Gaze2Instruct (G2I), a novel approach for predicting the intentions of people with SSMI. G2I leverages eye-gaze data and visual input to automatically generate natural language instructions, which can then be interpreted by existing language-conditioned robotics. By translating eye-gaze into versatile language commands, we enable intuitive interaction with assistive robots for individuals with SSMI, allowing for unstructured, real-time task execution without predefined grammar or task-specific solutions, leveraging the power of segmentation models and Multimodal Large Language Models (MLLM). Through a series of experiments, we demonstrate the effectiveness of our system in generating accurate and meaningful instructions, reducing cognitive load, and improving the ease of interaction for users with SSMI.

In the realm of stock prediction, machine learning models encounter considerable obstacles due to the inherent low signal-to-noise ratio and the nonstationary nature of financial markets. These challenges often result in spurious correlations and unstable predictive relationships, leading to poor performance of models when applied to out-of-sample (OOS) domains. To address these issues, we investigateDomain Generalizationtechniques, with a particular focus on causal representation learning to improve a prediction model's generalizability to OOS domains. By leveraging multi-factor models from econometrics, we introduce a novel error bound that explicitly incorporates causal relationships. In addition, we present the connection between the proposed error bound and market nonstationarity. We also develop aCausal Discoverytechnique to discover invariant feature representations, which effectively mitigates the proposed error bound, and the influence of spurious correlations on causal discovery is rigorously examined. Our theoretical findings are substantiated by numerical results, showcasing the effectiveness of our approach in enhancing the generalizability of stock prediction models.

Deep learning models are widely used for time series classification (TSC) due to their scalability and efficiency. However, their performance degrades under challenging data conditions such as class similarity, multimodal distributions, and noise. To address these limitations, we propose CDNet, a Contrastive Diffusion-based Network that enhances existing classifiers by generating informative positive and negative samples via a learned diffusion process. Unlike traditional diffusion models that denoise individual samples, CDNet learns transitions between samples—both within and across classes—through convolutional approximations of reverse diffusion steps. We introduce a theoretically grounded CNN-based mechanism to enable both denoising and mode coverage, and incorporate an uncertainty-weighted composite loss for robust training. Extensive experiments on the UCR Archive and simulated datasets demonstrate that CDNet significantly improves state-of-the-art (SOTA) deep learning classifiers, particularly under noisy, similar, and multimodal conditions.

Fault detection or anomaly detection relies heavily on learning from datasets where only normal samples are available, resulting in the emergence of numerous one-class classification (OCC) methods. However, learning discriminative deep representatives with good generalization from cross-domain positive samples remains challenging. Therefore, this work proposes an end-to-end framework, deep transfer one-class classification (DTOCC) for unsupervised fault detection, which combines adversarial generative OCC and distribution alignment from the perspective of manifold learning. Specifically, pseudo-negative samples are generated outside the positive manifold, facilitating the model to learn discrimination with respect to normal and anomaly. Further, cross-domain positive samples are aligned in log-Euclidean manifold space to enhance representation learning. Then, we provide the specific implementations for fault detection and validate its superiority through case studies on multiclass and run-to-failure datasets, simulating both offline and online scenarios.

Few-shot data collected from hoisting system suffer from inadequate information in the practical industries, which reduces the diagnostic accuracy of the data-driven-based fault diagnosis approaches. To overcome this problem, in this article, a self-supervised metalearning generative adversarial network algorithm is proposed. The purpose of the proposed algorithm is to determine the optimal initialization parameters of the model by training on various data generation tasks, thus accomplishing new data generation using only a small amount of training data. Specifically, a self-supervised strategy is proposed to improve the generalization performance of the proposed algorithm. Besides, the fault data of the hoisting system are collected for data generation, and the experimental results show that the proposed algorithm can determine the optimal initialization parameters under the condition of insufficient datasets. The effectiveness of the proposed algorithm for few-shot fault diagnosis is verified by using a mixture of real data and generated data.

We propose a new transfer learning algorithm between tasks with different dynamics. The proposed algorithm solves an Imitation from Observation problem (IfO) to ground the source environment to the target task before learning an optimal policy in the grounded environment. The learned policy is deployed in the target task without additional training. A particular feature of our algorithm is the employment of multiple rollout policies during training with a goal to ground the environment more globally; hence, it is named as Multi-Policy Grounding (MPG). The quality of final policy is further enhanced via ensemble policy learning. We demonstrate the superiority of the proposed algorithm analytically and numerically. Numerical studies show that the proposed multi-policy approach allows comparable grounding with single policy approach with a fraction of target samples, hence the algorithm is able to maintain the quality of obtained policy even as the number of interactions with the target environment becomes extremely small.

Recent studies on few-shot classification using transfer learning pose challenges to the effectiveness and efficiency of episodic meta-learning algorithms. Transfer learning approaches are a natural alternative, but they are restricted to few-shot classification. Moreover, little attention has been on the development of probabilistic models with well-calibrated uncertainty from few-shot samples, except for some Bayesian episodic learning algorithms. To tackle the aforementioned issues, we propose a new transfer learning method to obtain accurate and reliable models for few-shot regression and classification. The resulting method does not require episodic meta-learning and is called meta-free representation learning (MFRL). MFRL first finds low-rank representation generalizing well on meta-test tasks. Given the learned representation, probabilistic linear models are fine-tuned with few-shot samples to obtain models with well-calibrated uncertainty. The proposed method not only achieves the highest accuracy on a wide range of few-shot learning benchmark datasets but also correctly quantifies the prediction uncertainty. In addition, weight averaging and temperature scaling are effective in improving the accuracy and reliability of few-shot learning in existing meta-learning algorithms with a wide range of learning paradigms and model architectures.

Recently, there has been a significant growth in the development of rechargeable battery-powered devices such as electric vehicles, leading to an urgent need for reliable and safe batteries. The remaining useful life (RUL) is a critical health indicator of battery, which is defined as the remaining number of charge and recharge cycles before the state-of-health falls below a user-specified threshold under certain operating settings. Substantially, the RUL can be estimated by adaptive stochastic processes or advanced machine learning techniques. However, the existing approaches either assume over-simplified degradation pattern in accordance with physics laws leading to poor generalizability or act as a black box offering no interpretation. To address these limitations, in this article, we develop a pattern-driven degradation process by integrating a recursive Gaussian distribution with its mean learnt from a gated recurrent unit (GRU) driven degradation pattern to capture degradation fluctuation into the model. Due to the non-Markovian state transitions, a joint-learning sampling-based expectation maximization algorithm was developed to estimate model parameters based on historical observations. Finally, numerical studies using real battery data showed that the proposed method achieves over 3% and 40% higher accuracy in RUL prediction than the GRU and adaptive Wiener process, respectively.

Learning from Demonstrations (LfD) is a powerful approach for incorporating advice from experts in the form of demonstrations. However, demonstrations often come from multiple sub-optimal experts with conflicting goals, rendering them difficult to incorporate effectively in online settings. To address this, we formulate a quadratic program whose solution yields an adaptive weighting over experts, that can be used to sample experts with relevant goals. In order to compare different source and target task goals safely, we model their uncertainty using normal-inverse-gamma priors, whose posteriors are learned from demonstrations using Bayesian neural networks with a shared encoder. Our resulting approach, which we call Bayesian Experience Reuse, can be applied for LfD in static and dynamic decision-making settings. We demonstrate its effectiveness for minimizing multi-modal functions, and optimizing a high-dimensional supply chain with cost uncertainty, where it is also shown to improve upon the performance of the demonstrators' policies.

We focus on multiple experts performing a task in a Markov decision process (MDP) environment. A probabilistic assignment of trajectories to clusters and a mathematical framework which leverages the utility function are employed to jointly estimate the discount factor and reward. We treat the number of clusters as a hyperparameter which can be "freely" selected by the problem designer. In this work, we specifically treat the cluster of trajectories as a latent variable in the adapted maximum entropy inverse reinforcement learning (IRL) formulation; the introduction of this latent variable adds to the complexity of the IRL problem. To manage such complexity, we optimize a marginal log-likelihood function via Expectation Maximization. To test our approach, we have utilized behavioral data generated from three MDP environments. Experimental works show that our approach is promising towards the estimation of discount factors in IRL for non-interacting multiple experts.

Sample efficiency and risk-awareness are central to the development of practical reinforcement learning (RL) for complex decision-making. The former can be addressed by transfer learning, while the latter by optimizing some utility function of the return. However, the problem of transferring skills in a risk-aware manner is not well-understood. In this paper, we address the problem of transferring policies between tasks in a common domain that differ only in their reward functions, in which risk is measured by the variance of reward streams. Our approach begins by extending the idea of generalized policy improvement to maximize entropic utilities, thus extending the dynamic programming's policy improvement operation to sets of policies \emph{and} levels of risk-aversion. Next, we extend the idea of successor features (SF), a value function representation that decouples the environment dynamics from the rewards, to capture the variance of returns. Our resulting risk-aware successor features (RaSF) integrate seamlessly within the RL framework, inherit the superior task generalization ability of SFs, while incorporating risk into the decision-making. Experiments on a discrete navigation domain and control of a simulated robotic arm demonstrate the ability of RaSFs to outperform alternative methods including SFs, when taking the risk of the learned policies into account.

In reinforcement learning, agents that consider the context or current state when transferring source policies have been shown to outperform context-free approaches. However, existing approaches suffer from limitations, including sensitivity to sparse or delayed rewards and estimation errors in values. One important insight is that explicit learned models of the source dynamics, when available, could benefit contextual transfer in such settings. In this paper, we assume a family of tasks with shared sub-goals but different dynamics, and availability of estimated dynamics and policies for source tasks. To deal with possible estimation errors in dynamics, we introduce a novel Bayesian mixture-of-experts for learning state-dependent beliefs over source task dynamics that match the target dynamics using state transitions collected from the target task. The mixture is easy to interpret, is robust to estimation errors in dynamics, and is compatible with most RL algorithms. We incorporate it into standard policy reuse frameworks and demonstrate its effectiveness on benchmarks from OpenAI gym.

Potential based reward shaping is a powerful technique for accelerating convergence of reinforcement learning algorithms. Typically, such information includes an estimate of the optimal value function and is often provided by a human expert or other sources of domain knowledge. However, this information is often biased or inaccurate and can mislead many reinforcement learning algorithms. In this paper, we apply Bayesian Model Combination with multiple experts in a way that learns to trust a good combination of experts as training progresses. This approach is both computationally efficient and general, and is shown numerically to improve convergence across discrete and continuous domains and different reinforcement learning algorithms.