Multi-objective reinforcement learning (MORL) has shown great promise in many real-world applications. Existing MORL algorithms often aim to learn a policy that optimizes individual objective functions simultaneously with a given prior preference (or weights) on different objectives. However, these methods often suffer from the issue ofgradient conflictsuch that the objectives with larger gradients dominate the update direction, resulting in a performance degeneration on other objectives. In this paper, we develop a novel dynamic weighting multi-objective actor-critic algorithm (MOAC) under two options of sub-procedures named as conflict-avoidant (CA) and faster convergence (FC) in objective weight updates. MOAC-CA aims to find a CA update direction that maximizes the minimum value improvement among objectives, and MOAC-FC targets at a much faster convergence rate. We provide a comprehensive finite-time convergence analysis for both algorithms. We show that MOAC-CA can find a ϵ + ϵapp-accurate Pareto stationary policy using O(ϵ−5) samples, while ensuring a small ϵ+ √ϵapp-level CA distance (defined as the distance to the CA direction), where ϵappis the function approximation error. The analysis also shows that MOAC-FC improves the sample complexity to O(ϵ−3), but with a constant-level CA distance. Our experiments on MT10 demonstrate the improved performance of our algorithms over existing MORL methods with fixed preference.

Actor-critic (AC) is a powerful method for learning an optimal policy in reinforcement learning, where the critic uses algorithms, e.g., temporal difference (TD) learning with function approximation, to evaluate the current policy and the actor updates the policy along an approximate gradient direction using information from the critic. This paper provides the *tightest* non-asymptotic convergence bounds for both the AC and natural AC (NAC) algorithms. Specifically, existing studies show that AC converges to an $\epsilon+\varepsilon_{\text{critic}}$ neighborhood of stationary points with the best known sample complexity of $\mathcal{O}(\epsilon^{-2})$ (up to a log factor), and NAC converges to an $\epsilon+\varepsilon_{\text{critic}}+\sqrt{\varepsilon_{\text{actor}}}$ neighborhood of the global optimum with the best known sample complexity of $\mathcal{O}(\epsilon^{-3})$, where $\varepsilon_{\text{critic}}$ is the approximation error of the critic and $\varepsilon_{\text{actor}}$ is the approximation error induced by the insufficient expressive power of the parameterized policy class. This paper analyzes the convergence of both AC and NAC algorithms with compatible function approximation. Our analysis eliminates the term $\varepsilon_{\text{critic}}$ from the error bounds while still achieving the best known sample complexities. Moreover, we focus on the challenging single-loop setting with a single Markovian sample trajectory. Our major technical novelty lies in analyzing the stochastic bias due to policy-dependent and time-varying compatible function approximation in the critic, and handling the non-ergodicity of the MDP due to the single Markovian sample trajectory. Numerical results are also provided in the appendix.

Distributionally Robust Reinforcement Learning (DR-RL) aims to derive a policy optimizing the worst-case performance within a predefined uncertainty set. Despite extensive research, previous DR-RL algorithms have predominantly favored model-based approaches, with limited availability of model-free methods offering convergence guarantees or sample complexities. This paper proposes a model-free DR-RL algorithm leveraging the Multi-level Monte Carlo (MLMC) technique to close such a gap. Our innovative approach integrates a threshold mechanism that ensures finite sample requirements for algorithmic implementation, a significant departure from previous model-free algorithms. We adapt our algorithm to accommodate uncertainty sets defined by total variation, Chi-square divergence, and KL divergence, and provide finite sample analyses under all three cases. Remarkably, our algorithms represent the first model-free DR-RL approach featuring finite sample complexity for total variation and Chi-square divergence uncertainty sets, while also offering an improved sample complexity and broader applicability compared to existing model-free DR-RL algorithms for the KL divergence model. The complexities of our method establish the tightest results for all three uncertainty models in model-free DR-RL, underscoring the effectiveness and efficiency of our algorithm, and highlighting its potential for practical applications.

Message passing algorithm (MPA) scheme is employed to detect multiuser signals for a sparse code multiple access (SCMA) system. However, the huge complexity of the original MPA scheme makes it hard to be implemented in practice. In this paper, we propose an efficient low complexity detection scheme based on the Gaussian approximation. The original discrete messages on the factor graph are approximated as continuous Gaussian messages, which can avoid the complex marginalization operation of the original MPA scheme, such that the total complexity is reduced to scaling linearly with the codebook size. Besides simulation results demonstrate that our proposed scheme achieves almost the same bit error ratio (BER) performance with the original MPA scheme.

The daily precipitation datasets of the Qinghai-Tibetan plateau (QTP) are mainly assimilated from remote sensing products and in-situ observations. The accuracy of those datasets needs further improvement with environmental and meteorological factors. This paper selected the related environmental and meteorological factors as input; k-Nearest Neighbor (KNN), Multivariate Adaptive Regression Splines (MARS), Support Vector Machine (SVM), Multinomial Log-linear Models (MLM) and Artificial Neural Networks (ANN) as correction models; 112 upscaled daily precipitation observations from the standard meteorological stations as ground truth to correct the commonly used ITPCAS and CMORPH daily precipitation of the QTP. Results show that the KNN model has the highest correction accuracy. The distribution of the corrected ITPCAS precipitation is nearer to the spatial pattern of the precipitation over the QTP than the corrected CMORPH precipitation. The correction accuracy is influenced by the precipitation distribution pattern of the original dataset.

As the meteorological stations in the Qinghai-Tibetan plateau (QTP) are scarcely and unevenly distributed, daily precipitation datasets generated from observation data and remote sensing inversion models are not accurate. The data accuracy can be improved by environmental and meteorological factors. This study selected k-Nearest Neighbor (KNN), a machine learning model, to correct the commonly used ITPCAS precipitation data over the QTP by establishing the relationship between daily precipitation and environmental (elevation, slope, aspect, vegetation) as well as meteorological factors (air temperature, humidity, wind speed). Error analysis shows that the KNN-corrected ITPCAS precipitation is more accurate than the original one. The spatial distribution of the corrected ITPCAS precipitation agrees well with the precipitation distribution pattern of the QTP. The error distribution of the corrected ITPCAS precipitation shows significant seasonal and regional characteristics.

Sparse code multiple access (SCMA), as a promising non-orthogonal multiple access approach, can support massive access devices communications for cellular the Internet of Things (IoT). This paper focuses on the research of low complexity detection scheme for SCMA system. Although the original message passing algorithm (MPA) detector achieves nearly optimal bit error ratio (BER) performance for SCMA system, the high computation complexity makes it hard to implement in practice. To handle this problem, we propose a low complexity detection scheme based on the edge selection. The message passing from function nodes (FNs) to variable nodes (VNs) on factor graph model is updated only with partial selected edges, furthermore the information loss brought by the edge selection is compensated via the mean feedback. Numerical results demonstrate that the proposed scheme can achieve an affordable trade-off between BER performance and complexity for SCMA system.