Deep reinforcement learning (DRL) has achieved remarkable success in solving sequential decision-making problems across various domains. However, a critical challenge is sample inefficiency, especially in real-world environments with high-dimensional solution spaces due to continuous state and action spaces. Although off-policy actor-critic algorithms have been proposed to mitigate this issue, the gains in sample efficiency remain limited, as decision-making in these algorithms relies solely on the policy function that might not always yield optimal actions. To bridge the gap, we design a novel feedback-based decision-making mechanism (FADA) that incorporates a feedback mechanism into the actor-critic framework to enhance decision-making robustness. Specifically, FADA utilizes feedback from the value function (critic) to calibrate the decisions produced by the policy function (actor). More concretely, FADA comprises four integrated modules: a decision-space expansion module (DEM) to produce a pool of candidate actions, a critic-guided evaluation module (CGEM) that estimates the efficacy of the candidate actions, an adaptive selection module (ASM) that adaptively selects a set of elite actions based on the estimated efficacy and samples the final action, and an iterative refinement module (IRM) that improves the quality of elite actions. We evaluate our approach on multiple tasks in the DeepMind Control Suite, and the experimental results demonstrate a significant improvement in sample efficiency.

In model-based reinforcement learning (MBRL), the quality of simulated experiences is a critical bottleneck to effective policy learning. Existing research has primarily focused on reducing the generation errors of these simulated experiences but has largely ignored how the varying quality of these experiences impacts policy learning during their exploitation. To bridge this gap, we propose a novel quality-aware experience exploitation scheme, called QA2E, which dynamically exploits simulated experiences based on their assessed quality to enhance the effectiveness of model-based policy learning. Particularly, we develop a weighted Bellman backup approach to dynamically adjust the influence of simulated experiences on policy learning based on their assessed quality. Since directly measuring the quality is impractical, QA2E estimates it through the epistemic uncertainty derived from the prediction results of an ensemble of transition models. Experimental results demonstrate that QA2E significantly improves policy learning performance by more effectively exploiting simulated experiences.

With the onset of the big data era, designing efficient and effective machine learning algorithms to analyze large-scale data is in dire need. In practice, data is typically generated by multiple parties and stored in a geographically distributed manner, which spurs the study of distributed machine learning. Traditional master-worker type of distributed machine learning algorithms assumes a trusted central server and focuses on the privacy issue in linear learning models, while privacy in nonlinear learning models and security issues are not well studied. To address these issues, in this paper, we explore the blockchain technique to propose a decentralized privacy-preserving and secure machine learning system, called LearningChain, by considering a general (linear or nonlinear) learning model and without a trusted central server. Specifically, we design a decentralized Stochastic Gradient Descent (SGD) algorithm to learn a general predictive model over the blockchain. In decentralized SGD, we develop differential privacy based schemes to protect each party’s data privacy, and propose an l-nearest aggregation algorithm to protect the system from potential Byzantine attacks. We also conduct theoretical analysis on the privacy and security of the proposed LearningChain. Finally, we implement LearningChain on Etheurum and demonstrate its efficiency and effectiveness through extensive experiments.