This paper presents a novel approach to addressing the control challenges of underactuated systems, focusing on the swing-up and stabilisation tasks on the double pendulum system. We propose the Average-Reward Entropy Advantage Policy Optimisation (AR-EAPO), a model-free reinforcement learning (RL) algorithm that integrates the strengths of the average-reward RL and the maximum entropy RL (MaxEnt RL). The average reward criterion allows the use of a simple reward function by naturally promoting the longterm goals, at the same time MaxEnt RL encourages the robustness of the policy. We validate our approach through simulations, consistently outperforming standard RL baselines and traditional control methods. Also, we provide preliminary test results on real double pendulum hardware. Additional experiments on MuJoCo environments further demonstrate AR-EAPO's efficacy on general continuous control tasks. This work underscores the potential of the average-reward criterion in simplifying control design while achieving superior results.

Entropy Regularisation is a widely adopted technique that enhances policy optimisation performance and stability. Maximum entropy reinforcement learning (MaxEnt RL) regularises policy evaluation by augmenting the objective with an entropy term, showing theoretical benefits in policy optimisation. However, its practical application in straightforward direct policy gradient settings remains surprisingly underexplored. We hypothesise that this is due to the difficulty of managing the entropy reward in practice. This paper proposes Entropy Advantage Policy Optimisation (EAPO), a simple method that facilitates MaxEnt RL implementation by separately estimating task and entropy objectives. Our empirical evaluations demonstrate that extending Proximal Policy Optimisation (PPO) and Trust Region Policy Optimisation (TRPO) within the MaxEnt framework improves optimisation performance, generalisation, and exploration in various environments. Moreover, our method provides a stable and performant MaxEnt RL algorithm for discrete action spaces.

Hearthstone is a popular online collectible card game (CCG). Hearthstone imposes interesting challenges in developing a search algorithm for the game AI. As a CCG, it has a considerable amount of hidden information from each player’s private hand and deck. Moreover, the action space is full of stochastic actions compared to other similar games. That is, instead of a single move, each player is allowed to build a move sequence via various combinations of atomic actions. Therefore, when applying any heuristic search algorithm, the branching factor of the search space is extremely large. In this paper, we explore the use of Monte Carlo Tree Search (MCTS) with approaches to reduce the complexity of the search space and decide on the best strategy. First, we utilise state abstraction to present the search space as a Directed Acyclic Graph (DAG) and introduce a variant of Upper Confidence Bound for Trees (UCT) algorithm for the DAG. Next, we apply the sparse sampling algorithm to handle imperfect information and randomness and reduce the stochastic branching factor. This paper presents empirical evaluations of the proposed framework for Hearthstone and the experimental results suggest that our approach is well suited for developing a better AI agent.