Abstract The RoboCup 3D soccer simulation league serves as a competitive platform for showcasing innovation in autonomous humanoid robot agents through simulated soccer matches. Our team, FC Portugal, developed a new codebase from scratch in Python after RoboCup 2021. The team’s performance relies on a set of skills centered around novel unifying primitives and a custom, symmetry-extended version of the proximal policy optimization algorithm. Our methods have been thoroughly tested in official RoboCup matches, where FC Portugal has won the last two main competitions, in 2022 and 2023. This paper presents our training framework, as well as a timeline of skills developed using our skill-set-primitives, which considerably improve the sample efficiency and stability of skills, and motivate seamless transitions. We start with a significantly fast Sprint-Kick developed in 2021 and progress to the most recent skill set, including a multi-purpose omnidirectional walk, a dribble with unprecedented ball control, a solid kick, and a push skill. The push addresses low-level collision scenarios and high-level strategies to increase ball possession. We address the resource-intensive nature of this task through an innovative multi-agent learning approach. Finally, we release the team’s codebase to the RoboCup community, providing other teams with a robust and modern foundation upon which they can build new features.

Path Planning methods for the autonomous control of Unmanned Aerial Vehicle (UAV) swarms are on the rise due to the numerous advantages they bring. There are increasingly more scenarios where autonomous control of multiple UAVs is required. Most of these scenarios involve a large number of obstacles, such as power lines or trees. Despite these challenges, there are also several advantages; if all UAVs can operate autonomously, personnel expenses can be reduced. Additionally, if their flight paths are optimized, energy consumption is reduced, leaving more battery time for other operations. In this paper, a Reinforcement Learning-based system is proposed to solve this problem in environments with obstacles by utilizing Q-Learning. This method allows a model, in this case, an Artificial Neural Network, to self-adjust by learning from its mistakes and successes. Regardless of the map’s size or the number of UAVs in the swarm, the goal of these paths is to ensure complete coverage of an area with fixed obstacles for tasks like field prospecting. Setting goals or having any prior information apart from the provided map is not required. During the experimentation phase, five maps of varying sizes were used, each with different obstacles and a varying number of UAVs. To evaluate the quality of the results, the number of actions taken by each UAV to complete the task in each experiment was considered. The results indicate that the system achieves solutions with fewer movements as the number of UAVs increases. An increasing number of UAVs on a map lead to solutions in fewer moves. The results have been compared, and a statistical significance analysis has been conducted on the proposed model’s outcomes, demonstrating its capabilities. Thus, it is shown that a two-layer Artificial Neural Network used to implement a Q-Learning algorithm is sufficient to operate on maps with obstacles.

Meta-game balance is a crucial task in game development, and automation of this process could assist game developers by vastly reducing time costs. We explore and evaluate a meta-game balance model over the recently proposed VGC AI Competition Framework. We propose an adversarial model where team builder agents try to maximize their win rate by narrowing to the most optimal team configurations, resulting in a reduction of the diversity of Pokémon employed, while a balancing agent re-adapts the Pokémon inner attributes to incentive the team builder agents to incorporate a greater variety of Pokémon into their teams increasing the meta-game's overall diversity and balance. one Furthermore, we developed multiple team builder agents divided into two groups: the first group assumes that individual Pokémon advantages are the primary factor to determine the outcome of game matches; the second group also exploits the implicit synergy between teammates. These agents make use of meta-gaming, linear optimization, and evolutionary search to find strong combinations against the current meta-game. The strongest team builder is faced against the team meta-game balance agent for its evaluation. Deep learning is also employed to predict the outcome of matches and recommend constructive elements of teams.

Regardless of the goal of a game, it should be a pleasant and fun experience for its players. For some games to be enjoyable, the level of difficulty must be carefully calibrated, otherwise, players will feel bored or frustrated. Multiplayer scenarios in particular, where one player’s satisfaction might not translate to the enjoyment of other players and poses extra challenges in balancing the difficulty. The performance of one player is relative to the opponent, versus single-player scenarios where we can fully control the environment. We propose an AI automation framework for difficulty balancing in two-player games, where balancing is seen as a Reinforcement Learning task. A Game Master (GM) agent learns how to use handicap game mechanics, signaled by a reward function that evaluates a weighted combination of aesthetic criteria that encourages dramatization and allows a player in the lead to go back and a player in the rear to catch up, creating the desired rubber banding effect that balances out skill gaps. The quality of the games with the trained GM embedded is examined by measuring the same aesthetic criteria on the resulting games, and by analyzing the resulting changes in the game.

This work describes an Augmented Reality Serious Game (ARSG) focused on facilitating the process of acquiring new skills in individuals with motor disabilities. In particular, this game aims to help them control a robotic wheelchair. A racing track was used as a game narrative, including restriction areas, static and dynamic objects, obstacles and various signs. A user study with 20 participants was conducted to compare different methods used to place virtual content on the real-world environment while the user interacts with the game and control the wheelchair in the physical space: C1 - motion tracking using cloud anchors; C2 - offline motion tracking. Results suggest condition C1 is more precise and robust, while condition C2 seems to be easier to configure.

This work presents a framework for a new type of meta-game balance AI Competition based on Pokémon, Pokémon battles can be viewed as adversarial games played by AIs. Around these games, there is also a meta-game: which Pokémon to include in a team for battles, which moves to pick for every Pokémon in the team, etc. This meta-game is itself a game with a set of rules that govern which Pokémon and which moves are available in the roster that can be selected from, or which attributes (health points, damage, etc.) a Pokémon or moves should have. The aim of the framework is to facilitate competitions in creating the most balanced meta-game possible; one where there is a large variety of Pokémon and moves to choose from, and many possible combinations that are effective. AI agents could assist human designers in achieving strategically expressive meta-games, and this type of benchmark could incentivize game designers and researchers alike to advance knowledge on this type of domain.

This article proposes a general approach to solve the hand–eye calibration problem. The system is general since it is able to calibrate any number of cameras and, moreover, is able to simultaneously perform the calibration of several instances of the two common hand–eye calibration use cases: eye-on-hand and eye-to-base. The calibration is solved with a nonlinear least squares method, and the reprojection error is used as a metric to guide the optimization procedure. Our approach is seamlessly integrated with the robot operating system framework and allows for the interactive positioning of sensors and labeling of data, facilitating both the data acquisition and labeling and the calibration procedures. Results show that the proposed approach is able to handle any calibration use case with a minimal initial configuration. The approach is compared with several other state-of-the-art hand–eye calibration algorithms. Results show that the proposed approach produces very accurate calibrations when compared to the state of the art.

When compared with their single-agent counterpart, multi-agent systems have an additional set of challenges for reinforcement learning algorithms, including increased complexity, non-stationary environments, credit assignment, partial observability, and achieving coordination. Deep reinforcement learning has been shown to achieve successful policies through implicit coordination, but does not handle partial-observability. This paper describes a deep reinforcement learning algorithm, based on multi-agent actor-critic, that simultaneously learns action policies for each agent, and communication protocols that compensate for partial-observability and help enforce coordination. We also research the effects of noisy communication, where messages can be late, lost, noisy, or jumbled, and how that affects the learned policies. We show how agents are able to learn both high-level policies and complex communication protocols for several different partially-observable environments. We also show how our proposal outperforms other state-of-the-art algorithms that don't take advantage of communication, even with noisy communication channels.

Generating robust locomotion for a humanoid robot in the presence of disturbances is difficult because of its high number of degrees of freedom and its unstable nature. In this paper, we used the concept of Divergent Component of Motion (DCM) and propose an optimal closed-loop controller based on Linear-Quadratic-Gaussian to generate a robust and stable walking for humanoid robots. The biped robot dynamics has been approximated using the Linear Inverted Pendulum Model (LIPM). Moreover, we propose a controller to adjust the landing location of the swing leg to increase the withstanding level of the robot against a severe external push. The performance and also the robustness of the proposed controller is analyzed and verified by performing a set of simulations using MATLAB. The simulation results showed that the proposed controller is capable of providing a robust walking even in the presence of disturbances and in challenging situations.

This paper proposes a framework designed to generate a closed-loop walking engine for a humanoid robot. In particular, the core of this framework is an abstract dynamics model which is composed of two masses that represent the lower and the upper body of a humanoid robot. Moreover, according to the proposed dynamics model, the low-level controller is formulated as a Linear-Quadratic-Gaussian (LQG) controller that is able to robustly track the desired trajectories. Besides, this framework is fully parametric which allows using an optimization algorithm to find the optimum parameters. To examine the performance of the proposed framework, a set of simulation using a simulated Nao robot in the RoboCup 3D simulation environment has been carried out. Simulation results show that the proposed framework is capable of providing fast and reliable omnidirectional walking. After optimizing the parameters using genetic algorithm (GA), the maximum forward walking velocity that we have achieved was $80.5cm/s$.

In recent years, the artificial intelligence community has taken big strides in the application of reinforcement learning to games or similar environments using deep learning. From Atari to board games, including motor control or riddle solving, fairly generic deep learning algorithms can now achieve great policies by simply learning to play from experience, and minimal knowledge of the specific domain. However, these algorithms are very demanding in terms of time and hardware in order to achieve the results reported in the literature. So much so, that some algorithms would take years to achieve state-of-the-art performance in commodity hardware. Not only that, but even the learning environments can hinder the speed of the learning process, if they have not been performance optimized. In this paper, we evaluate a complex existing environment, and propose a performance-oriented version, which we call GeoFriends2. We describe the motivation behind the creation of our version, and how it is suitable for both single- and multi-agent reinforcement learning. We then use Asynchronous Deep Learning to create complex policies that can act as baselines for future research on this environment. We also describe a set of techniques that speed up the learning process such that tests can be run with commodity hardware in hours, and not weeks, and using much simpler network architectures.

CMA-ES is one of the most popular stochastic search algorithms. It performs favourably in many tasks without the need of extensive parameter tuning. The algorithm has many beneficial properties, including automatic step-size adaptation, efficient covariance updates that incorporates the current samples as well as the evolution path and its invariance properties. Its update rules are composed of well established heuristics where the theoretical foundations of some of these rules are also well understood. In this paper we will fully derive all CMA-ES update rules within the framework of expectation-maximisation-based stochastic search algorithms using information-geometric trust regions. We show that the use of the trust region results in similar updates to CMA-ES for the mean and the covariance matrix while it allows for the derivation of an improved update rule for the step-size. Our new algorithm, Trust-Region Co-variance Matrix Adaptation Evolution Strategy (TR-CMA-ES) is fully derived from first order optimization principles and performs favourably in compare to standard CMA-ES algorithm.

Many stochastic search algorithms are designed to optimize a fixed objective function to learn a task, i.e., if the objective function changes slightly, for example, due to a change in the situation or context of the task, relearning is required to adapt to the new context. For instance, if we want to learn a kicking movement for a soccer robot, we have to relearn the movement for different ball locations. Such relearning is undesired as it is highly inefficient and many applications require a fast adaptation to a new context/situation. Therefore, we investigate contextual stochastic search algorithms that can learn multiple, similar tasks simultaneously. Current contextual stochastic search methods are based on policy search algorithms and suffer from premature convergence and the need for parameter tuning. In this paper, we extend the well known CMA-ES algorithm to the contextual setting and illustrate its performance on several contextual tasks. Our new algorithm, called contextual CMA-ES, leverages from contextual learning while it preserves all the features of standard CMA-ES such as stability, avoidance of premature convergence, step size control and a minimal amount of parameter tuning.

Stochastic search algorithms are black-box optimizer of an objective function. They have recently gained a lot of attention in operations research, machine learning and policy search of robot motor skills due to their ease of use and their generality. Yet, many stochastic search algorithms require relearning if the task or objective function changes slightly to adapt the solution to the new situation or the new context. In this paper, we consider the contextual stochastic search setup. Here, we want to find multiple good parameter vectors for multiple related tasks, where each task is described by a continuous context vector. Hence, the objective function might change slightly for each parameter vector evaluation of a task or context. Contextual algorithms have been investigated in the field of policy search, however, the search distribution typically uses a parametric model that is linear in the some hand-defined context features. Finding good context features is a challenging task, and hence, non-parametric methods are often preferred over their parametric counter-parts. In this paper, we propose a non-parametric contextual stochastic search algorithm that can learn a non-parametric search distribution for multiple tasks simultaneously. In difference to existing methods, our method can also learn a context dependent covariance matrix that guides the exploration of the search process. We illustrate its performance on several non-linear contextual tasks.

Stochastic search algorithms are general black-box optimizers. Due to their ease of use and their generality, they have recently also gained a lot of attention in operations research, machine learning and policy search. Yet, these algorithms require a lot of evaluations of the objective, scale poorly with the problem dimension, are affected by highly noisy objective functions and may converge prematurely. To alleviate these problems, we introduce a new surrogate-based stochastic search approach. We learn simple, quadratic surrogate models of the objective function. As the quality of such a quadratic approximation is limited, we do not greedily exploit the learned models. The algorithm can be misled by an inaccurate optimum introduced by the surrogate. Instead, we use information theoretic constraints to bound the `distance' between the new and old data distribution while maximizing the objective function. Additionally the new method is able to sustain the exploration of the search distribution to avoid premature convergence. We compare our method with state of art black-box optimization methods on standard uni-modal and multi-modal optimization functions, on simulated planar robot tasks and a complex robot ball throwing task.The proposed method considerably outperforms the existing approaches.

In a soccer match, a cooperative behavior emerges from the combined execution of simple actions by players. A cooperative behavior can be planned if players are previously committed to its execution prior to its start or unplanned otherwise. The ability to reproduce some of these behaviors can be useful to help a team achieve better performances. This work presents an approach to identify and extract cooperative behaviors that start from set-pieces and lead to a goal while ball possession is kept. The representation of these behaviors is abstracted using a set-play definition language to promote their reusability. A set of game log files generated with the FC Portugal team and collected from the RoboCup 2010 2D simulated soccer competition were analyzed. The results achieved showed that 25% of the total goals scored originated from set-pieces which attests to the importance of performing this analysis. Several guidelines for the definition of future set-plays were also inferred. In the future, these behaviors shall be tested to infer which are capable of neutralizing an opponent's team strategy and maximize the creation of goal opportunities.

The paper presents the architecture and algorithms developed for Dom Dinis, a simple compass-guided robot built by the authors. This includes environment exploration, task planning and task execution. Environment exploration, based on repeating a reactive goal search, enables a progressive construction of a grid based map. Based on the (possibly incomplete) map, the robot is able to plan its tasks. The execution capabilities of the robot include exception handling. Essential to all these capabilities is the knowledge of the robot's position in the world. The position is computed based on tracking traversed distances and followed orientations. Orientation is given by a compass. Dom Dinis does not use wheel encoders at all.

The paper discusses some new hardware and software tools that can be used for the design of virtual circuits based on dynamically reconfigurable FPGAs. With the aid of these tools we can implement a system that requires some, hardware resources R/sub c/, on available hardware that has resources R/sub h/, where R/sub c/>R/sub h/. The main idea of the approach supported by these tools is the rational combination of FPGA capabilities with some proposed methods for producing a modifiable specification, together with a novel technique for architectural and logic synthesis, which has been incorporated into the new design environment.

No abstract available.

A set of parameters to assess the objective quality of visualizations of a voxel-based data set, produced using a ray caster, is proposed as a first step toward the evaluation of the overall quality of these visualizations. Results obtained using synthetic data and a simple implementation of a ray caster are presented. The final goal of this evaluation is the computation of "confidence indices" that could offer the user a "guided visualization", i.e. allow him/her to decide what are the "best" visualizations of a data set.