This paper presents a fuzzy set-based methodology to achieve simultaneous path adherence and local collision avoidance in autonomous navigation. It targets scenarios where a global path to the vehicle's destination is already planned but with imperfect knowledge of the obstacles in the environment and their dynamics. Assuming that the vehicle can localize itself and the obstacles around it, the proposed methodology incorporates the global path and the acquired real-time knowledge of the local obstacles by the vehicle to create a comprehensive fuzzy representation of the environment. This fuzzy representation is then utilized to assess the desirability of the vehicle states within an optimization framework that balances global path adherence and obstacle avoidance objectives. The proposed gradient-based solution to this optimization problem navigates the vehicle such that it maintains its global course toward the designated destination while avoiding collision with local obstacles. Extensive simulations on a mobile robot validate the method's efficacy in guiding vehicles along desired paths, maneuvering around obstacles with minimal deviation from the route, and negotiating local minima without oscillations. Additionally, simulation results highlight the proposed fuzzy set-based methodology's low computational complexity, real-time operation capability, and adaptability in handling complex geometries of paths and obstacles.

This paper introduces a novel approach to enhance the stability and efficiency of R-STDP in the context of federated learning. The primary objective is to stabilize the unbounded growth of R-STDP and make it more responsive to real-time changes. The methodology involves integrating R-STDP with Spiking Neural Networks and employing the norm of the neural network model for adjusting weighted aggregation in federated learning systems. The proposed method incorporates a mechanism where weights decay over time, depending on the duration since the agent last published its model. Additionally, the sampling time is dynamically adjusted based on the Euclidean norm, which measures the distance between the weight matrices of the agents and the server. The results demonstrate that the proposed event-triggered federated learning method significantly enhances learning speed and performance. At the same time, the dynamic aggregation interval efficiently reduces communication between the agents and the central server, especially after model convergence. This research presents a significant advancement in federated learning and offers a more stable, responsive, and efficient learning process.

We propose using a genetic algorithm to select hyperparameters in multiagent reinforcement learning (MARL) settings. In particular, we look at this in the context of cooperation and altruism. We show through the use of three continuous space games, that certain algorithmic hyperparameters are better suited to allow to agents learn altruistic behaviors. The agents learn using fuzzy actor critic learning algorithms in either a hierarchical structure or a single actor critic policy. The genetic algorithm selects the discount factors, the reward weights, and the standard deviation of noise applied to actor during learning. The genetic algorithm uses a fitness function based on the ratio of successful tests the group of agents can pass after training. This automated selection of these specific hyperparameters show that they are important for cooperation and also not trivial to select.

How do we know when a reinforcement learning policy needs to adapt? In non-stationary environments, agents must adapt and learn in environments that change dynamically. We propose a finite-horizon model-free solution using a hierarchical learning structure with fuzzy systems. The higher-level learning policy advises the lower-level policy when to start and stop learning based on the temporal differences calculated within the lower-level. Major differences in the temporal difference of each action produced by an agent may indicate environment change. This structure is tested with multi-agent differential games in both the cooperative and competitive aspect. Results show that this method is quick to notice and adapt the policy within relatively few learning episodes.

Pursuit-Evasion (PE) games are regarded as a major platform for game theory. In this kind of game, an agent called an evader tries to escape from another agent called a pursuer. Active Target Defense (ATD) is a derivative of PE games, attracting attention recently. In an ATD game, the evader, often called an invader, strives to capture a moving target. The pursuer, called a defender, tries to intercept the invader. This paper implements the Spike-Timing-Dependent Plasticity (STDP) algorithm to train two Spiking Neural Networks (SNNs) to find a suitable solution for the ATD problem in decentralized situations. One of the SNNs is used to control the invader, while the other controls the defender. The performance is compared with the analytical solution for the pedestrian model. The results showed that an SNN can learn the optimal capture point only using relative velocities and line of sight.

This paper addresses a game of guarding a territory with several invaders and several defenders. Although there are a plethora of studies for the single-invader single-defender scenario, or single-invader multi-defender case, there are few articles on the multi-invader case. The reason is the assignment problem. Each defender must know the policy for capturing each invader. In addition, each defender must choose an invader to capture during the game. We proposed a hierarchical reinforcement learning (HRL) process to solve the assignment problem in a multi-invader multi-defender game of guarding a territory. In the proposed method, a higher-level policy is able to change the assignment in the middle of the game as the game environment is changing. The proposed method is also examined on the game of active target defence. It is shown that the proposed method can solve the assignment problem without any knowledge of an external optimal policy, such as geometric approaches.

In the physical world, individuals compete against others within their own population or separately evolving populations. Robotic agents will soon face this coevolutionary and adversarial reality. Many challenges not encountered in the evolution of single populations are encountered in adversarial coevolution. These challenges can render single evolutionary approaches either less effective or ineffective. Most problems have a fundamental goal, but also feature secondary desirable objectives. Here, evader agents in the pursuit-evasion game that do not elude capture are effectively useless. Secondly, when applied to evolve real robots online in a coevolutionary context, non-optimal robots can be erratic and cause damage. Objective hierarchy can be used to define the importance of each objective, and promote quicker optimization of primary objectives such as evasion. The Evolutionary Inherited Neuromodulated Neurocontroller (EINN) method incorporates objective weighted ranking (OWR), a novel objective hierarchy method that promotes optimization of the primary objective in simultaneous multi-objective optimization. EINN is compared to the previously demonstrated Lamarckian-inherited Neuromodulated MultiObjective Evolutionary Neurocontroller (LNMOEN), and shown to be effective in a single evolutionary context.

The disparity in performance between simulated and real systems is a major problem in robotics and other fields. Simulating measurement noise is one method of reducing these performance differences. Here, we examine the effect of measurement noise on the behaviour and topology of evolved neuromodulated neurocontrollers applied to control evader agents in a pursuit-evasion game. Measurement noise in the form of a zero-mean, normally distributed random signal is applied to the evader’s radar range and angle signals. The results indicate that increasing the levels of measurement noise increases the number of generations required to evolve fit agents. Noise at the neurocontroller outputs is of lesser amplitude than that at the inputs, suggesting a low-pass filtering operation. When levels of measurement noise different to those with which they were evolved were applied to the neurocontrollers, greater amplitude in the measurement noise signal increased the average length of time required to capture the evader. When the level of measurement noise was changed during evolution, after a few generations of further evolution, the neurocontrollers were able to adapt to both increases and decreases in the amount of noise. The evolutionary neurocontrollers are robust to high levels of measurement noise and can adapt to large changes in noise amplitude. This suggests that the neurocontrollers will be robust when used in the field on real robots, and that they may be a good solution to bridging the gap between simulation and reality.

Autonomous vehicles in the pursuit-evasion game, subject to the effects of mass and drag, are controlled using an evolutionary multiobjective neuromodulated controller with unsupervised learning. Multiobjective evolution of network weights and topologies (NEAT-MODS) is extended with Lamarckian-inherited neuromodulated learning. NEAT-MODS is an NSGA-II augmented multiobjective neurocontroller that uses two conflicting objectives. By evolving pursuit agents optimized with the separate and conflicting objectives of `capturing evaders' and `minimizing energy consumption', efficient neurocontrollers can be evolved. NEAT-MODS uses a selection process that aims to ensure Pareto-optimal genotypic diversity and elitism. Neuromodulation is a biologically-inspired technique that can adapt the per-connection learning rates of synaptic plasticity. Lamarckian inheritance allows behaviours learned during parent generations to be passed on to their offspring. The capability of the design is demonstrated in a series of experiments with a simulated evolved vehicle pursuing a basic evader vehicle. It is shown that compact and efficient neurocontrollers for pursuer agents with nonzero mass and drag, capable of capturing an optimal evader while simultaneously minimizing energy consumption, are evolved.

This paper presents a machine-learning algorithm applied to a quadcopter application. We are proposing a fuzzy actor-critic learning (FACL) algorithm. This method enables a pursuer quadcopter to capture an evader quadcopter in the pursuit-evasion (PE) differential game. In this application, the pursuer learns its control strategies by interacting with evader and learning from past experiences. Both the critique and the actor are fuzzy inference systems (FIS). It is assumed that the pursuer knows only the instantaneous position and speed of the evader and vice versa. The FACL will generate the desired trajectory as the input for low-level controllers. Simulation results are presented for the PE differential game to demonstrate the practicality of our machine-learning algorithm.

In this paper, a deceptive version of guarding a territory in a grid world is proposed. Like the original version, a defender tries to intercept an invader before it invades the targets. However, the discerning invader can deceive the defender about its real goal so that it can improve its performance. On the other hand, the defender tries to confront the invader by guessing its true goal. A two-level policy is obtained via reinforcement learning (RL). In the lower level, the invader and the defender learn their optimal policies to invade or defend a particular territory. In the higher level, the invader learns which territory it should pretend to invade in order to manipulate the defender's belief function. A multiagent reinforcement learning (MARL) algorithm is implemented for obtaining the optimal policies via the minimax Q-learning algorithm at the lower level. Whereas for the higher-level policy a single-agent Q-learning algorithm is utilized. Results of different reward functions are compared. The results show that the invader can improve its performance by taking advantage of deception.

This paper presents a decentralized learning technique that enables two pursuers or more to capture a single evader in the pursuit-evasion (PE) differential games. Both the pursuers and the evader should learn their control strategies simultaneously by interacting with each other. The proposed learning technique uses a fuzzy actor-critic learning automaton (FACLA) algorithm together with the Kalman filter technique. Both the critique and the actor are fuzzy inference systems (FIS). The Kalman filter is used as an estimation method for the evader's next position. Then, the pursuers can find the evader's movement direction based on this estimation to avoid collision among them and to reduce the capture time. It is assumed that each pursuer knows only the instantaneous position of the evader and vice versa. Also, we assume that there is not any type of communication among them and each pursuer considers the other pursuers as part of its environment. So, the cooperation among pursuers to minimize the capture time can be done in a decentralized manner. For each player in the game, the proposed learning technique is used to autonomously tune the parameters of its fuzzy logic controller (FLC) to self-learn its control strategy. Simulation results for the case of three players PE differential game; two pursuers attempt to capture a single evader; are given to show the feasibility of our learning algorithm.

In this paper, we investigate the use of reinforcement learning to train the players in the game of guarding a territory. The game is played in the continuous domain. There are two players in the game: an invader and a guard. In our formulation, we set the guard to be 30 percent faster than the invader. We make the assumption that the players have no a priori knowledge of their optimal behavior. Therefore, the players will obtain these after learning. In other words, both players are simultaneously learning in the game. We introduce the Apollonius circle approach to determine the optimal solution of the guarding a territory game, when the guard is faster than the invader. We make use of the optimal solution of the game determined using the Apollonius circle approach to evaluate the learning performance of the players. We present simulation results and discuss the effectiveness of the approach.

In the context of the IEEE 1588 Precision Time Protocol (PTP), estimating the delay's bias is a problem that appears in both one-way (using transparent devices) or two-way message exchange mechanisms. For estimating the offset via the two-way message exchange mechanism, it is usually assumed that the expected value of delays in forward and reverse directions are equal. However, this is not a realistic assumption for packet-based wide area networks, where delays in down-link and up-link directions may have a significant difference. In this work, we propose a solution to estimate the random delay's bias and improve the synchronization accuracy of IEEE 1588. Our method is easy to implement and is compatible with the current version of the protocol. We compared our results to no bias correction and the Boot-strap method. In addition to the improvement in synchronization accuracy, our method allows us to update the slave clock recursively. The proposed method works well even in the presence of large frequency offsets and can also be implemented by using different filters.

This paper addresses the case of dual learning in the pursuit-evasion (PE) differential game and examines how fast the players can learn their default control strategies. The players should learn their default control strategies simultaneously by interacting with each other. Each player's learning process depends on the rewards received from its environment. The learning process is implemented using a two stage learning algorithm that combines the particle swarm optimization (PSO)-based fuzzy logic control (FLC) algorithm with the Q-Learning fuzzy inference system (QFIS) algorithm. The PSO algorithm is used as a global optimizer to autonomously tune the parameters of a fuzzy logic controller whereas the QFIS algorithm is used as a local optimizer. The two stage learning algorithm is compared through simulation with the default control strategy, the PSO-based FLC algorithm, and the QFIS algorithm. Simulation results show that the players are able to learn their default control strategies. Also, it shows that the two stage learning algorithm outperforms the PSO-based FLC algorithm and the QFIS algorithm with respect to the learning time.

This paper proposes a Delay Asymmetry Correction (DAC) Model to enhance clock synchronization protocols based on a Master-Slave arrangement such as the IEEE 1588 PTP protocol. The purpose of this work is to mitigate the effects of unpredictable packet delay variations (PDV), which may aggravate asymmetric link delays on timing packets, in order to improve the synchronization accuracy of the slave clock with respect to the master clock. The key idea in our work is to filter clock updates derived from the master-slave message exchange, using only good samples for slave clock updates. The proposed solution is implemented in Network Simulator 2 (NS-2.34). NS-2 test cases are implemented according to the ITU-T G.8261 document covering various network loads and network conditions. Our simulation results indicate that the proposed solution improves the slave accuracy significantly, achieving almost perfect synchronization accuracy in the presence of a wide range of network traffic loads, network congestions, and temporary network outage. Furthermore, when there is a routing path change due to the failure in the network, the solution also maintains high slave clock accuracy with respect to the master clock.

This paper investigates four methods of implementing a Q-Learning Fuzzy Inference System (QFIS) algorithm to autonomously tune the parameters of a fuzzy inference system. We use an actor-critique structure and we simulate mobile robots playing the differential form of the pursuit evasion game. Both the critique and the actor are fuzzy inference systems. The four methods come from the fact whether it is necessary to tune all the parameters (i.e. all the premise and the consequent parameters) of the critique and the actor or just tune their consequent parameters. The four methods are applied to three versions of the pursuit evasion games. In the first version just the pursuer is learning. In the second version, the evader uses its higher maneuverability and plays intelligently against a self-learning pursuer. In the final version, both the pursuer and the evader are learning. We evaluate which parameters are best to tune and which parameters have little impact on the performance.

The main contribution of this work is a novel machine reinforcement learning algorithm for problems where a Poissonian stochastic time delay is present in the agent's reinforcement signal. Despite the presence of the reinforcement noise, the algorithm can craft a suitable control policy for the agent's environment. The novel approach can deal with reinforcements which may be received out of order in time or may even overlap, which was not previously considered in the literature. The proposed algorithm is simulated and its performance is compared to a standard Q-learning algorithm. Through simulation, the proposed method is found to improve the performance of a learning agent in an environment with Poissonian-type stochastically delayed rewards.

A multi-agent policy iteration learning algorithm is proposed in this work. The Exponential Moving Average (EMA) mechanism is used to update the policy for a Q-learning agent so that it converges to an optimal policy against the policies of the other agents. The proposed EMA Q-learning algorithm is examined on a variety of matrix and stochastic games. Simulation results show that the proposed algorithm converges in a wider variety of situations than state-of-the-art multi-agent reinforcement learning (MARL) algorithms.

A two-way message exchange mechanism is at the core of many clock synchronization protocols. The basic concept of this method has been designed for symmetrical links, otherwise, depending on the level of asymmetric delays, large synchronization errors may occur. Many estimation based methods have been proposed so far for estimating the degree of asymmetry and reducing this error. However, many of them are solved for a specific distribution of link random delays. Moreover, imposing a high level of computational complexity upon network nodes is another problem of them. In this paper we introduce a recursive solution for improving the performance of the basic two-way message exchange mechanism. The advantage of this method is that it keeps the protocol simple and therefore easily implementable. The proposed method does not make any specific assumption about the distribution of the random delays. It also provides the network designer with some deployment flexibility, trading off convergence speed against stability.

In this paper, we improve on a previously proposed algorithm for exploring a 2-D environment with multiple robots using Potential Games. A potential game is a type of game that results in coordinated behaviours amongst players. This is done by enforcing strict rules for each player in selecting an action from its action set. As part of this game, we define a potential function for the game that is meaningful in terms of achieving the greater objective of exploring a space. Furthermore, an objective function is assigned for each player from this potential function. We then create an algorithm for the exploration of an obstacle-filled bounded space, and demonstrate through simulation how it outperforms an uncoordinated algorithm by reducing the time needed to uncover the space. This algorithm is then improved by having robots predict the future positions of all other robots.

Using `Simultaneous Localization and Mapping' (SLAM), mobile robots can become truly autonomous in the exploration of their environment. However, once these environments becomes too large, Multi-Robot SLAM becomes a requirement. This paper will outline how a mobile robot should decide when best to merge its maps with another robot's upon rendezvous, as opposed to doing so immediately. This decision will be based on the current status of the mapping particle filters and the current status of the environment. Using Reinforcement Learning, a model can be established and then trained upon to determine a policy capable of deciding when best to merge. This will allow the robot to incur less error during a merge compared to simply merging immediately. This policy is trained and validated using simulated mobile robot datasets.

We extend the potential-based shaping method from Markov decision processes to multi-player general-sum stochastic games. We prove that the Nash equilibria in a stochastic game remains unchanged after potential-based shaping is applied to the environment. The property of policy invariance provides a possible way of speeding convergence when learning to play a stochastic game.

This paper addresses the problem of tuning the input and the output parameters of a fuzzy logic controller. A novel technique that combines Q(λ)-learning with a fuzzy inference system as a function approximation is proposed. The system learns autonomously without supervision or a priori training data. The proposed technique is applied to two different differential games. The proposed technique is compared with the classical control strategy, Q(λ)-learning only, and the technique proposed in [1] in which a neural network is used as a function approximation for Q-learning. Computer simulations show the usefulness of the proposed technique.

This paper discusses learning in n-purser n-evader games. In a pursuit-evasion game, one player (the pursuer) pursues another one while the other (the evader) tries to escape. We assume that each player only knows the instantaneous position of the other players but at the same time none of them knows its control strategy nor the control strategy of the other players. Therefore, the players have to self-learn their control strategies on-line by interaction with each other. In this paper, we extend our previous work from learning in a single pursuit-evasion game to learning in a multi-pursuit-evasion game. We use the Q(λ)-learning based genetic fuzzy controller technique (QLBGFC) proposed in. The proposed technique combines reinforcement learning with both a fuzzy controller and genetic algorithms in a two-phase structure. In addition to the proposed QLBGFC, we construct a new Q-table that is responsible for learning the coupling process between the pursuers and the evaders. To test the performance of the proposed technique, it is compared with the optimal strategy of a single pursuit-evasion game. Computer simulations show the usefulness of the proposed technique.

This paper addresses the problem of tuning the input and the output parameters of a fuzzy logic controller. A novel technique that combines Q(λ)-learning with function approximation (fuzzy inference system) is proposed. The system learns autonomously without supervision or a priori training data. The proposed technique is applied to a pursuit-evasion differential game in which both the pursuer and the evader self-learn their control strategies. The proposed technique is compared with the classical control strategy, Q(λ)-learning only, and the technique proposed in in which a neural network is used as a function approximation for Q-learning. Computer simulations show the usefulness of the proposed technique.

This paper presents a new method of using hybrid intelligent systems to solve the problem of tuning the parameters of a fuzzy logic controller. Two different hybrid intelligent systems are introduced in this paper. Each system proposes learning in a two-stage iterative process. The first system combines a fuzzy logic controller with genetic algorithms to form the iterative genetic based fuzzy logic controller technique (IGBFLC). The second system combines a fuzzy logic controller with an adaptive network to form the iterative adaptive network fuzzy inference system (IANFIS). The proposed systems are applied to a model of pursuit-evasion game. In this model, we are seeking for the optimal strategy of the pursuer given that the evader plays its optimal strategy. The proposed systems are compared with the PD controller, the genetic-based fuzzy logic controller and the ANFIS technique. Computer simulations and results show that when compared to the optimal strategy, the proposed systems outperform the other techniques.

This paper presents a novel technique to tune the parameters of a fuzzy logic controller using a combination of reinforcement learning and genetic algorithms. The proposed technique is called a Q(λ)-learning based genetic fuzzy logic controller (QLBGFLC). The proposed technique is applied to a pursuit-evasion game in which the pursuer does not know its control strategy. We assume that we do not even have a simplistic PD controller strategy. The learning goal for the pursuer is to self-learn its control strategy. The pursuer should do that on-line by interaction with the environment; in this case the evader. Our proposed technique is compared with the optimal strategy, Q(λ)-learning only, and unsupervised genetic algorithm learning. Computer simulations show the usefulness of the proposed technique.

This paper mainly discusses learning in pursuit-evasion game. In the pursuit-evasion model, one robot pursues another one in a partially known environment. Partially known environment means that each robot knows the instant position of the other robot but at the same time none of them knows its control strategy. Therefore, both robots have to self-learn their control strategies on-line by interaction with each other. A new hybrid learning technique is proposed. The proposed technique combines reinforcement learning with both a fuzzy controller and genetic algorithms in a two-phase structure. The proposed technique is called a Q(¿)-learning based genetic fuzzy controller (QLBGFC). To test the performance of our proposed technique, it is compared with the optimal strategy, the Q(¿)-learning, and the reward-based genetic algorithms. Computer simulations show the usefulness of the proposed technique. In addition, the convergence and the boundedness of the Q-learning algorithm used in the proposed technique are shown.

In this paper a reinforcement fuzzy learning scheme for robots playing a differential game is derived. A differential game may be considered a Markov decision process in continuous time, with continuous states and actions. The robots receive reinforcements from the environment after they take an action; and this reinforcement is then used to adapt a fuzzy controller that stores the experience accumulated by the robot. Every calculation is done in a physical system based on microcontrollers to control the movement of the robots and sensors to measure their position and angle in a 2D-plane. Filters are also implemented to approximate the derivatives of the states. Experiments of a pursuer-evader game are provided in order to show the feasibility of the technique. It should be noted, though, that the technique may also be used in a multi-game environment.

This paper examines three methods of adaptive output feedback control for robotic manipulators. Implement ing output feedback for control, instead of full-state feedback, allows use of only the position information. The position can be measured quite accurately, while velocity and acceleration measurements tend to get corrupted by noise. As well, having only a position sensor reduces costs in producing the robot. The three methods examined each use some form of state estimation. The methods examined are: a method proposed by Lee and Khalil using a high-gain observer, Craig, Hsu, and Sastry’s method of adaptive robot control using a linear observer that we propose herein, and a method proposed by Gourdeau and Schwartz using an Extended Kalman Filter (EKF). The methods are all implemented in simulation and compared in both noise-free and noise-contaminated cases.

Image coding using multirate filter banks and wavelets is evaluated in this paper. The coding systems considered are based on the M-channel general lapped biorthgonal transform (GLBT) and the embedded zerotree wavelet (EZW) coding methods, as well as the baseline JPEG standard. The study concentrates on both coding efficiency and complexity. The tradeoff between efficiency and complexity of each coding system has been analyzed. The coding results show that the choice of a coding scheme depends mainly on the applications at hand.

Alternative methods are examined for estimating motion and structure of objects undergoing smooth maneuvers through measurements of feature positions in long, multiple-camera image sequences. Performance of extended Kalman filters are compared for Euler angle-axis, roll-pitch-yaw, and quaternion parameterizations of rotational motion. The angle-axis method was found to give the best overall performance with a computationally efficient implementation.

A globally stable direct adaptive control algorithm for robotic manipulators is proposed. The algorithm is based on a desired compensation adaptation law and control law such that the measurement noise is no longer correlated with the regression vector. Therefore the parameter estimates are not susceptible to parameter drift. Lyapunov theory is used to design both the control law and the adaptation law. Based on stability theory we establish sufficient conditions for the choice of the controller gains. These sufficient conditions are formulated directly from the bounds on the inertial parameters. The performance of the proposed algorithm is demonstrated by computer simulation. The proposed method is also compared to the methods proposed by Slotine and Li (1987), and Sadegh and Horowitz (1990).

An adaptive motion control scheme for robotic manipulators is presented. This is an adaptive computed torque method (CTM) that requires only position measurements. These measurements and the input torques are used in an extended Kalman filter (EKF) to estimate the inertial parameter of the full nonlinear robot model as well as the joint positions and velocities. These estimates are used by the CTM to generate the input torques. This combination of the EKF and the CTM was shown to result in a stable adaptive control scheme. Experimental results with a two-degree-of-freedom direct-drive robot illustrate the performance of this scheme.>