One promising approach to general-purpose artificial intelligence is neuroevolution, which has worked well on a number of problems from resource optimization to robot control. However, state-of-the-art neuroevolution algorithms like neuroevolution of augmenting topologies (NEAT) have surprising difficulty on problems that are fractured, i.e., where the desired actions change abruptly and frequently. Previous work demonstrated that bias and constraint (e.g., RBF-NEAT and Cascade-NEAT algorithms) can improve learning significantly on such problems. However, experiments in this paper show that relatively unrestricted algorithms (e.g., NEAT) still yield the best performance on problems requiring reactive control. Ideally, a single algorithm would be able to perform well on both fractured and unfractured problems. This paper introduces such an algorithm called SNAP-NEAT that uses adaptive operator selection to integrate strengths of NEAT, RBF-NEAT, and Cascade-NEAT. SNAP-NEAT is evaluated empirically on a set of problems ranging from reactive control to high-level strategy. The results show that SNAP-NEAT can adapt intelligently to the type of problem that it faces, thus laying the groundwork for learning algorithms that can be applied to a wide variety of problems.

Evolution of neural networks, or neuroevolution, bas been successful on many low-level control problems such as pole balancing, vehicle control, and collision warning. However, high-level strategy problems that require the integration of multiple sub-behaviors have remained difficult for neuroevolution to solve. This paper proposes the hypothesis that such problems are difficult because they are fractured: the correct action varies discontinuously as the agent moves from state to state. This hypothesis is evaluated on several examples of fractured high-level reinforcement learning domains. Standard neuroevolution methods such as NEAT indeed have difficulty solving them. However, a modification of NEAT that uses radial basis function (RBF) nodes to make precise local mutations to network output is able to do much better. These results provide a better understanding of the different types of reinforcement learning problems and the limitations of current neuroevolution methods. Thus, they lay the groundwork for creating the next generation of neuroevolution algorithms that can learn strategic high-level behavior in fractured domains.

Many serious automobile accidents could be avoided if drivers were warned of impending crashes before they occur. Creating such warning systems by hand, however, is a difficult and time-consuming task. This paper describes three advances toward evolving neural networks with NEAT (NeuroEvolution of Augmenting Topologies) to warn about such crashes in real-world environments. First, NEAT was evaluated in a complex, dynamic simulation with other cars, where it outperformed three hand-coded strawman warning policies and generated warning levels comparable with those of an open-road warning system. Second, warning networks were trained using raw pixel data from a simulated camera. Surprisingly, NEAT was able to generate warning networks that performed similarly to those trained with higher-level input and still outperformed the baseline hand-coded warning policies. Third, the NEAT approach was evaluated in the real world using a robotic vehicle testbed. Despite noisy and ambiguous sensor data, NEAT successfully evolved warning networks using both laser rangefinders and visual sensors. The results in this paper set the stage for developing warning networks for real-world traffic, which may someday save lives in real vehicles.

Many serious automobile accidents could be avoided if drivers were warned of impending crashes before they occurred. In this paper, a vehicle warning system is evolved to predict such crashes in the RARS driving simulator. The NeuroEvolution of Augmenting Topologies (NEAT) method is first used to evolve a neural network driver that can autonomously navigate a track without crashing. The network is subsequently impaired, resulting in a driver that occasionally makes mistakes and crashes. Using this impaired driver, a crash predictor is evolved that can predict how far in the future a crash is going to occur, information that can be used to generate an appropriate warning level. The main result is that NEAT can successfully evolve a warning system that takes into account the recent history of inputs and outputs, and therefore makes few errors. Experiments were also run to compare training offline from previously collected data with training online in the simulator. While both methods result in successful warning systems, offline training is both faster and more accurate. Thus, the results in this paper set the stage for developing crash predictors that are both accurate and able to adapt online, which may someday save lives in real vehicles.

Feature selection is the process of finding the set of inputs to a machine learning algorithm that will yield the best performance. Developing a way to solve this problem automatically would make current machine learning methods much more useful. Previous efforts to automate feature selection rely on expensive meta-learning or are applicable only when labeled training data is available. This paper presents a novel method called FS-NEAT which extends the NEAT neuroevolution method to automatically determine an appropriate set of inputs for the networks it evolves. By learning the network's inputs, topology, and weights simultaneously, FS-NEAT addresses the feature selection problem without relying on meta-learning or labeled data. Initial experiments in an autonomous car racing simulation demonstrate that FS-NEAT can learn better and faster than regular NEAT. In addition, the networks it evolves are smaller and require fewer inputs. Furthermore, FS-NEAT's performance remains robust even as the feature selection task it faces is made increasingly difficult.

This paper presents a machine learning approach to optimizing a quadrupedal trot gait for forward speed. Given a parameterized walk designed for a specific robot, we propose using a form of policy gradient reinforcement learning to automatically search the set of possible parameters with the goal of finding the fastest possible walk. We implement and test our approach on a commercially available quadrupedal robot platform, namely the Sony Aibo robot. After about three hours of learning, all on the physical robots and with no human intervention other than to change the batteries, the robots achieved a gait faster than any previously known gait known for the Aibo, significantly outperforming a variety of existing hand-coded and learned solutions.