A hallmark of intelligence is the ability to autonomously learn new flexible, cognitive behaviors - that is, behaviors where the appropriate action depends not just on immediate stimuli (as in simple reflexive stimulus-response associations), but on contextual information that must be adequately acquired, stored and processed. While many meta-learning algorithms can design agents that autonomously learn new tasks, cognitive tasks adds another level of learning and memory to typical ``learning-to-learn'' problems. Here we evolve neural networks, endowed with plastic connections and neuromodulation, over a sizable set of simple cognitive tasks adapted from a computational neuroscience framework. The resulting evolved networks can automatically modify their own connectivity to acquire a novel simple cognitive task, never seen during evolution, from stimuli and rewards alone, through the spontaneous operation of their evolved neural organization and plasticity system. Our results emphasize the importance of carefully considering the multiple learning loops involved in the emergence of intelligent behavior.

Continual lifelong learning requires an agent or model to learn many sequentially ordered tasks, building on previous knowledge without catastrophically forgetting it. Much work has gone towards preventing the default tendency of machine learning models to catastrophically forget, yet virtually all such work involves manually-designed solutions to the problem. We instead advocate meta-learning a solution to catastrophic forgetting, allowing AI to learn to continually learn. Inspired by neuromodulatory processes in the brain, we propose A Neuromodulated Meta-Learning Algorithm (ANML). It differentiates through a sequential learning process to meta-learn an activation-gating function that enables context-dependent selective activation within a deep neural network. Specifically, a neuromodulatory (NM) neural network gates the forward pass of another (otherwise normal) neural network called the prediction learning network (PLN). The NM network also thus indirectly controls selective plasticity (i.e. the backward pass of) the PLN. ANML enables continual learning without catastrophic forgetting at scale: it produces state-of-the-art continual learning performance, sequentially learning as many as 600 classes (over 9,000 SGD updates).

In this paper, we introduce a novel form of value function, $Q(s, s’)$, that expresses the utility of transitioning from a state $s$ to a neighboring state $s’$ and then acting optimally thereafter. In order to derive an optimal policy, we develop a forward dynamics model that learns to make next-state predictions that maximize this value. This formulation decouples actions from values while still learning off-policy. We highlight the benefits of this approach in terms of value function transfer, learning within redundant action spaces, and learning off-policy from state observations generated by sub-optimal or completely random policies. Code and videos are available at http://sites.google.com/view/qss-paper.

Continual learning is the problem of sequentially learning new tasks or knowledge while protecting previously acquired knowledge. However, catastrophic forgetting poses a grand challenge for neural networks performing such learning process. Thus, neural networks that are deployed in the real world often struggle in scenarios where the data distribution is non-stationary (concept drift), imbalanced, or not always fully available, i.e., rare edge cases. We propose a Differentiable Hebbian Consolidation model which is composed of a Differentiable Hebbian Plasticity (DHP) Softmax layer that adds a rapid learning plastic component (compressed episodic memory) to the fixed (slow changing) parameters of the softmax output layer; enabling learned representations to be retained for a longer timescale. We demonstrate the flexibility of our method by integrating well-known task-specific synaptic consolidation methods to penalize changes in the slow weights that are important for each target task. We evaluate our approach on the Permuted MNIST, Split MNIST and Vision Datasets Mixture benchmarks, and introduce an imbalanced variant of Permuted MNIST - a dataset that combines the challenges of class imbalance and concept drift. Our proposed model requires no additional hyperparameters and outperforms comparable baselines by reducing forgetting.

How can we build agents that keep learning from experience, quickly and efficiently, after their initial training? Here we take inspiration from the main mechanism of learning in biological brains: synaptic plasticity, carefully tuned by evolution to produce efficient lifelong learning. We show that plasticity, just like connection weights, can be optimized by gradient descent in large (millions of parameters) recurrent networks with Hebbian plastic connections. First, recurrent plastic networks with more than two million parameters can be trained to memorize and reconstruct sets of novel, high-dimensional (1000+ pixels) natural images not seen during training. Crucially, traditional non-plastic recurrent networks fail to solve this task. Furthermore, trained plastic networks can also solve generic meta-learning tasks such as the Omniglot task, with competitive results and little parameter overhead. Finally, in reinforcement learning settings, plastic networks outperform non-plastic equivalent in a maze exploration task. We conclude that differentiable plasticity may provide a powerful novel approach to the learning-to-learn problem.

Visual attention has many effects on neural responses, producing complex changes in firing rates, as well as modifying the structure and size of receptive fields, both in topological and feature space. Several existing models of attention suggest that these effects arise from selective modulation of neural inputs. However, anatomical and physiological observations suggest that attentional modulation targets higher levels of the visual system (such as V4 or MT) rather than input areas (such as V1). Here we propose a simple mechanism that explains how a top-down attentional modulation, falling on higher visual areas, can produce the observed effects of attention on neural responses. Our model requires only the existence of modulatory feedback connections between areas, and short-range lateral inhibition within each area. Feedback connections redistribute the top-down modulation to lower areas, which in turn alters the inputs of other higher-area cells, including those that did not receive the initial modulation. This produces firing rate modulations and receptive field shifts. Simultaneously, short-range lateral inhibition between neighboring cells produce competitive effects that are automatically scaled to receptive field size in any given area. Our model reproduces the observed attentional effects on response rates (response gain, input gain, biased competition automatically scaled to receptive field size) and receptive field structure (shifts and resizing of receptive fields both spatially and in complex feature space), without modifying model parameters. Our model also makes the novel prediction that attentional effects on response curves should shift from response gain to contrast gain as the spatial focus of attention drifts away from the studied cell.

Attention plays an important role in natural vision. Implementing realistic attentional processes in artificial vision systems could greatly improve their performance. However, existing models of attention do not adequately capture all of its complex effects on neural activity. In particular, existing models cannot reproduce recently reported effects such as shifting and scaling of receptive fields. Here we suggest that many of these effects arise naturally from feedback connections between visual areas (which redistribute top-down attentional modulation) and local, nonspecific short-range inhibition (which produce competition between stimuli that is automatically scaled to receptive field size). We show that a simple model with two reciprocally connected layers and short-range inhibition can generate many known effects of attention, including receptive field shift and resizing. Due to its conceptual simplicity, the model may be easily integrated into a broad range of computer vision systems.

It is often suggested that traditional models of artificial evolution, based on explicit, human-defined fitness functions, are fundamentally more restricted and less creative than natural evolution, in which no such constraint exists. After a discussion and refinement of this statement, we suggest a classification of evolutionary systems according to their evolutionary ldquocreativityrdquo. We describe an environment, called Evosphere, in which a population of 3D creatures interact, fight with each other, and evolve freely on the surface of a ldquomicroplanetrdquo. We demonstrate the onset of natural selection and adaptive evolution within this virtual world, both by visual inspection and statistical analysis. We show that the introduction of reproductively isolated species enriches the dynamics of the system, leading to simple evolutionary feedbacks among species.

We attempt to provide a comprehensive answer to the question of whether, and when, an arrow of complexity emerges in Darwinian evolution. We note that this expression can be interpreted in different ways, including a passive, incidental growth, or a pervasive bias towards complexification. We argue at length that an arrow of complexity does indeed occur in evolution, which can be most reasonably interpreted as the result of a passive trend rather than a driven one. What, then, is the role of evolution in the creation of this trend, and under which conditions will it emerge? In the later sections of this article we point out that when certain proper conditions (which we attempt to formulate in a concise form) are met, Darwinian evolution predictably creates a sustained trend of increase in maximum complexity (that is, an arrow of complexity) that would not be possible without it; but if they are not, evolution will not only fail to produce an arrow of complexity, but may actually prevent any increase in complexity altogether. We conclude that, with regard to the growth of complexity, evolution is very much a double-edged sword.

We introduce the N-strikes-out algorithm, a simple steady-state genetic algorithm for competitive coevolution. The algorithm can be summarised as follows: Run competitions between randomly chosen individuals, keep track of the number of defeats for each individual, and remove any individual which has been defeated N times. Naive application of the algorithm in 2-population problems leads to severe disengagement. We find that disengagement can be eliminated (for all tasks involving real-valued continuous scores) by determining 'victories' and 'defeats' between fellow members of the same species, using competitions against a single member of the opposing species as a point of comparison. We apply our algorithm to the "box-grabbing" problem for artificial 3D creatures introduced by Sims. We compare our algorithm with Sims' original Last Elite Opponent algorithm, and describe (and explain) different results obtained with two different implementations differing mainly by the harshness of their selection regimes.

We present the results of our replication of Karl Sims' work on the evolution of artificial creatures in a physically realistic 3D environment. We used standard McCulloch-Pitts neurons instead of a more complex set of ad hoc neurons, which we believe makes our model a more general tool for future experiments in artificial (co-)evolution. We provide a detailed description of our model and freely accessible source code. We describe our results both qualitatively and quantitatively, including an analysis of some evolved neural controllers. To the best of our knowledge, our work is the first replication of Sims' efforts to achieve results comparable to Sims' in efficiency and complexity, with standard neurons and realistic Newtonian physics