In the absence of external guidance, how can a robot learn to map the many raw pixels of high-dimensional visual inputs to useful action sequences? We propose here Continual Curiosity driven Skill Acquisition (CCSA). CCSA makes robots intrinsically motivated to acquire, store and reuse skills. Previous curiosity-based agents acquired skills by associating intrinsic rewards with world model improvements, and used reinforcement learning to learn how to get these intrinsic rewards. CCSA also does this, but unlike previous implementations, the world model is a set of compact low-dimensional representations of the streams of high-dimensional visual information, which are learned through incremental slow feature analysis. These representations augment the robot's state space with new information about the environment. We show how this information can have a higher-level (compared to pixels) and useful interpretation, for example, if the robot has grasped a cup in its field of view or not. After learning a representation, large intrinsic rewards are given to the robot for performing actions that greatly change the feature output, which has the tendency otherwise to change slowly in time. We show empirically what these actions are (e.g., grasping the cup) and how they can be useful as skills. An acquired skill includes both the learned actions and the learned slow feature representation. Skills are stored and reused to generate new observations, enabling continual acquisition of complex skills. We present results of experiments with an iCub humanoid robot that uses CCSA to incrementally acquire skills to topple, grasp and pick-place a cup, driven by its intrinsic motivation from raw pixel vision.

Consider a self-motivated artificial agent who is exploring a complex environment. Part of the complexity is due to the raw high-dimensional sensory input streams, which the agent needs to make sense of. Such inputs can be compactly encoded through a variety of means; one of these is slow feature analysis (SFA). Slow features encode spatiotemporal regularities, which are information-rich explanatory factors (latent variables) underlying the high-dimensional input streams. In our previous work, we have shown how slow features can be learned incrementally, while the agent explores its world, and modularly, such that different sets of features are learned for different parts of the environment (since a single set of regularities does not explain everything). In what order should the agent explore the different parts of the environment? Following Schmidhuber's theory of artificial curiosity, the agent should always concentrate on the area where it can learn the easiest-to-learn set of features that it has not already learned. We formalize this learning problem and theoretically show that, using our model, called curiosity-driven modular incremental slow feature analysis, the agent on average will learn slow feature representations in order of increasing learning difficulty, under certain mild conditions. We provide experimental results to support the theoretical analysis.

How can a humanoid robot autonomously learn and refine multiple sensorimotor skills as a byproduct of curiosity driven exploration, upon its high-dimensional unprocessed visual input? We present SKILLABILITY, which makes this possible. It combines the recently introduced Curiosity Driven Modular Incremental Slow Feature Analysis (Curious Dr. MISFA) with the well-known options framework. Curious Dr. MISFA's objective is to acquire abstractions as quickly as possible. These abstractions map high-dimensional pixel-level vision to a low-dimensional manifold. We find that each learnable abstraction augments the robot's state space (a set of poses) with new information about the environment, for example, when the robot is grasping a cup. The abstraction is a function on an image, called a slow feature, which can effectively discretize a high-dimensional visual sequence. For example, it maps the sequence of the robot watching its arm as it moves around, grasping randomly, then grasping a cup, and moving around some more while holding the cup, into a step function having two outputs: when the cup is or is not currently grasped. The new state space includes this grasped/not grasped information. Each abstraction is coupled with an option. The reward function for the option's policy (learned through Least Squares Policy Iteration) is high for transitions that produce a large change in the step-functionlike slow features. This corresponds to finding bottleneck states, which are known good subgoals for hierarchical reinforcement learning - in the example, the subgoal corresponds to grasping the cup. The final skill includes both the learned policy and the learned abstraction. SKILLABILITY makes our iCub the first humanoid robot to learn complex skills such as to topple or grasp an object, from raw high-dimensional video input, driven purely by its intrinsic motivations.

We introduce a novel algorithm called upper confidence - weighted learning (UCWL) for online multiclass learning from binary feedback (e.g., feedback that indicates whether the prediction was right or wrong). UCWL combines the upper confidence bound (UCB) framework with the soft confidence-weighted (SCW) online learning scheme. In UCB, each instance is classified using both score and uncertainty. For a given instance in the sequence, the algorithm might guess its class label primarily to reduce the class uncertainty. This is a form of informed exploration, which enables the performance to improve with lower sample complexity compared to the case without exploration. Combining UCB with SCW leads to the ability to deal well with noisy and nonseparable data, and state-of-the-art performance is achieved without increasing the computational cost. A potential application setting is human-robot interaction (HRI), where the robot is learning to classify some set of inputs while the human teaches it by providing only binary feedback—or sometimes even the wrong answer entirely. Experimental results in the HRI setting and with two benchmark datasets from other settings show that UCWL outperforms other state-of-the-art algorithms in the online binary feedback setting—and surprisingly even sometimes outperforms state-of-the-art algorithms that get full feedback (e.g., the true class label), whereas UCWL gets only binary feedback on the same data sequence.

We introduce a dynamic neural algorithm called Dynamic Neural (DN) SARSA(λ) for learning a behavioral sequence from delayed reward. DN-SARSA(λ) combines Dynamic Field Theory models of behavioral sequence representation, classical reinforcement learning, and a computational neuroscience model of working memory, called Item and Order working memory, which serves as an eligibility trace. DN-SARSA(λ) is implemented on both a simulated and real robot that must learn a specific rewarding sequence of elementary behaviors from exploration. Results show DN-SARSA(λ) performs on the level of the discrete SARSA(λ), validating the feasibility of general reinforcement learning without compromising neural dynamics.

Artificial curiosity tries to maximize learning progress. We apply this concept to a physical system. Our Katana robot arm curiously plays with wooden blocks, using vision, reaching, and grasping. It is intrinsically motivated to explore its world. As a by-product, it learns how to place blocks stably, and how to stack blocks.

We introduce here an incremental version of slow feature analysis (IncSFA), combining candid covariance-free incremental principal components analysis (CCIPCA) and covariance-free incremental minor components analysis (CIMCA). IncSFA's feature updating complexity is linear with respect to the input dimensionality, while batch SFA's (BSFA) updating complexity is cubic. IncSFA does not need to store, or even compute, any covariance matrices. The drawback to IncSFA is data efficiency: it does not use each data point as effectively as BSFA. But IncSFA allows SFA to be tractably applied, with just a few parameters, directly on high-dimensional input streams (e.g., visual input of an autonomous agent), while BSFA has to resort to hierarchical receptive-field-based architectures when the input dimension is too high. Further, IncSFA's updates have simple Hebbian and anti-Hebbian forms, extending the biological plausibility of SFA. Experimental results show IncSFA learns the same set of features as BSFA and can handle a few cases where BSFA fails.

In this paper, we propose an object learning system that incorporates sensory information from an automotive radar system and a video camera. The radar system provides coarse attention for the focus of visual analysis on relatively small areas within the image plane. The attended visual areas are coded and learned by a three-layer neural network utilizing what is called in-place learning: Each neuron is responsible for the learning of its own processing characteristics within the connected network environment, through inhibitory and excitatory connections with other neurons. The modeled bottom-up, lateral, and top-down connections in the network enable sensory sparse coding, unsupervised learning, and supervised learning to occur concurrently. This paper is applied to learn two types of encountered objects in multiple outdoor driving settings. Cross-validation results show that the overall recognition accuracy is above 95% for the radar-attended window images. In comparison with the uncoded representation and purely unsupervised learning (without top-down connection), the proposed network improves the overall recognition rate by 15.93% and 6.35%, respectively. The proposed system is also compared favorably with other learning algorithms. The result indicates that our learning system is the only one that is fit for incremental and online object learning in a real-time driving environment.

The Where-What Network 3 (WWN-3) is an artificial developmental network modeled after visual cortical pathways, for the purpose of attention and recognition in the presence of complex natural backgrounds. It is general-purpose and not pre-determined to detect a certain type of stimulus. It is a learning network, which develops its weights from images using a supervised paradigm and a local Hebbian learning algorithm. Attention has been thought of as bottom-up or top-down. This paper focuses on the biologically-inspired mechanisms of top-down attention in WWN-3, through top-down excitation that interacts with bottom-up activity at every layer within the network. Top-down excitation in WWN-3 can control the location of attention by imposing a certain location or disengaging from the current location. It can also control what type of object to search for. Paired layers and sparse coding deal with potential hallucination problems. Top-down attention in WWN occurs as soon as an action emerges at a motor layer, which could be imposed by a teacher or internally selected. Given two competing foregrounds in the same scene, WWN showed effective performance in all the attention modes tested.

A pathway in the central nervous system (CNS) is a path through which nervous signals are processed in an orderly fashion. A sensorimotor pathway starts from a sensory input and ends at a motor output, although almost all pathways are not simply unidirectional. In this paper, we introduce a simple, biologically inspired, unified computational model - Multi-layer In-place Learning Network (MILN), with a design goal to develop a recurrent network, as a function of sensorimotor signals, for open-ended learning of multiple sensorimotor tasks. The biologically motivated MILN provides automatic feature derivation and pathway refinement from the temporally real-time inputs. The work presented here is applied in the challenging application field of developing reactive behaviors from a video camera and a (noisy) radar range sensor for a vehicle-based robot in open, natural driving environments. An internal model of the agent’s experience of the environments is created and refined from the ground-up using a cell-centered model, based on the genomic equivalence principle. The outputs can be imposed by a teacher, at the same time as the learning is active. At any time instant, sensory information from the radar allows the system to focus its visual analysis on relatively small areas within the image plane (attention selection), in a computationally efficient way, suitable for real-time training. This system was trained with data from 10 different city and highway road environments, and cross validation shows that MILN was able to correctly recognize above 95% of the radar-extracted images from the multiple environments. The in-place learning mechanism compares with other learning algorithms favorably, as results of a comparison indicate that in-place learning is the only one to fit all the specified criteria of development of a general-purpose sensorimotor pathway.

The cerebral cortex uses a large number of top-down connections, but the roles of the top-down connections remain unclear. Through end-to-end (sensor-to-motor) multilayered networks that use three types of connections (bottom-up, lateral, and top-down), the new Topographic Class Grouping (TCG) mechanism shown in this paper explains how the top-down connections influence (1) the type of feature detectors (neurons) developed and (2) their placement in the neuronal plane. The top-down connections boost the variations in the neuronal between class directions during the training phase. The first outcome of this top-down boosted input space is the facilitation of the emergence of feature detectors that are purer, measured statistically by the average entropy of the neurons’ development. The relatively purer neurons are more “abstract,” i.e., characterizing class-specific (or motor-specific) input information, resulting in better classification rates. The second outcome of this top-down boosted input space is the increase of the distance between input samples that belong to different classes, resulting in a farther separation of neurons according to their class. Therefore, neurons that respond to the same class become relatively nearer. This results in TCG, measured statistically by a smaller within-class scatter of responses when the neuronal plane has a fixed size. Although these mechanisms are potentially applicable to any pattern recognition applications, we report quantitative effects of these mechanisms for 3D object recognition of center-normalized, background-controlled objects. TCG has enabled a significant reduction of the recognition errors.