Humans learn to recognize categories of objects, even when exposed to little language supervision. Behavioral studies and the successes of self-supervised learning (SSL) models suggest that this learning may hinge on modeling spatial regularities of visual features. However, SSL models rely on geometric image augmentations such as masking portions of an image or aggressively cropping it, which are not known to be performed by the brain. Here, we propose CO-SSL, an alternative to geometric image augmentations to model spatial co-occurrences. CO-SSL aligns local representations (before pooling) with a global image representation. Combined with a neural network endowed with small receptive fields, we show that it outperforms previous methods by up to on ImageNet-1k when not using cropping augmentations. In addition, CO-SSL can be combined with cropping image augmentations to accelerate category learning and increases the robustness to internal corruptions and small adversarial attacks. Overall, our work paves the way towards a new approach for modeling biological learning and developing self-supervised representations in artificial systems.

We present Hierarchical Residual Networks (HiResNets), deep convolutional neural networks with long-range residual connections between layers at different hierarchical levels.HiResNets draw inspiration on the organization of the mammalian brain by replicating the direct connections from subcortical areas to the entire cortical hierarchy.We show that the inclusion of hierarchical residuals in several architectures, including ResNets, results in a boost in accuracy and faster learning.A detailed analysis of our models reveals that they perform hierarchical compositionality by learning feature maps relative to the compressed representations provided by the skip connections.

Cortical networks are capable of unsupervised learning and spontaneous replay of complex temporal sequences. Endowing artificial spiking neural networks with similar learning abilities remains a challenge. In particular, it is unresolved how different plasticity rules can contribute to both learning and the maintenance of network stability during learning. Here we introduce a biologically inspired form of GABA-Modulated Spike Timing-Dependent Plasticity (GMS) and demonstrate its ability to permit stable learning of complex temporal sequences including natural language in recurrent spiking neural networks. Motivated by biological findings, GMS utilizes the momentary level of inhibition onto excitatory cells to adjust both the magnitude and sign of Spike Timing-Dependent Plasticity (STDP) of connections between excitatory cells. In particular, high levels of inhibition in the network cause depression of excitatory-to-excitatory connections. We demonstrate the effectiveness of this mechanism during several sequence learning experiments with character- and token-based text inputs as well as visual input sequences. We show that GMS maintains stability during learning and spontaneous replay and permits the network to form a clustered hierarchical representation of its input sequences. Overall, we provide a biologically inspired model of unsupervised learning of complex sequences in recurrent spiking neural networks.

Large-scale screening programs for vision impairments can incur substantial costs. Computer-based screening methods, which combine different measurements within a single system, can facilitate and reduce the costs of such programs. Here, we present a virtual reality (VR) software, which includes tests for the assessment of visual acuity, stereoacuity, eye misalignments, and interocular suppression as well as games targeting different visual functions that may serve as treatment methods. The software can be easily extended to incorporate new tests and games. We present a proof of concept demonstrating the functionality of the software and its applicability in individuals with impaired binocularity. We evaluate a stereoacuity test in VR based on disparity detection using contoured objects by comparing its results to those obtained by standard clinical tests, i.e., TNO, Randot, and Titmus, for 7 amblyopes and 6 healthy controls. We evaluate the applicability of a new VR-based suppression test in 10 amblyopes and 6 healthy individuals. For the latter, we exploit the effects of short-term monocular deprivation, which induce a change of ocular dominance. Finally, we outline technical limitations and discuss potential applications.

The vision transformer (ViT) utilizes a learnable position embedding (PE) to encode the location of an image patch.However, it is unclear if this learnable PE is vital and what its benefits are.This paper explores an alternative way of encoding patch locations that exploits prior knowledge about their spatial arrangement called circle relationship embedding (CRE).CRE considers the distance of image patches from the central patch based on the four-neighborhood to simplify the PE.Our experiments show that combining CRE with PE achieves better performance than using PE alone.The code for this paper can be downloaded at: https://github.com/trieschlab/CRE.

The electrical and computational properties of neurons in our brains are determined by a rich repertoire of membrane-spanning ion channels and elaborate dendritic trees. However, the precise reason for this inherent complexity remains unknown, given that simpler models with fewer ion channels are also able to functionally reproduce the behaviour of some neurons. Here, we stochastically varied the ion channel densities of a biophysically detailed dentate gyrus granule cell model to produce a large population of putative granule cells, comparing those with all 15 original ion channels to their reduced but functional counterparts containing only 5 ion channels. Strikingly, valid parameter combinations in the full models were dramatically more frequent at ~6% vs. ~1% in the simpler model. The full models were also more stable in the face of perturbations to channel expression levels. Scaling up the numbers of ion channels artificially in the reduced models recovered these advantages confirming the key contribution of the actual number of ion channel types. We conclude that the diversity of ion channels gives a neuron greater flexibility and robustness to achieve a target excitability.

The application of Deep Learning (DL) for medical diagnosis is often hampered by two problems. First, the amount of training data may be scarce, as it is limited by the number of patients who have acquired the condition. Second, the training data may be corrupted by various types of noise. Here, we study the problem of brain tumor detection from magnetic resonance spectroscopy (MRS) data, where both types of problems are prominent. To overcome these challenges, we propose a new method for training a deep neural network that distills particularly representative training examples and augments the training data by mixing these samples from one class with those from the same and other classes to create additional training samples. We demonstrate that this technique substantially improves performance, allowing our method to achieve human-expert-level accuracy with just a few thousand training examples.

Animals utilize a variety of active sensing mechanisms to perceive the world around them. Echolocating bats are an excellent model for the study of active auditory localization. The big brown bat (Eptesicus fuscus), for instance, employs active head roll movements during sonar prey tracking. The function of head rolls in sound source localization is not well understood. Here, we propose an echolocation model with multi-axis head rotation to investigate the effect of active head roll movements on sound localization performance. The model autonomously learns to align the bat's head direction towards the target. We show that a model with active head roll movements better localizes targets than a model without head rolls. Furthermore, we demonstrate that active head rolls also reduce the time required for localization in elevation. Finally, our model offers key insights to sound localization cues used by echolocating bats employing active head movements during echolocation.

Learning complex behaviors through reinforcement learning is particularly challenging when reward is only available upon successful completion of the full behavior. In manipulation robotics, so-called shaping rewards are often used to overcome this problem. However, these usually require human engineering or (partial)world models describing, e.g., the kinematics of the robot or high-level modules for perception. Here we propose an alternative method to learn an object palm-touching task through a weakly-supervised and stagewise learning of simpler tasks. First, the robot learns to fixate the object with its cameras. Second, the robot learns eye-hand coordination by learning to fixate its end effector. Third, using the previously acquired skills an informative shaping reward can be computed which facilitates efficient learning of the object palm-touching task. We demonstrate in simulation that learning the full task with this shaping reward is comparable to learning with an informative supervised reward.

Recent experiments have demonstrated that visual cortex engages in spatio-temporal sequence learning and prediction. The cellular basis of this learning remains unclear, however. Here we present a spiking neural network model that explains a recent study on sequence learning in the primary visual cortex of rats. The model posits that the sequence learning and prediction abilities of cortical circuits result from the interaction of spike-timing dependent plasticity (STDP) and homeostatic plasticity mechanisms. It also reproduces changes in stimulus-evoked multi-unit activity during learning. Furthermore, it makes precise predictions regarding how training shapes network connectivity to establish its prediction ability. Finally, it predicts that the adapted connectivity gives rise to systematic changes in spontaneous network activity. Taken together, our model establishes a new conceptual bridge between the structure and function of cortical circuits in the context of sequence learning and prediction.

Developmental robotics seeks to build robots that learn to interact with the environment largely autonomously. These robots can calibrate their sensorimotor competencies on their own, much like developing children. In this paper, we build a developmental model of image stabilization based on the active efficient coding (AEC) framework and apply the model to a real robotic platform. In the visual system of primates, the optokinetic response (OKR) and the vestibulo-ocular reflex (VOR) cooperate to ensure image stabilization during relative motion between the observer and the environment. Inspired by these biological findings, our model integrates visual, inertial and motor encoder sensory cues. The sensory processing and the motor policy co-develop. The visual processing is based on a sparse coding algorithm. Motor behavior is learned using reinforcement learning. Our results show that the stabilization performance is improved by integrating visual and inertial inputs. Importantly, the weighting between the two inputs is learned automatically as the robot interacts with the environment.

Due to its ability to learn complex behaviors in high-dimensional state-action spaces, deep reinforcement learning algorithms have attracted much interest in the robotics community. For a practical reinforcement learning implementation on a robot, it has to be provided with an informative reward signal that makes it easy to discriminate the values of nearby states. To address this issue, prior information, e.g. in the form of a geometric model, or human supervision are often assumed. This paper proposes a method to learn binocular fixations without such prior information. Instead, it uses an informative reward requiring little supervised information. The reward computation is based on an anomaly detection mechanism which uses convolutional autoencoders. These detectors estimate in a weakly supervised way an object's pixellic position. This position estimate is affected by noise, which makes the reward signal noisy. We first show that this affects both the learning speed and the resulting policy. Then, we propose a method to partially remove the noise using regression on the detection change given sensor data. The binocular fixation task is learned in a simulated environment on an object training set with various shapes and colors. The learned policy is compared with another one learned with a highly informative and noiseless reward signal. The tests are carried out on the training set and on a test set of new objects. We observe similar performances, showing that the environment-encoding step can replace the prior information.

Humans learn multisensory eye-hand coordination starting from infancy without supervision. For an example, they learn to track their hands by exploiting various sensory modalities, such as vision and proprioception. This integration occurs as they learn to perceive the world around them and their relationship to it. Most prior work has focused on the role of vision, as it is a primary sensory source for humans. However, it is interesting to study how vision and proprioception interact. We propose a system which combines visual and proprioceptive information to learn the eye-hand coordination skills that enable a robot to fixate its camera gaze on the end effector of its arm. In our model, visual cues are part of the feedback control loop, whereas proprioceptive cues are part of a feedforward control loop. Both controllers, as well as the sensory transform from raw visual information to a neural sensory representation are learned as the robot performs motor babbling movements. Visual information is encoded by sparse coding. The basis functions that emerge are similar to the receptive fields in the human visual cortex. An actor-critic reinforcement learning algorithm is used to drive eye motor neurons fusing visual and proprioceptive cues. We model and test the system in the iCub simulation environment. Our results suggest that these sensory modalities are capable of jointly learning model parameters to perform the tracking task. The evolved policy has characteristics that are qualitatively similar to the human oculomotor plant.

The ability to learn sequential behaviors is a fundamental property of our brains. Yet a long stream of studies including recent experiments investigating motor sequence learning in adult human subjects have produced a number of puzzling and seemingly contradictory results. In particular, when subjects have to learn multiple action sequences, learning is sometimes impaired by proactive and retroactive interference effects. In other situations, however, learning is accelerated as reflected in facilitation and transfer effects. At present it is unclear what the underlying neural mechanism are that give rise to these diverse findings. Here we show that a recently developed recurrent neural network model readily reproduces this diverse set of findings. The self-organizing recurrent neural network (SORN) model is a network of recurrently connected threshold units that combines a simplified form of spike-timing dependent plasticity (STDP) with homeostatic plasticity mechanisms ensuring network stability, namely intrinsic plasticity (IP) and synaptic normalization (SN). When trained on sequence learning tasks modeled after recent experiments we find that it reproduces the full range of interference, facilitation, and transfer effects. We show how these effects are rooted in the network's changing internal representation of the different sequences across learning and how they depend on an interaction of training schedule and task similarity. Furthermore, since learning in the model is based on fundamental neuronal plasticity mechanisms, the model reveals how these plasticity mechanisms are ultimately responsible for the network's sequence learning abilities. In particular, we find that all three plasticity mechanisms are essential for the network to learn effective internal models of the different training sequences. This ability to form effective internal models is also the basis for the observed interference and facilitation effects. This suggests that STDP, IP, and SN may be the driving forces behind our ability to learn complex action sequences.

Understanding the structure and dynamics of cortical connectivity is vital to understanding cortical function. Experimental data strongly suggest that local recurrent connectivity in the cortex is significantly non-random, exhibiting, for example, above-chance bidirectionality and an overrepresentation of certain triangular motifs. Additional evidence suggests a significant distance dependency to connectivity over a local scale of a few hundred microns, and particular patterns of synaptic turnover dynamics, including a heavy-tailed distribution of synaptic efficacies, a power law distribution of synaptic lifetimes, and a tendency for stronger synapses to be more stable over time. Understanding how many of these non-random features simultaneously arise would provide valuable insights into the development and function of the cortex. While previous work has modeled some of the individual features of local cortical wiring, there is no model that begins to comprehensively account for all of them. We present a spiking network model of a rodent Layer 5 cortical slice which, via the interactions of a few simple biologically motivated intrinsic, synaptic, and structural plasticity mechanisms, qualitatively reproduces these non-random effects when combined with simple topological constraints. Our model suggests that mechanisms of self-organization arising from a small number of plasticity rules provide a parsimonious explanation for numerous experimentally observed non-random features of recurrent cortical wiring. Interestingly, similar mechanisms have been shown to endow recurrent networks with powerful learning abilities, suggesting that these mechanism are central to understanding both structure and function of cortical synaptic wiring.

A number of unsupervised learning algorithms seeking to account for the receptive field properties of simple cells in the mammalian primary visual cortex have been proposed. Among these are principal component analysis and sparse coding. While it appears that the receptive field properties learned by sparse coding match those measured in cortical cells better than those learned by principal component analysis, it is still not clear why biological neural systems might prefer to use sparse codes. In this paper we explore another reason why sparse representations might be preferred over principal component analysis by studying the utility of different coding schemes in an adaptive behaving agent. We suggest that the qualitative properties of representations based on sparse coding are more stable in the presence of changes in the input statistics than those of representations based on principal component analysis. We demonstrate this by examining representations learned on binocular visual input with different disparity distributions. Our results show that in encoding retinal disparity, the properties of sparse codes are more stable, and that this has important implications in adaptive agents, where the statistics change over time. In particular, in an agent who jointly learns a representation for binocular visual inputs along with a vergence control policy, the learned behavior is unstable when actions are driven by PCA based representations, but stable and self-calibrating when driven by sparse coding based representations.

Even in the absence of sensory stimulation the brain is spontaneously active. This background "noise" seems to be the dominant cause of the notoriously high trial-to-trial variability of neural recordings. Recent experimental observations have extended our knowledge of trial-to-trial variability and spontaneous activity in several directions: 1. Trial-to-trial variability systematically decreases following the onset of a sensory stimulus or the start of a motor act. 2. Spontaneous activity states in sensory cortex outline the region of evoked sensory responses. 3. Across development, spontaneous activity aligns itself with typical evoked activity patterns. 4. The spontaneous brain activity prior to the presentation of an ambiguous stimulus predicts how the stimulus will be interpreted. At present it is unclear how these observations relate to each other and how they arise in cortical circuits. Here we demonstrate that all of these phenomena can be accounted for by a deterministic self-organizing recurrent neural network model (SORN), which learns a predictive model of its sensory environment. The SORN comprises recurrently coupled populations of excitatory and inhibitory threshold units and learns via a combination of spike-timing dependent plasticity (STDP) and homeostatic plasticity mechanisms. Similar to balanced network architectures, units in the network show irregular activity and variable responses to inputs. Additionally, however, the SORN exhibits sequence learning abilities matching recent findings from visual cortex and the network's spontaneous activity reproduces the experimental findings mentioned above. Intriguingly, the network's behaviour is reminiscent of sampling-based probabilistic inference, suggesting that correlates of sampling-based inference can develop from the interaction of STDP and homeostasis in deterministic networks. We conclude that key observations on spontaneous brain activity and the variability of neural responses can be accounted for by a simple deterministic recurrent neural network which learns a predictive model of its sensory environment via a combination of generic neural plasticity mechanisms.

Developmental robots require cognitive structures that can learn perception-action cycles via interactions with the environment. Here, we extend the efficient coding hypothesis, which has been used to model the development of sensory processing in isolation, to model the development of the perception-action cycle. Our extension combines sparse coding and reinforcement learning so that sensory processing and behavior co-develop to optimize a shared intrinsic motivational signal: the fidelity of the neural encoding of the sensory input under resource constraints. Applying this framework to a model of a robot actively observing a time-varying environment leads to the simultaneous development of visual smooth pursuit behavior and model neurons similar to cortical neurons selective to visual motion. We suggest that this general principle may form the basis for a unified and integrated approach to learning many other perception/action loops.

The information processing abilities of neural circuits arise from their synaptic connection patterns. Understanding the laws governing these connectivity patterns is essential for understanding brain function. The overall distribution of synaptic strengths of local excitatory connections in cortex and hippocampus is long-tailed, exhibiting a small number of synaptic connections of very large efficacy. At the same time, new synaptic connections are constantly being created and individual synaptic connection strengths show substantial fluctuations across time. It remains unclear through what mechanisms these properties of neural circuits arise and how they contribute to learning and memory. In this study we show that fundamental characteristics of excitatory synaptic connections in cortex and hippocampus can be explained as a consequence of self-organization in a recurrent network combining spike-timing-dependent plasticity (STDP), structural plasticity and different forms of homeostatic plasticity. In the network, associative synaptic plasticity in the form of STDP induces a rich-get-richer dynamics among synapses, while homeostatic mechanisms induce competition. Under distinctly different initial conditions, the ensuing self-organization produces long-tailed synaptic strength distributions matching experimental findings. We show that this self-organization can take place with a purely additive STDP mechanism and that multiplicative weight dynamics emerge as a consequence of network interactions. The observed patterns of fluctuation of synaptic strengths, including elimination and generation of synaptic connections and long-term persistence of strong connections, are consistent with the dynamics of dendritic spines found in rat hippocampus. Beyond this, the model predicts an approximately power-law scaling of the lifetimes of newly established synaptic connection strengths during development. Our results suggest that the combined action of multiple forms of neuronal plasticity plays an essential role in the formation and maintenance of cortical circuits.

Various optimality principles have been proposed to explain the characteristics of coordinated eye and head movements during visual orienting behavior. At the same time, researchers have suggested several neural models to underly the generation of saccades, but these do not include online learning as a mechanism of optimization. Here, we suggest an open-loop neural controller with a local adaptation mechanism that minimizes a proposed cost function. Simulations show that the characteristics of coordinated eye and head movements generated by this model match the experimental data in many aspects, including the relationship between amplitude, duration and peak velocity in head-restrained and the relative contribution of eye and head to the total gaze shift in head-free conditions. Our model is a first step towards bringing together an optimality principle and an incremental local learning mechanism into a unified control scheme for coordinated eye and head movements.

Although models based on independent component analysis (ICA) have been successful in explaining various properties of sensory coding in the cortex, it remains unclear how networks of spiking neurons using realistic plasticity rules can realize such computation. Here, we propose a biologically plausible mechanism for ICA-like learning with spiking neurons. Our model combines spike-timing dependent plasticity and synaptic scaling with an intrinsic plasticity rule that regulates neuronal excitability to maximize information transmission. We show that a stochastically spiking neuron learns one independent component for inputs encoded either as rates or using spike-spike correlations. Furthermore, different independent components can be recovered, when the activity of different neurons is decorrelated by adaptive lateral inhibition.

Information theory predicts the need for information maximization as sensory information must be compressed into a limited range of responses that spiking neurons can generate. We propose computational theory and learning rules based on information theory that lead to information maximization using intrinsic plasticity in a stochastically spiking neuron model. Computer simulations are used to verify the theoretical results. Further experiments show that the intrinsic plasticity rules described in this article lead to a desired exponential output distribution, firing-rate homeostasis, and adaptation to sensory deprivation in our model as observed in cortical neurons.

Several studies have suggested different cost functions to explain the kinematic characteristics of saccades. However, these studies do not present any neural implementation of the optimization procedure they use. Instead, they are based on optimal control theory approaches that provide a global analytical solution rather than a local adaptation scheme. In this study, we propose a model comprised of an open-loop neural controller and an adaptation unit. The neural controller receives the initial target position as input. The adaptation unit, which is the neural interpretation of a simple cost function, evaluates the optimality of this controller and induces weight changes in the controller via a local learning rule. Realistic saccades are obtained with the proposed model. We speculate that the superior colliculus and the cerebellum behave quite similar to our model's neural controller and adaptation unit.

Only a subset of available sensory information is useful for decision making. Classical models of the brain's sensory system, such as generative models, consider all elements of the sensory stimuli. However, only the action-relevant components of stimuli need to reach the motor control and decision making structures in the brain. To learn these action-relevant stimuli, the part of the sensory system that feeds into a motor control circuit needs some kind of relevance feedback. We propose a simple network model consisting of a feature learning (sensory) layer that feeds into a reinforcement learning (action) layer. Feedback is established by the reinforcement learner's temporal difference (delta) term modulating an otherwise Hebbian-like learning rule of the feature learner. Under this influence, the feature learning network only learns the relevant features of the stimuli, i.e. those features on which goal-directed actions are to be based. With the input preprocessed in this manner, the reinforcement learner performs well in delayed reward tasks. The learning rule approximates an energy function's gradient descent. The model presents a link between reinforcement learning and unsupervised learning and may help to explain how the basal ganglia receive selective cortical input.

Current models for learning feature detectors work on two timescales: on a fast timescale, the internal neurons' activations adapt to the current stimulus; on a slow timescale, the weights adapt to the statistics of the set of stimuli. Here we explore the adaptation of a neuron's intrinsic excitability, termed intrinsic plasticity, which occurs on a separate timescale. Here, a neuron maintains homeostasis of an exponentially distributed firing rate in a dynamic environment. We exploit this in the context of a generative model to impose sparse coding. With natural image input, localized edge detectors emerge as models of V1 simple cells. An intermediate timescale for the intrinsic plasticity parameters allows modeling aftereffects. In the tilt aftereffect, after a viewer adapts to a grid of a certain orientation, grids of a nearby orientation will be perceived as tilted away from the adapted orientation. Our results show that adapting the neurons' gain-parameter but not the threshold-parameter accounts for this effect. It occurs because neurons coding for the adapting stimulus attenuate their gain, while others increase it. Despite its simplicity and low maintenance, the intrinsic plasticity model accounts for more experimental details than previous models without this mechanism.

We propose a model of intrinsic plasticity for a continuous activation model neuron based on information theory. We then show how intrinsic and synaptic plasticity mechanisms interact and allow the neuron to discover heavy-tailed directions in the input. We also demonstrate that intrinsic plasticity may be an alternative explanation for the sliding threshold postulated in the BCM theory of synaptic plasticity. We present a theoretical analysis of the interaction of intrinsic plasticity with different Hebbian learning rules for the case of clustered inputs. Finally, we perform experiments on the "bars" problem, a popular nonlinear independent component analysis problem.

We present a system for the automatic interpretation of cluttered scenes containing multiple partly occluded objects in front of unknown, complex backgrounds. The system is based on an extended elastic graph matching algorithm that allows the explicit modeling of partial occlusions. Our approach extends an earlier system in two ways. First, we use elastic graph matching in stereo image pairs to increase matching robustness and disambiguate occlusion relations. Second, we use richer feature descriptions in the object models by integrating shape and texture with color features. We demonstrate that the combination of both extensions substantially increases recognition performance. The system learns about new objects in a simple one-shot learning approach. Despite the lack of statistical information in the object models and the lack of an explicit background model, our system performs surprisingly well for this very difficult task. Our results underscore the advantages of view-based feature constellation representations for difficult object recognition problems.

We describe the design of an anthropomorphic robot head intended as a research platform for studying autonomously learning active vision systems. The robot head closely mimics the major degrees of freedom of the human neck/eye apparatus and allows a number of facial expressions. We show that our robot head can shift its direction of gaze at speeds which come close to that of human saccades. Since our design only makes use of low cost consumer grade components, it paves the way for widespread use of anthropomorphic robot heads in science, education, health-care, and entertainment.

This paper explores the computational consequences of simultaneous in- trinsic and synaptic plasticity in individual model neurons. It proposes a new intrinsic plasticity mechanism for a continuous activation model neuron based on low order moments of the neuron's firing rate distribu- tion. The goal of the intrinsic plasticity mechanism is to enforce a sparse distribution of the neuron's activity level. In conjunction with Hebbian learning at the neuron's synapses, the neuron is shown to discover sparse directions in the input.

Summary form only given. We present a simple computational model of the emergence of gaze following behavior in infant caregiver interactions. We regard gaze following as a skill that infants acquire because they learn that monitoring their caregiver's direction of gaze allows them to predict where interesting objects/events in their environment are (Moore, 1996). In particular, we propose a specific "basic set" of mechanisms that are sufficient for gaze following to emerge (Fasel and Deak, 2002). This basic set comprises perceptual and motivational biases and habituation mechanisms driving the infant to look at and shift attention between "interesting" visual stimuli, a generic learning mechanism that learns behavioral strategies to satisfy these preferences, and a structured environment providing correlations between where caregivers look and where interesting stimuli are. We formalize these ideas in a simple model based on temporal difference learning. We analyze the model and demonstrate that a) the proposed basic set of mechanisms is indeed sufficient for gaze following to emerge and b) alterations of parameters of some of the basic set mechanisms motivated by findings on developmental disorders lead to impairments in the learning of gaze following that are typical of these disorders.

Our knowledge of the way that the visual system operates in everyday behavior has, until recently, been very limited. This information is critical not only for understanding visual function, but also for understanding the consequences of various kinds of visual impairment, and for the development of interfaces between human and artificial systems. The development of eye trackers that can be mounted on the head now allows monitoring of gaze without restricting the observer's movements. Observations of natural behavior have demonstrated the highly task-specific and directed nature of fixation patterns, and reveal considerable regularity between observers. Eye, head, and hand coordination also reveals much greater flexibility and task-specificity than previously supposed. Experimental examination of the issues raised by observations of natural behavior requires the development of complex virtual environments that can be manipulated by the experimenter at critical points during task performance. Experiments where we monitored gaze in a simulated driving environment demonstrate that visibility of task relevant information depends critically on active search initiated by the observer according to an internally generated schedule, and this schedule depends on learnt regularities in the environment. In another virtual environment where observers copied toy models we showed that regularities in the spatial structure are used by observers to control eye movement targeting. Other experiments in a virtual environment with haptic feedback show that even simple visual properties like size are not continuously available or processed automatically by the visual system, but are dynamically acquired and discarded according to the momentary task demands.

We are interested in saccade contingent scene updates where the visual information presented in a display is altered while a saccadic eye movement of an unconstrained, freely moving observer is in progress. Since saccades typically last only several tens of milliseconds depending on their size, this poses dif cult constraints on the latency of detection. We have integrated two complementary eye trackers in a virtual reality helmet to simultaneously 1) detect saccade onsets with very low latency and 2) track the gaze with high precision albeit higher latency. In a series of experiments we demonstrate the system s capability of detecting saccade onsets with suf ciently low latency to make scene changes while a saccade is still progressing. While the method was developed to facilitate studies of human visual perception and attention, it may nd interesting applications in human-computer interaction and computer graphics.

Sensory integration or sensor fusion -- the integration of information from different modalities, cues, or sensors -- is among the most fundamental problems of perception in biological and artificial systems. We propose a new architecture for adaptively integrating different cues in a self-organized manner. In Democratic Integration different cues agree on a result, and each cue adapts toward the result agreed on. In particular, discordant cues are quickly suppressed and recalibrated, while cues having been consistent with the result in the recent past are given a higher weight in the future. The architecture is tested in a face tracking scenario. Experiments show its robustness with respect to sudden changes in the environment as long as the changes disrupt only a minority of cues at the same time, although all cues may be disrupted at one time or another.

A computer vision system for person-independent recognition of hand postures against complex backgrounds is presented. The system is based on the elastic graph matching, which was extended to allow for combinations of different feature types at the graph nodes.

A mechanism for the self-organized integration of different adaptive cues is proposed. In democratic integration the cues agree on a result and each cue adapts towards the result agreed upon. A technical formulation of this scheme is employed in a face tracking system. The self-organized adaptivity lends itself to suppression and recalibration of discordant cues. Experiments show that the system is robust to sudden changes in the environment as long as the changes disrupt only a minority of cues at the same time, although all cues may be affected in the long run.

A system for the classification of hand postures against complex backgrounds in grey-level images is presented. The system employs elastic graph matching, which has already been successfully employed for the recognition of faces. Our system reaches 86.2% correct classification on our gallery of 239 images of ten postures against complex backgrounds. The system is robust with respect to certain variations in size of hand and shape of posture.