How does our motor system solve the problem of anticipatory control in spite of a wide spectrum of response dynamics from different musculo-skeletal systems, transport delays as well as response latencies throughout the central nervous system? To a great extent, our highly-skilled motor responses are a result of a reactive feedback system, originating in the brain-stem and spinal cord, combined with a feed-forward anticipatory system, that is adaptively fine-tuned by sensory experience and originates in the cerebellum. Based on that interaction we design the counterfactual predictive control (CFPC) architecture, an anticipatory adaptive motor control scheme in which a feed-forward module, based on the cerebellum, steers an error feedback controller with counterfactual error signals. Those are signals that trigger reactions as actual errors would, but that do not code for any current of forthcoming errors. In order to determine the optimal learning strategy, we derive a novel learning rule for the feed-forward module that involves an eligibility trace and operates at the synaptic level. In particular, our eligibility trace provides a mechanism beyond co-incidence detection in that it convolves a history of prior synaptic inputs with error signals. In the context of cerebellar physiology, this solution implies that Purkinje cell synapses should generate eligibility traces using a forward model of the system being controlled. From an engineering perspective, CFPC provides a general-purpose anticipatory control architecture equipped with a learning rule that exploits the full dynamics of the closed-loop system.

This letter offers a computational account of Pavlovian conditioning in the cerebellum based on active inference and predictive coding. Using eyeblink conditioning as a canonical paradigm, we formulate a minimal generative model that can account for spontaneous blinking, startle responses, and (delay or trace) conditioning. We then establish the face validity of the model using simulated responses to unconditioned and conditioned stimuli to reproduce the sorts of behavior that are observed empirically. The scheme's anatomical validity is then addressed by associating variables in the predictive coding scheme with nuclei and neuronal populations to match the (extrinsic and intrinsic) connectivity of the cerebellar (eyeblink conditioning) system. Finally, we try to establish predictive validity by reproducing selective failures of delay conditioning, trace conditioning, and extinction using (simulated and reversible) focal lesions. Although rather metaphorical, the ensuing scheme can account for a remarkable range of anatomical and neurophysiological aspects of cerebellar circuitry-and the specificity of lesion-deficit mappings that have been established experimentally. From a computational perspective, this work shows how conditioning or learning can be formulated in terms of minimizing variational free energy (or maximizing Bayesian model evidence) using exactly the same principles that underlie predictive coding in perception.

The cerebellum is a brain structure necessary for skilled motor behaviour and has a well understood and repetitive architecture. Such an architecture inspired the Marr-Albus-Ito theory of cerebellar learning, that provides an explanation for the acquisition of motor skills by the cerebellum. Numerous computational models inspired in such a theory have already been employed in robotic tasks. Here we look into one of the suggested roles of the cerebellum, the replacement of reflexes by anticipatory actions and we apply it to a robot navigation task. The acquisition of anticipatory actions has been thoroughly studied in the field of classical conditioning. Of particular interest is the so-called CS-intensity effect, an effect that links the rapidity of execution of an anticipatory protective action, the Conditioned Response (CR), to the intensity of a predictive signal, the Conditioning Stimulus (CS). We propose that the CS-intensity effect implements a built-in sensory-motor contingency that allows to carry over a skill learned in a safe and easy context, e.g., turning at slow velocity, to a more difficult one, e.g., a turning at a faster speed. We demonstrate this hypothesis in a series of experiments where a robot has to navigate a track that has a turn. We show that after being trained at a slow velocity, by means of the CS-intensity effect, the cerebellar controller modulates the turning such that its onset anticipates as the robot speed increases. Ultimately, through incremental learning, this generalization allows the robot to learn to navigate the track at its maximum speed.

According to Konorski's two phase theory of conditioning the associative processes underlying classical conditioning can be separated into a fast valence driven nonspecific learning systems (NLS) and a slow specific learning system (SLS). The theory states that the NLS elicits a non-specific state of arousal and that the SLS is responsible for the exact elicitation of a coordinated motor response. Based on biological evidence we propose the amygdala, the basal forebrain and the auditory cortex as an example of NLS and the cerebellum for the SLS. The performance of the model was tested applying the eye-blink paradigm of classical conditioning. The unconditioned stimulus induced amygdala stimulation of the nucleus basalis elicits plasticity in the NLS. This leads to an increased representation if the conditioned stimulus in the cortex. The plasticity of the cerebellar SLS is regulated by these increased cortical representation coding the behavioral importance of the conditioned stimulus. Here we provide a complete account of Konorskis proposal by integrating these two systems into a complete biologically-grounded computational model of the two-phase theory of classical conditioning.