Objective classification biomarkers that are developed using resting-state functional magnetic resonance imaging (rs-fMRI) data are expected to contribute to more effective treatment for psychiatric disorders. Unfortunately, no widely accepted biomarkers are available at present, partially because of the large variety of analysis pipelines for their development. In this study, we comprehensively evaluated analysis pipelines using a large-scale, multi-site fMRI dataset for major depressive disorder (MDD). We explored combinations of options in four sub-processes of the analysis pipelines: six types of brain parcellation, four types of functional connectivity (FC) estimations, three types of site-difference harmonization, and five types of machine-learning methods. A total of 360 different MDD classification biomarkers were constructed using the SRPBS dataset acquired with unified protocols (713 participants from four sites) as the discovery dataset, and datasets from other projects acquired with heterogeneous protocols (449 participants from four sites) were used for independent validation. We repeated the procedure after swapping the roles of the two datasets to identify superior pipelines, regardless of the discovery dataset. The classification results of the top 10 biomarkers showed high similarity, and weight similarity was observed between eight of the biomarkers, except for two that used both data-driven parcellation and FC computation. We applied the top 10 pipelines to the datasets of other psychiatric disorders (autism spectrum disorder and schizophrenia), and eight of the biomarkers exhibited sufficient classification performance for both disorders. Our results will be useful for establishing a standardized pipeline for classification biomarkers.

Our study explores how ecological aspects of motor learning enhance survival by improving movement efficiency and mitigating injury risks during task failures. Traditional motor control theories mainly address isolated body movements and often overlook these ecological factors. We introduce a novel computational motor control approach, incorporating ecological fitness and a strategy that alternates between success-driven movement efficiency and failure-driven safety, akin to win-stay/lose-shift tactics. In our experiments, participants performed squat-to-stand movements under novel force perturbations. They adapted effectively through various adaptive motor control mechanisms to avoid falls, reducing failure rates rapidly. The results indicate a high-level ecological controller in human motor learning that switches objectives between safety and movement efficiency, depending on failure or success. This approach is supported by policy learning, internal model adaptation, and adaptive feedback control. Our findings offer a comprehensive perspective on human motor control, integrating risk management in a hierarchical reinforcement learning framework for real-world environments.

Although the cerebellum is typically associated with supervised learning algorithms, it also exhibits extensive involvement in reward processing. In this study, we investigated the cerebellum's role in executing reinforcement learning algorithms, with a particular emphasis on essential reward-prediction errors. We employed the Q-learning model to accurately reproduce the licking responses of mice in a Go/No-go auditory-discrimination task. This method enabled the calculation of reinforcement learning variables, such as reward, predicted reward, and reward-prediction errors in each learning trial. Through tensor component analysis of two-photon Ca2+ imaging data from more than 6,000 Purkinje cells, we found that climbing fiber inputs of the two distinct components, which were specifically activated during Go and No-go cues in the learning process, showed an inverse relationship with predictive reward-prediction errors. Assuming bidirectional parallel-fiber Purkinje-cell synaptic plasticity, we constructed a cerebellar neural-network model with 5,000 spiking neurons of granule cells, Purkinje cells, cerebellar nuclei neurons, and inferior olive neurons. The network model qualitatively reproduced distinct changes in licking behaviors, climbing-fiber firing rates, and their synchronization during discrimination learning separately for Go/No-go conditions. We found that Purkinje cells in the two components could develop specific motor commands for their respective auditory cues, guided by the predictive reward-prediction errors from their climbing fiber inputs. These results indicate a possible role of context-specific actors in modular reinforcement learning, integrating with cerebellar supervised learning capabilities.

Humans have the amazing ability to learn the dynamics of the body and environment to develop motor skills. Traditional motor studies using arm reaching paradigms have viewed this ability as the process of 'internal model adaptation'. However, the behaviors have not been fully explored in the case when reaches fail to attain the intended target. Here we examined human reaching under two force fields types; one that induces failures (i.e., target errors), and the other that does not. Our results show the presence of a distinct failure-driven adaptation process that enables quick task success after failures, and before completion of internal model adaptation, but that can result in persistent changes to the undisturbed trajectory. These behaviors can be explained by considering a hierarchical interaction between internal model adaptation and the failure-driven adaptation of reach direction. Our findings suggest that movement failure is negotiated using hierarchical motor adaptations by humans.

The cerebellum is known to have an important role in sensing and execution of precise time intervals, but the mechanism by which arbitrary time intervals can be recognized and replicated with high precision is unknown. We propose a computational model in which precise time intervals can be identified from the pattern of individual spike activity in a population of parallel fibers in the cerebellar cortex. The model depends on the presence of repeatable sequences of spikes in response to conditioned stimulus input. We emulate granule cells using a population of Izhikevich neuron approximations driven by random but repeatable mossy fiber input. We emulate long-term depression (LTD) and long-term potentiation (LTP) synaptic plasticity at the parallel fiber to Purkinje cell synapse. We simulate a delay conditioning paradigm with a conditioned stimulus (CS) presented to the mossy fibers and an unconditioned stimulus (US) some time later issued to the Purkinje cells as a teaching signal. We show that Purkinje cells rapidly adapt to decrease firing probability following onset of the CS only at the interval for which the US had occurred. We suggest that detection of replicable spike patterns provides an accurate and easily learned timing structure that could be an important mechanism for behaviors that require identification and production of precise time intervals.

We previously proposed, on theoretical grounds, that the cerebellum must regulate the dimensionality of its neuronal activity during motor learning and control to cope with the low firing frequency of inferior olive neurons, which form one of two major inputs to the cerebellar cortex. Such dimensionality regulation is possible via modulation of electrical coupling through the gap junctions between inferior olive neurons by inhibitory GABAergic synapses. In addition, we previously showed in simulations that intermediate coupling strengths induce chaotic firing of inferior olive neurons and increase their information carrying capacity. However, there is no in vivo experimental data supporting these two theoretical predictions. Here, we computed the levels of synchrony, dimensionality, and chaos of the inferior olive code by analyzing in vivo recordings of Purkinje cell complex spike activity in three different coupling conditions: carbenoxolone (gap junctions blocker), control, and picrotoxin (GABA-A receptor antagonist). To examine the effect of electrical coupling on dimensionality and chaotic dynamics, we first determined the physiological range of effective coupling strengths between inferior olive neurons in the three conditions using a combination of a biophysical network model of the inferior olive and a novel Bayesian model averaging approach. We found that effective coupling co-varied with synchrony and was inversely related to the dimensionality of inferior olive firing dynamics, as measured via a principal component analysis of the spike trains in each condition. Furthermore, for both the model and the data, we found an inverted U-shaped relationship between coupling strengths and complexity entropy, a measure of chaos for spiking neural data. These results are consistent with our hypothesis according to which electrical coupling regulates the dimensionality and the complexity in the inferior olive neurons in order to optimize both motor learning and control of high dimensional motor systems by the cerebellum.

Brain is not a mere input-output information transformation system, but a dynamical system that generates spontaneous spatiotemporal patterns even without sensory inputs, executed movements, or cognitive tasks. These spontaneously generated patterns by brain dynamics are called spontaneous brain activities for experimental animals, and resting state brain activities for humans. The resting state brain activity of an individual contains much information about age, cognitive capability, mental disorder etc. By combining information decoding from brain activity and its neurofeedback in reinforcement learning paradigms, we can unconsciously control brain activity patterns corresponding to specific information. This leads to therapies of psychiatric disorders, unconscious manipulation of facial preferences, color qualia, confidence in decision making, increase of cognitive capability, etc. Ubicomp community can expect this technology will soon be available in much cheaper and lighter devices such as EEG and near infrared spectroscopy instead of heavy and expensive fMRI or MEG.

Reinforcement learning (RL) can provide a basic framework for autonomous robots to learn to control and maximize future cumulative rewards in complex environments. To achieve high performance, RL controllers must consider the complex external dynamics for movements and task (reward function) and optimize control commands. For example, a robot playing tennis and squash needs to cope with the different dynamics of a tennis or squash racket and such dynamic environmental factors as the wind. In addition, this robot has to tailor its tactics simultaneously under the rules of either game. This double complexity of the external dynamics and reward function sometimes becomes more complex when both the multiple dynamics and multiple reward functions switch implicitly, as in the situation of a real (multi-agent) game of tennis where one player cannot observe the intention of her opponents or her partner. The robot must consider its opponent's and its partner's unobservable behavioral goals (reward function). In this article, we address how an RL agent should be designed to handle such double complexity of dynamics and reward. We have previously proposed modular selection and identification for control (MOSAIC) to cope with nonstationary dynamics where appropriate controllers are selected and learned among many candidates based on the error of its paired dynamics predictor: the forward model. Here we extend this framework for RL and propose MOSAIC-MR architecture. It resembles MOSAIC in spirit and selects and learns an appropriate RL controller based on the RL controller's TD error using the errors of the dynamics (the forward model) and the reward predictors. Furthermore, unlike other MOSAIC variants for RL, RL controllers are not a priori paired with the fixed predictors of dynamics and rewards. The simulation results demonstrate that MOSAIC-MR outperforms other counterparts because of this flexible association ability among RL controllers, forward models, and reward predictors.

We propose a novel exoskeleton robot prototype aimed at a brain-machine interface and rehabilitation for postural control for elderly people, people with spinal cord injury, stroke patients, and others with similar needs. By arranging pneumatic muscles with electric motors in a optimal way, one can achieve both weight-reduction and torque-controllability. Its anthropomorphic design and torque-controllability enable users to implement and test various rehabilitation/compensation programs consistent with human motor control and learning mechanism. Hybrid drive itself is not new, but its specialized application to lightweight exoskeleton is novel. This paper reports the design and development of the robot, particularly addressing a hybrid drive for load-bearing tasks such as standing and postural maintenance. The experimental data as well as the attached videos demonstrate the effectiveness of the proposed system.

Interaction of a robot with dynamic environments would require continuous adaptation of force and impedance, which is generally not available in current robot systems. In contrast, humans learn novel task dynamics with appropriate force and impedance through the concurrent minimization of error and energy, and exhibit the ability to modify movement trajectory to comply with obstacles and minimize forces. This article develops a similar automatic motor behavior for a robot and reports experiments with a one degree-of-freedom system. In a postural control task, the robot automatically adapts torque to counter a slow disturbance and shifts to increasing its stiffness when the disturbance increases in frequency. In the presence of rigid obstacles, it refrains from increasing force excessively, and relaxes gradually to follow the obstacle, but comes back to the desired state when the obstacle is removed. A trajectory tracking task demonstrates that the robot is able to adapt to different loads during motion. On introduction of a new load, it increases its stiffness to adapt to the load quickly, and then relaxes once the adaptation is complete. Furthermore, in the presence of an obstacle, the robot adjusts its trajectory to go around it.

We present our ongoing effort to achieve compliant balancing to dynamic walking on our torque-controlled, human-sized, biped humanoid robot. Inspired by human musculoskeletal systems, our approach integrates full-body task-space force controllers with joint-space pattern generators on the commanded joint torque output to facilitate robust control performance, as well as the efficient online learning. With this approach various compliant and stable motions have been created in a constructive manner. We demonstrate the effectiveness of our approach by two folds of experiments: 1) Compliant double / single-support balancing and quasi-static walking on uneven terrain, which do not require any joint patters, 2) Fast and stable squat and dynamic walking by introducing joint-space pattern generators.

In this video presentation we introduce a 50 degrees of freedom humanoid robot, CB -ComputationalBrain[1]. CB is a humanoid robot created for exploring the underlying processing of the human brain while dealing with the real world. We place our investigations within real world contexts, as humans do. In so doing, we focus on utilising a system that is closer to humans - in sensing, kinematics configuration and performance. We present a full-body compliance controller that was developed for the motion control of our humanoid robot [2]. Our initial experimentation on our system includes: 1) full-body compliant control - physical interactions/balancing/motion control; 2) the integrated visual ocular-motor responses; 3) perception and control - reaching, foveation, and active object recognition; 4) our studies of Central Pattern Generator for walking.

Ten years have passed since the Japanese ‘Century of the Brain’ was promoted, and its most notable objective, the unique ‘Creating the Brain’ approach, has led us to apply a humanoid robot as a neuroscience tool. Here, we aim to understand the brain to the extent that we can make humanoid robots solve tasks typically solved by the human brain by using essentially the same principles. I postulate that this ‘Understanding the Brain by Creating the Brain’ approach is the only way to fully understand neural mechanisms in a rigorous sense. Even if we could create an artificial brain, we could not investigate its functions, such as vision or motor control, if we just let it float in incubation fluid in a jar. The brain must be connected to sensors and a motor apparatus so that it can interact with its environment. A humanoid robot controlled by an artificial brain, which is implemented as software based on computational models of brain functions, seems to be the most plausible candidate for this purpose, given currently available technology. With the slogan of ‘Understanding the Brain by Creating the Brain’, in the mid-80s we started to use robots for brain research (Miyamoto & Kawato 1988), and about 10 different kinds of robots have been used by our group at Osaka University’s Department of Biophysical Engineering, ATR Laboratories, ERATO Kawato Dynamic Brain Project (ERATO 1996–2001), and ICORP Kawato Computational Brain Project (ICOPR 2004–2009).

An increasing number of projects in neuroscience require statistical analysis of high-dimensional data, as, for instance, in the prediction of behavior from neural firing or in the operation of artificial devices from brain recordings in brain-machine interfaces. Although prevalent, classical linear analysis techniques are often numerically fragile in high dimensions due to irrelevant, redundant, and noisy information. We developed a robust Bayesian linear regression algorithm that automatically detects relevant features and excludes irrelevant ones, all in a computationally efficient manner. In comparison with standard linear methods, the new Bayesian method regularizes against overfitting, is computationally efficient (unlike previously proposed variational linear regression methods, is suitable for data sets with large numbers of samples and a very high number of input dimensions) and is easy to use, thus demonstrating its potential as a drop-in replacement for other linear regression techniques. We evaluate our technique on synthetic data sets and on several neurophysiological data sets. For these neurophysiological data sets we address the question of whether EMG data collected from arm movements of monkeys can be faithfully reconstructed from neural activity in motor cortices. Results demonstrate the success of our newly developed method, in comparison with other approaches in the literature, and, from the neurophysiological point of view, confirms recent findings on the organization of the motor cortex. Finally, an incremental, real-time version of our algorithm demonstrates the suitability of our approach for real-time interfaces between brains and machines.

We propose a framework for skill synthesis for robots that exploits the human capacity to learn novel control tasks. The conceptual idea is to incorporate the target robotic platform into the experimenter's body schema so that it can be controlled effortlessly as if the robot were a part of the body. Once this stage is achieved, the dexterity on a task exhibited with the new external limb -the robot- can be used for designing controllers for the task under consideration. This article exemplifies the proposed framework by showing the derivation of an effective open-loop controller that can manipulate two balls with the fingers of a 16-DOF robotic hand.

Recent studies have shown that multiple internal models are acquired in the cerebellum and that these can be switched under a given context of behavior. It has been proposed that long-term depression (LTD) of parallel fiber (PF)-Purkinje cell (PC) synapses forms the cellular basis of cerebellar learning, and that the presynaptically synthesized messenger nitric oxide (NO) is a crucial "gatekeeper" for LTD. Because NO diffuses freely to neighboring synapses, this volume learning is not input-specific and brings into question the biological significance of LTD as the basic mechanism for efficient supervised learning. To better characterize the role of NO in cerebellar learning, we simulated the sequence of electrophysiological and biochemical events in PF-PC LTD by combining established simulation models of the electrophysiology, calcium dynamics, and signaling pathways of the PC. The results demonstrate that the local NO concentration is critical for induction of LTD and for its input specificity. Pre- and postsynaptic coincident firing is not sufficient for a PF-PC synapse to undergo LTD, and LTD is induced only when a sufficient amount of NO is provided by activation of the surrounding PFs. On the other hand, above-adequate levels of activity in nearby PFs cause accumulation of NO, which also allows LTD in neighboring synapses that were not directly stimulated, ruining input specificity. These findings lead us to propose the hypothesis that NO represents the relevance of a given context and enables context-dependent selection of internal models to be updated. We also predict sparse PF activity in vivo because, otherwise, input specificity would be lost.

Computational neuroscience research, or more in general, system neuroscience research, is much more difficult than molecular research although it is essential to understanding brain mechanisms. The abstract description of the new paradigm is as follows. Suppose one has a computational theory, which postulates that some brain networks solve some computational problems and a specific brain locus contains a specific computational representation. Then one needs to extract this information either by some noninvasive method or unit recording, and manipulate this by altered computational algorithms derived from the theory. The altered or processed information is fed back to the brain by appropriate methods (e.g. visual or tactile feedbacks, TMS, electrical stimulation). Observed brain activity and behavior are now guaranteed causally linked. The author explains the background of this new paradigm and several possible applications.

An increasing number of projects in neuroscience requires the sta- tistical analysis of high dimensional data sets, as, for instance, in predicting behavior from neural firing or in operating artificial de- vices from brain recordings in brain-machine interfaces. Linear analysis techniques remain prevalent in such cases, but classical linear regression approaches are often numerically too fragile in high dimensions. In this paper, we address the question of whether EMG data collected from arm movements of monkeys can be faith- fully reconstructed with linear approaches from neural activity in primary motor cortex (M1). To achieve robust data analysis, we develop a full Bayesian approach to linear regression that auto- matically detects and excludes irrelevant features in the data, reg- ularizing against overfitting. In comparison with ordinary least squares, stepwise regression, partial least squares, LASSO regres- sion and a brute force combinatorial search for the most predictive input features in the data, we demonstrate that the new Bayesian method offers a superior mixture of characteristics in terms of reg- ularization against overfitting, computational efficiency and ease of use, demonstrating its potential as a drop-in replacement for other linear regression techniques. As neuroscientific results, our anal- yses demonstrate that EMG data can be well predicted from M1 neurons, further opening the path for possible real-time interfaces between brains and machines.

We propose a framework for learning biped locomotion using dynamical movement primitives based on nonlinear oscillators. In our previous work, we suggested dynamical movement primitives as a central pattern generator (CPG) to learn biped locomotion from demonstration. We introduced an adaptation algorithm for the frequency of the oscillators based on phase resetting at the instance of heel strike and entrainment between the phase oscillator and mechanical system using feedback from the environment. In this paper, we empirically explore the role of phase resetting in the proposed algorithm for robust biped locomotion. We demonstrate that phase resetting contributes to robustness against external perturbations and environmental changes by numerical simulations and experiments with a physical biped robot.

In this paper, we propose a new technique based on recent neuroimaging studies as a tool for the assessment of interactive systems. For this purpose, we analyze the mental process that takes place while human subjects learn to use new tools by using two different approaches. One is an experiment on task performance based on the conventional direct testing method of a user interface, and the other is an indirect method based on recent neuroimaging studies that indirectly estimate the process of interaction through the observation of the human brain activities. The results obtained from the direct experiment on performance evaluation are compared with those from the indirect analysis of the human brain activity, which is measured by a non-invasive neuroimaging measuring method. The process of acquisition of internal models while subjects learn to use new tools is also discussed.

We propose a modular reinforcement learning architecture for nonlinear, nonstationary control tasks, which we call multiple model-based reinforcement learning (MMRL). The basic idea is to decompose a complex task into multiple domains in space and time based on the predictability of the environmental dynamics. The system is composed of multiple modules, each of which consists of a state prediction model and a reinforcement learning controller. The "responsibility signal," which is given by the softmax function of the prediction errors, is used to weight the outputs of multiple modules, as well as to gate the learning of the prediction models and the reinforcement learning controllers. We formulate MMRL for both discrete-time, finite-state case and continuous-time, continuous-state case. The performance of MMRL was demonstrated for discrete case in a nonstationary hunting task in a grid world and for continuous case in a nonlinear, nonstationary control task of swinging up a pendulum with variable physical parameters.

Humans demonstrate a remarkable ability to generate accurate and appropriate motor behavior under many different and often uncertain environmental conditions. We previously proposed a new modular architecture, the modular selection and identification for control (MOSAIC) model, for motor learning and control based on multiple pairs of forward (predictor) and inverse (controller) models. The architecture simultaneously learns the multiple inverse models necessary for control as well as how to select the set of inverse models appropriate for a given environment. It combines both feedforward and feedback sensorimotor information so that the controllers can be selected both prior to movement and subsequently during movement. This article extends and evaluates the MOSAIC architecture in the following respects. The learning in the architecture was implemented by both the original gradient-descent method and the expectation-maximization (EM) algorithm. Unlike gradient descent, the newly derived EM algorithm is robust to the initial starting conditions and learning parameters. Second, simulations of an object manipulation task prove that the architecture can learn to manipulate multiple objects and switch between them appropriately. Moreover, after learning, the model shows generalization to novel objects whose dynamics lie within the polyhedra of already learned dynamics. Finally, when each of the dynamics is associated with a particular object shape, the model is able to select the appropriate controller before movement execution. When presented with a novel shape-dynamic pairing, inappropriate activation of modules is observed followed by on-line correction.