Recently, robust reinforcement learning (RL) methods designed to handle adversarial input observations have received significant attention, motivated by RL’s inherent vulnerabilities. While existing approaches have demonstrated reasonable success, addressing worst-case scenarios over long time horizons requires both minimizing the agent’s cumulative rewards for adversaries and training agents to counteract them through alternating learning. However, this process introduces mutual dependencies between the agent and the adversary, making interactions with the environment inefficient and hindering the development of off-policy methods. In this work, we propose a novel off-policy method that eliminates the need for additional environmental interactions by reformulating adversarial learning as a soft-constrained optimization problem. Our approach is theoretically supported by the symmetric property of policy evaluation between the agent and the adversary. The implementation is available at https://github.com/nakanakakosuke/VALT_SAC.

Contract bridge, a cooperative game characterized by imperfect information and multi-agent dynamics, poses significant challenges and serves as a critical benchmark in artificial intelligence (AI) research. Success in this domain requires agents to effectively cooperate with their partners. This study demonstrates that an appropriate combination of existing methods can perform surprisingly well in bridge bidding against WBridge5, a leading benchmark in the bridge bidding system and a multiple-time World Computer-Bridge Championship winner. Our approach is notably simple, yet it outperforms the current state-of-the-art methodologies in this field. Furthermore, we have made our code and models publicly available as open-source software. This initiative provides a strong starting foundation for future bridge AI research, facilitating the development and verification of new strategies and advancements in the field.

Abstract In this study, we propose a novel motion re‐targeting framework that provides natural motions of target robot character models similar to the given source motions of a different skeletal structure. The natural target motion requires satisfying kinematic constraints to show a similar motion to the source motion although the kinematical structure between the source and the target character models differ from each other. Simultaneously, the target motion should maintain physically plausible features such as keeping the balance of the target character model. To handle the issue, we utilize a simple physics model (an inverted‐pendulum‐on‐a‐cart model) during the motion re‐targeting process. By interpreting the source motion's balancing property via the pendulum model, the target motion inherits the balancing property of the source motion. The inheritance is derived by performing the motion analysis to extract the necessary parameters for re‐targeting the pendulum model's motion pattern and parameter learning to estimate the suitable parameters for the target character model. Based on the simple physics inheritance, the proposed framework provides balance‐preserving target motions, even applicable to the full‐body physics simulation or a real robot control. We validate the framework by experimenting with motion re‐targeting from animal character and human character source models to the quadruped‐ and humanoid‐type target models with Muaythai punching, kicking and walking motions. We also implement comparisons with the existing methods to clarify the enhancement.

We propose Pgx, a suite of board game reinforcement learning (RL) environments written in JAX and optimized for GPU/TPU accelerators. By leveraging JAX's auto-vectorization and parallelization over accelerators, Pgx can efficiently scale to thousands of simultaneous simulations over accelerators. In our experiments on a DGX-A100 workstation, we discovered that Pgx can simulate RL environments 10-100x faster than existing implementations available in Python. Pgx includes RL environments commonly used as benchmarks in RL research, such as backgammon, chess, shogi, and Go. Additionally, Pgx offers miniature game sets and baseline models to facilitate rapid research cycles. We demonstrate the efficient training of the Gumbel AlphaZero algorithm with Pgx environments. Overall, Pgx provides high-performance environment simulators for researchers to accelerate their RL experiments. Pgx is available at https://github.com/sotetsuk/pgx.

Numerous games have served as testbeds for artificial intelligence (AI) research to measure its progress. Mahjong is a highly challenging multi-agent imperfect information game with a vast player population. However, a challenge with using Mahjong as a testbed for AI is the lack of a publicly available framework that is fast, easy to use and implements popular rules for human players. We propose and describe Mjx, an open-source Mahjong framework, which implements one of the most popular Mahjong rules, riichi Mahjong (Japanese Mahjong). We compared the execution speed of Mjx with existing popular open-source software and demonstrated that it achieves 100x faster performance. Mjx is available at https://github.conmjx-project/mjx.

Abstract In this study, we focus on developing a motion synthesis framework that generates a natural transition motion between two different behaviours to interact with a moving object. Specifically, the proposed framework generates the transition motion, bridging from a locomotive behaviour to an object interaction behaviour. And, the transition motion should adapt to the spatio‐temporal variation of the target object in an online manner, so as to naturally connect the behaviours. To solve this issue, we propose a framework that combines a regression model and a transition motion planner. The neural network‐based regression model estimates the reference transition strategy to guide the reference pattern of the transitioning, adapted to the varying situation. The transition motion planner reconstructs the transition motion based on the reference pattern while considering dynamic constraints that avoid the footskate and interaction constraints. The proposed framework is validated to synthesize various transition motions while adapting to the spatio‐temporal variation of the object by using object grasping motion, and athletic motions in soccer.

Visual properties that primarily attract bottom-up attention are collectively referred to as saliency. In this study, to understand the neural activity involved in top-down and bottom-up visual attention, we aim to prepare pairs of natural and unnatural images with common saliency. For this purpose, we propose an image transformation method based on deep neural networks that can generate new images while maintaining the consistent feature map, in particular the saliency map. This is an ill-posed problem because the transformation from an image to its corresponding feature map could be many-to-one, and in our particular case, the various images would share the same saliency map. Although stochastic image generation has the potential to solve such ill-posed problems, the most existing methods focus on adding diversity of the overall style/touch information while maintaining the naturalness of the generated images. To this end, we developed a new image transformation method that incorporates higher-dimensional latent variables so that the generated images appear unnatural with less context information but retain a high diversity of local image structures. Although such high-dimensional latent spaces are prone to collapse, we proposed a new regularization based on Kullback-Leibler divergence to avoid collapsing the latent distribution. We also conducted human experiments using our newly prepared natural and corresponding unnatural images to measure overt eye movements and functional magnetic resonance imaging, and found that those images induced distinctive neural activities related to top-down and bottom-up attentional processing.

In behavioral learning, reward-related events are encoded into phasic dopamine (DA) signals in the brain. In particular, unexpected reward omission leads to a phasic decrease in DA (DA dip) in the striatum, which triggers long-term potentiation (LTP) in DA D2 receptor (D2R)-expressing spiny-projection neurons (D2 SPNs). While this LTP is required for reward discrimination, it is unclear how such a short DA-dip signal (0.5-2 s) is transferred through intracellular signaling to the coincidence detector, adenylate cyclase (AC). In the present study, we built a computational model of D2 signaling to determine conditions for the DA-dip detection. The DA dip can be detected only if the basal DA signal sufficiently inhibits AC, and the DA-dip signal sufficiently disinhibits AC. We found that those two requirements were simultaneously satisfied only if two key molecules, D2R and regulators of G protein signaling (RGS) were balanced within a certain range; this balance has indeed been observed in experimental studies. We also found that high level of RGS was required for the detection of a 0.5-s short DA dip, and the analytical solutions for these requirements confirmed their universality. The imbalance between D2R and RGS is associated with schizophrenia and DYT1 dystonia, both of which are accompanied by abnormal striatal LTP. Our simulations suggest that D2 SPNs in patients with schizophrenia and DYT1 dystonia cannot detect short DA dips. We finally discussed that such psychiatric and movement disorders can be understood in terms of the imbalance between D2R and RGS.

Animals remember temporal links between their actions and subsequent rewards. We previously discovered a synaptic mechanism underlying such reward learning in D1 receptor (D1R)-expressing spiny projection neurons (D1 SPN) of the striatum. Dopamine (DA) bursts promote dendritic spine enlargement in a time window of only a few seconds after paired pre- and post-synaptic spiking (pre-post pairing), which is termed as reinforcement plasticity (RP). The previous study has also identified underlying signaling pathways; however, it still remains unclear how the signaling dynamics results in RP. In the present study, we first developed a computational model of signaling dynamics of D1 SPNs. The D1 RP model successfully reproduced experimentally observed protein kinase A (PKA) activity, including its critical time window. In this model, adenylate cyclase type 1 (AC1) in the spines/thin dendrites played a pivotal role as a coincidence detector against pre-post pairing and DA burst. In particular, pre-post pairing (Ca2+ signal) stimulated AC1 with a delay, and the Ca2+-stimulated AC1 was activated by the DA burst for the asymmetric time window. Moreover, the smallness of the spines/thin dendrites is crucial to the short time window for the PKA activity. We then developed a RP model for D2 SPNs, which also predicted the critical time window for RP that depended on the timing of pre-post pairing and phasic DA dip. AC1 worked for the coincidence detector in the D2 RP model as well. We further simulated the signaling pathway leading to Ca2+/calmodulin-dependent protein kinase II (CaMKII) activation and clarified the role of the downstream molecules of AC1 as the integrators that turn transient input signals into persistent spine enlargement. Finally, we discuss how such timing windows guide animals' reward learning.

Neuroscientific studies have revealed that alpha oscillations in electroencephalography (EEG) signals play an active role in the cognitive aspects of human daily life. In this paper, we developed an artificial encoding model to mimic the dynamic brain visual processing system in terms of alpha oscillations. We analyzed the induced alpha power during free viewing of natural videos and extracted relevant features in a participant-independent way. Meanwhile, the visual characteristics that would trigger involuntary attention in the early stage of visual processing were extracted from videos and further employed to estimate the alpha fluctuations at every 0.5 s. We compared the encoding performances across 19 electrode channel locations and selected an optimal encoding model with a support vector regression. The results demonstrated a promising brain encoding model in EEG signals, that could further contribute to the development of brain-computer interface and visual design.

We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adversarial training, our method defines the adversarial direction without label information and is hence applicable to semi-supervised learning. Because the directions in which we smooth the model are only "virtually" adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward- and back-propagations. In our experiments, we applied VAT to supervised and semi-supervised learning tasks on multiple benchmark datasets. With a simple enhancement of the algorithm based on the entropy minimization principle, our VAT achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10.

The reproducibility of embryonic development is remarkable, although molecular processes are intrinsically stochastic at the single-cell level. How the multicellular system resists the inevitable noise to acquire developmental reproducibility constitutes a fundamental question in developmental biology. Toward this end, we focused on vertebrate somitogenesis as a representative system, because somites are repeatedly reproduced within a single embryo whereas such reproducibility is lost in segmentation clock gene-deficient embryos. However, the effect of noise on developmental reproducibility has not been fully investigated, because of the technical difficulty in manipulating the noise intensity in experiments. In this study, we developed a computational model of ERK-mediated somitogenesis, in which bistable ERK activity is regulated by an FGF gradient, cell-cell communication, and the segmentation clock, subject to the intrinsic noise. The model simulation generated our previous in vivo observation that the ERK activity was distributed in a step-like gradient in the presomitic mesoderm, and its boundary was posteriorly shifted by the clock in a stepwise manner, leading to regular somite formation. Here, we showed that this somite regularity was robustly maintained against the noise. Removing the clock from the model predicted that the stepwise shift of the ERK activity occurs at irregular timing with irregular distance owing to the noise, resulting in somite size variation. This model prediction was recently confirmed by live imaging of ERK activity in zebrafish embryos. Through theoretical analysis, we presented a mechanism by which the clock reduces the inherent somite irregularity observed in clock-deficient embryos. Therefore, this study indicates a novel role of the segmentation clock in noise-resistant developmental reproducibility.

A major goal of contemporary neuroscience research is to map the structural connectivity of mammalian brain using microscopy imaging data. In this context, the reconstruction of densely labeled axons from two-photon microscopy images is a challenging and important task. The visually overlapping, crossing, and often strongly distorted images of the axons allow many ambiguous interpretations to be made. We address the problem of tracking axons in densely labeled samples of neurons in large image data sets acquired from marmoset brains. Our high-resolution images were acquired using two-photon microscopy and they provided whole brain coverage, occupying terabytes of memory. Both the image distortions and the large data set size frequently make it impractical to apply present-day neuron tracing algorithms to such data due to the optimization of such algorithms to the precise tracing of either single or sparse sets of neurons. Thus, new tracking techniques are needed. We propose a probabilistic axon tracking algorithm (PAT). PAT tackles the tracking of axons in two steps: locally (L-PAT) and globally (G-PAT). L-PAT is a probabilistic tracking algorithm that can tackle distorted, cluttered images of densely labeled axons. L-PAT divides a large micrograph into smaller image stacks. It then processes each image stack independently before mapping the axons in each image to a sparse model of axon trajectories. G-PAT merges the sparse L-PAT models into a single global model of axon trajectories by minimizing a global objective function using a probabilistic optimization method. We demonstrate the superior performance of PAT over standard approaches on synthetic data. Furthermore, we successfully apply PAT to densely labeled axons in large images acquired from marmoset brains.

Ensemble discriminative tracking utilizes a committee of classifiers, to label data samples, which are in turn, used for retraining the tracker to localize the target using the collective knowledge of the committee. Committee members could vary in their features, memory update schemes, or training data, however, it is inevitable to have committee members that excessively agree because of large overlaps in their version space. To remove this redundancy and have an effective ensemble learning, it is critical for the committee to include consistent hypotheses that differ from one-another, covering the version space with minimum overlaps. In this study, we propose an online ensemble tracker that directly generates a diverse committee by generating an efficient set of artificial training. The artificial data is sampled from the empirical distribution of the samples taken from both target and background, whereas the process is governed by query-by-committee to shrink the overlap between classifiers. The experimental results demonstrate that the proposed scheme outperforms conventional ensemble trackers on public benchmarks.

Living tissues undergo deformation during morphogenesis. In this process, cells generate mechanical forces that drive the coordinated cell motion and shape changes. Recent advances in experimental and theoretical techniques have enabled in situ measurement of the mechanical forces, but the characterization of mechanical properties that determine how these forces quantitatively affect tissue deformation remains challenging, and this represents a major obstacle for the complete understanding of morphogenesis. Here, we proposed a non-invasive reverse-engineering approach for the estimation of the mechanical properties, by combining tissue mechanics modeling and statistical machine learning. Our strategy is to model the tissue as a continuum mechanical system and to use passive observations of spontaneous tissue deformation and force fields to statistically estimate the model parameters. This method was applied to the analysis of the collective migration of Madin-Darby canine kidney cells, and the tissue flow and force were simultaneously observed by the phase contrast imaging and traction force microscopy. We found that our monolayer elastic model, whose elastic moduli were reverse-engineered, enabled a long-term forecast of the traction force fields when given the tissue flow fields, indicating that the elasticity contributes to the evolution of the tissue stress. Furthermore, we investigated the tissues in which myosin was inhibited by blebbistatin treatment, and observed a several-fold reduction in the elastic moduli. The obtained results validate our framework, which paves the way to the estimation of mechanical properties of living tissues during morphogenesis.

Animals are able to reach a desired state in an environment by controlling various behavioral patterns. Identification of the behavioral strategy used for this control is important for understanding animals' decision-making and is fundamental to dissect information processing done by the nervous system. However, methods for quantifying such behavioral strategies have not been fully established. In this study, we developed an inverse reinforcement-learning (IRL) framework to identify an animal's behavioral strategy from behavioral time-series data. We applied this framework to C. elegans thermotactic behavior; after cultivation at a constant temperature with or without food, fed worms prefer, while starved worms avoid the cultivation temperature on a thermal gradient. Our IRL approach revealed that the fed worms used both the absolute temperature and its temporal derivative and that their behavior involved two strategies: directed migration (DM) and isothermal migration (IM). With DM, worms efficiently reached specific temperatures, which explains their thermotactic behavior when fed. With IM, worms moved along a constant temperature, which reflects isothermal tracking, well-observed in previous studies. In contrast to fed animals, starved worms escaped the cultivation temperature using only the absolute, but not the temporal derivative of temperature. We also investigated the neural basis underlying these strategies, by applying our method to thermosensory neuron-deficient worms. Thus, our IRL-based approach is useful in identifying animal strategies from behavioral time-series data and could be applied to a wide range of behavioral studies, including decision-making, in other organisms.

A discriminative ensemble tracker employs multiple classifiers, each of which casts a vote on all of the obtained samples. The votes are then aggregated in an attempt to localize the target object. Such method relies on collective competence and the diversity of the ensemble to approach the target/non-target classification task from different views. However, by updating all of the ensemble using a shared set of samples and their final labels, such diversity is lost or reduced to the diversity provided by the underlying features or internal classifiers' dynamics. Additionally, the classifiers do not exchange information with each other while striving to serve the collective goal, i.e., better classification. In this study, we propose an active collaborative information exchange scheme for ensemble tracking. This, not only orchestrates different classifiers towards a common goal but also provides an intelligent update mechanism to keep the diversity of classifiers and to mitigate the shortcomings of one with the others. The data exchange is optimized with regard to an ensemble uncertainty utility function, and the ensemble is updated via co-training. The evaluations demonstrate promising results realized by the proposed algorithm for the real-world online tracking.

Adaptive tracking-by-detection is a popular approach to track arbitrary objects in various situations. Such approaches treat tracking as a classification task and constantly update the object model. The update procedure requires a set of labeled examples, where samples are collected from the last observation, and then labeled. However, these intermediate steps typically follow a set of heuristic rules for labeling and uninformed search in the sample space, which decrease the effectiveness of model update. In this study, we present a framework for adaptive tracking that utilizes active learning for effective sample selection and labeling them. The active sampler employs a committee of randomized-classifiers to select the most informative samples and query their label from an auxiliary detector with a long-term memory. The committee is then updated with the obtained labels. Experiments show that our algorithm outperforms state-of-the-art trackers on various benchmark videos.

Uncertainty of fear conditioning is crucial for the acquisition and extinction of fear memory. Fear memory acquired through partial pairings of a conditioned stimulus (CS) and an unconditioned stimulus (US) is more resistant to extinction than that acquired through full pairings; this effect is known as the partial reinforcement extinction effect (PREE). Although the PREE has been explained by psychological theories, the neural mechanisms underlying the PREE remain largely unclear. Here, we developed a neural circuit model based on three distinct types of neurons (fear, persistent and extinction neurons) in the amygdala and medial prefrontal cortex (mPFC). In the model, the fear, persistent and extinction neurons encode predictions of net severity, of unconditioned stimulus (US) intensity, and of net safety, respectively. Our simulation successfully reproduces the PREE. We revealed that unpredictability of the US during extinction was represented by the combined responses of the three types of neurons, which are critical for the PREE. In addition, we extended the model to include amygdala subregions and the mPFC to address a recent finding that the ventral mPFC (vmPFC) is required for consolidating extinction memory but not for memory retrieval. Furthermore, model simulations led us to propose a novel procedure to enhance extinction learning through re-conditioning with a stronger US; strengthened fear memory up-regulates the extinction neuron, which, in turn, further inhibits the fear neuron during re-extinction. Thus, our models increased the understanding of the functional roles of the amygdala and vmPFC in the processing of uncertainty in fear conditioning and extinction.

Tubular shaped networks appear not only in medical images like X-ray-, time-of-flight MRI- or CT-angiograms but also in microscopic images of neuronal networks. We present EMILOVE (Efficient Monte-carlo Image-analysis for the Location Of Vascular Entity), a novel modeling algorithm for tubular networks in biomedical images. The model is constructed using tablet shaped particles and edges connecting them. The particles encode the intrinsic information of tubular structure, including position, scale and orientation. The edges connecting the particles determine the topology of the networks. For simulated data, EMILOVE was able to accurately extract the tubular network. EMILOVE showed high performance in real data as well; it successfully modeled vascular networks in real cerebral X-ray and time-of-flight MRI angiograms. We also show some promising, preliminary results on microscopic images of neurons.

This work proposes a model for human habituation while riding a robotic wheelchair. We present and describe the concept of human navigational habituation which we define as the human habituation to repetitively riding a robotic wheelchair. The approach models habituation in terms of preferred linear velocity based on the experience of riding a wheelchair. We argue that preferred velocity changes as the human gets used to riding on the wheelchair. Inexperienced users initially prefer to ride at a slow moderate pace, however the longer they ride they prefer to speed up to a certain comfort level and find initial slower velocities to be tediously “too slow” for their experience level. The proposed habituation model provides passenger preferred velocity based on experience. Human biological measurements, galvanic skin conductance, and participant feedback demonstrate the preference for habituation velocity control over fixed velocity control. To our knowledge habituation modeling is new in the field of autonomous navigation and robotics.

Crosstalk between neurons and glia may constitute a significant part of information processing in the brain. We present a novel method of statistically identifying interactions in a neuron-glia network. We attempted to identify neuron-glia interactions from neuronal and glial activities via maximum-a-posteriori (MAP)-based parameter estimation by developing a generalized linear model (GLM) of a neuron-glia network. The interactions in our interest included functional connectivity and response functions. We evaluated the cross-validated likelihood of GLMs that resulted from the addition or removal of connections to confirm the existence of specific neuron-to-glia or glia-to-neuron connections. We only accepted addition or removal when the modification improved the cross-validated likelihood. We applied the method to a high-throughput, multicellular in vitro Ca2+ imaging dataset obtained from the CA3 region of a rat hippocampus, and then evaluated the reliability of connectivity estimates using a statistical test based on a surrogate method. Our findings based on the estimated connectivity were in good agreement with currently available physiological knowledge, suggesting our method can elucidate undiscovered functions of neuron-glia systems.

The noninvasive brain-machine interface (BMI) is anticipated to be an effective tool of communication not only in laboratory settings but also in our daily livings. The direct communication channel created by BMI can assist aging societies and the handicapped and improve human welfare. In this paper we propose and experiment a BMI framework that combines BMI with a robotic house and autonomous robotic wheelchair. Autonomous navigation is achieved by placing waypoints within the house and, from the user side, the user performs BMI to give commands to the house and wheelchair. The waypoint framework can offer essential services to the user with an effectively improved information-transfer rate and is an excellent examples of the fusion of data measured by sensors in the house, which can offer insight into further studies.

In reinforcement learning, large state and action spaces make the estimation of value functions impractical, so a value function is often represented as a linear combination of basis functions whose linear coefficients constitute parameters to be estimated. However, preparing basis functions requires a certain amount of prior knowledge and is, in general, a difficult task. To overcome this difficulty, an adaptive basis function construction technique has been proposed by Keller recently, but it requires excessive computational cost. We propose an efficient approach to this difficulty, in which the problem of approximating the value function is decomposed into a number of subproblems, each of which can be solved with small computational cost. Computer experiments show that the CPU time needed by our method is much smaller than that by the existing method.

We propose a new reconstruction procedure for X-ray computed tomography (CT) based on Bayesian modeling. We utilize the knowledge that the human body is composed of only a limited number of materials whose CT values are roughly known in advance. Although the exact Bayesian inference of our model is intractable, we propose an efficient algorithm based on the variational Bayes technique. Experiments show that the proposed method performs better than the existing methods in severe situations where samples are limited or metal is inserted into the body.

We propose a framework for expanding a given image using an interpolator that is trained in advance with training data, based on sparse bayesian estimation for determining the optimal and compact support for efficient image expansion. Experiments on test data show that learned interpolators are compact yet superior to classical ones.

Image expansion by linear filtering is attractive and widely used because of its simplicity and efficiency, and many interpolation methods fall in this category. In this study, we model filtering as linear regression from low- to high-resolution color image patches, and propose a learning-based design method of image expansion filters based on sparse Bayesian estimation. Sparseness is imposed on the filter coefficients to obtain compact supports. Image expansion is formulated as the problem of finding the predictive mean of a high-resolution patch given a low-resolution patch to expand. Since an exact evaluation of the predictive distribution is difficult, variational methods are employed to derive an efficient algorithm. Experiments on test data show that good generalization performance is obtained based on sparse filters and that color modeling improves the expansion quality.

In this letter, we present new methods of multiclass classification that combine multiple binary classifiers. Misclassification of each binary classifier is formulated as a bit inversion error with probabilistic models by making an analogy to the context of information transmission theory. Dependence between binary classifiers is incorporated into our model, which makes a decoder a type of Boltzmann machine. We performed experimental studies using a synthetic data set, data sets from the UCI repository, and bioinformatics data sets, and the results show that the proposed methods are superior to the existing multiclass classification methods.

Multiclass classification is one of the fundamental tasks in bioinformatics and typically arises in cancer diagnosis studies by gene expression profiling. There have been many studies of aggregating binary classifiers to construct a multiclass classifier based on one-versus-the-rest (1R), one-versus-one (11), or other coding strategies, as well as some comparison studies between them. However, the studies found that the best coding depends on each situation. Therefore, a new problem, which we call the "optimal coding problem," has arisen: how can we determine which coding is the optimal one in each situation? To approach this optimal coding problem, we propose a novel framework for constructing a multiclass classifier, in which each binary classifier to be aggregated has a weight value to be optimally tuned based on the observed data. Although there is no a priori answer to the optimal coding problem, our weight tuning method can be a consistent answer to the problem. We apply this method to various classification problems including a synthesized data set and some cancer diagnosis data sets from gene expression profiling. The results demonstrate that, in most situations, our method can improve classification accuracy over simple voting heuristics and is better than or comparable to state-of-the-art multiclass predictors.

Reinforcement learning (RL) methods based on least-squares temporal difference (LSTD) have been developed recently and have shown good practical performance. However, the quality of their estimation has not been well elucidated. In this article, we discuss LSTD-based policy evaluation from the new view-point of semiparametric statistical inference. In fact, the estimator can be obtained from a particular estimating function which guarantees its convergence to the true value asymptotically, without specifying a model of the environment. Based on these observations, we 1) analyze the asymptotic variance of an LSTD-based estimator, 2) derive the optimal estimating function with the minimum asymptotic estimation variance, and 3) derive a suboptimal estimator to reduce the computational burden in obtaining the optimal estimating function.

Advances in time-lapse fluorescence microscopy have enabled us to directly observe dynamic cellular phenomena. Although the techniques themselves have promoted the understanding of dynamic cellular functions, the vast number of images acquired has generated a need for automated processing tools to extract statistical information. A problem underlying the analysis of time-lapse cell images is the lack of rigorous methods to extract morphodynamic properties. Here, we propose an algorithm called edge evolution tracking (EET) to quantify the relationship between local morphological changes and local fluorescence intensities around a cell edge using time-lapse microscopy images. This algorithm enables us to trace the local edge extension and contraction by defining subdivided edges and their corresponding positions in successive frames. Thus, this algorithm enables the investigation of cross-correlations between local morphological changes and local intensity of fluorescent signals by considering the time shifts. By applying EET to fluorescence resonance energy transfer images of the Rho-family GTPases Rac1, Cdc42, and RhoA, we examined the cross-correlation between the local area difference and GTPase activity. The calculated correlations changed with time-shifts as expected, but surprisingly, the peak of the correlation coefficients appeared with a 6-8 min time shift of morphological changes and preceded the Rac1 or Cdc42 activities. Our method enables the quantification of the dynamics of local morphological change and local protein activity and statistical investigation of the relationship between them by considering time shifts in the relationship. Thus, this algorithm extends the value of time-lapse imaging data to better understand dynamics of cellular function.

Sound production process is often expressed by a source-filter model, which assumes that sound signals are generated by convolution of a source signal and a synthesis filter. Although the source-filter model has been widely used, simultaneous estimation of source signals and synthesis filters is difficult due to its inherent indeterminacy. To reduce the indeterminacy, we propose a state-space model that utilizes temporal continuity of pitch and loudness. From the assumption that the synthesis filter contains the timbre-like instrument-specific features while the source signal represents time-variant components such as pitch and loudness, we can estimate the parameters of the synthesis filter and use them for instrument identification. The instrument identification experiments showed comparable or higher accuracy than the existing instrument identification methods with less number of parameters. This result supports the possibility to develop a reliable estimation method of a dynamic source-filter model.

In this article, we propose a new method of multi-class classification in the framework of error-correcting output coding (ECOC). Misclassification of each binary classifier is formulated as a bit inversion error with a probabilistic model for each class and dependence between binary classifiers is incorporated into our model, which makes a decoder, a type of Boltzmann machine. Experimental studies using a synthetic dataset and datasets from UCI repository are performed, and the results show that the proposed method is superior to other existing multi-class classification methods.

In bioinformatics it is often desirable to combine data from various measurement sources and thus structured feature vectors are to be analyzed that possess different intrinsic blocking characteristics (e.g., different patterns of missing values, obser- vation noise levels, effective intrinsic dimensionalities). We propose a new ma- chine learning tool, heterogeneous component analysis (HCA), for feature extrac- tion in order to better understand the factors that underlie such complex structured heterogeneous data. HCA is a linear block-wise sparse Bayesian PCA based not only on a probabilistic model with block-wise residual variance terms but also on a Bayesian treatment of a block-wise sparse factor-loading matrix. We study vari- ous algorithms that implement our HCA concept extracting sparse heterogeneous structure by obtaining common components for the blocks and specific compo- nents within each block. Simulations on toy and bioinformatics data underline the usefulness of the proposed structured matrix factorization concept.

Games constitute a challenging domain of reinforcement learning (RL) for acquiring strategies because many of them include multiple players and many unobservable variables in a large state space. The difficulty of solving such realistic multiagent problems with partial observability arises mainly from the fact that the computational cost for the estimation and prediction in the whole state space, including unobservable variables, is too heavy. To overcome this intractability and enable an agent to learn in an unknown environment, an effective approximation method is required with explicit learning of the environmental model. We present a model-based RL scheme for large-scale multiagent problems with partial observability and apply it to a card game, hearts. This game is a well-defined example of an imperfect information game and can be approximately formulated as a partially observable Markov decision process (POMDP) for a single learning agent. To reduce the computational cost, we use a sampling technique in which the heavy integration required for the estimation and prediction can be approximated by a plausible number of samples. Computer simulation results show that our method is effective in solving such a difficult, partially observable multiagent problem.

Independent component analysis (ICA) is currently the most popularly used approach to blind source separation (BSS), the problem of recovering unknown source signals when their mixtures are observed but the actual mixing process is unknown. Many ICA algorithms assume that a fixed set of source signals consistently exists in mixtures throughout the time-series to be examined. However, real-world signals often have such difficult nonstationarity that each source signal abruptly appears or disappears, thus the set of active sources dynamically changes with time. In this paper, we propose switching ICA (SwICA), which focuses on such situations. The proposed approach is based on the noisy ICA formulated as a generative model. We employ a special type of hidden Markov model (HMM) to represent such prior knowledge that the source may abruptly appear or disappear with time. The special HMM setting t hen provides an effect ofvariable selection in a dynamic way. We use the variational Bayes (VB) method to derive an effective approximation of Bayesian inference for this model. In simulation experiments using artificial and realistic source signals, the proposed method exhibited performance superior to existing methods, especially in the presence of noise. The compared methods include the natural-gradient ICA with a nonholonomic constraint, and the existing ICA method incorporating an HMM source model, which aims to deal with general nonstationarities that may exist in source signals. In addition, the proposed method could successfully recover the source signals even when the total number of true sources was overestimated or was larger than that of mixtures. We also propose a modification of the basic Markov model into a semi-Markov model, and show that the semi-Markov one is more effective for robust estimation of the source appearance.

Recently, many researchers on humanoid robotics are interested in quasi-passive-dynamic walking (quasi-PDW) which is similar to human walking. It is desirable that control parameters in quasi-PDW are automatically adjusted because robots often suffer from changes in their physical parameters and the surrounding environment. Reinforcement learning (RL) can be a key technology to this adaptability, and it has been shown that RL realizes quasi-PDW in a simulation study. To apply the existing method to controlling real robots, however, requires further improvement to accelerate its learning, otherwise the robots will break down before acquiring appropriate controls. To accelerate the learning, this study employs off-policy natural actor-critic (off-NAC), and applies it to an acquisition problem of quasi-PDW. The most important feature of the off-NAC is that it reuses the samples that has already been obtained by previous controllers. This study also shows an adaptive method of the learning rate. Simulation as well as real experiments demonstrate that fast and stable learning of quasi-PDW of an unstable biped robot can be realized by our modified off-NAC

BACKGROUND: Various statistical scores have been proposed for evaluating the significance of genes that may exhibit differential expression between two or more controlled conditions. However, in many clinical studies to detect clinical marker genes for example, the conditions have not necessarily been controlled well, thus condition labels are sometimes hard to obtain due to physical, financial, and time costs. In such a situation, we can consider an unsupervised case where labels are not available or a semi-supervised case where labels are available for a part of the whole sample set, rather than a well-studied supervised case where all samples have their labels. RESULTS: We assume a latent variable model for the expression of active genes and apply the optimal discovery procedure (ODP) proposed by Storey (2005) to the model. Our latent variable model allows gene significance scores to be applied to unsupervised and semi-supervised cases. The ODP framework improves detectability by sharing the estimated parameters of null and alternative models of multiple tests over multiple genes. A theoretical consideration leads to two different interpretations of the latent variable, i.e., it only implicitly affects the alternative model through the model parameters, or it is explicitly included in the alternative model, so that the interpretations correspond to two different implementations of ODP. By comparing the two implementations through experiments with simulation data, we have found that sharing the latent variable estimation is effective for increasing the detectability of truly active genes. We also show that the unsupervised and semi-supervised rating of genes, which takes into account the samples without condition labels, can improve detection of active genes in real gene discovery problems. CONCLUSION: The experimental results indicate that the ODP framework is effective for hypotheses including latent variables and is further improved by sharing the estimations of hidden variables over multiple tests.

In cluster analyses, setting the scale parameter which is implicitly related to the complexity of the data distribution is an important issue; different scale values lead to different results and hence cause different interpretation. In this study, we discuss a framework of multi-scale clustering, where clustering is done with multiple scale values and then the obtained results are compiled into a visually appropriate form to observe overall structures of the clusters. For such purpose, a brick view method is proposed in this study. The construction of a brick view diagram consists of a reindexing procedure of clusters obtained with various scale values and a sorting procedure of samples so as to minimize the distortion defined based on the multiple clustering results. Although some popular clustering methods, such as K-means, spherical K-means, and hierarchical clustering, have been used within the multi-scale framework, we introduce mean-shift clustering based on the kernel density estimation for directional data. We evaluate our approach and existing hierarchical clustering by using an artificial data set and a real data set of gene expression profiles. The results show global structures of distributions can be observed well and in a stable manner, in the brick view diagram.

A class of biped locomotion called passive dynamic walking (PDW) has been recognized to be efficient in energy consumption and a key to understand human walking. Although PDW is sensitive to the initial condition and disturbances, some studies of quasi-PDW, which introduces supplementary actuators, are reported to overcome the sensitivity. In this article, for realization of the quasi-PDW, an on-line learning scheme of a feedback controller based on a policy gradient reinforcement learning method is proposed. Computer simulations show that the parameter in a quasi-PDW controller is automatically tuned by our method utilizing the passivity of the robot dynamics. The obtained controller is robust against variations in the slope gradient to some extent.

In this letter, we propose a noisy nonlinear version of independent component analysis (ICA). Assuming that the probability density function (p. d. f.) of sources is known, a learning rule is derived based on maximum likelihood estimation (MLE). Our model involves some algorithms of noisy linear ICA (e. g., Bermond & Cardoso, 1999) or noise-free nonlinear ICA (e. g., Lee, Koehler, & Orglmeister, 1997) as special cases. Especially when the nonlinear function is linear, the learning rule derived as a generalized expectation-maximization algorithm has a similar form to the noisy ICA algorithm previously presented by Douglas, Cichocki, and Amari (1998). Moreover, our learning rule becomes identical to the standard noise-free linear ICA algorithm in the noiseless limit, while existing MLE-based noisy ICA algorithms do not rigorously include the noise-free ICA. We trained our noisy nonlinear ICA by using acoustic signals such as speech and music. The model after learning successfully simulates virtual pitch phenomena, and the existence region of virtual pitch is qualitatively similar to that observed in a psychoacoustic experiment. Although a linear transformation hypothesized in the central auditory system can account for the pitch sensation, our model suggests that the linear transformation can be acquired through learning from actual acoustic signals. Since our model includes a cepstrum analysis in a special case, it is expected to provide a useful feature extraction method that has often been given by the cepstrum analysis.

We propose a novel approach to acquire the optimal policy for a continuous Markov decision process. Based on an analogy from statistical mechanics, we introduce a variational free energy over a policy. A good policy can be obtained by minimizing the variational free energy. According to our approach, the optimal policy in linear quadratic regulator problems can be obtained by using Kalman filtering and smoothing techniques. Even in non-linear problems, a semi-optimal policy can be obtained by Monte Carlo technique with a Gaussian process method.

This paper discusses temporal reasoning with respect to constraints on two concurrent sequences of events. If two given sequences of events can be mapped into one sequence that satisfies a given constraint, then the constraint is said to be consistent. First, we mention that the consistency of such constraints is NP-complete. Then we introduce the notion of graph representations of constraints. If a graph representation of a given constraint c can be constructed in polynomial time, then the consistency of c is decidable in polynomial time. However, it is shown that the graph representability of a given c is coNP-complete. Next, we propose a subclass CDC$^{\neq} of constraints such that for each constraint c in CDC$^{\neq}$, a graph representation of c can be constructed in polynomial time. The expressive power of CDC$^{\neq}$ is incomparable to any other subclasses of constraints for which the consistency problem is known to be tractable.

We formulate an automatic strategy acquisition problem for the multi-agent card game "hearts" as a reinforcement learning (RL) problem. Since there are often a lot of unobservable cards in this game, RL is approximately dealt with in the framework of a partially observable Markov decision process (POMDP). This article presents a POMDP-RL method based on estimation of unobservable state variables and prediction of actions of the opponent agents. Simulation results show our model-based POMDP-RL method is applicable to a realistic multi-agent problem.

MOTIVATION: Gene expression profile analyses have been used in numerous studies covering a broad range of areas in biology. When unreliable measurements are excluded, missing values are introduced in gene expression profiles. Although existing multivariate analysis methods have difficulty with the treatment of missing values, this problem has received little attention. There are many options for dealing with missing values, each of which reaches drastically different results. Ignoring missing values is the simplest method and is frequently applied. This approach, however, has its flaws. In this article, we propose an estimation method for missing values, which is based on Bayesian principal component analysis (BPCA). Although the methodology that a probabilistic model and latent variables are estimated simultaneously within the framework of Bayes inference is not new in principle, actual BPCA implementation that makes it possible to estimate arbitrary missing variables is new in terms of statistical methodology. RESULTS: When applied to DNA microarray data from various experimental conditions, the BPCA method exhibited markedly better estimation ability than other recently proposed methods, such as singular value decomposition and K-nearest neighbors. While the estimation performance of existing methods depends on model parameters whose determination is difficult, our BPCA method is free from this difficulty. Accordingly, the BPCA method provides accurate and convenient estimation for missing values. AVAILABILITY: The software is available at http://hawaii.aist-nara.ac.jp/~shige-o/tools/.

This article presents a new theoretical framework to consider the dynamics of a stochastic spiking neuron model with general membrane response to input spike. We assume that the input spikes obey an inhomogeneous Poisson process. The stochastic process of the membrane potential then becomes a gaussian process. When a general type of the membrane response is assumed, the stochastic process becomes a Markov-gaussian process. We present a calculation method for the membrane potential density and the firing probability density. Our new formulation is the extension of the existing formulation based on diffusion approximation. Although the single Markov assumption of the diffusion approximation simplifies the stochastic process analysis, the calculation is inaccurate when the stochastic process involves a multiple Markov property. We find that the variation of the shape of the membrane response, which has often been ignored in existing stochastic process studies, significantly affects the firing probability. Our approach can consider the reset effect, which has been difficult to deal with by analysis based on the first passage time density.

In this article, we discuss the reconstruction of chaotic dynamics in a partial observation situation. As a function approximator, we employ a normalized Gaussian network (NGnet), which is trained by an on-line EM algorithm. In order to deal with the partial observation, we propose a new embedding method based on smoothing filters, which is called integral embedding. The NGnet is trained to learn the dynamical system in the integral coordinate space. Experimental results show that the trained NGnet is able to reproduce a chaotic attractor that well approximates the complexity and instability of the original chaotic attractor, even when the data involve large noise. In comparison with our previous method using delay coordinate embedding, this new method is more robust to noise and faster in learning.

In this article, we propose a new reinforcement learning (RL) method for a system having continuous state and action spaces. Our RL method has an architecture like the actor-critic model. The critic tries to approximate the Q-function, which is the expected future return for the current state-action pair. The actor tries to approximate a stochastic soft-max policy defined by the Q-function. The soft-max policy is more likely to select an action that has a higher Q-function value. The online EM algorithm is used to train the critic and the actor. We apply this method to two control problems. Computer simulations show that our method is able to acquire fairly good control in the two tasks after a few learning trials.

In this article, we propose new analog neural approaches to combinatorial optimization problems, in particular, quadratic assignment problems (QAPs). Our proposed methods are based on an analog version of the lambda-opt heuristics, which simultaneously changes assignments for lambda elements in a permutation. Since we can take a relatively large lambda value, our new methods can achieve a middle-range search over possible solutions, and this helps the system neglect shallow local minima and escape from local minima. In experiments, we have applied our methods to relatively large-scale (N = 80 - 150) QAPs. Results have shown that our new methods are comparable to the present champion algorithms; for two benchmark problems, they are obtain better solutions than the previous champion algorithms.

A normalized gaussian network (NGnet) (Moody & Darken, 1989) is a network of local linear regression units. The model softly partitions the input space by normalized gaussian functions, and each local unit linearly approximates the output within the partition. In this article, we propose a new on-line EMalgorithm for the NGnet, which is derived from the batch EMalgorithm (Xu, Jordan, &Hinton 1995), by introducing a discount factor. We show that the on-line EM algorithm is equivalent to the batch EM algorithm if a specific scheduling of the discount factor is employed. In addition, we show that the on-line EM algorithm can be considered as a stochastic approximation method to find the maximum likelihood estimator. A new regularization method is proposed in order to deal with a singular input distribution. In order to manage dynamic environments, where the input-output distribution of data changes over time, unit manipulation mechanisms such as unit production, unit deletion, and unit division are also introduced based on probabilistic interpretation. Experimental results show that our approach is suitable for function approximation problems in dynamic environments. We also apply our on-line EM algorithm to robot dynamics problems and compare our algorithm with the mixtures-of-experts family.