The inference of the causal relationship between a pair of observed variables is a fundamental problem in science, and most existing approaches are based on one single causal model. In practice, however, observations are often collected from multiple sources with heterogeneous causal models due to certain uncontrollable factors, which renders causal analysis results obtained by a single model skeptical. In this paper, we generalize the Additive Noise Model (ANM) to a mixture model, which consists of a finite number of ANMs, and provide the condition of its causal identifiability. To conduct model estimation, we propose Gaussian Process Partially Observable Model (GPPOM), and incorporate independence enforcement into it to learn latent parameter associated with each observation. Causal inference and clustering according to the underlying generating mechanisms of the mixture model are addressed in this work. Experiments on synthetic and real data demonstrate the effectiveness of our proposed approach.

Although nonstationary data are more common in the real world, most existing causal discovery methods do not take nonstationarity into consideration. In this letter, we propose a kernel embedding-based approach, ENCI, for nonstationary causal model inference where data are collected from multiple domains with varying distributions. In ENCI, we transform the complicated relation of a cause-effect pair into a linear model of variables of which observations correspond to the kernel embeddings of the cause-and-effect distributions in different domains. In this way, we are able to estimate the causal direction by exploiting the causal asymmetry of the transformed linear model. Furthermore, we extend ENCI to causal graph discovery for multiple variables by transforming the relations among them into a linear nongaussian acyclic model. We show that by exploiting the nonstationarity of distributions, both cause-effect pairs and two kinds of causal graphs are identifiable under mild conditions. Experiments on synthetic and real-world data are conducted to justify the efficacy of ENCI over major existing methods.

In this letter, we study the confounder detection problem in the linear model, where the target variable [Formula: see text] is predicted using its [Formula: see text] potential causes [Formula: see text]. Based on an assumption of a rotation-invariant generating process of the model, recent study shows that the spectral measure induced by the regression coefficient vector with respect to the covariance matrix of [Formula: see text] is close to a uniform measure in purely causal cases, but it differs from a uniform measure characteristically in the presence of a scalar confounder. Analyzing spectral measure patterns could help to detect confounding. In this letter, we propose to use the first moment of the spectral measure for confounder detection. We calculate the first moment of the regression vector-induced spectral measure and compare it with the first moment of a uniform spectral measure, both defined with respect to the covariance matrix of [Formula: see text]. The two moments coincide in nonconfounding cases and differ from each other in the presence of confounding. This statistical causal-confounding asymmetry can be used for confounder detection. Without the need to analyze the spectral measure pattern, our method avoids the difficulty of metric choice and multiple parameter optimization. Experiments on synthetic and real data show the performance of this method.

In this paper, we deal with the problem of inferring causal relations for multidimensional data. Based on the postulate that the distribution of the cause and the conditional distribution of the effect given cause are generated independently, we show that the covariance matrix of the mean embedding of the cause in reproducing kernel Hilbert space (RKHS) is free independent with the covariance matrix of the conditional embedding of the effect given cause. This, called freeness condition, induces a cause-effect asymmetry that a designed measurement is 0 in the causal direction but smaller than 0 in the anticausal direction, and it uncovers the causal direction. One important novel aspect of this paper is that we interpret the independence as a freeness condition between covariance matrices of RKHS distribution embeddings, and it has a wide applicability. We show that our freeness condition-based inference method succeeds in scenarios like additive noise cases, where other methods fail, by theoretical analysis and experimental results.

In this article, we deal with the problem of inferring causal directions when the data are on discrete domain. By considering the distribution of the cause [Formula: see text] and the conditional distribution mapping cause to effect [Formula: see text] as independent random variables, we propose to infer the causal direction by comparing the distance correlation between [Formula: see text] and [Formula: see text] with the distance correlation between [Formula: see text] and [Formula: see text]. We infer that X causes Y if the dependence coefficient between [Formula: see text] and [Formula: see text] is smaller. Experiments are performed to show the performance of the proposed method.

In this article, we deal with the causal discovery problem on discrete data. First, we present a causal discovery method for traditional additive noise models that identifies the causal direction by analyzing the supports of the conditional distributions. Then, we present a causal mixture model to address the problem that the function transforming cause to effect varies across the observations. We propose a novel method called Support Analysis (SA) for causal discovery with the mixture model. Experiments using synthetic and real data are presented to demonstrate the performance of our proposed algorithm.

Causal discovery via the asymmetry between the cause and the effect has proved to be a promising way to infer the causal direction from observations. The basic idea is to assume that the mechanism generating the cause distribution p(x) and that generating the conditional distribution p(y|x) correspond to two independent natural processes and thus p(x) and p(y|x) fulfill some sort of independence condition. However, in many situations, the independence condition does not hold for the anticausal direction; if we consider p(x, y) as generated via p(y)p(x|y), then there are usually some contrived mutual adjustments between p(y) and p(x|y). This kind of asymmetry can be exploited to identify the causal direction. Based on this postulate, in this letter, we define an uncorrelatedness criterion between p(x) and p(y|x) and, based on this uncorrelatedness, show asymmetry between the cause and the effect in terms that a certain complexity metric on p(x) and p(y|x) is less than the complexity metric on p(y) and p(x|y). We propose a Hilbert space embedding-based method EMD (an abbreviation for EMbeDding) to calculate the complexity metric and show that this method preserves the relative magnitude of the complexity metric. Based on the complexity metric, we propose an efficient kernel-based algorithm for causal discovery. The contribution of this letter is threefold. It allows a general transformation from the cause to the effect involving the noise effect and is applicable to both one-dimensional and high-dimensional data. Furthermore it can be used to infer the causal ordering for multiple variables. Extensive experiments on simulated and real-world data are conducted to show the effectiveness of the proposed method.

Causal discovery for high-dimensional observations is a useful tool in many fields such as climate analysis and financial market analysis. A linear Trace method has been proposed to identify the causal direction between two linearly coupled high-dimensional observations X and Y. However, in reality, the relations between X and Y are usually nonlinear and consequently the linear Trace method may fail. In this paper, we propose a method to infer the nonlinear causal relations for two high-dimensional observations X and Y. The idea is to map the observations to high dimensional Reproducing Kernel Hilbert Space (RKHS) such that the nonlinear relations become simple linear ones. We show that the linear Trace condition holds for the causal direction but it is violated for the anti-causal direction in RKHS. Based on this theoretical result, we develop a simple algorithm to infer the causal direction for nonlinearly coupled causal pairs. Synthetic data and real world data experiments are conducted to show the effectiveness of our proposed method.

LiNGAM has been successfully applied to some real-world causal discovery problems. Nevertheless, causal sufficiency is assumed; that is, there is no latent confounder of the observations, which may be unrealistic for real-world problems. Taking into the consideration latent confounders will improve the reliability and accuracy of estimations of the real causal structures. In this letter, we investigate a model called linear nongaussian acyclic models in the presence of latent gaussian confounders (LiNGAM-GC) which can be seen as a specific case of lvLiNGAM. This model includes the latent confounders, which are assumed to be independent gaussian distributed and statistically independent of the disturbances. To tackle the causal discovery problem of this model, first we propose a pairwise cumulant-based measure of causal directions for cause-effect pairs. We prove that in spite of the presence of latent gaussian confounders, the causal direction of the observed cause-effect pair can be identified under the mild condition that the disturbances are simultaneously supergaussian or subgaussian. We propose a simple and efficient method to detect the violation of this condition. We extend our work to multivariate causal network discovery problems. Specifically we propose algorithms to estimate the causal network structure, including causal ordering and causal strengths, using an iterative root finding-removing scheme based on pairwise measure. To address the redundant edge problem due to the finite sample size effect, we develop an efficient bootstrapping-based pruning algorithm. Experiments on synthetic data and real-world data have been conducted to show the applicability of our model and the effectiveness of our proposed algorithms.

In conventional causal discovery, structural equation models (SEM) are directly applied to the observed variables, meaning that the causal effect can be represented as a function of the direct causes themselves. However, in many real world problems, there are significant dependencies in the variances or energies, which indicates that causality may possibly take place at the level of variances or energies. In this paper, we propose a probabilistic causal scale-mixture model with spatiotemporal variance dependencies to represent a specific type of generating mechanism of the observations. In particular, the causal mechanism including contemporaneous and temporal causal relations in variances or energies is represented by a Structural Vector AutoRegressive model (SVAR). We prove the identifiability of this model under the non-Gaussian assumption on the innovation processes. We also propose algorithms to estimate the involved parameters and discover the contemporaneous causal structure. Experiments on synthesis and real world data are conducted to show the applicability of the proposed model and algorithms.

Many applications naturally involve time series data and the vector autoregression (VAR), and the structural VAR (SVAR) are dominant tools to investigate relations between variables in time series. In the first part of this work, we show that the SVAR method is incapable of identifying contemporaneous causal relations for Gaussian process. In addition, least squares estimators become unreliable when the scales of the problems are large and observations are limited. In the remaining part, we propose an approach to apply Bayesian network learning algorithms to identify SVARs from time series data in order to capture both temporal and contemporaneous causal relations, and avoid high-order statistical tests. The difficulty of applying Bayesian network learning algorithms to time series is that the sizes of the networks corresponding to time series tend to be large, and high-order statistical tests are required by Bayesian network learning algorithms in this case. To overcome the difficulty, we show that the search space of conditioning sets d-separating two vertices should be a subset of the Markov blankets. Based on this fact, we propose an algorithm enabling us to learn Bayesian networks locally, and make the largest order of statistical tests independent of the scales of the problems. Empirical results show that our algorithm outperforms existing methods in terms of both efficiency and accuracy.

Dependency analysis is a typical approach for Bayesian network learning, which infers the structures of Bayesian networks by the results of a series of conditional independence (CI) tests. In practice, testing independence conditioning on large sets hampers the performance of dependency analysis algorithms in terms of accuracy and running time for the following reasons. First, testing independence on large sets of variables with limited samples is not stable. Second, for most dependency analysis algorithms, the number of CI tests grows at an exponential rate with the sizes of conditioning sets, and the running time grows of the same rate. Therefore, determining how to reduce the number of CI tests and the sizes of conditioning sets becomes a critical step in dependency analysis algorithms. In this article, we address a two-phase algorithm based on the observation that the structures of Markov random fields are similar to those of Bayesian networks. The first phase of the algorithm constructs a Markov random field from data, which provides a close approximation to the structure of the true Bayesian network; the second phase of the algorithm removes redundant edges according to CI tests to get the true Bayesian network. Both phases use Markov blanket information to reduce the sizes of conditioning sets and the number of CI tests without sacrificing accuracy. An empirical study shows that the two-phase algorithm performs well in terms of accuracy and efficiency.

Many applications naturally involve time series data, and the vector auto regression (VAR) and the structural VAR (SVAR) are dominant tools to investigate relations between variables in time series. In the first part of this work, we show that the SVAR method is incapable of identifying contemporaneous causal relations when data follow Gaussian distributions. In addition, least squares estimators become unreliable when the scales of the problems are large and observations are limited. In the remaining part, we propose an approach to apply Bayesian network learning algorithms to identify SVARs from time series data in order to capture both temporal and contemporaneous causal relations and avoid high-order statistical tests. The difficulty of applying Bayesian network learning algorithms to time series is that the sizes of the networks corresponding to time series tend to be large and high-order statistical tests are required by Bayesian network learning algorithms in this case. To overcome the difficulty, we show that the search space of conditioning sets d-separating two vertices should be subsets of Markov blankets. Based on this fact, we propose an algorithm learning Bayesian networks locally and making the largest order of statistical tests independent of the scales of the problems. Empirical results show that our algorithm outperforms existing methods in terms of both efficiency and accuracy.

Permutation indeterminacy and scaling ambiguity occur in ICA and they are particularly problematic in time-frequency domain separation of convolutive mixtures. The quality of separation is severely degraded if these two problems are not well addressed. In this paper, we propose new approaches to solve the permutation indeterminacy and scaling ambiguity in the separation of convolutive mixture in frequency domain. We first apply Short Time Fourier Transform to the observed signals in order to transform the convolutive mixing in time domain to instantaneous mixing in time-frequency domain. A fixed-point algorithm with test of saddle point is adopted to derive the separated components in each frequency bin. To solve the permutation problem,we propose a new matching algorithm for this purpose. First we use discrete Haar Wavelet Transform to extract the feature vectors from the magnitude waveforms of the separated components and use Singular Value Decomposition to achieve dimension reduction. The permutation problem is solved by clustering the feature vectors using the new matching algorithm which is a combination of basic K-means and Hungarian algorithm. To solve the scaling ambiguity problem, we treat it as an overcomplete problem and realize it by maximizing the posterior of the scaling factor. Finally, experiments are conducted using benchmark data to present the effectiveness and performance of our proposed algorithms.

Bayesian network learning algorithms have been widely used for causal discovery since the pioneer work [13,18]. Among all existing algorithms, three-phase dependency analysis algorithm (TPDA) [5] is the most efficient one in the sense that it has polynomial-time complexity. However, there are still some limitations to be improved. First, TPDA depends on mutual information-based conditional independence (CI) tests, and so is not easy to be applied to continuous data. In addition, TPDA uses two phases to get approximate skeletons of Bayesian networks, which is not efficient in practice. In this paper, we propose a two-phase algorithm with partial correlation-based CI tests: the first phase of the algorithm constructs a Markov random field from data, which provides a close approximation to the structure of the true Bayesian network; at the second phase, the algorithm removes redundant edges according to CI tests to get the true Bayesian network. We show that two-phase algorithm with partial correlation-based CI tests can deal with continuous data following arbitrary distributions rather than only Gaussian distribution.

This paper concerns the discovery of Order Preserving Clusters (OP-Clusters) in microarray data, in each of which a subset of genes induce a similar linear ordering along a subset of conditions. By converting each gene expression vector into an ordered label sequence, we transfer the problem into finding frequent orders appearing in the sequence set. We present two heuristic algorithms Growing Prefix and Suffix (GPS) and Growing Frequent Position (GFP), to solve this problem. Their performance is evaluated empirically using synthetic and real microarray data. The result shows our approaches are effective and efficient and outperform existing methods in many aspects. The two proposed algorithms, GPS and GFP, both have good scale-up properties with the dimension of the dataset and the size of the clusters. They have comparable performance, albeit GPS gets higher precision, whereas GFP has lower computation cost.

Nonlinear ICA may not result in nonlinear blind source separation, since solutions to nonlinear ICA are highly non-unique. In practice, the nonlinearity in the data generation procedure is usually not strong. Thus it is reasonable to select the solution with the mixing procedure close to linear. In this paper we propose to solve nonlinear ICA with the "minimal nonlinear distortion" principle. This is achieved by incorporating a regularization term to minimize the mean square error between the mixing mapping and the best-fitting linear one. As an application, the proposed method helps to identify linear, non-Gaussian, and acyclic causal models when mild nonlinearity exists in the data generation procedure. Using this method to separate daily returns of a set of stocks, we successfully identify their linear causal relations. The resulting causal relations give some interesting insights into the stock market.

Blind separation of convolutive mixtures by minimizing the mutual information between output sequences can avoid the side effect of temporally whitening the outputs, but it involves the score function difference, whose estimation may be problematic when the data dimension is greater than two. This greatly limits the application of this method. Fortunately, for separating convolutive mixtures, pairwise independence of outputs leads to their mutual independence. As an implementation of this idea, we propose a way to separate convolutive mixtures by enforcing pairwise independence. This approach can be applied to separate convolutive mixtures of a moderate number of sources.

Over the past decade, application of reinforcement learning (RL) in asset allocation and portfolio management has attracted much attention. However, most classical RL algorithms do not take risk into account, which may lead to treacherous trading decisions. In this paper, we propose a risk-averse RL method, named reward adjustment reinforcement learning. Our method incorporates risk to the classical RL framework by adjusting the reward with a risk penalty obtained from the GARCH model. This approach is generally easy in implementation and analysis when compared with other risk-averse models. Analysis is given to reveal the connection between our method and existing risk-averse RL methods. Experiment results on artificial data and real data in Hong Kong stock market are provided to compare the performances of our method and risk-sensitive RL algorithm and to illustrate the superiority of our method on generalization performance.

Subband decomposition ICA (SDICA), an extension of ICA, assumes that each source is represented as the sum of some independent subcomponents and dependent subcomponents, which have different frequency bands. In this article, we first investigate the feasibility of separating the SDICA mixture in an adaptive manner. Second, we develop an adaptive method for SDICA, namely band-selective ICA (BS-ICA), which finds the mixing matrix and the estimate of the source independent subcomponents. This method is based on the minimization of the mutual information between outputs. Some practical issues are discussed. For better applicability, a scheme to avoid the high-dimensional score function difference is given. Third, we investigate one form of the overcomplete ICA problems with sources having specific frequency characteristics, which BS-ICA can also be used to solve. Experimental results illustrate the success of the proposed method for solving both SDICA and the over-complete ICA problems.

In independent component analysis (ICA), when using the one-unit contrast function optimization approach to estimate independent components one by one, the constraint of uncorrelatedness between independent components prevents the algorithm from converging to the previously found components. A popular way to achieve uncorrelatedness is the Gram-Schmidt-like decorrelation scheme. In fact, uncorrelatedness between independent components can be achieved by reducing the degree of freedom in the unknown parameter set of the de-mixing matrix. In this letter, we propose to exploit the dimension-reduction technique to exactly enforce uncorrelatedness between difference independent components. The advantage of this method is that dimension reduction of the observations and de-mixing weight vectors makes the computation complexity lower and produces a faster convergence. Hence, our method results in a faster algorithm in computation of ICA.

The linear mixture model has been investigated in most articles tackling the problem of blind source separation. Recently, several articles have addressed a more complex model: blind source separation (BSS) of post-nonlinear (PNL) mixtures. These mixtures are assumed to be generated by applying an unknown invertible nonlinear distortion to linear instantaneous mixtures of some independent sources. The gaussianization technique for BSS of PNL mixtures emerged based on the assumption that the distribution of the linear mixture of independent sources is gaussian. In this letter, we review the gaussianization method and then extend it to apply to PNL mixture in which the linear mixture is close to gaussian. Our proposed method approximates the linear mixture using the Cornish-Fisher expansion. We choose the mutual information as the independence measurement to develop a learning algorithm to separate PNL mixtures. This method provides better applicability and accuracy. We then discuss the sufficient condition for the method to be valid. The characteristics of the nonlinearity do not affect the performance of this method. With only a few parameters to tune, our algorithm has a comparatively low computation. Finally, we present experiments to illustrate the efficiency of our method.

In this paper, firstly we show that the problem of blind inversion of Wiener systems is a special case of blind separation of post-nonlinear instantaneous mixtures approximately, and derive the learning rule for the former problem using this relationship. Secondly, we review the Gaussianization method for blind inversion of Wiener systems. Based on the fact that the convolutive mixture is close to Gaussian, this method roughly approximates the convolutive mixture by a Gaussian variable and constructs the inverse nonlinearity easily. Thirdly, in order to improve the performance, the Cornish-Fisher expansion is exploited to model the latent convolutive mixture, and then the extended Gaussianization method is developed. We show that the performance of our method is insensitive to the nonlinearity in the Wiener system. Experimental results are presented to illustrate the validity and efficiency of our method.

In this paper, we use block-diagonal matrix of approximate the Hessian matrix in the Levenberg Marquardt method during the training of recurrent neural networks. We analyze the weight updating strategies and the groupings of the weights associated with the approximation. Two weight updating strategies, namely asynchronous and synchronous updating methods are investigated. Asynchronous method updated weights of one block at a time while synchronous method updates all weights at the same time. Variations of these two methods, which involve the determination of two parameters mu and lambda, are examined. Four weight grouping methods, correlation blocks, k-unit blocks, layer blocks and arbitrary blocks are investigated and compared. Their computational complexity, approximation ability, and training time is analyzed. Comparing with the original Levenberg Marquardt method, the block-diagonal approximation methods give substantial improvement in training time without degrading the generalization ability.

Holistic parsers offer a viable alternative to traditional algorithmic parsers. They have good generalization performance and are robust inherently. In a holistic parser, parsing is achieved by mapping the connectionist representation of the input sentence to the connectionist representation of the target parse tree directly. Little prior knowledge of the underlying parsing mechanism thus needs to be assumed. However, it also makes holistic parsing difficult to understand. In this article, an analysis is presented for studying the operations of the confluent preorder parser (CPP). In the analysis, the CPP is viewed as a dynamical system, and holistic parsing is perceived as a sequence of state transitions through its state-space. The seemingly one-shot parsing mechanism can thus be elucidated as a step-by-step inference process, with the intermediate parsing decisions being reflected by the states visited during parsing. The study serves two purposes. First, it improves our understanding of how grammatical errors are corrected by the CPP. The occurrence of an error in a sentence will cause the CPP to deviate from the normal track that is followed when the original sentence is parsed. But as the remaining terminals are read, the two trajectories will gradually converge until finally the correct parse tree is produced. Second, it reveals that having systematic parse tree representations alone cannot guarantee good generalization performance in holistic parsing. More important, they need to be distributed in certain useful locations of the representational space. Sentences with similar trailing terminals should have their corresponding parse tree representations mapped to nearby locations in the representational space. The study provides concrete evidence that encoding the linearized parse trees as obtained via preorder traversal can satisfy such a requirement.

Recursive least squares (RLS)-based algorithms are a class of fast online training algorithms for feedforward multilayered neural networks (FMNNs). Though the standard RLS algorithm has an implicit weight decay term in its energy function, the weight decay effect decreases linearly as the number of learning epochs increases, thus rendering a diminishing weight decay effect as training progresses. In this paper, we derive two modified RLS algorithms to tackle this problem. In the first algorithm, namely, the true weight decay RLS (TWDRLS) algorithm, we consider a modified energy function whereby the weight decay effect remains constant, irrespective of the number of learning epochs. The second version, the input perturbation RLS (IPRLS) algorithm, is derived by requiring robustness in its prediction performance to input perturbations. Simulation results show that both algorithms improve the generalization capability of the trained network.

We use the block-diagonal matrix to approximate the Hessian matrix in the Levenberg Marquardt method for the training of recurrent neural networks. Substantial improvement of the training time over the original Levenberg Marquardt method is observed without degrading the generalization ability. Three weight grouping methods, correlation blocks, k-unit blocks and layer blocks were investigated and compared. Their computational complexity, approximation ability, and training time are analyzed.

Ensemble of networks has been proven to give better prediction result than a single network. Two commonly used methods of determining the ensemble weights are simple average ensemble method and the generalized ensemble method. In the paper, we propose a weighted least square ensemble network. The major difference between this method and the other ensemble methods is that we do not assume that neither individual training data nor networks in the ensemble are independent and uncorrelated. Two variances of this model are also introduced, which require fewer computations. The sunspot data was used as a benchmark test of the proposed methods. From the result, we find that for the correlation ensemble, one variance of the weighted least square method gave the best ensemble weightings.

We propose the block-diagonal matrix to approximate the Hessian matrix in the Levenberg-Marquardt method in the training of neural networks. Two weight updating strategies, namely asynchronous and synchronous updating methods, were investigated. Asynchronous method updates weights of one block at a time while synchronous method updates all weights at the same time. Variations of these two methods, which involves the determination of the parameters /spl mu/ and /spl lambda/, are examined.

Connectionist holistic parsing offers a viable and attractive alternative to traditional algorithmic parsers. With exposure to a limited subset of grammatical sentences and their corresponding parse trees only, a holistic parser is capable of learning inductively the grammatical regularity underlying the training examples that affects the parsing process. In the past, various connectionist parsers have been proposed. Each approach had its own unique characteristics, and yet some techniques were shared in common. In this article, various dimensions underlying the design of a holistic parser are explored, including the methods to encode sentences and parse trees, whether a sentence and its corresponding parse tree share the same representation, the use of confluent inference, and the inclusion of phrases in the training set. Different combinations of these design factors give rise to different holistic parsers. In succeeding discussions, we scrutinize these design techniques and compare the performances of a few parsers on language parsing, including the confluent preorder parser, the backpropagation parsing network, the XERIC parser of Berg (1992), the modular connectionist parser of Sharkey and Sharkey (1992), Reilly's (1992) model, and their derivatives. Experiments are performed to evaluate their generalization capability and robustness. The results reveal a number of issues essential for building an effective holistic parser.

Pruning a neural network to a reasonable smaller size, and if possible to give a better generalization, has long been investigated. Conventionally the common technique of pruning is based on considering error sensitivity measure, and the nature of the problem being solved is usually stationary. In this article, we present an adaptive pruning algorithm for use in a nonstationary environment. The idea relies on the use of the extended Kalman filter (EKF) training method. Since EKF is a recursive Bayesian algorithm, we define a weight-importance measure in term of the sensitivity of a posteriori probability. Making use of this new measure and the adaptive nature of EKF, we devise an adaptive pruning algorithm called adaptive Bayesian pruning. Simulation results indicate that in a noisy nonstationary environment, the proposed pruning algorithm is able to remove network redundancy adaptively and yet preserve the same generalization ability.

Recently we have proposed a simple circuit of winner-take-all (WTA) neural network. Assuming no external input, we have derived an analytic equation for its network response time. In this paper, we further analyze the network response time for a class of winner-take-all circuits involving self-decay and show that the network response time of such a class of WTA is the same as that of the simple WTA model.

We present a variant to the DCT-based block algorithm proposed in Hsu and Wu (1996) for signal embedding in digital images. Instead of inter-block relations, our algorithm uses intra-block relations to generate the watermarked image. We describe the algorithm and its performance against translation and cropping. The features of our method are: (1) the watermark is perceptually invisible; (2) little loss of relevant information of original image; (3) the watermark can be retrieved by using a secret key; and (4) the watermark is robust against translation and area cropping.

Pruning is one of the effective techniques for improving the generalization error of neural networks. Existing pruning techniques are derived mainly from the viewpoint of energy minimization, which is commonly used in gradient-based learning methods. In recurrent networks, extended Kalman filter (EKF)-based training has been shown to be superior to gradient-based learning methods in terms of speed. This article explains a pruning procedure for recurrent neural networks using EKF training. The sensitivity of a posterior probability is used as a measure of the importance of a weight instead of error sensitivity since posterior probability density is readily obtained from this training method. The pruning procedure is tested using three problems: (1) the prediction of a simple linear time series, (2) the identification of a nonlinear system, and (3) the prediction of an exchange-rate time series. Simulation results demonstrate that the proposed pruning method is able to reduce the number of parameters and improve the generalization ability of a recurrent network.

This paper describes work on developing a large vocabulary speech database for Cantonese. As a major Chinese dialect, Cantonese is spoken by tens of millions of people in Southern China and Hong Kong. It is very different from Mandarin or Putonghua in phonology, phonetics, vocabulary and grammatical structure. A speech database specially designed for Cantonese is urgently needed for the design, implementation and performance evaluation of various speech recognition systems. The proposed database contains a large number of speech utterances which include isolated syllables, polysyllabic words and phonetically rich sentences. It covers most of the intra-syllable and inter-syllable acoustic variations.

In this paper, syntactic parsing is discussed in the context of connectionism, a new model, the confluent preorder parser (CPP), is proposed which exemplifies the holistic parsing paradigm. Holistic parsing has the advantage that little knowledge has to be assumed concerning the detailed parsing algorithm. This algorithm is often unkown or debatable, especially when human language understanding is concerned. In the CPP, syntactic parsing is achieved by transforming from the connectionist representation of the sentence to the connectionist representation of the preorder traversal of its parse tree, instead of to the representation of the parse tree itself. As revealed by the simulation experiments, generalization performance is excellent (as high as 90%). Also, the CPP is capable of parsing erroneous sentences and resolving lexical category ambiguities. A systematic study is conducted to explore the range of factors which can affect the effectiveness of the system. The error-recovery capability is especially useful in natural language processing when incomplete or even ungrammatical sentences must be dealt with.

Forgetting learning is an incremental learning rule in associative memories. With it, the recent learning items can be encoded, and the old learning items will be forgotten. In this article, we analyze the storage behavior of bidirectional associative memory (BAM) under the forgetting learning. That is, “Can the most recent k learning item be stored as a fixed point?” Also, we discuss how to choose the forgetting constant in the forgetting learning such that the BAM can correctly store as many as possible of the most recent learning items. Simulation is provided to verify the theoretical analysis.

In the transmission of vector quantized data, the vector quantizer and the communication system are usually designed separately. With such an approach, the channel noise results in significant degradations in the performance of the vector quantizer. To solve this problem, we should properly create the mapping from the codebook of the quantizer to the channel signal set of the communication system. This paper proposes a new approach to construct such a mapping based on the ordering property of the self-organizing feature map (SOFM). We use the neighborhood structure of the SOFM and the neighborhood structure of the channel signal set to construct the mapping. Simulation results confirm that the proposed approach is robust with respect to channel noise.

In this paper, a bidirectional associative memory (BAM) model with second-order connections, namely second-order bidirectional associative memory (SOBAM), is first reviewed. The stability and statistical properties of the SOBAM are then examined. We use an example to illustrate that the stability of the SOBAM is not guaranteed. For this result, we cannot use the conventional energy approach to estimate its memory capacity. Thus, we develop the statistical dynamics of the SOBAM. Given that a small number of errors appear in the initial input, the dynamics shows how the number of errors varies during recall. We use the dynamics to estimate the memory capacity, the attraction basin, and the number of errors in the retrieved items. Extension of the results to higher-order bidirectional associative memories is also discussed.

This paper presents an algorithm to form a topographic map resembling to the self-organizing map. The idea stems on defining an energy function which reveals the local correlation between neighboring neurons. The larger the value of the energy function, the higher the correlation of the neighborhood neurons. On this account, the proposed algorithm is defined as the gradient ascent of this energy function. Simulations on two-dimensional maps are illustrated.

Describes a new method of utilizing recurrent neural networks (RNNs) for speech modeling and speech recognition. For each particular speech unit, a fully connected recurrent neural network is built such that the static and dynamic speech characteristics are represented simultaneously by a specific temporal pattern of neuron activation states. By using the temporal RNN output, an input utterance can be represented as a number of stationary speech segments, which may be related to the basic phonetic components of the speech unit. An efficient self-supervised training algorithm has been developed for the RNN speech model. The segmentation for input utterances and the statistical modeling for individual phonetic segments are performed interactively in this training process. Some experimental results are used to demonstrate how the proposed RNN speech model can be used effectively for automatic recognition of isolated speech utterances.

In our previous work #1#, a novel method of utilizing a set of fully connected recurrent neural networks #RNNs# for speech modeling has been proposed. Despite the e#ectiveness of the RNN model in characterizing individual speech units, the system performs less satisfactorily for speech recognition due to poor discrimination between models. In this paper, an e#cient discriminative training procedure is developed for the RNN based recognition system. By using discriminative training, each RNN speech model is adjusted to reduce its distance from the designated speech unit while increase distances from the others. In addition, a duration-screening process is introduced to enhance the discriminating power of the recognition system. Speaker-dependent recognition experiments have been carried out for 1# 11 isolated Cantonese digits, 2# 58 very confusing Cantonese CV syllables, and 3# 20 English isolated words. The recognition rates attained are 90.9#, 86.7# and 93.5# respectively. I. Int...

Tone identification is essential for the recognition of the Chinese language, specifically far Cantonese which is well known for being very rich in tones. The paper presents an efficient method for tone recognition of isolated Cantonese syllables. Suprasegmental feature parameters are extracted from the voiced portion of a monosyllabic utterance and a three-layer feedforward neural network is used to classify these feature vectors. Using a phonologically complete vocabulary of 234 distinct syllables, the recognition accuracy for single-speaker and multispeaker is given by 89.0% and 87.6% respectively.>

The stability, capacity and statistical dynamics of second-order bidirectional associative memory (BAM) are presented here. We first use an example to illustrate that the state of second-order BAR I may converge to limited cycles. When error in the retrieved pairs is not allowed, a lower bound of memory capacity is derived. That is O(min(n/sup 2//(log n),p/sup 2//(log p))) where n and p are the dimensions of the library pairs. Since the state of second-order BAM may converge to limited cycles, the conventional method cannot be used to estimate its memory capacity when small errors in the retrieval pairs are allowed. Hence, the statistical dynamics of second-order BAM is introduced: starting with an initial state close to the library pairs, how the confidence interval of the number of errors changes during recalling. From the dynamics, the attraction basin, memory capacity, and final error in the retrieval pairs can be estimated. Also, some numerical results are given. Finally, an extension of the results to higher-order BAM is discussed.>

A novel method using neural networks for translational invariant object recognition is described in this paper. The objective is to enable the recognition of objects in any shifted position when the objects are presented to the network in only one standard location during the training procedure. With the presence of multiple or overlapped objects in the scene, translational invariant object recognition is a very difficult task. Noise corruption of the image creates another difficulty. In this paper, a novel approach is proposed to tackle this problem, using neural networks with the consideration of multiple objects and the presence of noise. This method utilizes the secondary responses activated by the backpropagation network. A confirmative network is used to obtain the object identification and location, based on these secondary responses. Experimental results were used to demonstrate the ability of this approach.