When the information source is a continuous distribution and the rate-distortion function is strictly larger than the Shannon lower bound, the explicit evaluation of the rate-distortion function is not straightforward. We evaluate the rate-distortion function for an independent identically distributed gamma source with respect to the absolute-log distortion measure. The logarithmic transformation reduces this rate-distortion problem to that under the absolute distortion measure. Extending the explicit evaluation of the rate-distortion function for the Gaussian sources, we obtain the parametric form of the rate-distortion function. We show that the optimal distribution of reconstruction consists of a continuous component enclosed by left and right discrete components, and the left discrete component vanishes when the acceptable distortion is small. We further extend the result for a wider class of source distributions.

We discuss a nonparametric estimation method for the mixing distributions in mixture models. The problem is formalized as a minimization of a one-parameter objective functional, which becomes the maximum likelihood estimation or the kernel vector quantization in special cases. Generalizing the theorem for the nonparametric maximum likelihood estimation, we prove the existence and discreteness of the optimal mixing distribution and provide an algorithm to calculate it. It is demonstrated that with an appropriate choice of the parameter, the proposed method is less prone to overfitting than the maximum likelihood method. We further discuss the connection between the unifying estimation framework and the rate-distortion problem.

We evaluate the rate-distortion function for the i.i.d. gamma sources with respect to the absolute-log distortion measure. The logarithmic transformation reduces this rate-distortion problem to that under the absolute distortion measure. Extending the explicit evaluation of the rate-distortion function for the Gaussian sources, we obtain the parametric form of the rate-distortion function. We show that the optimal distribution of reconstruction consists of a continuous component enclosed by left and right discrete components and the left discrete component vanishes when the allowed distortion is small.

This issue marks the beginning of the IEEE Transactions on Neural Networks and Learning Systems (TNNLS). By adding "Learning Systems" to the title, we now state explicitly the scope of the Transactions to include neural networks as well as related learning systems. This issue marks a new era in the history of our Transactions. The Transactions is now ready to face the challenges of the next 10-20 years. With the evolution of the fields of neural networks in particular and computational intelligence in general, the IEEE Transactions on Neural Networks and Learning Systems will continue to grow and to succeed in this ever-changing world. Also included are a few comments about the review process of TNN manuscripts and the introduction of 14 new TNNLS Associate Editors. Short biographies are included for the new Associate Editors.

The information transfer through a single neuron is a fundamental information processing in the brain. This paper studies the information-theoretic capacity of a single neuron by treating the neuron as a communication channel. Two different models are considered. The temporal coding model of a neuron as a communication channel assumes the output is ¿ where ¿ is a gamma-distributed random variable corresponding to the interspike interval, that is, the time it takes for the neuron to fire once. The rate coding model is similar; the output is the actual rate of firing over a fixed period of time. We prove that for both models, the capacity achieving distribution has only a finite number of probability mass points. This allows us to compute numerically the capacity of a neuron. Our capacity results are in a plausible range based on biological evidence to date.

Information transfer through a single neuron is a fundamental component of information processing in the brain, and computing the information channel capacity is important to understand this information processing. The problem is difficult since the capacity depends on coding, characteristics of the communication channel, and optimization over input distributions, among other issues. In this letter, we consider two models. The temporal coding model of a neuron as a communication channel assumes the output is tau where tau is a gamma-distributed random variable corresponding to the interspike interval, that is, the time it takes for the neuron to fire once. The rate coding model is similar; the output is the actual rate of firing over a fixed period of time. Theoretical studies prove that the distribution of inputs, which achieves channel capacity, is a discrete distribution with finite mass points for temporal and rate coding under a reasonable assumption. This allows us to compute numerically the capacity of a neuron. Numerical results are in a plausible range based on biological evidence to date.

This article proposes a new method to improve the reception of the digital satellite broadcasting on a moving vehicle. There are two major factors which make the reception difficult, the Doppler shift and the fading channel. The proposed method does not rely on any diversity receptions nor pilot sequences but uses the channel estimation and stochastic inference. First, Doppler shift is estimated and removed. Then the fading channel is represented with a graphical model, and the parameters are estimated by maximum likelihood estimation. The inference of the code words based on the graphical model is computed and used for decoding. We confirmed the improvements through experiments with numerical simulations and real data. Computational cost is reasonable for real time system and the results are promising.

Turbo and LDPC (low-density parity check) codes are simple and new types of error correction codes which give a powerful and practical performance of error correction. Although experimental results show their efficacy, further theoretical analysis is necessary, which is not straightforward. We have built a unified framework of turbo and LDPC codes based on information geometry. The framework helps our intuitive understanding of the codes and opens a new prospect of further analysis. We have revealed some properties of these codes in the proposed framework. This paper summarizes the results.

Belief propagation (BP) is a universal method of stochastic reasoning. It gives exact inference for stochastic models with tree interactions and works surprisingly well even if the models have loopy interactions. Its performance has been analyzed separately in many fields, such as AI, statistical physics, information theory, and information geometry. This article gives a unified framework for understanding BP and related methods and summarizes the results obtained in many fields. In particular, BP and its variants, including tree reparameterization and concave-convex procedure, are reformulated with information-geometrical terms, and their relations to the free energy function are elucidated from an information-geometrical viewpoint. We then propose a family of new algorithms. The stabilities of the algorithms are analyzed, and methods to accelerate them are investigated.

Since the proposal of turbo codes in 1993, many studies have appeared on this simple and new type of codes which give a powerful and practical performance of error correction. Although experimental results strongly support the efficacy of turbo codes, further theoretical analysis is necessary, which is not straightforward. It is pointed out that the iterative decoding algorithm of turbo codes shares essentially similar ideas with low-density parity-check (LDPC) codes, with Pearl's belief propagation algorithm applied to a cyclic belief diagram, and with the Bethe approximation in statistical physics. Therefore, the analysis of the turbo decoding algorithm will reveal the mystery of those similar iterative methods. In this paper, we recapture and extend the geometrical framework initiated by Richardson to the information geometrical framework of dual affine connections, focusing on both of the turbo and LDPC decoding algorithms. The framework helps our intuitive understanding of the algorithms and opens a new prospect of further analysis. We reveal some properties of these codes in the proposed framework, including the stability and error analysis. Based on the error analysis, we finally propose a correction term for improving the approximation.

Two array signal processing techniques are combined with independent component analysis (ICA) to enhance the performance of blind separation of acoustic signals in a reflective environment. The first technique is the subspace method which reduces the effect of room reflection when the system is used in a room. Room reflection is one of the biggest problems in blind source separation (BSS) in acoustic environments. The second technique is a method of solving permutation. For employing the subspace method, ICA must be used in the frequency domain, and precise permutation is necessary for all frequencies. In this method, a physical property of the mixing matrix, i.e., the coherency in adjacent frequencies, is utilized to solve the permutation. The experiments in a meeting room showed that the subspace method improved the rate of automatic speech recognition from 50% to 68% and that the method of solving permutation achieves performance that closely approaches that of the correct permutation, differing by only 4% in recognition rate.

Two array signal processing techniques are combined with independent component analysis to enhance the performance of blind separation of acoustic signals in a reflective environment such as rooms. The first technique is the subspace method which reduces the effect of room reflection. The second technique is a method of solving the permutation, in which the coherency of the mixing matrix in adjacent frequencies is utilized.

The mystery of belief propagation (BP) decoder, especially of the turbo decoding, is studied from information geometrical viewpoint. The loopy belief network (BN) of turbo codes makes it difficult to obtain the true “belief” by BP, and the characteristics of the algorithm and its equilib- rium are not clearly understood. Our study gives an intuitive understand- ing of the mechanism, and a new framework for the analysis. Based on the framework, we reveal basic properties of the turbo decoding.

We report a result of perturbation analysis on decoding error of the belief propagation decoder for Gallager codes. The analysis is based on infor- mation geometry, and it shows that the principal term of decoding error at equilibrium comes from the m-embedding curvature of the log-linear submanifold spanned by the estimated pseudoposteriors, one for the full marginal, and K for partial posteriors, each of which takes a single check into account, where K is the number of checks in the Gallager code. It is then shown that the principal error term vanishes when the parity-check matrix of the code is so sparse that there are no two columns with overlap greater than 1.

ICA (independent component analysis) is a technique for analyzing multi-variant data. Lots of results are reported in the field of neurobiological data analysis such as EEG (electroencephalography), MRI (magnetic resonance imaging), and MEG (magnetoencephalography) using ICA. But there still remain problems. In most of the neurobiological data, there is a large amount of noise, and the number of independent components is unknown which gives difficulties for many ICA algorithms. We discuss an approach to separate noise-contaminated data without knowing the number of independent components. The idea is to replace PCA (principal component analysis), which is used as the preprocessing of many ICA algorithms, with factor analysis. In the new preprocessing, the number of the sources and the amount of the noise are estimated. After the preprocessing, an ICA algorithm is used to estimate the separation matrix and mixing system. Through experiments with MEG data, we show this approach is effective.