Batch normalization (BN) has proven to be an effective algorithm for deep neural network training by normalizing the input to each neuron and reducing the internal covariate shift. The space of weight vectors in the BN layer can be naturally interpreted as a Riemannian manifold, which is invariant to linear scaling of weights. Following the intrinsic geometry of this manifold provides a new learning rule that is more efficient and easier to analyze. We also propose intuitive and effective gradient clipping and regularization methods for the proposed algorithm by utilizing the geometry of the manifold. The resulting algorithm consistently outperforms the original BN on various types of network architectures and datasets.

Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given that connectionist temporal classification (CTC) is utilized as an objective of learning an MDRNN for sequence labeling, the non-convexity of CTC poses a problem when applying HF to the network. As a solution, a convex approximation of CTC is formulated and its relationship with the EM algorithm and the Fisher information matrix is discussed. An MDRNN up to a depth of 15 layers is successfully trained using HF, resulting in an improved performance for sequence labeling.

This letter looks at the physics behind sensory data by identifying the parameters that govern the physical system and estimating them from sensory observations. We extend Takens's delay-embedding theorem to the dynamical systems controlled by parameters. An embedding of the product space of the phase and the parameter spaces of the dynamical system can be obtained by the delay-embedding map, provided that the parameter of the dynamical system changes slowly. The reconstruction error is bounded for slowly varying parameters. A manifold learning technique is applied to the embedding obtained to extract a low-dimensional global coordinate system representing the product space. The phase space of the deterministic dynamics can be contracted by using the adjacency relationship in time, which enables recovery of only the parameter space. As examples, the manifolds of synthetic and real-world vowels with time-varying fundamental frequency (F(0)) are analyzed, and the F(0) contours are extracted by an unsupervised algorithm. Experimental results show that the proposed method leads to robust performance under various noise conditions and rapid changes of F(0) compared with the current state-of-the-art F(0) estimation algorithms.

BACKGROUND: Nonnegative matrix factorization (NMF) is a feature extraction method that has the property of intuitive part-based representation of the original features. This unique ability makes NMF a potentially promising method for biological sequence analysis. Here, we apply NMF to fold recognition and remote homolog detection problems. Recent studies have shown that combining support vector machines (SVM) with profile-profile alignments improves performance of fold recognition and remote homolog detection remarkably. However, it is not clear which parts of sequences are essential for the performance improvement. RESULTS: The performance of fold recognition and remote homolog detection using NMF features is compared to that of the unmodified profile-profile alignment (PPA) features by estimating Receiver Operating Characteristic (ROC) scores. The overall performance is noticeably improved. For fold recognition at the fold level, SVM with NMF features recognize 30% of homolog proteins at > 0.99 ROC scores, while original PPA feature, HHsearch, and PSI-BLAST recognize almost none. For detecting remote homologs that are related at the superfamily level, NMF features also achieve higher performance than the original PPA features. At > 0.90 ROC50 scores, 25% of proteins with NMF features correctly detects remotely related proteins, whereas using original PPA features only 1% of proteins detect remote homologs. In addition, we investigate the effect of number of positive training examples and the number of basis vectors on performance improvement. We also analyze the ability of NMF to extract essential features by comparing NMF basis vectors with functionally important sites and structurally conserved regions of proteins. The results show that NMF basis vectors have significant overlap with functional sites from PROSITE and with structurally conserved regions from the multiple structural alignments generated by MUSTANG. The correlation between NMF basis vectors and biologically essential parts of proteins supports our conjecture that NMF basis vectors can explicitly represent important sites of proteins. CONCLUSION: The present work demonstrates that applying NMF to profile-profile alignments can reveal essential features of proteins and that these features significantly improve the performance of fold recognition and remote homolog detection.

A high-performance fundamental frequency estimator inspired by the human auditory system is presented. It uses a cubic-root transform and contrast enhancement on the spectral domain to enhance weak harmonics and suppress noise. The proposed weighted unbiasing process reduces multiple or submultiple pitch errors efficiently. Evaluation is performed under the presence of white noise, babble noise, and telephone channel distortion. The results show that the proposed method performs equivalent to or better than previous state-of-the-art methods in the majority of the noise and channel conditions.