The projected belief network (PBN) is a deep layered generative network with tractable likelihood function (LF) and can be used as a Bayesian classifier by training a separate model on each data class, and then classifying based on maximum likelihood (ML). Unlike other generative models with tractable LF, the PBN can share an embodiment and network parameters with a feed-forward classifier network. By training a PBN with a cost function that combines LF with classifier cross-entropy, its network weights can be ``aligned" to the decision boundaries separating the data class from other classes. This results in a Bayesian classifier that rivals state of the art discriminative classifiers. These claims are backed up by classification experiments involving spectrograms of spoken keywords and handwritten characters.

In this paper, a Neural network is derived from first principles, assuming only that each layer begins with a linear dimension-reducing transformation. The approach appeals to the principle of Maximum Entropy (MaxEnt) to find the posterior distribution of the input data of each layer, conditioned on the layer output variables. This posterior has a well-defined mean, the conditional mean estimator, that is calculated using a type of neural network with theoretically-derived activation functions similar to sigmoid, softplus, and relu. This implicitly provides a theoretical justification for their use. A theorem that finds the conditional distribution and conditional mean estimator under the MaxEnt prior is proposed, unifying results for special cases. Combining layers results in an auto-encoder with conventional feed-forward analysis network and a type of linear Bayesian belief network in the reconstruction path.

This paper addresses the duality between the deterministic feed-forward neural networks (FF-NNs) and linear Bayesian networks (LBNs), which are the generative stochastic models representing probability distributions over the visible data based on a linear function of a set of latent (hidden) variables. The maximum entropy principle is used to define a unique generative model corresponding to each FF-NN, called projected belief network (PBN). The FF-NN exactly recovers the hidden variables of the dual PBN. The large- N asymptotic approximation to the PBN has the familiar structure of an LBN, with the addition of an invertible nonlinear transformation operating on the latent variables. It is shown that the exact nature of the PBN depends on the range of the input (visible) data details for the three cases of input data range are provided. The likelihood function of the PBN is straightforward to calculate, allowing it to be used as a generative classifier. An example is provided in which a generative classifier based on the PBN has comparable performance to a deep belief network in classifying handwritten characters. In addition, several examples are provided that demonstrate the duality relationship, for example, by training networks from either side of the duality.

In this paper, we present a novel exponentially embedded families (EEF) based classification method, in which the probability density function (PDF) on raw data is estimated from the PDF on features. With the PDF construction, we show that class-specific features can be used in the proposed classification method, instead of a common feature subset for all classes as used in conventional approaches. We apply the proposed EEF classifier for text categorization as a case study and derive an optimal Bayesian classification rule with class-specific feature selection based on the Information Gain score. The promising performance on real-life data sets demonstrates the effectiveness of the proposed approach and indicates its wide potential applications.

In this paper, we present a Bayesian classification approach for automatic text categorization using class-specific features. Unlike conventional text categorization approaches, our proposed method selects a specific feature subset for each class. To apply these class-specific features for classification, we follow Baggenstoss's PDF Projection Theorem (PPT) to reconstruct the PDFs in raw data space from the class-specific PDFs in low-dimensional feature subspace, and build a Bayesian classification rule. One noticeable significance of our approach is that most feature selection criteria, such as Information Gain (IG) and Maximum Discrimination (MD), can be easily incorporated into our approach. We evaluate our method's classification performance on several real-world benchmarks, compared with the state-of-the-art feature selection approaches. The superior results demonstrate the effectiveness of the proposed approach and further indicate its wide potential applications in data mining.

Recent theoretical advances in class-dependent feature extraction are reviewed. These advances, culminating in the multi-resolution HMM (MR-HMM) statistical model are proposed for the detection and classification of transient signals that are composed of diverse components with widely varying structure and resolution.

This letter is concerned with time-series analysis using overlapped processing windows shaded using the Hanning window function, such as is used in short-time Fourier transform (STFT) analysis. We present a special case where different analysis window sizes produce equivalent outputs where equivalence is defined by the existence of an orthonormal linear transformation relating the two analyses. We apply the concept to the problem of detecting pulses of unknown duration in Gaussian noise. We also demonstrate how the method can be used to apply the PDF projection theorem to shaded overlapped processing windows.

In this paper, we derive a technique for analysis of local distortions which affect data in real-world applications. In the paper, we focus on image data, specifically handwritten characters. Given a reference image and a distorted copy of it, the method is able to efficiently determine the rotations, translations, scaling, and any other distortions that have been applied. Because the method is robust, it is also able to estimate distortions for two unrelated images, thus determining the distortions that would be required to cause the two images to resemble each other. The approach is based on a polynomial series expansion using matrix powers of linear transformation matrices. The technique has applications in pattern recognition in the presence of distortions.

We derive an algorithm similar to the well-known Baum-Welch (1970) algorithm for estimating the parameters of a hidden Markov model (HMM). The new algorithm allows the observation PDF of each state to be defined and estimated using a different feature set. We show that estimating parameters in this manner is equivalent to maximizing the likelihood function for the standard parameterization of the HMM defined on the input data space. The processor becomes optimal if the state-dependent feature sets are sufficient statistics to distinguish each state individually from a common state.

In this paper, a new algorithm based on the Baum-Welch algorithm for estimating the parameters of a hidden Markov model (HMM) is presented. It allows each state to be observed using a different set of features rather than relying on a common feature set. Each feature set is chosen to be a sufficient statistic for discrimination of the given state from a common "white-noise" state. Comparison of likelihood values is possible through the use of likelihood ratios. The new algorithm is the same in theory as the algorithm based on a common feature set, but without the necessity of estimating high-dimensional probability density functions (PDFs). A simulated data example is provided showing superior performance over the conventional HMM.

We present a new approach to the design of probabilistic classifiers. Rather than working with a common high-dimensional feature vector the classifier is written in terms of separate feature vectors chosen specifically for each class and their low-dimensional PDFs. While sufficiency is not a requirement, if the feature vectors are sufficient to distinguish the corresponding class from a common (null) hypothesis, the method is equivalent to the maximum a posteriori probability classifier. The method has applications to speech, image, and general pattern recognition problems.

In the unlabeled data problem, data contains signals from various sources whose identities are not known apriori, yet the parameters of the individual sources must be estimated. To do this optimally, it is necessary to optimize the data PDF which may be modeled as a mixture density, jointly over the parameters of all the signal models. This can present a problem of enormous complexity if the number of signal classes is large. This paper describes a algorithm for jointly estimating the parameters of the various signal types, each with different parameterizations and associated sufficient statistics. In doing so, it maximizes the likelihood function of all the parameters jointly, but does so without incurring the full dimensionality of the problem. It allows lower-dimensional sufficient statistics to be utilized for each signal model, yet still achieves joint optimality. It uses an extension of the class-specific decomposition of the Bayes minimum error probability classifier.