We introduce *Error Broadcast and Decorrelation* (EBD), a novel learning framework for neural networks that addresses credit assignment by directly broadcasting output errors to individual layers, circumventing weight transport of backpropagation. EBD is rigorously grounded in the stochastic orthogonality property of Minimum Mean Square Error estimators. This fundamental principle states that the error of an optimal estimator is orthogonal to functions of the input. Guided by this insight, EBD defines layerwise loss functions that directly penalize correlations between layer activations and output errors, thereby establishing a principled foundation for error broadcasting. This theoretically sound mechanism naturally leads to the experimentally observed three-factor learning rule and integrates with biologically plausible frameworks to enhance performance and plausibility. Numerical experiments demonstrate EBD’s competitive or better performance against other error-broadcast methods on benchmark datasets. Our findings establish EBD as an efficient, biologically plausible, and principled alternative for neural network training.

We introduce a Bayesian perspective for the structured matrix factorization problem. The proposed framework provides a probabilistic interpretation for existing geometric methods based on determinant minimization. We model input data vectors as linear transformations of latent vectors drawn from a distribution uniform over a particular domain reflecting structural assumptions, such as the probability simplex in Nonnegative Matrix Factorization and polytopes in Polytopic Matrix Factorization. We represent the rows of the linear transformation matrix as vectors generated independently from a normal distribution whose covariance matrix is inverse Wishart distributed. We show that the corresponding maximum a posteriori estimation problem boils down to the robust determinant minimization approach for structured matrix factorization, providing insights about parameter selections and potential algorithmic extensions.

The backpropagation algorithm has experienced remarkable success in training large-scale artificial neural networks; however, its biological plausibility has been strongly criticized, and it remains an open question whether the brain employs supervised learning mechanisms akin to it. Here, we propose correlative information maximization between layer activations as an alternative normative approach to describe the signal propagation in biological neural networks in both forward and backward directions. This new framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm. The coordinate descent-based optimization of the corresponding objective, combined with the mean square error loss function for fitting labeled supervision data, gives rise to a neural network structure that emulates a more biologically realistic network of multi-compartment pyramidal neurons with dendritic processing and lateral inhibitory neurons. Furthermore, our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths, a significant critique against the plausibility of the conventional backpropagation algorithm. This is achieved by leveraging two alternative, yet equivalent forms of the correlative mutual information objective. These alternatives intrinsically lead to forward and backward prediction networks without weight symmetry issues, providing a compelling solution to this long-standing challenge.

Polytopic matrix factorization (PMF) is a recently introduced matrix decomposition method in which the data vectors are modeled as linear transformations of samples from a polytope. The successful recovery of the original factors in the generative PMF model is conditioned on the "identifiability" of the chosen polytope. In this article, we investigate the problem of determining the identifiability of a polytope. The identifiability condition requires the polytope to be permutation-and/or-sign-only invariant. We show how this problem can be efficiently solved by using a graph automorphism algorithm. In particular, we show that checking only the generating set of the linear automorphism group of a polytope, which corresponds to the automorphism group of an edge-colored complete graph, is sufficient. This property prevents checking all the elements of the permutation group, which requires factorial algorithm complexity. We demonstrate the feasibility of the proposed approach through some numerical experiments.

We introduce a new information maximization (infomax) approach for the blind source separation problem. The proposed framework provides an information-theoretic perspective for determinant maximization-based structured matrix factorization methods such as nonnegative and polytopic matrix factorization. For this purpose, we use an alternative joint entropy measure based on the log-determinant of covariance, which we refer to as log-determinant (LD) entropy. The corresponding (LD) mutual information between two vectors reflects a level of their correlation. We pose the infomax BSS criterion as the maximization of the LD-mutual information between the input and output of the separator under the constraint that the output vectors lie in a presumed domain set. In contrast to the ICA infomax approach, the proposed information maximization approach can separate both dependent and independent sources. Furthermore, we can provide a finite sample guarantee for the perfect separation condition in the noiseless case.

Extraction of latent sources of complex stimuli is critical for making sense of the world. While the brain solves this blind source separation (BSS) problem continuously, its algorithms remain unknown. Previous work on biologically-plausible BSS algorithms assumed that observed signals are linear mixtures of statistically independent or uncorrelated sources, limiting the domain of applicability of these algorithms. To overcome this limitation, we propose novel biologically-plausible neural networks for the blind separation of potentially dependent/correlated sources. Differing from previous work, we assume some general geometric, not statistical, conditions on the source vectors allowing separation of potentially dependent/correlated sources. Concretely, we assume that the source vectors are sufficiently scattered in their domains which can be described by certain polytopes. Then, we consider recovery of these sources by the Det-Max criterion, which maximizes the determinant of the output correlation matrix to enforce a similar spread for the source estimates. Starting from this normative principle, and using a weighted similarity matching approach that enables arbitrary linear transformations adaptable by local learning rules, we derive two-layer biologically-plausible neural network algorithms that can separate mixtures into sources coming from a variety of source domains. We demonstrate that our algorithms outperform other biologically-plausible BSS algorithms on correlated source separation problems.

Self-supervised learning allows AI systems to learn effective representations from large amounts of data using tasks that do not require costly labeling. Mode collapse, i.e., the model producing identical representations for all inputs, is a central problem to many self-supervised learning approaches, making self-supervised tasks, such as matching distorted variants of the inputs, ineffective. In this article, we argue that a straightforward application of information maximization among alternative latent representations of the same input naturally solves the collapse problem and achieves competitive empirical results. We propose a self-supervised learning method, CorInfoMax, that uses a second-order statistics-based mutual information measure that reflects the level of correlation among its arguments. Maximizing this correlative information measure between alternative representations of the same input serves two purposes: (1) it avoids the collapse problem by generating feature vectors with non-degenerate covariances; (2) it establishes relevance among alternative representations by increasing the linear dependence among them. An approximation of the proposed information maximization objective simplifies to a Euclidean distance-based objective function regularized by the log-determinant of the feature covariance matrix. The regularization term acts as a natural barrier against feature space degeneracy. Consequently, beyond avoiding complete output collapse to a single point, the proposed approach also prevents dimensional collapse by encouraging the spread of information across the whole feature space. Numerical experiments demonstrate that CorInfoMax achieves better or competitive performance results relative to the state-of-the-art SSL approaches.

Polytopic Matrix Factorization (PMF) is introduced as a flexible data decomposition tool with potential applications in unsupervised learning. PMF assumes a generative model where observations are lossless linear mixtures of some samples drawn from a particular polytope. Assuming that these samples are sufficiently scattered inside the polytope, a determinant maximization based criterion is used to obtain latent polytopic factors from the corresponding observations. This article aims to characterize all eligible polytopic sets that are suitable for the PMF framework. In particular, we show that any polytope whose set of vertices have only permutation and/or sign invariances qualifies for PMF framework. Such a rich set of possibilities enables elastic modeling of independent/dependent latent factors with combination of features such as relatively sparse/antisparse subvectors, mixture of signed/nonnegative components with optionally prescribed domains.

An important problem encountered by both natural and engineered signal processing systems is blind source separation. In many instances of the problem, the sources are bounded by their nature and known to be so, even though the particular bound may not be known. To separate such bounded sources from their mixtures, we propose a new optimization problem, Bounded Similarity Matching (BSM). A principled derivation of an adaptive BSM algorithm leads to a recurrent neural network with a clipping nonlinearity. The network adapts by local learning rules, satisfying an important constraint for both biological plausibility and implementability in neuromorphic hardware.

In this article, we propose a Bounded Component Analysis (BCA) approach for the separation of the convolutive mixtures of sparse sources. The corresponding algorithm is derived from a geometric objective function defined over a completely deterministic setting. Therefore, it is applicable to sources which can be independent or dependent in both space and time dimensions. We show that all global optima of the proposed objective are perfect separators. We also provide numerical examples to illustrate the performance of the algorithm.

We propose “compressed training adaptive equalization” as a novel framework to reduce the quantity of training symbols in a communication packet. It is a semi-blind approach for communication systems employing time-domain/frequency-domain equalizers, and founded upon the idea of exploiting the magnitude boundedness of digital communication symbols. The corresponding algorithms are derived by combining the least-squares-cost-function measuring the training symbol reconstruction performance and the infinity-norm of the equalizer outputs as the cost for enforcing the special constellation boundedness property along the whole packet. In addition to providing a framework for developing effective adaptive equalization algorithms based on convex optimization, the proposed method establishes a direct link with compressed sensing by utilizing the duality of the ℓ1and ℓ∞norms. This link enables the adaptation of recently emerged ℓ1-norm-minimization-based algorithms and their analysis to the channel equalization problem. In particular, we show for noiseless/low noise scenarios, the required training length is on the order of the logarithm of the channel spread. Furthermore, we provide approximate performance analysis by invoking the recent MSE results from the sparsity-based data processing literature. Provided examples illustrate the significant training reductions by the proposed approach and demonstrate its potential for high bandwidth systems with fast mobility.

We introduce compressed training adaptive equalization as a novel approach for reducing number of training symbols in a communication packet. The proposed semi-blind approach is based on the exploitation of the special magnitude bounded-ness of communication symbols. The algorithms are derived from a special convex optimization setting based on l∞norm. The corresponding framework has a direct link with the com-pressive sensing literature established by invoking the duality between l1and l∞norms. Through this Link, it is possible to adapt various research results in sparse signal processing literature to adaptive equalization problem. In fact, through utilization of such a link, we show that the amount of training data needed is in the order of the logarithm of the channel spread (or equalizer length) in the fractionally spaced equalization scenario. The numerical experiments provided validates the analytical results and the potentials of the proposed approach.

Bounded component analysis (BCA) is a framework that can be considered as a more general framework than independent component analysis (ICA) under the boundedness constraint on sources. Using this framework, it is possible to separate dependent as well as independent components from their mixtures. In this paper, as an extension of a recently introduced instantaneous BCA approach, we introduce a family of convolutive BCA criteria and corresponding algorithms. We prove that the global optima of the proposed criteria, under generic BCA assumptions, are equivalent to a set of perfect separators. The algorithms introduced in this paper are capable of separating not only the independent sources but also the sources that are dependent/correlated in both component (space) and sample (time) dimensions. Therefore, under the condition that the sources are bounded, they can be considered as extended convolutive ICA algorithms with additional dependent/correlated source separation capability. Furthermore, they have potential to provide improvement in separation performance, especially for short data records. This paper offers examples to illustrate the space-time correlated source separation capability through a copula distribution-based example. In addition, a frequency-selective Multiple Input Multiple Output equalization example demonstrates the clear performance advantage of the proposed BCA approach over the state-of-the-art ICA-based approaches in setups involving convolutive mixtures of digital communication sources.

Bounded Component Analysis is a new framework for Blind Source Separation problem. It allows separation of both dependent and independent sources under the assumption about the magnitude boundedness of sources. This article proposes a novel Bounded Component Analysis optimization setting for the separation of the convolutive mixtures of sources as an extension of a recent geometric framework introduced for the instantaneous mixing problem. It is shown that the global maximizers of this setting are perfect separators. The article also provides the iterative algorithm corresponding to this setting and the numerical examples to illustrate its performance especially for separating convolutive mixtures of sources that are correlated in both space and time dimensions.

Bounded Component Analysis (BCA) has recently been introduced as an alternative method for the Blind Source Separation problem. Under the generic assumption on source boundedness, BCA provides a flexible framework for the separation of dependent (even correlated) as well as independent sources. This article provides a family of algorithms derived based on the geometric picture implied by the founding assumptions of the BCA approach. We also provide a numerical example demonstrating the ability of the proposed algorithms to separate mixtures of some dependent sources.

In the area of combination of adaptive filters, two main approaches, namely convex and affine combinations have been introduced. In this article, the relation between these two approaches is investigated. First, the problem of obtaining optimal convex combination coefficients is formulated as the projection of the optimal affine combination weights to the unit simplex in a weighted inner product space. Based on this formulation the closed form expressions for optimal combination weights and target MSE levels are obtained for two and three branch cases.

We study the convergence behavior of independent component analysis (ICA) algorithms that are based on the contrast function maximization and that employ symmetric orthogonalization method to guarantee the orthogonality property of the search matrix. In particular, the characterization of the critical points of the corresponding optimization problem and the stationary points of the conventional gradient ascent and fixed point algorithms are obtained. As an interesting and a useful feature of the symmetrical orthogonalization method, we show that the use of symmetric orthogonalization enables the monotonic convergence for the fixed point ICA algorithms that are based on the convex contrast functions.

We introduce a novel adaptive paraunitary approach to be used for the blind deconvolution of all deconvolvable MIMO mixing systems with memory The proposed adaptive approach is based on the use of alternating projections technique for the enforcement of the paraunitary constraint. The use of this approach enables extension of various instantaneous blind source separation (BSS) approaches to handle the convolutive BSS case. Three such methods, namely FastICA, multi user kurtosis and BSS for bounded magnitude signals are provided to illustrate the use of this approach

A novel blind source separation approach for channels with and without memory is introduced. The proposed approach makes use of a pre-whitening procedure to convert the original convolutive channel into a lossless and memoryless one. Then, a blind subgradient algorithm, which corresponds to an l/sub /spl infin// norm based criterion, is used for the separation of sources. The proposed separation algorithm exploits the assumed boundedness of the original sources and it has a simple update rule. The typical performance of the algorithm is illustrated through simulation examples where separation is achieved with only small numbers of iterations.

An iterative peak-to-average power ratio (PAR) reduction algorithm for discrete multi tone (DMT) based systems, such as OFDM and VDSL, is introduced. The proposed algorithm uses reserved or unused tones to minimize the l/sub /spl infin// norm of the DMT symbol vector iteratively based on a subgradient optimization technique. The resulting iterative algorithm has a very simple update rule and, therefore, a low computational complexity. Furthermore, the PSD level constraints can be easily incorporated into the algorithm. The proposed algorithm's performance is illustrated for an OFDM system with 256 carriers. It is shown that a high PAR reduction is achieved, especially for the cases where the PAR reduction tones are allowed to exceed the PSD mask level.

A novel blind equalization method, based on a subgradient search over a convex cost surface, is proposed. This is an alternative to the existing iterative blind equalization approaches such as the constant modulus algorithm (CMA) which mostly suffer from the convergence problems caused by their non-convex cost functions. The proposed method is an iterative algorithm, for both real and complex constellations, with a very simple update rule that minimizes the l/sub /spl infin// norm of the equalizer output under a linear constraint on the equalizer coefficients. The algorithm has a nice convergence behavior, attributed to the convex l/sub /spl infin// cost surface. Examples are provided to illustrate the algorithm's performance.

In conventional full duplex wireline systems digital echo cancellers are commonly used to suppress echo. For applications with asymmetric rates the complexity and performance of the echo canceller depend on the implementation of the rate matching function. When the transmit rate is lower than the receive rate, the traditional approach of resampling before filtering is inefficient. We show that efficient implementation can be obtained by reversing the order of resampling and filtering. We propose two new echo canceller structures based on scalar and vector error signals and develop associated adaptive algorithms.

We approach finite impulse response (FIR) equalization problem from an H ∞ perspective. First, we formulate the calculation of the optimal H ∞ performance for a given equalization setting as a semidefinite programming (SDP) problem. H ∞ criterion provides a set of FIR equalizers with different optimality properties. Among these, we formulate the calculation of risk-sensitive or minimum entropy FIR filter as the constrained analytic centring problem and mixed H 2 / H ∞ problem as another SDP. We provide examples to illustrate the procedures we described.

We approach FIR equalization problem from an H∞ perspective. \nFirst, we formulate the calculation of the optimal H∞ performance for a given equalization setting as a semidefinite programming (SDP) problem. H∞ criterion provides a set of FIR equalizers with different optimality properties. \nAmong \nthese, \nwe \nformulate \nthe \ncalculation \nof risk sensitive \nor \nminimum \nentropy \nFIR \nfilter \nas \nthe \nconstrained analytic centring \nproblem \nand \nmixed \nH2/H" \nproblem \nas \nanother \nSDP. We \nprovide an \nexample \nto \nil- \nlustrate the \nprocedures \nwe \ndescribed.

As an alternative to existing techniques and algorithms, we investigate the merit of the H-infinity approach to the equalization of communication channels. We first look at causal H-infinity equalization problem and then look at the improvement due to finite delay. By introducing the risk sensitive property, we compare the average performance of the central H-infinity equalizer with the MMSE equalizer in equalizing minimum phase channels.