Probabilistic modeling is a core challenge in statistical machine learning. Tensor-based probabilistic graph methods address interpretability and stability concerns encountered in neural network approaches and allow tractable inference (e.g., marginal inference and conditional inference). In this paper, we introduce tensor ring decomposition for density estimation, which reduces the number of permutation candidates compared to existing methods, while simultaneously enhancing expressive power and maintaining tractable inference. Different non-negative strategies for density function results in two variants: Born TRDE offers simpler inference and sampling but with slightly lower accuracy, while Energy TRDE, though more complex, achieves superior performance. Furthermore, a mixture model that incorporates multiple permutation candidates with adaptive weights is designed, resulting in increased expressive flexibility and comprehensiveness. Unlike existing methods that focus on finding a single optimal permutation, our approach, inspired by ensemble learning, demonstrates that combining multiple suboptimal permutations can yield superior results. Experiments demonstrate that the proposed approach excels in estimating probability density functions and sampling, capturing intricate details with competitive or superior performance compared to existing state-of-the-art (SOTA) tractable density methods.

Topology optimization plays a crucial role in designing efficient and manufacturable structures. Traditional methods often yield free-form voids that, although providing design flexibility, introduce significant manufacturing challenges and require extensive post-processing. Conversely, feature-mapping topology optimization reduces post-processing efforts by constructing topologies using predefined geometric features. Nevertheless, existing approaches are significantly constrained by the limited set of geometric features available, the variety of parameters that each type of geometric feature can possess, and the necessity of employing differentiable signed distance functions. In this paper, we present a novel method that combines Neural Heaviside Signed Distance Functions (Heaviside SDFs) with structured latent shape representations to generate manufacturable voids directly within the optimization framework. Our architecture incorporates encoder and decoder networks to effectively approximate the Heaviside function and facilitate optimization within a unified latent space, thus addressing the feature diversity limitations of current feature-mapping techniques. Experimental results validate the effectiveness of our approach in balancing structural compliance, offering a new pathway to CAD-integrated design with minimal human intervention.

We developed a new tensor-based technique for connectivity analysis and applied it to the EEG data of 10 professional eSports players and 10 novices (control group) collected during 4 different oddball paradigms. The proposed technique utilizes a low-rank approximation of the Granger Causal autoregression with a single temporal filter, thus reducing the number of parameters and improving the convergence rate. Results showed that the temporal filter converges to a Morlet wavelet and establishes a strong connection between channels in the parietal cortex and sustainable negative connectivity between the frontal and occipital cortex, which corresponds to visual search potentials. Professional players also had significantly more prominent and faster ERP responses, which is consistent with the previous research.

Tensor decomposition of convolutional and fully-connected layers is an effective way to reduce parameters and FLOP in neural networks. Due to memory and power consumption limitations of mobile or embedded devices, the quantization step is usually necessary when pre-trained models are deployed. A conventional post-training quantization approach applied to networks with decomposed weights yields a drop in accuracy. This motivated us to develop an algorithm that finds tensor approximation directly with quantized factors and thus benefit from both compression techniques while keeping the prediction quality of the model. Namely, we propose to use Alternating Direction Method of Multipliers (ADMM) for Canonical Polyadic (CP) decomposition with factors whose elements lie on a specified quantization grid. We compress neural network weights with a devised algorithm and evaluate it’s prediction quality and performance. We compare our approach to state-of-the-art post-training quantization methods and demonstrate competitive results and high flexibility in achiving a desirable quality-performance tradeoff.

We present a novel procedure for optimization based on the combination of efficient quantized tensor train representation and a generalized maximum matrix volume principle.We demonstrate the applicability of the new Tensor Train Optimizer (TTOpt) method for various tasks, ranging from minimization of multidimensional functions to reinforcement learning.Our algorithm compares favorably to popular gradient-free methods and outperforms them by the number of function evaluations or execution time, often by a significant margin.

This paper proposes a constrained canonical polyadic (CP) tensor decomposition method with low-rank factor matrices. In this way, we allow the CP decomposition with high rank while keeping the number of the model parameters small. First, we propose an algorithm to decompose the tensors into factor matrices of given ranks. Second, we propose an algorithm which can determine the ranks of the factor matrices automatically, such that the fitting error is bounded by a user- selected constant. The algorithms are verified on the decomposition of a tensor of the MNIST hand-written image dataset.

Weighted canonical polyadic (CP) tensor decomposition appears in a wide range of applications. A typical situation where the weighted decomposition is needed is when some tensor elements are unknown, and the task is to fill in the missing elements under the assumption that the tensor admits a low-rank model. The traditional methods for large-scale decomposition tasks are based on alternating least-squares methods or gradient methods. Second-order methods might have significantly better convergence, but so far they were used only on small tensors. The proposed Krylov-Levenberg-Marquardt method enables to do second-order-based iterations even in large-scale decomposition problems, with or without weights. We show in simulations that the proposed technique can outperform existing state-of-the-art algorithms in some scenarios.

Neurons selective for faces exist in humans and monkeys. However, characteristics of face cell receptive fields are poorly understood. In this theoretical study, we explore the effects of complexity, defined as algorithmic information (Kolmogorov complexity) and logical depth, on possible ways that face cells may be organized. We use tensor decompositions to decompose faces into a set of components, called tensorfaces, and their associated weights, which can be interpreted as model face cells and their firing rates. These tensorfaces form a high-dimensional representation space in which each tensorface forms an axis of the space. A distinctive feature of the decomposition algorithm is the ability to specify tensorface complexity. We found that low-complexity tensorfaces have blob-like appearances crudely approximating faces, while high-complexity tensorfaces appear clearly face-like. Low-complexity tensorfaces require a larger population to reach a criterion face reconstruction error than medium- or high-complexity tensorfaces, and thus are inefficient by that criterion. Low-complexity tensorfaces, however, generalize better when representing statistically novel faces, which are faces falling beyond the distribution of face description parameters found in the tensorface training set. The degree to which face representations are parts based or global forms a continuum as a function of tensorface complexity, with low and medium tensorfaces being more parts based. Given the computational load imposed in creating high-complexity face cells (in the form of algorithmic information and logical depth) and in the absence of a compelling advantage to using high-complexity cells, we suggest face representations consist of a mixture of low- and medium-complexity face cells.

The canonical polyadic decomposition (CPD) is a convenient and intuitive tool for tensor factorization; however, for higher order tensors, it often exhibits high computational cost and permutation of tensor entries, and these undesirable effects grow exponentially with the tensor order. Prior compression of tensor in-hand can reduce the computational cost of CPD, but this is only applicable when the rank R of the decomposition does not exceed the tensor dimensions. To resolve these issues, we present a novel method for CPD of higher order tensors, which rests upon a simple tensor network of representative inter-connected core tensors of orders not higher than 3. For rigor, we develop an exact conversion scheme from the core tensors to the factor matrices in CPD and an iterative algorithm of low complexity to estimate these factor matrices for the inexact case. Comprehensive simulations over a variety of scenarios support the proposed approach.

Decompositions of tensors into factor matrices, which interact through a core tensor, have found numerous applications in signal processing and machine learning. A more general tensor model that represents data as an ordered network of subtensors of order-2 or order-3 has, so far, not been widely considered in these fields, although this so-called tensor network (TN) decomposition has been long studied in quantum physics and scientific computing. In this article, we present novel algorithms and applications of TN decompositions, with a particular focus on the tensor train (TT) decomposition and its variants. The novel algorithms developed for the TT decomposition update, in an alternating way, one or several core tensors at each iteration and exhibit enhanced mathematical tractability and scalability for large-scale data tensors. For rigor, the cases of the given ranks, given approximation error, and the given error bound are all considered. The proposed algorithms provide well-balanced TT-decompositions and are tested in the classic paradigms of blind source separation from a single mixture, denoising, and feature extraction, achieving superior performance over the widely used truncated algorithms for TT decomposition.

Canonical polyadic (CP) tensor decomposition is an important task in many applications. Many times, the true tensor rank is not known, or noise is present, and in such situations, different existing CP decomposition algorithms provide very different results. In this letter, we introduce a notion of sensitivity of CP decomposition and suggest to use it as a side criterion (besides the fitting error) to evaluate different CP decomposition results. Next, we propose a novel variant of a Krylov-Levenberg-Marquardt CP decomposition algorithm which may serve for CP decomposition with a constraint on the sensitivity. In simulations, we decompose order-4 tensors that come from convolutional neural networks. We show that it is useful to combine the CP decomposition algorithms with an error-preserving correction.

Canonical polyadic decomposition (CPD), also known as PARAFAC, is a representation of a given tensor as a sum of rank-one tensors. Traditional method for accomplishing CPD is the alternating least squares (ALS) algorithm. This algorithm is easy to implement with very low computational complexity per iteration. A disadvantage is that in difficult scenarios, where factor matrices in the decomposition contain nearly collinear columns, the number of iterations needed to achieve convergence might be very large. In this paper, we propose a modification of the algorithm which has similar complexity per iteration as ALS, but in difficult scenarios it needs a significantly lower number of iterations.

Decomposition of symmetric tensors has found numerous applications in blind sources separation, blind identification, clustering, and analysis of social interactions. In this paper, we consider fourth order symmetric tensors, and its symmetric tensor decomposition. By imposing unit-length constraints on components, we resort the optimisation problem to the constrained eigenvalue decomposition in which eigenvectors are represented in form of rank-1 matrices. To this end, we develop an augmented Lagrangian algorithm with simple update rules. The proposed algorithm has been compared with the Trust-Region solver over manifold, and achieved higher success rates. The algorithm is also validated for blind identification, and achieves more stable results than the ALSCAF algorithm.

Canonical polyadic decomposition of tensor is to approximate or express the tensor by sum of rank-1 tensors. When all or almost all components of factor matrices of the tensor are highly collinear, the decomposition becomes difficult. Algorithms, e.g., the alternating algorithms, require plenty of iterations, and may get stuck in false local minima. This paper proposes a novel method for such decompositions. The method injects one or a few rank-1 tensors into the data tensor in order to control the decompositions of the rank-expanded data, while still preserving the estimation accuracy of the original tensor. To achieve this, we develop a method to automatically generate the injected tensor which satisfies a specific estimation accuracy such that this tensor should not dominate rank-1 tensors of the data tensor, but is still able to be retrieved with a sufficient accuracy. Simulations on tensors with highly collinear factor matrices will illustrate efficiency of the proposed injecting method.

Canonical polyadic decomposition (CPD), also known as parallel factor analysis, is a representation of a given tensor as a sum of rank-one components. Traditional method for accomplishing CPD is the alternating least squares (ALS) algorithm. Convergence of ALS is known to be slow, especially when some factor matrices of the tensor contain nearly collinear columns. We propose a novel variant of this technique, in which the factor matrices are partitioned into blocks, and each iteration jointly updates blocks of different factor matrices. Each partial optimization is quadratic and can be done in closed form. The algorithm alternates between different random partitionings of the matrices. As a result, a faster convergence is achieved. Another improvement can be obtained when the method is combined with the enhanced line search of Rajihet al.Complexity per iteration is between those of the ALS and the Levenberg–Marquardt (damped Gauss–Newton) method. It is important, however, that the idea of alternating quadratic optimization with partitioned factor matrices is general and can be applied to other variants of the tensor decomposition problems, e.g., when non-Gaussian additive noise is considered.

In this paper, we propose a low-rank tensor deconvolution problem which seeks multiway replicative patterns and corresponding activating tensors of rank-1. An alternating least squares (ALS) algorithm has been derived for the model to sequentially update loading components and the patterns. In addition, together with a good initialisation method using tensor diagonalization, the update rules have been implemented with a low cost using fast inversion of block Toeplitz matrices as well as an efficient update strategy. Experiments show that the proposed model and the algorithm are promising in feature extraction and clustering.

We propose algorithms for Tucker tensor decomposition, which can avoid computing singular value decomposition or eigenvalue decomposition of large matrices as in the work-horse higher order orthogonal iteration (HOOI) algorithm. The novel algorithms require computational cost of O(I3R), which is cheaper than O(I3R + IR4+ R6) of HOOI for multilinear rank-(R, R, R) tensors of size I × I × I.

CANDECOMP/PARAFAC tensor decomposition (CPD) approximates multiway data by rank-1 tensors. Unlike matrix decomposition, the procedure which estimates the best rank-R tensor approximation through R sequential best rank-1 approximations does not work for tensors, because the deflation does not always reduce the tensor rank. In this paper we propose a novel deflation method for the problem in which rank R does not exceed the tensor dimensions. A rank-R CPD can be performed through (R − 1) rank-1 reductions. At each deflation stage, the residue tensor is constrained to have a reduced multilinear rank.

Various tensor decompositions use different arrangements of factors to explain multi-way data. Components from different decompositions can vary in the number of parameters. Allowing a model to contain components from different decompositions results in a combinatoric number of possible models. Model selection balances approximation error and the number of parameters, but due to the number of possible models, post-hoc model selection is infeasible. Instead, we incrementally build a model. This approach is analogous to sparse coding with a union of dictionaries. The proposed greedy approach can estimate a model consisting of a combination of tensor decompositions.

We propose a novel decomposition approach to impute missing values in tensor data. The method uses smaller scale multiway patches to model the whole data or a small volume encompassing the observed missing entries. Simulations on color images show that our method can recover color images using only 5–10% of pixels, and outperforms other available tensor completion methods.

A novel approach is proposed to extract high-rank patterns from multiway data. The method is useful when signals comprise collinear components or complex structural patterns. Alternating least squares and multiplication algorithms are developed for the new model with/without non negativity constraints. Experimental results on synthetic data and real-world dataset confirm the validity of the proposed model and algorithms.

In this paper, a novel implementation of the damped Gauss-Newton algorithm (also known as Levenberg-Marquart) for the CANDECOMP-PARAFAC (CP) tensor decomposition is proposed. The method is based on a fast inversion of the approximate Hessian for the problem. It is shown that the inversion can be computed on O(NR6) operations, where N and R is the tensor order and rank, respectively. It is less than in the best existing state-of-the art algorithm with O(N3R6) operations. The damped Gauss-Newton algorithm is suitable namely for difficult scenarios, where nearly-colinear factors appear in several modes simultaneously. Performance of the method is shown on decomposition of large tensors (100 × 100 × 100 and 100 × 100 × 100 × 100) of rank 5 to 90.

Non-negative Canonical Polyadic decomposition (NCPD) and non-negative Tucker decomposition (NTD) were compared for extracting the multi-domain feature of visual mismatch negativity (vMMN), a small event-related potential (ERP), for the cognitive research. Since signal-to-noise ratio in vMMN is low, NTD outperformed NCPD. Moreover, we proposed an approach to select the multi-domain feature of an ERP among all extracted features and discussed determination of numbers of extracted components in NCPD and NTD regarding the ERP context.

Blind separation of underdetermined instantaneous mixtures is a popular solution to inverse problems encountered in audio or biomedical applications where the number of sources exceeds the number of sensors. There are two non-equivalent tasks: to identify the mixing matrix and to separate the original sources. In this paper, we focus on the latter task by proposing a novel beamformer that minimizes the theoretical mean square error distance between the separated and original signals. The beamformer has two stages: one for the estimation of signals and one for their refinement. Within the former stage, the signals are assumed to be random and locally stationary, while the latter stage is based on a semi-deterministic model. The experiments prove superior performance of the proposed method compared to conventional MMSE beamforming.

Classification methods have been widely applied in most brain computer interfaces (BCIs) that control devices for better quality of life. Most existing classification methods for P300-based BCIs extract features based on temporal structure related to P300 components of event-related potentials (ERPs). Some others exploit the spatial distribution of ERPs optimally selected by recursive channel elimination. However, none of them employed multilinear structures which exploit hidden features in P300-based BCI data. In this paper, we propose a new feature extraction method based on tensor decomposition for ERP-based BCIs. The method seeks an optimal feature subspace simultaneously spanned by temporal and spatial bases, and additional bases which indicate a variant of ERPs obtained by different degrees of polynomial fittings. The proposed method has been evaluated by both the BCI competition III data set II and the affective face driven paradigm data set, and achieved 92% and 95% classification accuracies respectively, which were better than those of most existing P300-based BCI algorithms.

A novel blind deconvolution is proposed to seek for basis patterns and their location maps inside a nonnegative data matrix. Basis patterns can have different sizes, and shift in independent directions. Moreover, the location maps can be low-rank or rank-one matrices composed by two relatively small and tall matrices or by two vectors. A general framework to solve this problem together with algorithms are introduced. The experiments on music and texture decomposition will confirm performance of our method, and of the proposed algorithms.

Through exploiting temporal, spectral, time-frequency representations, and spatial properties of mismatch negativity (MMN) simultaneously, this study extracts a multi-domain feature of MMN mainly using non-negative tensor factorization. In our experiment, the peak amplitude of MMN between children with reading disability and children with attention deficit was not significantly different, whereas the new feature of MMN significantly discriminated the two groups of children. This is because the feature was derived from multi-domain information with significant reduction of the heterogeneous effect of datasets.

The multiplicative algorithms are well-known for nonnegative matrix and tensor factorizations. The ALS algorithm for canonical decomposition (CP) has been proved as a "work horse" algorithm for general multiway data. Unfortunately, for CP with nonnegativity constraints, this algorithm with a rectifier (projection) may not converge to the desired solution without additional regularization parameters in matrix inverses. The hierarchical ALS algorithm improves the performance of the ALS algorithm, outperforms the multiplicative algorithm. However, NTF algorithms can face problem with collinear or bias data. In this paper, we propose a novel algorithm which overwhelmingly outperforms all the multiplicative, and (H)ALS algorithms. By solving the nonnegative quadratic programming problems, a general algorithm of the HALS has been derived and experimentally confirmed its validity and high performance for normal and difficult bench marks, and for real-world EEG dataset.

Alternating optimization algorithms for canonical polyadic decomposition (with/without nonnegative constraints) often accompany update rules with low computational cost, but could face problems of swamps, bottlenecks, and slow convergence. All-at-once algorithms can deal with such problems, but always demand significant temporary extra-storage, and high computational cost. In this paper, we propose an all-at-once algorithm with low complexity for sparse and nonnegative tensor factorization based on the damped Gauss-Newton iteration. Especially, for low-rank approximations, the proposed algorithm avoids building up Hessians and gradients, reduces the computational cost dramatically. Moreover, we proposed selection strategies for regularization parameters. The proposed algorithm has been verified to overwhelmingly outperform “state-of-the-art” NTF algorithms for difficult benchmarks, and for real-world application such as clustering of the ORL face database.

Nonnegative Matrix/Tensor factorization (NMF/NTF) have been used in the study of EEG, and the fit (explained variation) is often used to evaluate the performance of a nonnegative decomposition algorithm. However, this parameter only reveals the information derived from the mathematical model and just exhibits the reliability of the algorithms, and the property of EEG can not be reflected. If fits of two algorithms are identical, it is necessary to examine whether the desired components extracted by them are identical too. In order to verify this doubt, we performed NMF and NTF on the same dataset of an auditory event-related potentials (ERPs), and found that the identical fits of NMF and NTF under the hierarchical alternating least squares algorithms corresponded to different desired ERPs extracted by NMF and NTF, moreover, NTF contributed the ERP with much better timing and spectral properties. Such analysis implies that to combine the fit and property of the desired ERP component together helps evaluate the performance of NMF and NTF algorithms in the study of ERPs.