Machine learning, particularly deep learning, typically achieves high facial emotion image recognition accuracy benefiting from multiple labeled data. However, the datasets usually contain insufficient labeled samples and numerous unlabeled data since human labeling is a costly endeavor. For semi-supervised learning of these datasets, self-training procedure solely based on the visual features of images fails to comprehensively understand the intricate high-level semantic features. Since EEG signals contain not only visual information related to the visual stimulus but also emotional information related to brain activity, they are highly suitable as supervisory signals for labeling unlabeled facial emotion images. In this study, we specifically employ EEG signals evoked by visual image stimuli in conjunction with EEGNet3D to learn a discriminative EEG class representation manifold of brain activity. The one-hot class label is replaced with the EEG class representation as the supervisory to train the base model. Then, better pseudo-labeling is achieved using the base model in the EEG class representation manifold. Based on pseudo-labeling results, the utilization of unlabeled data is further improved. Interestingly, our findings reveal that when utilizing EEG class representations as supervisory information for the base model, the base model demonstrates a learning pattern that involves focusing more on the eye area when making judgments about emotions. This behavior closely resembles how the human brain decodes emotions. Experiments show that the performance of the proposed method can be effectively enhanced by combining labeled and pseudo-labeled images. Further experiments demonstrate that our method exhibits strong generalization abilities when applied to new image datasets and other visual networks.

Recent text-to-image (T2I) diffusion models have demonstrated remarkable capabilities in visual synthesis, yet their performance heavily relies on the quality of input prompts. However, optimizing discrete prompts remains challenging because the discrete nature of tokens prevents the direct application of the gradient descent method and the vast search space of possible token combinations. As a result, existing approaches either suffer from quantization errors when employing continuous optimization techniques or be- come trapped in local optima due to coordinate-wise greedy search. In this paper, we propose STEPS, a novel Sequential probability Tensor Estimation approach for hard Prompt Search. Our method reformulates discrete prompt optimization as a sequential probability tensor estimation problem, leveraging the inherent low-rank characteristics to address the curse of dimensionality. To further improve the computational efficiency, we develop a memory-bounded sampling approach that shrinks the prompt space without the iteration step dependency while preserving sequential optimization dynamics. Extensive experiments on various public datasets demonstrate that our method consistently outperforms existing approaches in T2I generation, cross-model prompt transferability, and harmful prompt optimization, validating the effectiveness of the proposed framework.

Hyperspectral pansharpening refers to fusing a panchromatic image (PAN) and a low-resolution hyperspectral image (LR-HSI) to obtain a high-resolution hyperspectral image (HR-HSI). Recently, guiding pre-trained diffusion models (DMs) has demonstrated significant potential in this area, leveraging their powerful representational abilities while avoiding complex training processes. However, these DMs are often trained on RGB images, not well-suited for pansharpening tasks, limited in adapting to the hyperspectral images. In this work, we propose a novel guided diffusion scheme with zero-shot guidance and neural spatialspectral decomposition (NSSD) to iteratively generate the RGB detail image and map the RGB detail image to target HR-HSI. Specifically, zero-shot guidance employs an auxiliary neural network that trained only with a PAN and LR-HSI to guide pre-trained DMs in generating the RGB detail image, informed by specific prior knowledge. Then, NSSD establishes a spectral mapping from the generated RGB detail image to the final HR-HSI. Extensive experiments are conducted on Pavia, Washington DC, Chukusei, and FR1 datasets to demonstrate that the proposed method significantly enhances the performance of DMs for hyperspectral pansharpening tasks, outperforming existing methods across multiple metrics and achieving improvements in visualization results. The code is available at https://github.com/Jin-liangXiao/DM-zs.

Parameter-Efficient Fine-Tuning (PEFT) of text-to-image models has become an increasingly popular technique with many applications. Among the various PEFT methods, Low-Rank Adaptation (LoRA) and its variants have gained significant attention due to their effectiveness, enabling users to fine-tune models with limited computational resources. However, the approximation gap between the low-rank assumption and desired fine-tuning weights prevents the simultaneous acquisition of ultra-parameter-efficiency and better performance. To reduce this gap and further improve the power of LoRA, we propose a new PEFT method that combines two classes of adaptations, namely, transform and residual adaptations. In specific, we first apply a full-rank and dense transform to the pre-trained weight. This learnable transform is expected to align the pre-trained weight as closely as possible to the desired weight, thereby reducing the rank of the residual weight. Then, the residual part can be effectively approximated by more compact and parameter-efficient structures, with a smaller approximation error. To achieve ultra-parameter-efficiency in practice, we design highly flexible and effective tensor decompositions for both the transform and residual adaptations. Additionally, popular PEFT methods such as DoRA can be summarized under this transform plus residual adaptation scheme. Experiments are conducted on fine-tuning Stable Diffusion models in subject-driven and controllable generation. The results manifest that our method can achieve better performances and parameter efficiency compared to LoRA and several baselines.

Class-Incremental Learning (CIL) has gained considerable attention due to its capacity to accommodate new classes during learning. Replay-based methods demonstrate state-of-the-art performance in CIL but suffer from high memory consumption to save a set of old exemplars for revisiting. To address this challenge, many memory-efficient replay methods have been developed by exploiting image compression techniques. However, the gains are often bittersweet when pixel-level compression methods are used. Here, we present a simple yet efficient approach that employs tensor decomposition to address these limitations. This method fully exploits the low intrinsic dimensionality and pixel correlation of images to achieve high compression efficiency while preserving sufficient discriminative information, significantly enhancing performance. We also introduce a hybrid exemplar selection strategy to improve the representativeness and diversity of stored exemplars. Extensive experiments across datasets with varying resolutions consistently demonstrate that our approach substantially boosts the performance of baseline methods, showcasing strong generalization and robustness.

Tensor regression is a powerful tool for analyzing complex multi-dimensional data in fields such as neuroimaging and spatiotemporal analysis, but its effectiveness is often hindered by insufficient sample sizes. To overcome this limitation, we adopt a transfer learning strategy that leverages knowledge from related source tasks to improve performance in data-scarce target tasks. This approach, however, introduces additional challenges including model shifts, covariate shifts, and decentralized data management. We propose the Low-Rank Tensor Transitions (LoRT) framework, which incorporates a novel fusion regularizer and a two-step refinement to enable robust adaptation while preserving low-tubal-rank structure. To support decentralized scenarios, we extend LoRT to D-LoRT, a distributed variant that maintains statistical efficiency with minimal communication overhead. Theoretical analysis and experiments on tensor regression tasks, including compressed sensing and completion, validate the robustness and versatility of the proposed methods. These findings indicate the potential of LoRT as a robust method for tensor regression in settings with limited data and complex distributional structures.

As the length of input text increases, the key-value (KV) cache in LLMs imposes prohibitive GPU memory costs and limits long-context inference on resource constrained devices. Existing approaches, such as KV quantization and pruning, reduce memory usage but suffer from numerical precision loss or suboptimal retention of key-value pairs. In this work, Low Rank Query and Key attention (LRQK) is introduced, a two-stage framework that jointly decomposes full-precision query and key matrices into compact rank-\(r\) factors during the prefill stage, and then employs these low-dimensional projections to compute proxy attention scores in \(\mathcal{O}(lr)\) time at each decode step. By selecting only the top-\(k\) tokens and a small fixed set of recent tokens, LRQK employs a mixed GPU-CPU cache with a hit-and-miss mechanism where only missing full-precision KV pairs are transferred, thereby preserving exact attention outputs while reducing CPU-GPU data movement. Extensive experiments on the RULER and LongBench benchmarks with LLaMA-3-8B and Qwen2.5-7B demonstrate that LRQK matches or surpasses leading sparse-attention methods in long context settings, while delivering significant memory savings with minimal accuracy loss. Our code is available at \url{https://github.com/tenghuilee/LRQK}.

Despite the growing success of transform-based tensor models such as the t-product, their underlying geometric principles remain poorly understood. Classical differential geometry, built on real-valued function spaces, is not well suited to capture the algebraic and spectral structure induced by transform-based tensor operations. In this work, we take an initial step toward a geometric framework for tensors equipped with tube-wise multiplication via orthogonal transforms. We introduce the notion of smooth t-manifolds, defined as topological spaces locally modeled on structured tensor modules over a commutative t-scalar ring. This formulation enables transform-consistent definitions of geometric objects, including metrics, gradients, Laplacians, and geodesics, thereby bridging discrete and continuous tensor settings within a unified algebraic-geometric perspective. On this basis, we develop a statistical procedure for testing whether tensor data lie near a low-dimensional t-manifold, and provide nonasymptotic guarantees for manifold fitting under noise. We further establish approximation bounds for tensor neural networks that learn smooth functions over t-manifolds, with generalization rates determined by intrinsic geometric complexity. This framework offers a theoretical foundation for geometry-aware learning in structured tensor spaces and supports the development of models that align with transform-based tensor representations.

Giant AI models, typically trained and deployed centrally in the cloud, demand significant computational resources, posing privacy risks for the Internet of Things (IoT), particularly in the era of 6G-driven connectivity. federated learning (FL) mitigates this by enabling local training and server-side aggregation, fostering 6G Symbiotic IoT while preserving privacy in 6G networks. However, data heterogeneity (DH) in multimodal settings remains a formidable challenge, degrading model performance. While prior studies attribute DH to uneven data distributions, our empirical analysis reveals that hard samples also drive DH, manifesting across both local and global models. To address this, we propose MCMFL, a multimodal FL framework leveraging Monte Carlo dropout to quantify sample uncertainty and identify hard samples. Exploiting 6G’s excellent capabilities, MCMFL optimizes the local loss function and introduces MC dropout-based aggregation, a robust aggregation algorithm, enhancing the model’s resilience to hard samples. Extensive experiments show that MCMFL demonstrates superior performance, outperforming baseline aggregation methods by up to 5.38% on CIFAR-100, leading local enhancement baselines by 3.14% on TinyImageNet-200 and achieving the highest score of 4.79 in MTBenchmark for large language model. By shifting the focus from data distribution to sample-level uncertainty, MCMFL provides a novel framework for deploying large AI models via FL in IoT scenarios, mitigating the critical challenge of DH and enhancing model robustness.

Unsupervised learning provides efficient analytical tools for data-centric Internet of Things (IoT) applications. Nonnegative matrix factorization (NMF) is a fundamental tool in unsupervised machine learning, offering interpretable and part-based feature representations. While NMF is provably robust to bounded additive perturbations, the emergence of adversarial attacks highlights the need to reassess the certified robustness under strategically crafted perturbations. In this work, we propose a novel attack method that more effectively disrupts NMF factorization than existing adversarial methods by incorporating two key strategies: i) imposing low-rank constraints to guide perturbations toward principal subspaces; and ii) relaxing nonnegativity constraints on perturbations to inject negative components that effectively alter original feature additivity. We present analyses including ablation studies, convergence performance, and computational complexity. Extensive experiments on benchmark datasets show that the proposed method effectively attacks both vanilla NMF and existing adversarial NMF variants by disrupting the factorization and degrading performance in downstream tasks such as feature extraction and clustering.

Large language models have demonstrated predictable scaling behaviors with respect to model parameters and training data. This study investigates whether a similar scaling relationship exist for vision-language models with respect to the number of vision tokens. A mathematical framework is developed to characterize a relationship between vision token number and the expected divergence of distance between vision-referencing sequences. The theoretical analysis reveals two distinct scaling regimes: sublinear scaling for less vision tokens and linear scaling for more vision tokens. This aligns with model performance relationships of the form \(S(n) \approx c / n^{\alpha(n)}\), where the scaling exponent relates to the correlation structure between vision token representations. Empirical validations across multiple vision-language benchmarks show that model performance matches the prediction from scaling relationship. The findings contribute to understanding vision token scaling in transformers through a theoretical framework that complements empirical observations.

Recently, tensor singular value decomposition (t-SVD)-based methods were proposed to solve the low-rank tensor completion (LRTC) problem, which has achieved unprecedented success on image and video inpainting tasks. The t-SVD is limited to process third-order tensors. When faced with higher-order tensors, it reshapes them into third-order tensors, leading to the destruction of interdimensional correlations. To address this limitation, this letter introduces a tproductinduced Tucker decomposition (tTucker) model that replaces the mode product in Tucker decomposition with t-product, which jointly extends the ideas of t-SVD and high-order SVD. This letter defines the rank of the tTucker decomposition and presents an LRTC model that minimizes the induced Schatten-p norm. An efficient alternating direction multiplier method (ADMM) algorithm is developed to optimize the proposed LRTC model, and its effectiveness is demonstrated through experiments conducted on both synthetic and real data sets, showcasing excellent performance.

While tensor ring (TR) decomposition methods have been extensively studied, the determination of TR-ranks remains a challenging problem, with existing methods being typically sensitive to the determination of the starting rank (i.e., the first rank to be optimized). Moreover, current methods often fail to adaptively determine TR-ranks in the presence of noisy and incomplete data, and exhibit computational inefficiencies when handling high-dimensional data. To address these issues, we propose an odd-dimensional unfolding method for the effective determination of TR-ranks. This is achieved by leveraging the symmetry of the TR model and the bound rank relationship in TR decomposition. In addition, we employ the singular value thresholding algorithm to facilitate the adaptive determination of TR-ranks and use randomized sketching techniques to enhance the efficiency and scalability of the method. Extensive experimental results in rank identification, data denoising, and completion demonstrate the potential of our method for a broad range of applications.

Amid a rapidly evolving information era, large-scale multimodal knowledge discovery in computational social systems emerges as a key research domain. Large language models (LLMs) play a crucial role in this field, providing contextual understanding and task adaptability. Yet, centralized training of LLM raises privacy concerns. Federated learning (FL) offers a distributed alternative, but it struggles with data heterogeneity and security issues related to model parameters. To this end, we propose a robust aggregation method that leverages the relative total distance of models to improve global model performance in heterogeneous settings, complemented by Cheon-Kim-Kim-Song (CKKS) encryption to secure parameters against parameter stealing without performance loss. Extensive numeric results show our approach excels in LLM testing, scoring 3.74 on MTBenchmark and 8.17 on Vicuna, outperforming state-of-the-art FL methods against data heterogeneity challenges. It also achieves consistent gains on image datasets such as SVHN, CIFAR10, MNIST, TinyImageNet200, and CIFAR100, TinyImageNet200. In summary, our method offers an effective solution for secure multimodal data analysis in computational social systems.

Recently, the tensor nuclear norm (TNN)-based tensor robust principle component analysis (TRPCA) has achieved impressive performance in multidimensional data processing. The underlying assumption in TNN is the low-rankness of frontal slices of the tensor in the transformed domain (e.g., Fourier domain). However, the low-rankness assumption is usually violative for real-world multidimensional data (e.g., video and image) due to their intrinsically nonlinear structure. How to effectively and efficiently exploit the intrinsic structure of multidimensional data remains a challenge. In this article, we first suggest a kernelized TNN (KTNN) by leveraging the nonlinear kernel mapping in the transform domain, which faithfully captures the intrinsic structure (i.e., implicit low-rankness) of multidimensional data and is computed at a lower cost by introducing kernel trick. Armed with KTNN, we propose a tensor robust kernel PCA (TRKPCA) model for handling multidimensional data, which decomposes the observed tensor into an implicit low-rank component and a sparse component. To tackle the nonlinear and nonconvex model, we develop an efficient alternating direction method of multipliers (ADMM)-based algorithm. Extensive experiments on real-world applications collectively verify that TRKPCA achieves superiority over the state-of-the-art RPCA methods.

The tensor recurrent model is a family of nonlinear dynamical systems, of which the recurrence relation consists of a -fold (called degree- ) tensor product. Despite such models frequently appearing in advanced recurrent neural networks (RNNs), to this date, there are limited studies on their long memory properties and stability in sequence tasks. In this article, we propose a fractional tensor recurrent model, where the tensor degree is extended from the discrete domain to the continuous domain, so it is effectively learnable from various datasets. Theoretically, we prove that a large degree is essential to achieve the long memory effect in a tensor recurrent model, yet it could lead to unstable dynamical behaviors. Hence, our new model, named fractional tensor recurrent unit (fTRU), is expected to seek the saddle point between long memory property and model stability during the training. We experimentally show that the proposed model achieves competitive performance with a long memory and stable manners in several forecasting tasks compared to various advanced RNNs.

The recently proposed tensor tubal rank has been witnessed to obtain extraordinary success in real-world tensor data completion. However, existing works usually fix the transform orientation along the third mode and may fail to turn multidimensional low-tubal-rank structure into account. To alleviate these bottlenecks, we introduce two unfolding induced tensor nuclear norms (TNNs) for the tensor completion (TC) problem, which naturally extends tensor tubal rank to high-order data. Specifically, we show how multidimensional low-tubal-rank structure can be captured by utilizing a novel balanced unfolding strategy, upon which two TNNs, namely, overlapped TNN (OTNN) and latent TNN (LTNN), are developed. We also show the immediate relationship between the tubal rank of unfolding tensor and the existing tensor network (TN) rank, e.g., CANDECOMP/PARAFAC (CP) rank, Tucker rank, and tensor ring (TR) rank, to demonstrate its efficiency and practicality. Two efficient TC models are then proposed with theoretical guarantees by analyzing a unified nonasymptotic upper bound. To solve optimization problems, we develop two alternating direction methods of multipliers (ADMM) based algorithms. The proposed models have been demonstrated to exhibit superior performance based on experimental findings involving synthetic and real-world tensors, including facial images, light field images, and video sequences.

Multimodal sarcasm analysis is one of the most challenging research branch of the sentiment analysis area, due to the presence of cross-modality incongruity. However, existing works mainly attend to the coarse-grained incongruity analysis, and totally ignore the sentiment semantic coupling issue. This indeed limits the discriminate capability and robustness of the sarcasm analysis model. In order to address the above issue, we propose a novel Fine-grained Semantic Disentanglement Network (FSDN). Specifically, the intra-modality semantic disentanglement is performed to investigate the more intrinsic semantic cues of the same modality. Additionally, the inter-modality semantic disentanglement is leveraged to simultaneously facilitate the common and intrinsic semantic cues across modalities. Furthermore, the dual-spatial semantic interaction block is presented to explore the long-range cross-spatial semantic context between the obtained verbal and non-verbal semantic space with the global view. The above semantic disentanglement processes with both local and global views significantly unleash much more robustness even for the sarcasm case consisting of multiple semantic message. Various experiments indicate that the FSDN can receive state-of-the-art or competitive performance.

Conventional Outlier Robust Tensor Decomposition (ORTD) approaches generally represent sparse outlier corruption within a specific mode. However, such an assumption, which may hold for matrices, proves inadequate when applied to high-order tensors. In the tensor domain, the outliers are prone to be corrupted in multiple modes simultaneously. Addressing this limitation, this study proposes a novel ORTD approach by recovering low-rank tensors contaminated by outliers spanning multiple modes. In particular, we conceptualize outliers within high-order tensors as latent tensor group sparsity by decomposing the corrupted tensor into a sum of multiple latent components, where each latent component is exclusive to outliers within a particular direction. Thus, it can effectively mitigate the outlier corruptions prevalent in high-order tensors across multiple modes. To theoretically guarantee recovery performance, we rigorously analyze a non-asymptotic upper bound of the estimation error for the proposed ORTD approach. In the optimization process, we develop an efficient alternate direction method of multipliers (ADMM) algorithm. Empirical validation of the approach's efficacy is undertaken through comprehensive experimentation.

In numerous applications, binary reactions or event counts are observed and stored within high-order tensors. Tensor decompositions (TDs) serve as a powerful tool to handle such high-dimensional and sparse data. However, many traditional TDs are explicitly or implicitly designed based on the Gaussian distribution, which is unsuitable for discrete data. Moreover, most TDs rely on predefined multi-linear structures, such as CP and Tucker formats. Therefore, they may not be effective enough to handle complex real-world datasets. To address these issues, we propose ENTED, an Efficient Nonparametric TEnsor Decomposition for binary and count tensors. Specifically, we first employ a nonparametric Gaussian process (GP) to replace traditional multi-linear structures. Next, we utilize the Pólya-Gamma augmentation which provides a unified framework to establish conjugate models for binary and count distributions. Finally, to address the computational issue of GPs, we enhance the model by incorporating sparse orthogonal variational inference of inducing points, which offers a more effective covariance approximation within GPs and stochastic natural gradient updates for nonparametric models. We evaluate our model on several real-world tensor completion tasks, considering binary and count datasets. The results manifest both better performance and computational advantages of the proposed model.

Tensor network (TN) representation is a powerful technique for computer vision and machine learning. TN structure search (TN-SS) aims to search for a customized structure to achieve a compact representation, which is a chal-lenging NP-hard problem. Recent “sampling-evaluation”-based methods require sampling an extensive collection of structures and evaluating them one by one, resulting in pro-hibitively high computational costs. To address this issue, we propose a novel TN paradigm, named SVD-inspired TN decomposition (SVDinsTN), which allows us to efficiently solve the TN-SS problem from a regularized modeling per-spective, eliminating the repeated structure evaluations. To be specific, by inserting a diagonal factor for each edge of the fully-connected TN, SVDinsTN allows us to calculate TN cores and diagonal factors simultaneously, with the factor sparsity revealing a compact TN structure. In theory, we prove a convergence guarantee for the proposed method. Experimental results demonstrate that the proposed method achieves approximately 100 ~ 1000 times acceleration compared to the state-of-the-art TN-SS methods while maintaining a comparable level of representation ability.

In this paper, we present a novel hierarchical approach to adversarial attacks targeting Graph Neural Networks (GNNs), tailored to overcome the complexities inherent in large-scale poisoning attacks. Traditional global attack strategies often fail to yield effective results on extensive graph structures. Our innovative method implements a divide-and-conquer tactic, clustering nodes based on their embeddings and forming coarse-grained graphs from these clusters. We initiate perturbations at this coarse level, gradually honing them in more detailed, finer-grained graphs, while keeping non-essential nodes grouped. By employing meta-gradients derived from these refined graphs, we pinpoint critical edges for perturbation, thereby vastly simplifying the process and reducing the intricacy involved in manipulating large-scale graphs. This hierarchical strategy not only enhances the efficacy of the attacks but also maintains operational efficiency across expansive network structures.

The deep neural networks are known to be vulnerable to well-designed adversarial attacks. The most successful defense technique based on adversarial training (AT) can achieve optimal robustness against particular attacks but cannot generalize well to unseen attacks. Another effective defense technique based on adversarial purification (AP) can enhance generalization but cannot achieve optimal robustness. Meanwhile, both methods share one common limitation on the degraded standard accuracy. To mitigate these issues, we propose a novel pipeline to acquire the robust purifier model, named Adversarial Training on Purification (AToP), which comprises two components: perturbation destruction by random transforms (RT) and purifier model fine-tuned (FT) by adversarial loss. RT is essential to avoid overlearning to known attacks, resulting in the robustness generalization to unseen attacks, and FT is essential for the improvement of robustness. To evaluate our method in an efficient and scalable way, we conduct extensive experiments on CIFAR-10, CIFAR-100, and ImageNette to demonstrate that our method achieves optimal robustness and exhibits generalization ability against unseen attacks.

In adversarial defense, adversarial purification can be viewed as a special generation task with the purpose to remove adversarial attacks and diffusion models excel in adversarial purification for their strong generative power. With different predetermined generation requirements, various types of guidance have been proposed, but few of them focuses on adversarial purification. In this work, we propose to guide diffusion models for adversarial purification using contrastive guidance. We theoretically derive the proper noise level added in the forward process diffusion models for adversarial purification from a feature learning perspective. For the reverse process, it is implied that the role of contrastive loss guidance is to facilitate the evolution towards the signal direction. From the theoretical findings and implications, we design the forward process with the proper amount of Gaussian noise added and the reverse process with the gradient of contrastive loss as the guidance of diffusion models for adversarial purification. Empirically, extensive experiments on CIFAR-10, CIFAR-100, the German Traffic Sign Recognition Benchmark and ImageNet datasets with ResNet and WideResNet classifiers show that our method outperforms most of current adversarial training and adversarial purification methods by a large improvement.

Deep Multi-view Clustering (DMVC) stands out as a widely adopted technique aiming at enhanced clustering performance by leveraging diverse data sources. However, the critical issue of vulnerability to adversarial attacks is unexplored due to the lack of well-defined attack objectives. To fill this crucial gap, this paper is the first work to investigate the possibility of adversarial attacks on DMVC models. Specifically, we introduce an adversarial attack with Generative Adversarial Networks (GANs) with the aim to maximally change the complementarity and consistency of multiple views, thus leading to wrong clustering. Building upon this adversarial context, in the realm of defense, we propose a novel Adversarially Robust Deep Multi-View Clustering by leveraging adversarial training. Based on the analysis from an information-theoretic perspective, we design an Attack Mitigator that provides a foundation to guarantee the adversarial robustness of our DMVC models. Experiments conducted on multi-view datasets confirmed that our attack framework effectively reduces the clustering performance of the target model. Furthermore, our proposed adversarially robust method is also demonstrated to be an effective defense against such attacks. This work is a pioneer in exploring adversarial threats and advancing both theoretical understanding and practical strategies for robust multi-view clustering. Code is available at https://github.com/libertyhhn/AR-DMVC.

Tensor networks are efficient for extremely high-dimensional representation, but their model selection, known as tensor network structure search (TN-SS), is a challenging problem. Although several works have targeted TN-SS, most existing algorithms are manually crafted heuristics with poor performance, suffering from the curse of dimensionality and local convergence. In this work, we jump out of the box, studying how to harness large language models (LLMs) to automatically discover new TN-SS algorithms, replacing the involvement of human experts. By observing how human experts innovate in research, we model their common workflow and propose an automatic algorithm discovery framework called tnGPS. The proposed framework is an elaborate prompting pipeline that instruct LLMs to generate new TN-SS algorithms through iterative refinement and enhancement. The experimental results demonstrate that the algorithms discovered by tnGPS exhibit superior performance in benchmarks compared to the current state-of-the-art methods. Our code is available at https://github.com/ChaoLiAtRIKEN/tngps.

With the advancement of neural networks, diverse methods for neural Granger causality have emerged, which demonstrate proficiency in handling complex data, and nonlinear relationships. However, the existing framework of neural Granger causality has several limitations. It requires the construction of separate predictive models for each target variable, and the relationship depends on the sparsity on the weights of the first layer, resulting in challenges in effectively modeling complex relationships between variables as well as unsatisfied estimation accuracy of Granger causality. Moreover, most of them cannot grasp full-time Granger causality. To address these drawbacks, we propose a **J**acobian **R**egularizer-based **N**eural **G**ranger **C**ausality (**JRNGC**) approach, a straightforward yet highly effective method for learning multivariate summary Granger causality and full-time Granger causality by constructing a single model for all target variables. Specifically, our method eliminates the sparsity constraints of weights by leveraging an input-output Jacobian matrix regularizer, which can be subsequently represented as the weighted causal matrix in the post-hoc analysis. Extensive experiments show that our proposed approach achieves competitive performance with the state-of-the-art methods for learning summary Granger causality and full-time Granger causality while maintaining lower model complexity and high scalability.

In multi-output regression, we identify a previously neglected challenge that arises from the inability of training distribution to cover all combinations of input features, leading to combinatorial distribution shift (CDS). To the best of our knowledge, this is the first work to formally define and address this problem. We tackle it through a novel tensor decomposition perspective, proposing the Functional t-Singular Value Decomposition (Ft-SVD) theorem which extends the classical tensor SVD to infinite and continuous feature domains, providing a natural tool for representing and analyzing multi-output functions. Within the Ft-SVD framework, we formulate the multi-output regression problem under CDS as a low-rank tensor estimation problem under the missing not at random (MNAR) setting, and introduce a series of assumptions about the true functions, training and testing distributions, and spectral properties of the ground-truth embeddings, making the problem more tractable. To address the challenges posed by CDS in multi-output regression, we develop a tailored Double-Stage Empirical Risk Minimization (ERM-DS) algorithm that leverages the spectral properties of the embeddings and uses specific hypothesis classes in each frequency component to better capture the varying spectral decay patterns. We provide rigorous theoretical analyses that establish performance guarantees for the ERM-DS algorithm. This work lays a preliminary theoretical foundation for multi-output regression under CDS.

Diffusion models are effective purification methods, where the noises or adversarial attacks are removed using generative approaches before pre-existing classifiers conducting classification tasks. However, the efficiency of diffusion models is still a concern, and existing solutions are based on knowledge distillation which can jeopardize the generation quality because of the small number of generation steps. Hence, we propose TendiffPure as a tensorized and compressed diffusion model for purification. Unlike the knowledge distillation methods, we directly compress U-Nets as backbones of diffusion models using tensor-train decomposition, which reduces the number of parameters and captures more spatial information in multi-dimensional data such as images. The space complexity is reduced from O ( N 2 ) to O ( NR 2 ) with R ≤ 4 as the tensor-train rank and N as the number of channels. Experimental results show that TendiffPure can more efficiently obtain high-quality purification results and outperforms the baseline purification methods on CIFAR-10, Fashion-MNIST, and MNIST datasets for two noises and one adversarial attack.

Recently, fusing low-resolution hyperspectral images (LR-HSIs) with high-resolution multispectral images (HR-MSIs) to obtain high-resolution HSI (HR-HSI) has become an emerging study. In this letter, Bayesian nonlocal canonical polyadic (CP) factorization (BNCPF) is proposed for fusing LR-HSI with HR-MSI, which applies CP factorization on the nonlocal tensors of HR-HSI. Compared with the vanilla scheme of applying CP factorization on HR-HSI, the nonlocal tensors reveal balanced low-rank properties along different modes, and thus CP factorization can better capture their intrinsic low-rankness. To avoid the immense CP-ranks selection on the nonlocal tensors, we develop a sparse Bayesian framework for automatic rank determination. For parameters estimation, we adopt the alternating direction method of multipliers (ADMMs) for the maximum a posteriori (MAP) estimator optimization. Experimental results verify the superiority and rank-robustness of the proposed method.

Learning a comprehensive representation from multiview data is crucial in many real-world applications. Multiview representation learning (MRL) based on nonnegative matrix factorization (NMF) has been widely adopted by projecting high-dimensional space into a lower order dimensional space with great interpretability. However, most prior NMF-based MRL techniques are shallow models that ignore hierarchical information. Although deep matrix factorization (DMF)-based methods have been proposed recently, most of them only focus on the consistency of multiple views and have cumbersome clustering steps. To address the above issues, in this article, we propose a novel model termed deep autoencoder-like NMF for MRL (DANMF-MRL), which obtains the representation matrix through the deep encoding stage and decodes it back to the original data. In this way, through a DANMF-based framework, we can simultaneously consider the multiview consistency and complementarity, allowing for a more comprehensive representation. We further propose a one-step DANMF-MRL, which learns the latent representation and final clustering labels matrix in a unified framework. In this approach, the two steps can negotiate with each other to fully exploit the latent clustering structure, avoid previous tedious clustering steps, and achieve optimal clustering performance. Furthermore, two efficient iterative optimization algorithms are developed to solve the proposed models both with theoretical convergence analysis. Extensive experiments on five benchmark datasets demonstrate the superiority of our approaches against other state-of-the-art MRL methods.

Tensor completion is a fundamental tool for incomplete data analysis, where the goal is to predict missing entries from partial observations. However, existing methods often make the explicit or implicit assumption that the observed entries are noise-free to provide a theoretical guarantee of exact recovery of missing entries, which is quite restrictive in practice. To remedy such drawback, this article proposes a novel noisy tensor completion model, which complements the incompetence of existing works in handling the degeneration of high-order and noisy observations. Specifically, the tensor ring nuclear norm (TRNN) and least-squares estimator are adopted to regularize the underlying tensor and the observed entries, respectively. In addition, a nonasymptotic upper bound of estimation error is provided to depict the statistical performance of the proposed estimator. Two efficient algorithms are developed to solve the optimization problem with convergence guarantee, one of which is specially tailored to handle large-scale tensors by replacing the minimization of TRNN of the original tensor equivalently with that of a much smaller one in a heterogeneous tensor decomposition framework. Experimental results on both synthetic and real-world data demonstrate the effectiveness and efficiency of the proposed model in recovering noisy incomplete tensor data compared with state-of-the-art tensor completion models.

Adversarial training is an effective way to defend deep neural networks (DNN) against adversarial examples. However, there are atypical samples that are rare and hard to learn, or even hurt DNNs' generalization performance on test data. In this paper, we propose a novel algorithm to reweight the training samples based on self-supervised techniques to mitigate the negative effects of the atypical samples. Specifically, a memory bank is built to record the popular samples as prototypes and calculate the memorization weight for each sample, evaluating the "typicalness" of a sample. All the training samples are reweigthed based on the proposed memorization weights to reduce the negative effects of atypical samples. Experimental results show the proposed method is flexible to boost state-of-the-art adversarial training methods, improving both robustness and standard accuracy of DNNs.

Tensor network (TN) is a powerful framework in machine learning, but selecting a good TN model, known as TN structure search (TN-SS), is a challenging and computationally intensive task. The recent approach TNLS (Li et al., 2022) showed promising results for this task. However, its computational efficiency is still unaffordable, requiring too many evaluations of the objective function. We propose TnALE, a surprisingly simple algorithm that updates each structure-related variable alternately by local enumeration, greatly reducing the number of evaluations compared to TNLS. We theoretically investigate the descent steps for TNLS and TnALE, proving that both the algorithms can achieve linear convergence up to a constant if a sufficient reduction of the objective is reached in each neighborhood. We further compare the evaluation efficiency of TNLS and TnALE, revealing that $\Omega(2^K)$ evaluations are typically required in TNLS for reaching the objective reduction, while ideally $O(KR)$ evaluations are sufficient in TnALE, where $K$ denotes the dimension of search space and $R$ reflects the “low-rankness” of the neighborhood. Experimental results verify that TnALE can find practically good TN structures with vastly fewer evaluations than the state-of-the-art algorithms.

Tensor decompositions (TDs) serve as a powerful tool for analyzing multiway data. Traditional TDs incorporate prior knowledge about the data into the model, such as a directed generative process from latent factors to observations. In practice, selecting proper structural or distributional assumptions beforehand is crucial for obtaining a promising TD representation. However, since such prior knowledge is typically unavailable in real-world applications, choosing an appropriate TD model can be challenging. This paper aims to address this issue by introducing a flexible TD framework that discards the structural and distributional assumptions, in order to learn as much information from the data. Specifically, we construct a TD model that captures the joint probability of the data and latent tensor factors through a deep energy-based model (EBM). Neural networks are then employed to parameterize the joint energy function of tensor factors and tensor entries. The flexibility of EBM and neural networks enables the learning of underlying structures and distributions. In addition, by designing the energy function, our model unifies the learning process of different types of tensors, such as static tensors and dynamic tensors with time stamps. The resulting model presents a doubly intractable nature due to the presence of latent tensor factors and the unnormalized probability function. To efficiently train the model, we derive a variational upper bound of the conditional noise-contrastive estimation objective that learns the unnormalized joint probability by distinguishing data from conditional noises. We show advantages of our model on both synthetic and several real-world datasets.

Multi-channel learning has gained significant attention in recent applications, where neural networks with t-product layers (t-NNs) have shown promising performance through novel feature mapping in the transformed domain. However, despite the practical success of t-NNs, the theoretical analysis of their generalization remains unexplored. We address this gap by deriving upper bounds on the generalization error of t-NNs in both standard and adversarial settings. Notably, it reveals that t-NNs compressed with exact transformed low-rank parameterization can achieve tighter adversarial generalization bounds compared to non-compressed models. While exact transformed low-rank weights are rare in practice, the analysis demonstrates that through adversarial training with gradient flow, highly over-parameterized t-NNs with the ReLU activation can be implicitly regularized towards a transformed low-rank parameterization under certain conditions. Moreover, this paper establishes sharp adversarial generalization bounds for t-NNs with approximately transformed low-rank weights. Our analysis highlights the potential of transformed low-rank parameterization in enhancing the robust generalization of t-NNs, offering valuable insights for further research and development.

Attention-based networks currently identify their effectiveness in multimodal sentiment analysis. However, existing methods ignore the redundancy of auxiliary modalities. More importantly, existing methods only attend to top-down attention (static process) or down-top attention (implicit process), leading to the coarse-grained multimodal sentiment context. In this paper, during the preprocessing period, we first propose the multimodal dynamic enhanced block to capture the intra-modality sentiment context. This can effectively decrease the intra-modality redundancy of auxiliary modalities. Furthermore, the bi-direction attention block is proposed to capture fine-grained multimodal sentiment context via the novel bi-direction multimodal dynamic routing mechanism. Specifically, the bi-direction attention block first highlights the explicit and low-level multimodal sentiment context. Then, the low-level multimodal context is transmitted to a carefully designed bi-direction multimodal dynamic routing procedure. This allows us to dynamically update and investigate high-level and much more fine-grained multimodal sentiment contexts. The experiments demonstrate that our fusion network can achieve state-of-the-art performance. Notably, our model outperforms the best baseline on the metric ‘Acc-7’ with an improvement of 6.9%.

Tensor-ring (TR) decomposition is a powerful tool for exploiting the low-rank property of multiway data and has been demonstrated great potential in a variety of important applications. In this article, non-negative TR (NTR) decomposition and graph-regularized NTR (GNTR) decomposition are proposed. The former equips TR decomposition with the ability to learn the parts-based representation by imposing non-negativity on the core tensors, and the latter additionally introduces a graph regularization to the NTR model to capture manifold geometry information from tensor data. Both of the proposed models extend TR decomposition and can be served as powerful representation learning tools for non-negative multiway data. The optimization algorithms based on an accelerated proximal gradient are derived for NTR and GNTR. We also empirically justified that the proposed methods can provide more interpretable and physically meaningful representations. For example, they are able to extract parts-based components with meaningful color and line patterns from objects. Extensive experimental results demonstrated that the proposed methods have better performance than state-of-the-art tensor-based methods in clustering and classification tasks.

Since a large proportion of real-world data is made of different representations or views, learning on data represented with multiple views (e.g., numerous types of features or modalities) has garnered considerable attention recently. Nonnegative matrix factorization (NMF) has been widely adopted for multi-view learning due to its great interpretability. We focus on unsupervised multi-view data representation in this paper and propose a novel framework termed Deep Autoencoder-like NMF (DANMF-MDR), which learns an intact representation by simultaneously exploring multi-view complementary and consistent information. Furthermore, an efficient iterative optimization algorithm is developed to solve the proposed model. Experimental results on three real-world multi-view datasets demonstrate that ours performs better than the SOTA multi-view NMF-based MDR approaches.

Tensor network (TN) methods have proven their considerable potential in deterministic regression and classification related paradigms, but remain underexplored in probabilistic settings. To this end, we introduce a variational inference framework for supervised learning in the context of TNs, referred to as the Bayesian Tensor Network (BTN). This is achieved by making use of the multi-linear nature of tensor networks which allows us to construct a structured variational model which scales linearly with data dimensionality. The so imposed low rank structure on the tensor mean and Kronecker separability of the local covariances makes it possible to efficiently induce weight dependencies in the posterior distribution. This is shown to enhance model expressiveness at a drastically lower parameter complexity compared to the standard mean-field approach. A comprehensive validation of the proposed framework demonstrates the competitiveness of BTNs against existing structured Bayesian neural network approaches, while exhibiting enhanced interpretability, computational efficiency, and ability to yield credibility intervals.

Recent works put much effort into tensor network structure search (TN-SS), aiming to select suitable tensor network (TN) structures, involving the TN-ranks, formats, and so on, for the decomposition or learning tasks. In this paper, we consider a practical variant of TN-SS, dubbed TN permutation search (TN-PS), in which we search for good mappings from tensor modes onto TN vertices (core tensors) for compact TN representations. We conduct a theoretical investigation of TN-PS and propose a practically-efficient algorithm to resolve the problem. Theoretically, we prove the counting and metric properties of search spaces of TN-PS, analyzing for the first time the impact of TN structures on these unique properties. Numerically, we propose a novel meta-heuristic algorithm, in which the searching is done by randomly sampling in a neighborhood established in our theory, and then recurrently updating the neighborhood until convergence. Numerical results demonstrate that the new algorithm can reduce the required model size of TNs in extensive benchmarks, implying the improvement in the expressive power of TNs. Furthermore, the computational cost for the new algorithm is significantly less than that in (Li and Sun, 2020).

The heart of multimodal learning research lies the challenge of effectively exploiting fusion representations among multiple modalities.However, existing two-way cross-modality unidirectional attention could only exploit the intermodal interactions from one source to one target modality. This indeed fails to unleash the complete expressive power of multimodal fusion with restricted number of modalities and fixed interactive direction.In this work, the multiway multimodal transformer (MMT) is proposed to simultaneously explore multiway multimodal intercorrelations for each modality via single block rather than multiple stacked cross-modality blocks. The core idea of MMT is the multiway multimodal attention, where the multiple modalities are leveraged to compute the multiway attention tensor. This naturally benefits us to exploit comprehensive many-to-many multimodal interactive paths. Specifically, the multiway tensor is comprised of multiple interconnected modality-aware core tensors that consist of the intramodal interactions. Additionally, the tensor contraction operation is utilized to investigate intermodal dependencies between distinct core tensors.Essentially, our tensor-based multiway structure allows for easily extending MMT to the case associated with an arbitrary number of modalities. Taking MMT as the basis, the hierarchical network is further established to recursively transmit the low-level multiway multimodal interactions to high-level ones. The experiments demonstrate that MMT can achieve state-of-the-art or comparable performance.

In the recent decade, multidimensional or tenso-rial time series have drawn increasing attention for their rich spatial and temporal information. In data collection, missing tensorial time instances can always occur at any random time steps, which cause the time steps to be not equally spaced, i.e., irregular. Furthermore, it is very likely that the tensorial time series are influenced by other tensor factors, yet from external resources. These driving tensor factors are referred to as tensor controls, whereas the influenced tensorial time series are named as tensor responses. Existing methods either cannot predict tensor responses using all available past tensor controls, or are not able to directly model continuous-time tensor-valued processes, incurring the curse-of-dimensionality issue. Therefore, we propose tensor neural controlled differential equations (TENCDEs) which incorporate tensor controls and explicitly define the continuous-time tensor-valued processes for tensor controls and tensor responses aiming at resolving irregular tensorial time series issues. TENCDE also effectively captures both spatial and temporal information, and avoids the curse-of-dimensionality issue. The unique solution and the consistency and convergence of solutions are also guaranteed. The experiments demonstrate the proposed TENCDE outperforms the existing models including the state-of-the-art methods.

Electroencephalography (EEG) provides access to neuronal dynamics non-invasively with millisecond resolution, rendering it a viable method in neuroscience and healthcare. However, its utility is limited as current EEG technology does not generalize well across domains (i.e., sessions and subjects) without expensive supervised re-calibration. Contemporary methods cast this transfer learning (TL) problem as a multi-source/-target unsupervised domain adaptation (UDA) problem and address it with deep learning or shallow, Riemannian geometry aware alignment methods. Both directions have, so far, failed to consistently close the performance gap to state-of-the-art domain-specific methods based on tangent space mapping (TSM) on the symmetric, positive definite (SPD) manifold.Here, we propose a machine learning framework that enables, for the first time, learning domain-invariant TSM models in an end-to-end fashion. To achieve this, we propose a new building block for geometric deep learning, which we denote SPD domain-specific momentum batch normalization (SPDDSMBN). A SPDDSMBN layer can transform domain-specific SPD inputs into domain-invariant SPD outputs, and can be readily applied to multi-source/-target and online UDA scenarios. In extensive experiments with 6 diverse EEG brain-computer interface (BCI) datasets, we obtain state-of-the-art performance in inter-session and -subject TL with a simple, intrinsically interpretable network architecture, which we denote TSMNet. Code: https://github.com/rkobler/TSMNet

We make an attempt to understand convolutional neural network by exploring the relationship between (deep) convolutional neural networks and Volterra convolutions. We propose a novel approach to explain and study the overall characteristics of neural networks without being disturbed by the horribly complex architectures. Specifically, we attempt to convert the basic structures of a convolutional neural network (CNN) and their combinations to the form of Volterra convolutions. The results show that most of convolutional neural networks can be approximated in the form of Volterra convolution, where the approximated proxy kernels preserve the characteristics of the original network. Analyzing these proxy kernels may give valuable insight about the original network. Based on this setup, we present methods to approximate the order-zero and order-one proxy kernels, and verify the correctness and effectiveness of our results.

Non-local low-rank tensor approximation has been developed as a state-of-the-art method for hyperspectral image (HSI) restoration, which includes the tasks of denoising, compressed HSI reconstruction and inpainting. Unfortunately, while its restoration performance benefits from more spectral bands, its runtime also substantially increases. In this paper, we claim that the HSI lies in a global spectral low-rank subspace, and the spectral subspaces of each full band patch group should lie in this global low-rank subspace. This motivates us to propose a unified paradigm combining the spatial and spectral properties for HSI restoration. The proposed paradigm enjoys performance superiority from the non-local spatial denoising and light computation complexity from the low-rank orthogonal basis exploration. An efficient alternating minimization algorithm with rank adaptation is developed. It is done by first solving a fidelity term-related problem for the update of a latent input image, and then learning a low-dimensional orthogonal basis and the related reduced image from the latent input image. Subsequently, non-local low-rank denoising is developed to refine the reduced image and orthogonal basis iteratively. Finally, the experiments on HSI denoising, compressed reconstruction, and inpainting tasks, with both simulated and real datasets, demonstrate its superiority with respect to state-of-the-art HSI restoration methods.

Tensor robust principal component analysis (TRPCA) is a fundamental model in machine learning and computer vision. Recently, tensor train (TT) decomposition has been verified effective to capture the global low-rank correlation for tensor recovery tasks. However, due to the large-scale tensor data in real-world applications, existing TRPCA models often suffer from high computational complexity. In this letter, we propose an efficient TRPCA under hybrid model of Tucker and TT. Specifically, in theory we reveal that TT nuclear norm (TTNN) of the original big tensor can be equivalently converted to that of a much smaller tensor via a Tucker compression format, thereby significantly reducing the computational cost of singular value decomposition (SVD). Numerical experiments on both synthetic and real-world tensor data verify the superiority of the proposed model.

Deep multitask learning (MTL) shares beneficial knowledge across participating tasks, alleviating the impacts of extreme learning conditions on their performances such as the data scarcity problem. In practice, participators stemming from different domain sources often have varied complexities and input sizes, for example, in the joint learning of computer vision tasks with RGB and grayscale images. For adapting to these differences, it is appropriate to design networks with proper representational capacities and construct neural layers with corresponding widths. Nevertheless, most of the state-of-the-art methods pay little attention to such situations, and actually fail to handle the disparities. To work with the dissimilitude of tasks' network designs, this article presents a distributed knowledge-sharing framework called tensor ring multitask learning (TRMTL), in which the relationship between knowledge sharing and original weight matrices is cut up. The framework of TRMTL is flexible, which is not only capable of sharing knowledge across heterogenous networks but also able to jointly learn tasks with varied input sizes, significantly improving performances of data-insufficient tasks. Comprehensive experiments on challenging datasets are conducted to empirically validate the effectiveness, efficiency, and flexibility of TRMTL in dealing with the disparities in MTL.

Deep image prior (DIP), which uses a deep convolutional network (ConvNet) structure as an image prior, has attracted wide attention in computer vision and machine learning. DIP empirically shows the effectiveness of the ConvNet structures for various image restoration applications. However, why the DIP works so well is still unknown. In addition, the reason why the convolution operation is useful in image reconstruction, or image enhancement is not very clear. This study tackles this ambiguity of ConvNet/DIP by proposing an interpretable approach that divides the convolution into "delay embedding" and "transformation" (i.e., encoder-decoder). Our approach is a simple, but essential, image/tensor modeling method that is closely related to self-similarity. The proposed method is called manifold modeling in embedded space (MMES) since it is implemented using a denoising autoencoder in combination with a multiway delay-embedding transform. In spite of its simplicity, MMES can obtain quite similar results to DIP on image/tensor completion, super-resolution, deconvolution, and denoising. In addition, MMES is proven to be competitive with DIP, as shown in our experiments. These results can also facilitate interpretation/characterization of DIP from the perspective of a "low-dimensional patch-manifold prior."

The popular tensor train (TT) and tensor ring (TR) decompositions have achieved promising results in science and engineering. However, TT and TR decompositions only establish an operation between adjacent two factors and are highly sensitive to the permutation of tensor modes, leading to an inadequate and inflexible representation. In this paper, we propose a generalized tensor decomposition, which decomposes an Nth-order tensor into a set of Nth-order factors and establishes an operation between any two factors. Since it can be graphically interpreted as a fully-connected network, we named it fully-connected tensor network (FCTN) decomposition. The superiorities of the FCTN decomposition lie in the outstanding capability for characterizing adequately the intrinsic correlations between any two modes of tensors and the essential invariance for transposition. Furthermore, we employ the FCTN decomposition to one representative task, i.e., tensor completion, and develop an efficient solving algorithm based on proximal alternating minimization. Theoretically, we prove the convergence of the developed algorithm, i.e., the sequence obtained by it globally converges to a critical point. Experimental results substantiate that the proposed method compares favorably to the state-of-the-art methods based on other tensor decompositions.

Jiajia Tang, Kang Li, Xuanyu Jin, Andrzej Cichocki, Qibin Zhao, Wanzeng Kong. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Latent factor models are canonical tools to learn low-dimensional and linear embedding of original data. Traditional latent factor models are based on low-rank matrix factorization of covariance matrices. However, for higher-order data with multiple modes, i.e., tensors, this simple treatment fails to take into account the mode-specific relations. This ignorance leads to inefficiency in analysis of complex structures as well as poor data compression ability. In this paper, unlike covariance matrices, we investigate high-order covariance tensor directly by exploiting tensor ring (TR) format and propose the Bayesian TR latent factor model, which can represent complex multi-linear correlations and achieves efficient data compression. To overcome the difficulty of finding the optimal TR-ranks and simultaneously imposing sparsity on loading coefficients, a multiplicative Gamma process (MGP) prior is adopted to automatically infer the ranks and obtain sparsity. Then, we establish an efficient parameter-expanded EM algorithm to learn the maximum a posteriori (MAP) estimate of model parameters. Finally, we evaluate our model on covariance estimation, latent factor learning and image inpainting problems.

Tensor-power (TP) recurrent model is a family of non-linear dynamical systems, of which the recurrence relation consists of a p-fold (a.k.a., degree-p) tensor product. Despite such the model frequently appears in the advanced recurrent neural networks (RNNs), to this date there is limited study on its memory property, a critical characteristic in sequence tasks. In this work, we conduct a thorough investigation of the memory mechanism of TP recurrent models. Theoretically, we prove that a large degree p is an essential condition to achieve the long memory effect, yet it would lead to unstable dynamical behaviors. Empirically, we tackle this issue by extending the degree p from discrete to a differentiable domain, such that it is efficiently learnable from a variety of datasets. Taken together, the new model is expected to benefit from the long memory effect in a stable manner. We experimentally show that the proposed model achieves competitive performance compared to various advanced RNNs in both the single-cell and seq2seq architectures.

Tensor decomposition (TD) has shown promising performance in image completion and denoising. Existing methods always aim to decompose one tensor into latent factors or core tensors by optimizing a particular cost function based on a specific tensor model. These algorithms iteratively learn the optima from random initialization given any individual tensor, resulting in slow convergence and low efficiency. In this paper, we propose an efficient TD algorithm that aims to learn a global mapping from input tensors to latent core tensors, under the assumption that the mappings of multiple tensors might be shared or highly correlated. To this end, we train a deep neural network (DNN) to model the global mapping and then apply it to decompose a newly given tensor with high efficiency. Furthermore, the initial values of DNN are learned based on meta-learning methods. By leveraging the pretrained core tensor DNN, our proposed method enables us to perform TD efficiently and accurately. Experimental results demonstrate the significant improvements of our method over other TD methods in terms of speed and accuracy.

Information steganography is a family of techniques that hide secret messages into a carrier; thus, the messages can only be extracted by receivers with a correct key Although many approaches have been proposed to achieve this purpose, historically, it is a difficult problem to conceal a large amount of information without occasioning human perceptible changes. In this paper, we explore the room introduced by the low-rank property of natural signals (i.e., images, audios), and propose a training-free model for efficient information steganography, which provides a capacity of hiding full-size images into carriers of the same spatial resolution. The key of our method is to randomly shuffle the secrets and carry out a simple reduction summation with the carrier. On the other hand, the secret images can be reconstructed by solving a convex optimization problem similar to the ordinary tensor decomposition. In the experimental analysis, we carry out two tasks: concealing a full-RGB-color image into a gray-scale image; concealing images into music signals. The results confirm the ability of our model to handle massive secret payloads. The code of our paper is provided in https://github.com/minogame/icassp-SIC.

In high dimensional tensorial time series prediction, it is highly desired to preserve spatial structural and underlying continuous sequential information in modelling. Existing methods either destroy the spatial structure and require a large number of parameters, such as neural ordinary differential equations (neural ODEs), or cannot capture the temporal information, especially for unevenly spaced time series, such as tensorial neural network families. Hence we propose Tensor Neural Ordinary Differential Equations (TENODEs) to address these issues. The dynamics of data is modelled by a tensorial neural network (TNN) consisting of the Tucker structure. Compared with neural ODEs, it reduces the number of parameters from$O(I^{2N})$to$O(NI^{2})$. To further ease the computational cost, we also propose a TENODE with further dimensionality reduction which produces a low-dimensional representation of the aforementioned two pieces of information and is projected to the target space by a 1-layer TNN. In each layer of the RHS for TENODE, the weight matrices should maintain Lipschitzness according to the Picard's theorem. Hence TENODEs are consistent and converge. Multiplication of weight matrices together can affect the optimisation stability: when values of some weights are enlarged, values of the other weights are correspondingly decreased, with the loss unchanged. This may create infinite solution spaces. We thus constrain weights to be orthogonal and solve this scaling issue to enhance the optimisation process. In consequence, the proposed TEN-ODEs successfully preserve data spatial and latent continuous sequential information. Experimental results demonstrate the efficacy of our proposed methods over competitive baselines.

Image restoration with corruptions from combined multiple types of distortion is a challenging and practical problem. Recent studies show that a promising technique is to perform parallel “operations” to handle different types of distortion, which can be modeled by a deep neural network framework. However, the reconstruction may be dominated by a small number of operations due to the heterogeneous features generated by different operations. To handle this issue, we introduce a tensor 1×1 convolutional layer by leveraging high-order tensor fusion, which can not only harmonize the heterogeneous features but also take high order statistical information into account. To efficiently learn the large-scale kernel tensor resulted from the tensor product, we employ tensor network to represent kernels, which is able to convert the exponential growth of the dimension to linear growth. Armed with this new layer, we propose high-order operation-wise attention network for the task of multi-distorted image restoration. The experimental results demonstrated that the proposed method outperforms the method with vanilla 1 × 1 convolutional layer in several typical tasks and is promising for more difficult tasks. Code is available at https://github.com/ZihaoH/High-order-OWAN.

Low-rank tensor completion (LRTC) aims to impute the missing entries from partially observed tensor data, among which low-rankness is of vital importance to get satisfactory results. In this letter, we propose a hierarchical low-rank factorization framework for high-order tensors. For the first layer, the low TR rank is exploited, and for the second layer the low-rankness of each TR core is further considered. With the hierarchical model, the low-rankness of the original tensor can be fully utilized and thus achieving better completion performance. Experimental results on synthetic data and on inpainting tasks using various datasets demonstrate the superior performance and efficiency of our proposed method as compared to the state-of-the-art algorithms.

Tensor-ring (TR) decomposition has recently attracted considerable attention in solving the low-rank tensor completion (LRTC) problem. However, due to an unbalanced unfolding scheme used during the update of core tensors, the conventional TR-based completion methods usually require a large TR rank to achieve the optimal performance, which leads to high computational cost in practical applications. To overcome this drawback, we propose a new method to exploit the low TR-rank structure in this article. Specifically, we first introduce a balanced unfolding operation called tensor circular unfolding, by which the relationship between TR rank and the ranks of tensor unfoldings is theoretically established. Using this new unfolding operation, we further propose an algorithm to exploit the low TR-rank structure by performing parallel low-rank matrix factorizations to all circularly unfolded matrices. To tackle the problem of nonuniform missing patterns, we apply a row weighting trick to each circularly unfolded matrix, which significantly improves the adaptive ability to various types of missing patterns. The extensive experiments have demonstrated that the proposed algorithm can achieve outstanding performance using a much smaller TR rank compared with the conventional TR-based completion algorithms; meanwhile, the computational cost is reduced substantially.

Exact recovery of tensor decomposition (TD) methods is a desirable property in both unsupervised learning and scientific data analysis. The numerical defects of TD methods, however, limit their practical applications on real-world data. As an alternative, convex tensor decomposition (CTD) was proposed to alleviate these problems, but its exact-recovery property is not properly addressed so far. To this end, we focus on latent convex tensor decomposition (LCTD), a practically widely-used CTD model, and rigorously prove a sufficient condition for its exact-recovery property. Furthermore, we show that such property can be also achieved by a more general model than LCTD. In the new model, we generalize the classic tensor (un-)folding into reshuffling operation, a more flexible mapping to relocate the entries of the matrix into a tensor. Armed with the reshuffling operations and exact-recovery property, we explore a totally novel application for (generalized) LCTD, i.e., image steganography. Experimental results on synthetic data validate our theory, and results on image steganography show that our method outperforms the state-of-the-art methods.

Low-rank tensor recovery has been widely applied to computer vision and machine learning. Recently, tubal nuclear norm (TNN) based optimization is proposed with superior performance as compared to other tensor nuclear norms. However, one major limitation is its orientation sensitivity due to low-rankness strictly defined along tubal orientation and it cannot simultaneously model spectral low-rankness in multiple orientations. To this end, we introduce two new tensor norms called OITNN-O and OITNN-L to exploit multi-orientational spectral low-rankness for an arbitrary K-way (K ≥ 3) tensors. We further formulate two robust tensor decomposition models via the proposed norms and develop two algorithms as the solutions. Theoretically, we establish non-asymptotic error bounds which can predict the scaling behavior of the estimation error. Experiments on real-world datasets demonstrate the superiority and effectiveness of the proposed norms.

Epileptic focus localization in patients with epileptic seizures is essential when surgery is needed. Recent studies show that this can be done automatically using machine learning approaches. However, well-designed feature extraction methods are often computationally demanding, requiring a large amount of data labeled by physicians, which is time consuming and impractical. In this paper, we firstly introduce a one-dimensional convolutional neural network (1D-CNN) model for epileptic seizure focus detection which avoids the manual, time-consuming feature extraction Moreover, to reduce the necessary number of training samples, we introduce an approach for data augmentation. The experimental results demonstrate the efficiency of the proposed method, with a nearly 3% improvement in performance using the data enhancement method compared to the best result obtained using the traditional feature extraction method.

In tensor completion tasks, the traditional low-rank tensor decomposition models suffer from the laborious model selection problem due to their high model sensitivity. In particular, for tensor ring (TR) decomposition, the number of model possibilities grows exponentially with the tensor order, which makes it rather challenging to find the optimal TR decomposition. In this paper, by exploiting the low-rank structure of the TR latent space, we propose a novel tensor completion method which is robust to model selection. In contrast to imposing the low-rank constraint on the data space, we introduce nuclear norm regularization on the latent TR factors, resulting in the optimization step using singular value decomposition (SVD) being performed at a much smaller scale. By leveraging the alternating direction method of multipliers (ADMM) scheme, the latent TR factors with optimal rank and the recovered tensor can be obtained simultaneously. Our proposed algorithm is shown to effectively alleviate the burden of TR-rank selection, thereby greatly reducing the computational cost. The extensive experimental results on both synthetic and real-world data demonstrate the superior performance and efficiency of the proposed approach against the state-of-the-art algorithms.

There has been an increased interest in multimodal language processing including multimodal dialog, question answering, sentiment analysis, and speech recognition.However, naturally occurring multimodal data is often imperfect as a result of imperfect modalities, missing entries or noise corruption.To address these concerns, we present a regularization method based on tensor rank minimization.Our method is based on the observation that high-dimensional multimodal time series data often exhibit correlations across time and modalities which leads to low-rank tensor representations.However, the presence of noise or incomplete values breaks these correlations and results in tensor representations of higher rank.We design a model to learn such tensor representations and effectively regularize their rank.Experiments on multimodal language data show that our model achieves good results across various levels of imperfection.

Non-local low-rank tensor approximation has been developed as a state-of-the-art method for hyperspectral image (HSI) denoising. Unfortunately, while their denoising performance benefits little from more spectral bands, the running time of these methods significantly increases. In this paper, we claim that the HSI lies in a global spectral low-rank subspace, and the spectral subspaces of each full band patch groups should lie in this global low-rank subspace. This motivates us to propose a unified spatial-spectral paradigm for HSI denoising. As the new model is hard to optimize, An efficient algorithm motivated by alternating minimization is developed. This is done by first learning a low-dimensional orthogonal basis and the related reduced image from the noisy HSI. Then, the non-local low-rank denoising and iterative regularization are developed to refine the reduced image and orthogonal basis, respectively. Finally, the experiments on synthetic and both real datasets demonstrate the superiority against the stateof-the-art HSI denoising methods.

Low-rank matrix completion (LRMC) is a classical model in both computer vision (CV) and machine learning, and has been successfully applied to various real applications. In the recent CV tasks, the completion is usually employed on the variants of data, such as "non-local" or filtered, rather than their original forms. This fact makes that the theoretical analysis of the conventional LRMC is no longer suitable in these applications. To tackle this problem, we propose a more general framework for LRMC, in which the linear transformations of the data are taken into account. We rigorously prove the identifiability of the proposed model and show an upper bound of the reconstruction error. Furthermore, we derive an efficient completion algorithm by using augmented Lagrangian multipliers and the sketching trick. In the experiments, we apply the proposed method to the classical image inpainting problem and achieve the state-of-the-art results.

Although the convolutional neural networks (CNNs) have become popular for various image processing and computer vision tasks recently, it remains a challenging problem to reduce the storage cost of the parameters for resource-limited platforms. In the previous studies, tensor decomposition (TD) has achieved promising compression performance by embedding the kernel of a convolutional layer into a low-rank subspace. However the employment of TD is naively on the kernel or its specified variants. Unlike the conventional approaches, this paper shows that the kernel can be embedded into more general or even random low-rank subspaces. We demonstrate this by compressing the convolutional layers via randomly-shuffled tensor decomposition (RsTD) for a standard classification task using CIFAR-10. In addition, we analyze how the spatial similarity of the training data influences the low-rank structure of the kernels. The experimental results show that the CNN can be significantly compressed even if the kernels are randomly shuffled. Furthermore, the RsTD-based method yields more stable classification accuracy than the conventional TD-based methods in a large range of compression ratios.

Tensor ring (TR) decomposition has been successfully used to obtain the state-of-the-art performance in the visual data completion problem. However, the existing TR-based completion methods are severely non-convex and computationally demanding. In addition, the determination of the optimal TR rank is a tough work in practice. To overcome these drawbacks, we first introduce a class of new tensor nuclear norms by using tensor circular unfolding. Then we theoretically establish connection between the rank of the circularly unfolded matrices and the TR ranks. We also develop an efficient tensor completion algorithm by minimizing the proposed tensor nuclear norm. Extensive experimental results demonstrate that our proposed tensor completion method outperforms the conventional tensor completion methods in the image video in-painting problem with striped missing values.

Dimensionality reduction is an essential technique for multiway large-scale data, i.e., tensor. Tensor ring (TR) decomposition has become popular due to its high representation ability and flexibility. However, the traditional TR decomposition algorithms suffer from high computational cost when facing large-scale data. In this paper, taking advantages of the recently proposed tensor random projection method, we propose two TR decomposition algorithms. By employing random projection on every mode of the large-scale tensor, the TR decomposition can be processed at a much smaller scale. The simulation experiment shows that the proposed algorithms are 4–25 times faster than traditional algorithms without loss of accuracy, and our algorithms show superior performance in deep learning dataset compression and hyperspectral image reconstruction experiments compared to the other randomized algorithms.

Tensor train decomposition is a powerful representation for high-order tensors, which has been successfully applied to various machine learning tasks in recent years. In this paper, we study a more generalized tensor decomposition with a ring-structured network by employing circular multilinear products over a sequence of lower-order core tensors. We refer to such tensor decomposition as tensor ring (TR) representation. Our goal is to introduce learning algorithms including sequential singular value decompositions and blockwise alternating least squares with adaptive tensor ranks. Experimental results demonstrate the effectiveness of the TR model and the learning algorithms. In particular, we show that the structure information and high-order correlations within a 2D image can be captured efficiently by employing an appropriate tensorization and TR decomposition.

Tensor-based multimodal fusion techniques have exhibited great predictive performance. However, one limitation is that existing approaches only consider bilinear or trilinear pooling, which fails to unleash the complete expressive power of multilinear fusion with restricted orders of interactions. More importantly, simply fusing features all at once ignores the complex local intercorrelations, leading to the deterioration of prediction. In this work, we first propose a polynomial tensor pooling (PTP) block for integrating multimodal features by considering high-order moments, followed by a tensorized fully connected layer. Treating PTP as a building block, we further establish a hierarchical polynomial fusion network (HPFN) to recursively transmit local correlations into global ones. By stacking multiple PTPs, the expressivity capacity of HPFN enjoys an exponential growth w.r.t. the number of layers, which is shown by the equivalence to a very deep convolutional arithmetic circuits. Various experiments demonstrate that it can achieve the state-of-the-art performance.

MOTIVATION: Genome-wide identification of the transcriptomic responses of human cell lines to drug treatments is a challenging issue in medical and pharmaceutical research. However, drug-induced gene expression profiles are largely unknown and unobserved for all combinations of drugs and human cell lines, which is a serious obstacle in practical applications. RESULTS: Here, we developed a novel computational method to predict unknown parts of drug-induced gene expression profiles for various human cell lines and predict new drug therapeutic indications for a wide range of diseases. We proposed a tensor-train weighted optimization (TT-WOPT) algorithm to predict the potential values for unknown parts in tensor-structured gene expression data. Our results revealed that the proposed TT-WOPT algorithm can accurately reconstruct drug-induced gene expression data for a range of human cell lines in the Library of Integrated Network-based Cellular Signatures. The results also revealed that in comparison with the use of original gene expression profiles, the use of imputed gene expression profiles improved the accuracy of drug repositioning. We also performed a comprehensive prediction of drug indications for diseases with gene expression profiles, which suggested many potential drug indications that were not predicted by previous approaches. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Time-series remote sensing (RS) images are often corrupted by various types of missing information such as dead pixels, clouds, and cloud shadows that significantly influence the subsequent applications. In this paper, we introduce a new low-rank tensor decomposition model, termed tensor ring (TR) decomposition, to the analysis of RS data sets and propose a TR completion method for the missing information reconstruction. The proposed TR completion model has the ability to utilize the low-rank property of time-series RS images from different dimensions. To further explore the smoothness of the RS image spatial information, total-variation regularization is also incorporated into the TR completion model. The proposed model is efficiently solved using two algorithms, the augmented Lagrange multiplier (ALM) and the alternating least square (ALS) methods. The simulated and real-data experiments show superior performance compared to other state-of-the-art low-rank related algorithms.

Multidimensional data (i.e., tensors) with missing entries are common in practice. Extracting features from incomplete tensors is an important yet challenging problem in many fields such as machine learning, pattern recognition, and computer vision. Although the missing entries can be recovered by tensor completion techniques, these completion methods focus only on missing data estimation instead of effective feature extraction. To the best of our knowledge, the problem of feature extraction from incomplete tensors has yet to be well explored in the literature. In this paper, we therefore tackle this problem within the unsupervised learning environment. Specifically, we incorporate low-rank tensor decomposition with feature variance maximization (TDVM) in a unified framework. Based on orthogonal Tucker and CP decompositions, we design two TDVM methods, TDVM-Tucker and TDVM-CP, to learn low-dimensional features viewing the core tensors of the Tucker model as features and viewing the weight vectors of the CP model as features. TDVM explores the relationship among data samples via maximizing feature variance and simultaneously estimates the missing entries via low-rank Tucker/CP approximation, leading to informative features extracted directly from observed entries. Furthermore, we generalize the proposed methods by formulating a general model that incorporates feature regularization into low-rank tensor approximation. In addition, we develop a joint optimization scheme to solve the proposed methods by integrating the alternating direction method of multipliers with the block coordinate descent method. Finally, we evaluate our methods on six real-world image and video data sets under a newly designed multiblock missing setting. The extracted features are evaluated in face recognition, object/action classification, and face/gait clustering. Experimental results demonstrate the superior performance of the proposed methods compared with the state-of-the-art approaches.

In this paper, we aim at the problem of tensor data completion. Tensor-train decomposition is adopted because of its powerful representation ability and linear scalability to tensor order. We propose an algorithm named Sparse Tensor-train Optimization (STTO) which considers incomplete data as sparse tensor and uses first-order optimization method to find the factors of tensor-train decomposition. Our algorithm is shown to perform well in simulation experiments at both low-order cases and high-order cases. We also employ a ten-sorization method to transform data to a higher-order form to enhance the performance of our algorithm. The results of image recovery experiments in various cases manifest that our method outperforms other completion algorithms. Especially when the missing rate is very high, e.g., 90% to 99%, our method is significantly better than the state-of-the-art methods.

In this work, we consider the task of classifying binary positive-unlabeled (PU) data. The existing discriminative learning based PU models attempt to seek an optimal reweighting strategy for U data, so that a decent decision boundary can be found. However, given limited P data, the conventional PU models tend to suffer from overfitting when adapted to very flexible deep neural networks. In contrast, we are the first to innovate a totally new paradigm to attack the binary PU task, from perspective of generative learning by leveraging the powerful generative adversarial networks (GAN). Our generative positive-unlabeled (GenPU) framework incorporates an array of discriminators and generators that are endowed with different roles in simultaneously producing positive and negative realistic samples. We provide theoretical analysis to justify that, at equilibrium, GenPU is capable of recovering both positive and negative data distributions. Moreover, we show GenPU is generalizable and closely related to the semi-supervised classification. Given rather limited P data, experiments on both synthetic and real-world dataset demonstrate the effectiveness of our proposed framework. With infinite realistic and diverse sample streams generated from GenPU, a very flexible classifier can then be trained using deep neural networks.

In this work, we introduce a new generalized nonlinear tensor regression framework called kernel-based multiblock tensor partial least squares (KMTPLS) for predicting a set of dependent tensor blocks from a set of independent tensor blocks through the extraction of a small number of common and discriminative latent components. By considering both common and discriminative features, KMTPLS effectively fuses the information from multiple tensorial data sources and unifies the single and multiblock tensor regression scenarios into one general model. Moreover, in contrast to multilinear model, KMTPLS successfully addresses the nonlinear dependencies between multiple response and predictor tensor blocks by combining kernel machines with joint Tucker decomposition, resulting in a significant performance gain in terms of predictability. An efficient learning algorithm for KMTPLS based on sequentially extracting common and discriminative latent vectors is also presented. Finally, to show the effectiveness and advantages of our approach, we test it on the real-life regression task in computer vision, i.e., reconstruction of human pose from multiview video sequences.

Electroencephalogram (EEG) based brain-computer interface (BCI) has been proved to be an effective communication way between human brain and external devices. In order to effectively recover the cortical dynamics from the EEG signals and improve the classification performance, plenty of studies focused on constructing subject-specific spatial and spectral filters, achieving considerable improvement in classification accuracy. However, almost all the approaches aimed to find one common subspace for projection of all the samples in different classes. Studies have shown that active channels and frequency information were not only subject-dependent but also class-dependent. Thus the variety of class-dependent spatial and spectral characteristics can provide further discriminative information for classification. In this paper, we proposed a tensor-based method which attempted to seek individual spatial and spectral subspaces for each class by which each class was projected into its own subspace separately such that they were easily to be classified. Finally, we added a regularization term in this model to avoid overfitting. We evaluated the effectiveness and robustness of the proposed method on two different datasets including one widely-used benchmark EEG dataset collected from healthy subjects and one self-collected EEG dataset collected from stroke patients. The results demonstrated its superior performance.

CANDECOMP/PARAFAC (CP) tensor factorization of incomplete data is a powerful and useful data analysis technique. This method can achieve the purpose of tensor completion through explicitly capturing the multilinear latent factors. Recently, a CP factorization based on a hierarchical probabilistic model has been proposed which is used fully Bayesian theory by incorporating a sparsity-inducing prior over multiple latent factors and the appropriate hyper-priors over all hyper-parameters. In this way, the rank of tensor can be determined automatically instead of traditional manual assignment. This method has been applied into image inpainting and facial image synthesis effectively. However, there is no research on the application in EEG signal processing of this method. Moreover, the EEG data loss often occurs during experiment recording period. In this paper, we used this newer data analysis method for processing EEG data set from P300 experiment including data completion under different levels of data missing and classification analysis on the recovered data. The experiment result shows that this method has a good processing performance on incomplete EEG signal.

High accuracy of electroencephalogram (EEG) classification can hardly be achieved if the signals are contaminated by severe artefacts. One helpless way to avoid such artefacts is usually to directly discard the severely disturbed EEG segments. This study considers a more elegant way that tries to recover the disturbed segments from other undisturbed segments. The possible artefacts in EEG are treated as missing values. A Bayesian tensor factorization (BTF) based method is proposed to implement EEG completion for artefact removal. By specifying a sparsity-inducing hierarchical prior, the underlying low-rank tensor is discovered from incomplete EEG tensor with automatically inferred model parameters. The EEG missing values are effectively predicted with robustness to overfitting. Effectiveness of the BTF algorithm is demonstrated on EEG data recorded from seven subjects in a brain-computer interface paradigm based on event-related potentials.

Reducing noise in a video sequence is of vital important in many real-world applications. One popular method is block matching collaborative filtering. However, the main drawback of this method is that noise standard deviation for the whole video sequence is known in advance. In this paper, we present a tensor based denoising framework that considers 3D patches instead of 2D patches. By collecting the similar 3D patches non-locally, we employ the low-rank tensor decomposition for collaborative filtering. Since we specify the non-informative prior over the noise precision parameter, the noise variance can be inferred automatically from observed video data. Therefore, our method is more practical, which does not require knowing the noise variance. The experimental on video denoising demonstrates the effectiveness of our proposed method.

With the increasing availability of various sensor technologies, we now have access to large amounts of multiblock (also called multiset, multirelational, or multiview) data that need to be jointly analyzed to explore their latent connections. Various component analysis methods have played an increasingly important role for the analysis of such coupled data. In this article, we first provide a brief review of existing matrix-based (two-way) component analysis methods for the joint analysis of such data with a focus on biomedical applications. Then, we discuss their important extensions and generalization to multiblock multiway (tensor) data. We show how constrained multiblock tensor decomposition methods are able to extract similar or statistically dependent common features that are shared by all blocks, by incorporating the multiway nature of data. Special emphasis is given to the flexible common and individual feature analysis of multiblock data with the aim to simultaneously extract common and individual latent components with desired properties and types of diversity. Illustrative examples are given to demonstrate their effectiveness for biomedical data analysis.

Regularization has been one of the most popular approaches to prevent overfitting in electroencephalogram (EEG) classification of brain-computer interfaces (BCIs). The effectiveness of regularization is often highly dependent on the selection of regularization parameters that are typically determined by cross-validation (CV). However, the CV imposes two main limitations on BCIs: 1) a large amount of training data is required from the user and 2) it takes a relatively long time to calibrate the classifier. These limitations substantially deteriorate the system's practicability and may cause a user to be reluctant to use BCIs. In this paper, we introduce a sparse Bayesian method by exploiting Laplace priors, namely, SBLaplace, for EEG classification. A sparse discriminant vector is learned with a Laplace prior in a hierarchical fashion under a Bayesian evidence framework. All required model parameters are automatically estimated from training data without the need of CV. Extensive comparisons are carried out between the SBLaplace algorithm and several other competing methods based on two EEG data sets. The experimental results demonstrate that the SBLaplace algorithm achieves better overall performance than the competing algorithms for EEG classification.

We propose a generative model for robust tensor factorization in the presence of both missing data and outliers. The objective is to explicitly infer the underlying low-CANDECOMP/PARAFAC (CP)-rank tensor capturing the global information and a sparse tensor capturing the local information (also considered as outliers), thus providing the robust predictive distribution over missing entries. The low-CP-rank tensor is modeled by multilinear interactions between multiple latent factors on which the column sparsity is enforced by a hierarchical prior, while the sparse tensor is modeled by a hierarchical view of Student-t distribution that associates an individual hyperparameter with each element independently. For model learning, we develop an efficient variational inference under a fully Bayesian treatment, which can effectively prevent the overfitting problem and scales linearly with data size. In contrast to existing related works, our method can perform model selection automatically and implicitly without the need of tuning parameters. More specifically, it can discover the groundtruth of CP rank and automatically adapt the sparsity inducing priors to various types of outliers. In addition, the tradeoff between the low-rank approximation and the sparse representation can be optimized in the sense of maximum model evidence. The extensive experiments and comparisons with many state-of-the-art algorithms on both synthetic and real-world data sets demonstrate the superiorities of our method from several perspectives.

In multi-data learning, it is usually assumed that common latent factors exist among multi-datasets, but it may lead to deteriorated performance when datasets are heterogeneous and unbalanced. In this paper, we propose a novel common structure for multi-data learning. Instead of common latent factors, we assume that datasets share Common Adjacency Graph (CAG) structure, which is more robust to heterogeneity and unbalance of datasets. Furthermore, we utilize CAG structure to develop a new method for multi-tensor completion, which exploits the common structure in datasets to improve the completion performance. Numerical results demostrate that the proposed method not only outperforms state-of-the-art methods for video in-painting, but also can recover missing data well even in cases that conventional methods are not applicable.

Motor imagery-based brain-computer interfaces (BCIs) training has been proved to be an effective communication system between human brain and external devices. A practical problem in BCI-based systems is how to correctly and efficiently identify and extract subject-specific features from the blurred scalp electroencephalography (EEG) and translate those features into device commands in order to control external devices. In real BCI-based applications, we usually define frequency bands and channels configuration that related to brain activities beforehand. However, a steady configuration usually loses effects due to individual variability among different subjects in practical applications. In this study, a robust tensor-based method is proposed for a multiway discriminative subspace extraction from tensor-represented EEG data, which performs well in motor imagery EEG classification without the prior neurophysiologic knowledge like channels configuration and active frequency bands. Motor imagery EEG patterns in spatial-spectral-temporal domain are detected directly from the multidimensional EEG, which may provide insights to the underlying cortical activity patterns. Extensive experiment comparisons have been performed on a benchmark dataset from the famous BCI competition III as well as self-acquired data from healthy subjects and stroke patients. The experimental results demonstrate the superior performance of the proposed method over the contemporary methods.

CANDECOMP/PARAFAC (CP) tensor factorization of incomplete data is a powerful technique for tensor completion through explicitly capturing the multilinear latent factors. The existing CP algorithms require the tensor rank to be manually specified, however, the determination of tensor rank remains a challenging problem especially for CP rank . In addition, existing approaches do not take into account uncertainty information of latent factors, as well as missing entries. To address these issues, we formulate CP factorization using a hierarchical probabilistic model and employ a fully Bayesian treatment by incorporating a sparsity-inducing prior over multiple latent factors and the appropriate hyperpriors over all hyperparameters, resulting in automatic rank determination. To learn the model, we develop an efficient deterministic Bayesian inference algorithm, which scales linearly with data size. Our method is characterized as a tuning parameter-free approach, which can effectively infer underlying multilinear factors with a low-rank constraint, while also providing predictive distributions over missing entries. Extensive simulations on synthetic data illustrate the intrinsic capability of our method to recover the ground-truth of CP rank and prevent the overfitting problem, even when a large amount of entries are missing. Moreover, the results from real-world applications, including image inpainting and facial image synthesis, demonstrate that our method outperforms state-of-the-art approaches for both tensor factorization and tensor completion in terms of predictive performance.

Nonnegative Tucker decomposition (NTD) is a powerful tool for the extraction of nonnegative parts-based and physically meaningful latent components from high-dimensional tensor data while preserving the natural multilinear structure of data. However, as the data tensor often has multiple modes and is large scale, the existing NTD algorithms suffer from a very high computational complexity in terms of both storage and computation time, which has been one major obstacle for practical applications of NTD. To overcome these disadvantages, we show how low (multilinear) rank approximation (LRA) of tensors is able to significantly simplify the computation of the gradients of the cost function, upon which a family of efficient first-order NTD algorithms are developed. Besides dramatically reducing the storage complexity and running time, the new algorithms are quite flexible and robust to noise, because any well-established LRA approaches can be applied. We also show how nonnegativity incorporating sparsity substantially improves the uniqueness property and partially alleviates the curse of dimensionality of the Tucker decompositions. Simulation results on synthetic and real-world data justify the validity and high efficiency of the proposed NTD algorithms.

Classification of multichannel electroencephalogram (EEG) recordings during motor imagination has been exploited successfully for brain-computer interfaces (BCI). Frequency bands and channels configuration that relate to brain activities associated with BCI tasks are often pre-decided as default in EEG analysis without deliberations. However, a steady configuration usually loses effects due to individual variability across different subjects in practical applications. In this paper, we propose an adaptive boosting algorithm in a unifying theoretical framework to model the usually predetermined spatial-spectral configurations into variable preconditions, and further introduce a novel heuristic of stochastic gradient boost for training base learners under these preconditions. We evaluate the effectiveness and robustness of our proposed algorithm based on two data sets recorded from diverse populations including the healthy people and stroke patients. The results demonstrate its superior performance.

We present a novel framework for tensor valued Gaussian processes (GP) regression, which exploits a covariance function defined on tensor representation of data inputs. In this way, we bring together the powerful GP methods supported by Bayesian inference and higher-order tensor analysis techniques into one framework. This enables us to account for the underlying structure of data within the model, providing a powerful framework for structural data analysis, such as 3D video sequences. To this end, we propose a new kernel function with tensor arguments under the assumption of generative models, in the form of product kernels where a symmetrical Kullback-Leibler divergence measure is exploited to define the covariance function for tensorial data. A fully Bayesian treatment is employed to estimate the hyperparameters and infer the predictive distributions. Simulation results on both the synthetic data and a real world application of estimating the crowd size from 3D videos demonstrate the effectiveness of the proposed framework.

Non-negative tensor factorization (NTF) has been successfully applied to analyze event-related potentials (ERPs), and shown superiority in terms of capturing multi-domain features. However, the time-frequency representation of ERPs by higher-order tensors are usually large-scale, which prevents the popularity of most tensor factorization algorithms. To overcome this issue, we introduce a non-negative canonical polyadic decomposition (NCPD) based on low-rank approximation (LRA) and hierarchical alternating least square (HALS) techniques. We applied NCPD (LRAHALS and benchmark HALS) and CPD to extract multi-domain features of a visual ERP. The features and components extracted by LRAHALS NCPD and HALS NCPD were very similar, but LRAHALS NCPD was 70 times faster than HALS NCPD. Moreover, the desired multi-domain feature of the ERP by NCPD showed a significant group difference (control versus depressed participants) and a difference in emotion processing (fearful versus happy faces). This was more satisfactory than that by CPD, which revealed only a group difference.

Two main issues for event-related potential (ERP) classification in brain-computer interface (BCI) application are curse-of-dimensionality and bias-variance tradeoff, which may deteriorate classification performance, especially with insufficient training samples resulted from limited calibration time. This study introduces an aggregation of sparse linear discriminant analyses (ASLDA) to overcome these problems. In the ASLDA, multiple sparse discriminant vectors are learned from differently l1-regularized least-squares regressions by exploiting the equivalence between LDA and least-squares regression, and are subsequently aggregated to form an ensemble classifier, which could not only implement automatic feature selection for dimensionality reduction to alleviate curse-of-dimensionality, but also decrease the variance to improve generalization capacity for new test samples. Extensive investigation and comparison are carried out among the ASLDA, the ordinary LDA and other competing ERP classification algorithms, based on different three ERP datasets. Experimental results indicate that the ASLDA yields better overall performance for single-trial ERP classification when insufficient training samples are available. This suggests the proposed ASLDA is promising for ERP classification in small sample size scenario to improve the practicability of BCI.

As tensors provide a natural and efficient representation of multidimensional structured data, in this paper, we consider probabilistic multinomial probit classification for tensor-variate inputs with Gaussian processes (GP) priors placed over the latent function. In order to take into account the underlying multimodes structure information within the model, we propose a framework of probabilistic product kernels for tensorial data based on a generative model assumption. More specifically, it can be interpreted as mapping tensors to probability density function space and measuring similarity by an information divergence. Since tensor kernels enable us to model input tensor observations, the proposed tensor-variate GP is considered as both a generative and discriminative model. Furthermore, a fully variational Bayesian treatment for multiclass GP classification with multinomial probit likelihood is employed to estimate the hyperparameters and infer the predictive distributions. Simulation results on both synthetic data and a real world application of human action recognition in videos demonstrate the effectiveness and advantages of the proposed approach for classification of multiway tensor data, especially in the case that the underlying structure information among multimodes is discriminative for the classification task.

We present a new supervised tensor regression method based on multi-way array decompositions and kernel machines. The main issue in the development of a kernel-based framework for tensorial data is that the kernel functions have to be defined on tensor-valued input, which here is defined based on multi-mode product kernels and probabilistic generative models. This strategy enables taking into account the underlying multilinear structure during the learning process. Based on the defined kernels for tensorial data, we develop a kernel-based tensor partial least squares approach for regression. The effectiveness of our method is demonstrated by a real-world application, i.e., the reconstruction of 3D movement trajectories from electrocorticography signals recorded from a monkey brain.

Non-negative Canonical Polyadic decomposition (NCPD) and non-negative Tucker decomposition (NTD) were compared for extracting the multi-domain feature of visual mismatch negativity (vMMN), a small event-related potential (ERP), for the cognitive research. Since signal-to-noise ratio in vMMN is low, NTD outperformed NCPD. Moreover, we proposed an approach to select the multi-domain feature of an ERP among all extracted features and discussed determination of numbers of extracted components in NCPD and NTD regarding the ERP context.

Integration of brain-computer interface (BCI) technique and assistive device is one of chief and promising applications of BCI system. With BCI technique, people with disabilities do not have to communicate with external environment through traditional and natural pathways like peripheral nerves and muscles, and could achieve it only by their brain activities. In this paper, we designed an electroencephalogram (EEG)-based wheelchair which can be steered by users' own thoughts without any other involvements. We evaluated the feasibility of BCI-based wheelchair in terms of accuracies and real-world testing. The results demonstrate that our BCI wheelchair is of good performance not only in accuracy, but also in practical running testing in a real environment. This fact implies that people can steer wheelchair only by their thoughts, and may have a potential perspective in daily application for disabled people.

A new generalized multilinear regression model, termed the higher order partial least squares (HOPLS), is introduced with the aim to predict a tensor (multiway array) Y from a tensor X through projecting the data onto the latent space and performing regression on the corresponding latent variables. HOPLS differs substantially from other regression models in that it explains the data by a sum of orthogonal Tucker tensors, while the number of orthogonal loadings serves as a parameter to control model complexity and prevent overfitting. The low-dimensional latent space is optimized sequentially via a deflation operation, yielding the best joint subspace approximation for both X and Y. Instead of decomposing X and Y individually, higher order singular value decomposition on a newly defined generalized cross-covariance tensor is employed to optimize the orthogonal loadings. A systematic comparison on both synthetic data and real-world decoding of 3D movement trajectories from electrocorticogram signals demonstrate the advantages of HOPLS over the existing methods in terms of better predictive ability, suitability to handle small sample sizes, and robustness to noise.

Through exploiting temporal, spectral, time-frequency representations, and spatial properties of mismatch negativity (MMN) simultaneously, this study extracts a multi-domain feature of MMN mainly using non-negative tensor factorization. In our experiment, the peak amplitude of MMN between children with reading disability and children with attention deficit was not significantly different, whereas the new feature of MMN significantly discriminated the two groups of children. This is because the feature was derived from multi-domain information with significant reduction of the heterogeneous effect of datasets.

A multilinear subspace regression model based on so called latent variable decomposition is introduced. Unlike standard regression methods which typically employ matrix (2D) data representations followed by vector subspace transformations, the proposed approach uses tensor subspace transformations to model common latent variables across both the independent and dependent data. The proposed approach aims to maximize the correlation between the so derived latent variables and is shown to be suitable for the prediction of multidimensional dependent data from multidimensional independent data, where for the estimation of the latent variables we introduce an algorithm based on Multilinear Singular Value Decomposition (MSVD) on a specially defined cross-covariance tensor. It is next shown that in this way we are also able to unify the existing Partial Least Squares (PLS) and N-way PLS regression algorithms within the same framework. Simulations on benchmark synthetic data confirm the advantages of the proposed approach, in terms of its predictive ability and robustness, especially for small sample sizes. The potential of the proposed technique is further illustrated on a real world task of the decoding of human intracranial electrocorticogram (ECoG) from a simultaneously recorded scalp electroencephalograph (EEG).

The Common Spatial Patterns (CSP) algorithm has been widely used in EEG classification and Brain Computer Interface (BCI). In this paper, we propose a multilinear formulation of the CSP, termed as TensorCSP or Common Tensor Discriminant Analysis (CTDA) for high-order tensor data. As a natural extension of CSP, the proposed algorithm uses the analogous optimization criteria in CSP and a new framework for simultaneous optimization of projection matrices on each mode based on tensor analysis theory is developed. Experimental results demonstrate that our proposed algorithm is able to improve classification accuracy of multi-class motor imagery EEG.

A major challenge in applying machine learning methods to Brain-Computer Interfaces (BCIs) is to overcome the on-line non-stationarity of the data blocks. An effective BCI system should be adaptive to and robust against the dynamic variations in brain signals. One solution to it is to adapt the model parameters of BCI system online. However, CSP is poor at adaptability since it is a batch type algorithm. To overcome this, in this paper, we propose the Incremental Common Spatial Pattern (ICSP) algorithm which performs the adaptive feature extraction on-line. This method allows us to perform the online adjustment of spatial filter. This procedure helps the BCI system robust to possible non-stationarity of the EEG data. We test our method to data from BCI motor imagery experiments, and the results demonstrate the good performance of adaptation of the proposed algorithm.