Modern neural networks (NNs), while effective at learning representations from given samples and handling downstream pattern recognition tasks, typically contain tens to hundreds of millions of parameters. The growth in NN size motivates ongoing research on effective network compression with the purpose of reducing the computational burden without significantly sacrificing the model performance. It is especially critical when deploying NNs on resource-constrained devices where the computation and storage efficiency are highly concerned. A promising and currently popular solution to model compression is to replace the NN weight matrix with its low-rank tensor approximation, i.e., implementing an efficient latent factorization of tensors (LFT) process on the NNs parameters. Based on thorough investigations into the state-of-the-art LFT-based model compression methods, this survey 1) provides a comprehensive review on the latest research progress of LFT-based model compression methods on various NNs (e.g., Convolutional NNs, Recurrent NNs, and Transformers); 2) summarizes a number of widely-used LFT toolboxes; 3) evaluates LFT methods for model compression on a variety of main-stream NN backbones; and 4) discusses the development trend of LFT-based model compression technique. This survey aims to provide a systematic and comprehensive survey concerning the LFT-based model compression methods to artificial intelligence researchers and engineers, thereby promoting further research development in this crucial field.

The widespread application of cloud services make users pay more attention to Quality of Service (QoS). Generally, the user cannot call all services simultaneously to obtain corresponding QoS data and can only choose a service from a few known data, thus it’s critical to predict unknown QoS values. A third-order tensor can model temporal patterns of QoS data, and studies indicate that the tensor latent factor analysis models based on Canonical Polyadic (CP) decomposition can effectively capture temporal patterns to predict unknown data in QoS. However, the existing CP decomposition-based models limit their learning ability since rank-one tensors contain less structure information, which results in low prediction accuracy. Therefore, this paper proposes a Biased Block Term Tensor Decomposition (BBTTD) model to achieve high accuracy for temporal pattern-aware QoS prediction. It mainly adopts the following three-fold ideas: (a) implementing a tensor learning model by adopting the block term decomposition in rank-([Formula: see text], [Formula: see text], 1) terms; (b) proposing the bias block term tensors to enhance the model’s prediction accuracy; (c) designing a nonnegative multiplication update algorithm to learning model parameters. Extensive experiments on two public dynamic QoS datasets demonstrate that BBTTD has higher prediction accuracy compared with several QoS prediction models.

Spatio-temporal data like a Dynamically Weighted Directed Network (DWDN) are ubiquitous in bigdata applications like an intelligent recommender system. They commonly illustrate the complex yet dynamic interactions between tremendous nodes, as well as contain rich knowledge regarding the involved nodes’ behavioral patterns with time dynamics. On the other hand, since the nodes constantly increase as the time accumulates, a DWDN becomes High-Dimensional and Incomplete (HDI) due to the limited interactions, widely-spread time slots and huge node count. Moreover, the inner temporal-spatio patterns exhibit strong nonlinearity, making it very difficult to grasp them from HDI data. To address this critical issue, this paper proposes a novel Tensor Causal Convolution Network (TCCN) model with three-fold ideas: a) innovatively building a feedforward tensor neural network with the incorporation of the Latent Factorization of Tensors (LFT) principle to model complex nonlinear interactions in an HDI DWDN efficiently; b) establishing a Tensor Causal Convolution (TCC) structure, which is able to accurately fuse the time-varying information from node interactions with high scalability; and c) developing a neighborhood regularization scheme to boost the local structural representation, thus capturing spatial dependencies among tremendous nodes. Extensively experimental results on ten real DWDNs from real bigdata applications evidently demonstrate that our proposed TCCN model outperforms several state-of-the-art models in both representation learning accuracy and convergence ability. It provides a highly-efficient representation learning approach for diverse bigdata applications.

Cryptocurrency services, as one of the most successful applications of blockchain technology, have recently garnered significant attention from the graph learning community. Its large-scale dynamic transaction records contain a variety of behavioral patterns and rich knowledge involving accounts, making the dynamic cryptocurrency transaction network embedding (DCTNE) a hot, yet thorny research topic. As the trading accounts increase and time accumulates, considerable transaction services are dispersed into various time slots, leading to very sparse transaction data within a time slot, that is, the transaction service data is high-dimensional and incomplete (HDI). To efficiently mine high-value knowledge from HDI data, this article proposes a proximal-ADMM-incorporated nonnegative latent-factorization-of-tensors (PNL) model for DCTNE that adopts threefold ideas: 1) incorporating the proximal terms into the alternating-direction-method-of-multipliers (ADMMs)-based learning scheme to reduce the oscillations for high estimation accuracy and fast convergence; 2) implementing a parallel training process with hyperparameter self-adaptation for high computational efficiency; and 3) proving that the proximal-incorporated learning scheme guarantees the convergence to a Karush–Kuhn–Tucker (KKT) stationary point. Experimental results on eight real-world DCTNs show that the PNL significantly outperforms several state-of-the-art (SOTA) models, demonstrating not only high efficiency and accuracy in performing DCTNE, but also strong potential to enhance the operational reliability and stability of cryptocurrency transaction systems.

A dynamic directed graph (DDG) can describe complex dynamic interactions among massive entities, for example, traffic transmissions in a metropolitan area network (MAN), in a natural way. Due to the rapid expansion of a network, it is impossible to capture all the interactions at each time slot, making a resultant DDG be high-dimensional and incomplete (HDI). A nonnegative latent factorization of tensors (NLFT) model has proven to be highly efficient in extracting desired knowledge from an HDI DDG. Nevertheless, an existing NLFT model attempts to be easily affected by the instantaneous data fluctuations. Motivated by this discovery, this article innovatively proposes a convolution bias-incorporated NLFT (CB-NLFT) model with threefold ideas: 1) utilizing the Tucker decomposition framework for accurately representing the complex patterns hidden in an HDI DDG; 2) establishing a novel convolution bias scheme for precisely depicting the instantaneous data fluctuations; and 3) theoretically proving the CB-NLFT model’s convergence. Extensively empirical studies on six real-world datasets illustrate that the proposed CB-NLFT achieves significantly higher accuracy and computational efficiency when addressing the representation learning to a DDG in comparison with state-of-the-art models.

A Dynamically Weighted Directed Network (DWDN) fundamentally illustrates the complex interactions among massive nodes from a big-data-oriented application, like the dynamic interactions among numerous terminals in a metropolitan network management system (MNMS). A High-Dimensional and Incomplete (HDI) tensor is able to flexibly quantize it, where lots of entries are missing primarily due to the impossibility in discovering the full interactions among numerous nodes. Such an HDI tensor can be effectively represented by a Latent Factorization of Tensors (LFT) model for extracting useful knowledge like potential links from it, while existing LFT models commonly adopt general regularization schemes without considering an HDI tensor's imbalanced known data, which impairs their generality. To address this issue, this paper develops an Fine-grained Regularized Nonnegative Latent factorization of tensors (FRNL) model based on two-fold ideas: a) innovatively proposing an Swish-p-based and fine-grained regularization scheme where the regularization effects acting on individual latent feature is proportional to its related instance count for precisely representing the imbalanced distribution of an HDI tensor's known data; b) implementing the self-adaptation of the model hyper-parameters via a fuzzy controller to achieve high practicability. The convergence ability of FRNL is justified theoretically. Experimental studies on eight DWDNs emerging from a real MNMS illustrate that compared with state-of-the-art LFT models, the proposed FRNL model obtains significantly higher learning accuracy and computational efficiency in representing a DWDN.

Complete and valid spatio-temporal traffic data play a vital role in intelligent transportation systems applications, such as congestion avoidance and route guidance. However, traffic data from real-world scenarios is usually incomplete or corrupted due to communication or sensors malfunctions, which makes the traffic analytics difficult. Since traffic data contains complex spatio-temporal patterns, it is very challenging to develop an efficient learning model that can accurately recover incomplete traffic data. To tackle this issue, this work propose a Tucker Decomposition-based Latent factorization of tensors (TDL) model with two interesting ideas: 1) modeling spatio-temporal traffic data as an incomplete third-order tensor and building a Tucker decomposition based learning objective according to the density-oriented principle for precisely recovering missing traffic data; and 2) adopting a proportional-integral-derivative (PID) control principle-incorporated parameters learning scheme for achieving high computational efficiency. Empirical studies on four traffic speed datasets generated from different cities demonstration that the proposed TDL model achieves significant performance gain in both accuracy and computational efficiency compared with state-of-the-art models.

A High-Dimensional and Incomplete (HDI) tensor is frequently encountered in a big data-related application concerning the complex dynamic interactions among numerous entities. Traditional tensor factorization-based models cannot handle an HDI tensor efficiently, while existing latent factorization of tensors models are all linear models unable to model an HDI tensor's nonlinearity. Motivated by this critical discovery, this paper proposes a Neural Latent Factorization of Tensors model, which provides a novel approach to nonlinear Canonical Polyadic decomposition on an HDI tensor. It is implemented with three-fold interesting ideas: a) adopting the density-oriented modeling principle to build rank-one tensor series with high computational efficiency and affordable storage cost; b) treating each rank-one tensor as a hidden neuron to achieve an efficient neural network structure; and c) developing an adaptive backward propagation (ABP) learning scheme for efficient model training. Experimental results on six HDI tensors from a real system demonstrate that compared with state-of-the-art models, the proposed model achieves significant performance gain in both convergence rate and accuracy. Hence, it is of great significance in performing challenging HDI tensor analysis.

A dynamically weighted directed network (DWDN) is frequently encountered in various big data-related applications like a terminal interaction pattern analysis system (TIPAS) concerned in this study. It consists of large-scale dynamic interactions among numerous nodes. As the involved nodes increase drastically, it becomes impossible to observe their full interactions at each time slot, making a resultant DWDN High Dimensional and Incomplete (HDI). An HDI DWDN, in spite of its incompleteness, contains rich knowledge regarding involved nodes various behavior patterns. To extract such knowledge from an HDI DWDN, this paper proposes a novel Alternating direction method of multipliers (ADMM)-based Nonnegative Latent-factorization of Tensors (ANLT) model. It adopts three-fold ideas: a) building a data density-oriented augmented Lagrangian function for efficiently handling an HDI tensors incompleteness and nonnegativity; b) splitting the optimization task in each iteration into an elaborately designed subtask series where each one is solved based on the previously solved ones following the ADMM principle to achieve fast convergence; and c) theoretically proving that its convergence is guaranteed with its efficient learning scheme. Experimental results on six DWDNs from real applications demonstrate that the proposed ANLT outperforms state-of-the-art models significantly in both computational efficiency and prediction accuracy.

Dynamic relationships are frequently encountered in big data and services computing-related applications, like dynamic data of user-side QoS in Web services. They are modeled into a high-dimensional and sparse (HiDS) tensor, which contain rich knowledge regarding temporal patterns. A non-negative latent factorization of tensors (NLFT) model is very effective in extracting such patterns from an HiDS tensor. However, it commonly suffers from overfitting with improper regularization schemes. To address this issue, this article investigates NLFT models with diversified regularization schemes. Six regularized NLFT models, i.e.,$L_{2}, L_{1}$, elastic net, log, dropout, and swish-regularized ones, are proposed and carefully investigated. Moreover, owing to their diversified regularization designs, they possess strong model diversity to achieve an effective ensemble. Empirical studies on HiDS QoS tensors from real applications demonstrate that compared with state-of-the-art models, the proposed ones better describe the temporal patterns hidden in an HiDS tensor, thereby achieving significantly higher prediction accuracy for missing data. Moreover, their ensemble further outperforms each of them in terms of prediction accuracy for missing QoS data.

Temporally dynamic QoS data are commonly encountered in large-scale cloud services environments. They can be quantized into a high-dimensional and incomplete (HDI) tensor defined on the user×service×time. Despite its HDI nature, it contains various temporal patterns highly helpful in representing involved users and services. A latent factorization of tensors (LFT) model is able to discover such patterns from an HDI tensor, while its model generality cannot be ensured due to the complex structure and incomplete data of an HDI tensor. To address this issues, this paper proposes an Instance-frequency-weighted regularized, Nonnegative and Adaptive LFT (INAL) model with three-fold ideas: a) adopting the principle of data density-oriented modeling to reduce the computation and storage complexity; b) refining the regularization effects on each latent factor with its relevant instance-frequency for illustrating the imbalanced distribution of known data in an HDI tensor; and c) making its hyper-parameter self-adaptive via incorporating the principle of a particle swarm optimization (PSO) algorithm into the training process, thereby achieving a highly adaptive and practical model. Empirical studies on two dynamic QoS datasets from real applications demonstrate that compared with state-of-the-art models, the proposed model achieves significant gain in prediction accuracy for unobserved dynamic QoS data and achieves highly competitive computational efficiency.

A big-data-related application like a Terminal Interaction Pattern Analysis System (TIPAS) commonly involves massive entities interacting with each other dynamically. Such interactions can be represented by a Dynamically Weighted Directed Network (DWDN). A large number of involved entities results in a high-dimensional and incomplete (HDI) network, which can be represented by an HDI tensor with numerous missing entries. In spite of its HDI nature, this tensor contains much useful knowledge regarding various desired patterns like unobserved links in DWDN. However, due to its extremely high dimensionality and low data density, it is very challenging to build a learning model that can precisely represent an HDI tensor. To address this issue, this work proposes a Neural Latent Factorization of Tensors (NeuLFoT) model with three interesting ideas: a) adopting the principle of density-oriented modeling and Canonical Polyadic tensor factorization to build rank-one tensor series relying on three-dimensional latent factors for precisely representing an HDI tensor’s known data; b) treating the obtained rank-one tensors as neurons to form a novel neural tensor network model; and c) proposing a novel Backward Propagation algorithm for Latent factorization of tensors (BPL) to ensure high training efficiency. Experimental results on two large-scale DWDNs generated by a real TIPAS demonstrate that compared with state-of-the-art models, the proposed model achieves significant gain in prediction accuracy for missing links of a DWDN and achieves highly competitive computational efficiency.

A nonnegative latent factorization of tensors (NLFT) model precisely represents the temporal patterns hidden in multichannel data emerging from various applications. It often adopts a single latent factor-dependent, nonnegative and multiplicative update on tensor (SLF-NMUT) algorithm. However, learning depth in this algorithm is not adjustable, resulting in frequent training fluctuation or poor model convergence caused by overshooting. To address this issue, this study carefully investigates the connections between the performance of an NLFT model and its learning depth via SLF-NMUT to present a joint learning-depth-adjusting scheme for it. Based on this scheme, a Depth-adjusted Multiplicative Update on tensor algorithm is innovatively proposed, thereby achieving a novel depth-adjusted nonnegative latent-factorization-of-tensors (DNL) model. Empirical studies on two industrial data sets demonstrate that compared with the state-of-the-art NLFT models, a DNL model achieves significant accuracy gain when performing missing data estimation on a high-dimensional and incomplete tensor with high efficiency.Note to Practitioners—Multichannel data are often encountered in various big-data-related applications. It is vital for a data analyzer to correctly capture the temporal patterns hidden in them for efficient knowledge acquisition and representation. This article focuses on analyzing temporal QoS data, which is a representative kind of multichannel data. To correctly extract their temporal patterns, an analyzer should correctly describe their nonnegativity. Such a purpose can be achieved by building a nonnegative latent factorization of tensors (NLFT) model relying on a single latent factor-dependent, nonnegative and multiplicative update on tensor (SLF-NMUT) algorithm. But its learning depth is not adjustable, making an NLFT model frequently suffer from severe fluctuations in its training error or even fail to converge. To address this issue, this study carefully investigates the learning rules for an NLFT model’s decision parameters using an SLF-NMUT and proposes a joint learning-depth-adjusting scheme. This scheme manipulates the multiplicative terms in SLF-NMUT-based learning rules linearly and exponentially, thereby making the learning depth adjustable. Based on it, this study builds a novel depth-adjusted nonnegative latent-factorization-of-tensors (DNL) model. Compared with the existing NLFT models, a DNL model better represents multichannel data. It meets industrial needs well and can be used to achieve high performance in data analysis tasks like temporal-aware missing data estimation

Quality-of-service (QoS) data vary over time, making it vital to capture the temporal patterns hidden in such dynamic data for predicting missing ones with high accuracy. However, currently latent factor (LF) analysis-based QoS-predictors are mostly defined on static QoS data without the consideration of such temporal dynamics. To address this issue, this paper presents a biased non-negative latent factorization of tensors (BNLFTs) model for temporal pattern-aware QoS prediction. Its main idea is fourfold: 1) incorporating linear biases into the model for describing QoS fluctuations; 2) constraining the model to be non-negative for describing QoS non-negativity; 3) deducing a single LF-dependent, non-negative, and multiplicative update scheme for training the model; and 4) incorporating an alternating direction method into the model for faster convergence. The empirical studies on two dynamic QoS datasets from real applications show that compared with the state-of-the-art QoS-predictors, BNLFT represents temporal patterns more precisely with high computational efficiency, thereby achieving the most accurate predictions for missing QoS data.