Transformer-based text-image matching model, known as CLIP, has garnered significant attention owing to its exceptional performance in text-image retrieval tasks and downstream applications. However, the interpret-ability of CLIP remains underexplored. Existing interpretation methods for Transformers often struggle with incomplete and unreliable attributions within the image and text modalities, respectively. In this paper, we propose a fine-grained interpretation method, termed Class Activation Values (CAV), to provide lucid and faithful visual explanations for CLIP-based text-image retrievals. Specifically, we systematically perform multi-scale accumulation and fusion of class-specific gradients and activation value features to generate high-definition explanations for the image encoder. Furthermore, we present element-wise gradient-based weights to attribute fine-grained relevance between value features and output similarity within the text encoder. The proposed CAV is capable of simultaneously rendering detailed and credible explanations due to its precise feature attribution. Extensive qualitative and quantitative experiments are conducted on the ImageNet-1k and MS COCO datasets, and the experimental results demonstrate that CAV outperforms state-of-the-art interpretation methods in both faithfulness and localization assessments across image and text modalities.

Account shuffling is a crucial method to solve the problems of high cross-shard transaction (TX) ratio and load imbalance in sharded blockchain. However, most existing methods primarily focus on account partitioning, with insufficient attention to account migration, resulting in limited improvements in throughput and latency. Therefore, we propose an efficient account shuffling mechanism based on lightweight migration in sharded blockchain (ASLM-Shard). Specifically, we first propose a migration-aware label propagation algorithm (MA-LPA) to improve the effect of account partitioning by balancing the relationship among account migration overhead, cross-shard TX ratio and load imbalance. Then, we adopt a sparse Merkle tree (SMT) to store account states to support flexible state verification, and propose a transaction-aware lightweight account migration (TLAM) method that leverages a “Lock-Mint” strategy to minimize migration costs while ensuring security. Extensive experimental results show that, compared with the SOTA baseline, ASLM-Shard improves system throughput by up to 27.8% and reduces TX latency by up to 81.1% when the account partitioning strategy is fixed, it also achieves up to 17.9% higher throughput and 23.4% lower latency when the migration method is fixed.

With the explosion of social media platforms, a substantial amount of data is generated from social information network. Tensor-based multi-modal clustering methods have been widely applied in various scenarios of social information network by mining potential correlative relationships from large-scale heterogeneous data. Nevertheless, the accuracy and efficiency of tensor-based multi-modal clustering methods are seriously restricted by noise data and the curse of dimensionality. Therefore, this paper presents a Tucker-based multi-modal clustering (TuMC) and an improved TuMC (ITuMC) to enhance the accuracy and efficiency of multi-modal clustering. First, we propose two Tucker-based attribute weight ranking learning approaches to calculate weight tensor efficiently. Then, we present a calculation approach for Tucker-based selective weighted tensor distance (SWTD) and a TuMC method. Meanwhile, an ITuMC method is explored by optimizing the calculation efficiency of the SWTD to further improve clustering speed. Finally, we present a Tucker-based multi-modal clustering and service framework for social information network. Extensive experimental results based on social Geolife GPS trajectory and electricity consumption datasets demonstrate that the TuMC and ITuMC methods can cluster multi-source heterogeneous data with both higher accuracy and efficiency under complex social information network by DVI, AR and execution time measurement.

Graph contrastive learning (GCL) has gained significant attention in the self-supervised learning field, which aims to maximize the mutual information of the same sample on different augmented views. How to balance the efficiency and performance of GCL model is a challenging issue. In existing GCL models, the performance is commonly limited by manual graph augmentation. Meanwhile, selecting substantial negative samples results in increased execution time. Therefore, this paper focuses on proposing a Multi-view Negative-Free Contrastive Learning on Adaptive graph augmentation (MNFCLA) method. First, we innovatively combine attention-based adaptive graph augmentation with non-negative sampling to obtain reliable augmented views and improve the computational efficiency of the model. Moreover, the GCL methods typically employ two augmented views, leading to model performance decrease. We design a multi-view Barlow Twins to calculate the cross correlation among multiple view embeddings, improving the model’s performance and robustness. Extensive experimental results on five node classification tasks demonstrate that the proposed MNFCLA model is superior to other baselines in computational efficiency and classification accuracy.

With advancements of diffusion models in generating high-quality, diverse, and creative content, subject-driven image editing based on diffusion models has been growing attention. However, it is challenging to perform precise control in image editing as minor changes in the input can lead to significant differences in generated concepts and content. Prevalent methods face significant limitations in terms of large numbers of trainable parameters and inference latency. Hence, we propose an efficient tensor-based fine-tuning method, Tucker/TT-based Fine-Tuning (TuckerFT and TTFT), for subject-driven image editing on diffusion models. Specifically, we first provide Tucker/TT-based Fine-tuning Convolutional Adapter and Attention Adapter to achieve parameter-efficient fine-tuning in convolutional and attention layers of diffusion models. Additionally, we further present a more sound initializing strategy for the Tucker/TT-based Adapters to guarantee steady training. Extensive experimental results demonstrate that TuckerFT/TTFT surpasses state-of-the-art methods on subject similarity, trainable parameters, and training duration.

As the visual interpretations for convolutional neural networks (CNNs), backpropagation attribution methods have been garnering growing attention. Nevertheless, majority of these methods merely concentrate on the ultimate convolutional layer, leading to tiny and concentrated interpretations that fail to adequately clarify the model-central attention. Therefore, we propose a precise attribution method (i.e., Holistic-CAM) for high-definition visual interpretation in the holistic stage of CNNs. Specifically, we first present weighted positive gradients to guarantee the sanity of interpretations in shallow layers and leverage multi-scale fusion to improve the resolution across the holistic stage. Then, we further propose a denoising strategy based on the fundamental scale component to eliminate the faithless attribution derived from fusing larger-scale features. The proposed method is capable of simultaneously rendering fine-grained and faithful interpretations for CNNs from shallow to deep layers. Extensive experimental results demonstrate that Holistic-CAM outperforms state-of-the-art methods on common-used benchmarks, including deletion and insertion, energy-based point game as well as Remove and Debias on ImageNet-1k, it also passes the sanity check easily.

Transportation big data generated from various Internet of Things devices have the feature of muti-source and heterogeneous. To efficiently represent and analyze these ubiquitous transportation big data, tensor and tensor-based data analysis methods have been widely adopted in recent years. As a tensor-based machine learning method, high-order dominant Z-eigen decomposition (HODZED) in multivariate multi-order Markov model is suitable for multi-modal transportation prediction. However, massive transportation data are usually generated in a streaming way and the transportation system requires frequent updates. To avoid recalculating the history data and provide immediate prediction, we propose a tensor train (TT) based incremental HODZED (TT-IHODZED) method. Concretely, we first present an incremental HODZED (IHODZED) method to update the dominant Z-eigentensor in multivariate multi-order Markov model. Then, TT-based tensor operations are adopted to IHODZED to speed up calculations, especially the repeated Einstein products. Furthermore, to solve the TT-based high-order linear equations in TT-IHODZED method, we also propose a TT-based biconjugate gradient stabilized (TT-HOBiCGS) algorithm. Experimental results based on real-world and synthetic datasets show that, compared to HODZED method, TT-IHODZED significantly improves computation efficiency up to 10 times while keeping the same or even better prediction accuracy.

In the era of big data, data security poses a paramount challenge in dealing with large-scale data in cloud computing environments. Traditional encryption methods such as RSA and Paillier face the problems that, as the data volume increases, the encryption consumes too much computation resource and the storage size of ciphertext is too large. To tackle these problems, this paper explores secure and lightweight tensor train (TT) based encryption methods for large-scale data in federated cloud environments. First, we develop a novel encryption method called Orthogonalized TT Encryption (OTT-EPT), which can encrypt large-scale plaintext data into a list of small-size ciphertext (TT cores) and store them in different clouds. Second, to tackle the unique characteristics of TT decomposition that adjacent TT cores have the same rank, which could be exploited by attackers to threaten encrypted data, we further propose a Noise-Added Orthogonalized TT Encryption (NOTT-EPT) method. NOTT-EPT can change arbitrary TT-rank by adding noise TT-cores on specific order. Furthermore, we present a TT-based encryption framework and a suite of secure homomorphic operation protocols for large-scale data computation in federated cloud environments. The experimental results indicate that, compared to state-of-the-art methods, our proposed methods not only ensure the integrity and security of encrypted data but also reduce both time and space consumption by about 1000 times.

Combining data-driven approaches and physical laws for industrial trajectory prediction can improve the performance of industrial applications such as path planning of robots and route selection of vehicles in transportation systems. In the era of industrial Big Data, the application of industrial trajectory prediction based on the physical Markov process and tensor model has attracted much attention. To synchronously improve the prediction accuracy and computational efficiency, this article proposes a tensor-train (TT)-based multiuser multivariate multiorder (3M) physical Markov prediction approach for multimodal industrial trajectory pattern mining. First, we propose a TT-based unified product calculation rule with its scalable computation approach based on decomposed TT cores to speed up the execution efficiency. Then, a TT-based 3M (TT-3M) Markov transition approach is presented. Furthermore, we put forward a TT-based power method to calculate the stationary joint eigentensor (SJE) and an SJE-based multimodal prediction algorithm to mine concealed trajectory patterns. Experimental results based on real GPS trajectory dataset show that compared with the original tensor-based 3M approach, the TT-3M approach can improve the computational efficiency up to three times and reduce the storage space proportion to a minimum of 1‰ while ensuring basically consistent prediction accuracy.

Predicting the future traffic flows by applying deep learning methods has become an alternative way for transportation network management. Combining recurrent neural networks (RNNs) with tensor to implement accurate predictions has drawn intensive attention. However, traditional RNNs cannot deal with the high-order traffic flow data and capture their inherent structural relationship to provide accurate multi-modal prediction services. Therefore, this article focuses on proposing a series of tensor-based RNNs (T-RNNs) and a T-RNNs based multi-modal prediction approach (TMMP) to provide accurate prediction services. First, we propose three types of T-RNNs including tensor-based vanilla RNN, tensor-based long short-term memory (T-LSTM) and tensor-based gated recurrent unit (T-GRU), in which the input, output and weights are arbitrary high-order tensors. Then, to compress the weight parameters, we further propose two compact TT-based GRU (TT-GRU) and Tucker-based GRU (Tucker-GRU) models by applying tensor train (TT) and Tucker decompositions to T-GRU model. Afterwards, based on the high-order output tensor generated by T-RNNs, a TMMP approach is proposed to achieve the accurate predictions under various scenarios. Extensive experimental results on the metro traffic flow dataset demonstrate that the proposed TMMP approach can improve the traffic flow prediction accuracy by at most 25.29 percentage compared with the traditional MSE-based approaches. Meanwhile, compared with the T-GRU model, the TT-GRU model can compress the number of parameters by 200~780 times. The proposed T-RNNs and TMMP approach can adapt to different application scenarios and can be used to improve the efficiency of transportation management.

Excavating human's temporal and spatial regularities hidden in trajectory data and predicting users' mobility patterns are conducive to providing proactive smart services for people. Combining Markov transition and tensor theories to improve the prediction performance has proved to be effective. However, the existing state-of-the-art multivariate multiorder Markov model neglects the mutual influence among different users. In a practical trajectory system, people's mobility patterns are influenced by their social relationships. Therefore, this article focuses on proposing a novel multiuser multivariate multiorder Markov model and a multimodal user mobility pattern prediction approach. First, we construct two concrete Markov trajectory transition models based on the single-user multivariate multiorder Markov model. Then, we propose a multiuser multivariate multiorder Markov model, including the influence model of multiple users and the multiuser Markov trajectory transition model. Afterward, two unified product-based power methods are developed to calculate the stationary joint eigentensor (SJE) for single-user and multiuser multivariate multiorder Markov models. Furthermore, an SJE-based multimodal prediction approach is proposed to realize precise mobility pattern prediction. Finally, we conduct a series of experiments based on real-world GPS trajectory data set to verify the performance of the proposed approaches. Experimental results demonstrate that the proposed multiuser multivariate multiorder Markov-based multimodal prediction approach can improve the trajectory prediction accuracy by highest up to 31.10% points compared with the Z-eigen-based approach.

Tensor-based big data analysis approaches are effectively exploited to handle multisource and heterogeneous cyber-physical-social big data generated from diverse spaces. However, the curse of dimensionality seriously restricts their widespread exploitation, especially under edge/fog computing environments. To alleviate the dilemma, we attempt to present a set of tensor-train (TT)-based tensor operations with their scalable computations and then propose a novel TT-based big data processing framework under edge/fog computing environments. Specifically, in this article, we first summarize and present a set of TT-based tensor operations by converting the original high-order tensor operation to a series of low-order (second- or third-order) TTcore-based operations. Then, we propose a two-layer scalable TT-based computation architecture, including inter-TTcore and intra-TTcore scalable models. Afterward, according to various scalable models, a series of scalable TT-based tensor computations (STT-TCs) with their complexity analysis are proposed in detail. Finally, we propose a novel TT-based big data processing framework to adapt to edge/fog computing environments. We conduct extensive experiments based on both random data sets and real-world ubiquitous bus traffic data sets. Experimental results demonstrate that the proposed STT-TCs can significantly improve computation efficiency and are suitable for edge/fog computing environments.

Online education and e-learning have vigorously sprung up and produced massive educational data in a streaming way. It is very challenging to acquire the appropriate learning resources and the suitable learning partners from the streaming-updated educational big data. This article aims to provide sustainable smart educational services including precise personalized recommendation and adaptive clustering under different contexts by correlatively analyzing the global educational data from multiple dimensions via incremental tensor decomposition. First, a tensor-based recommendation and service framework for the streaming educational big data is developed. Then three local tensors concerning learners, resources, and learning records are constructed and further fused to an integrated global learner-resource tensor. Afterward, we present an incremental tensor-based correlative analysis and personalized recommendation (ITCA-PR) algorithm to recommend appropriate resources under various contexts. Besides, we also propose an incremental tensor-based adaptive clustering and community recommendation (ITAC-CR) algorithm to recommend suitable learning partners under various contexts and accordingly construct adaptive learning communities. Extensive experimental results demonstrate that the ITCA-PR algorithm outperforms state-of-the-art improved collaborative filtering algorithms by F-score measurement and the ITAC-CR algorithm has a better clustering performance by DVI measurement. These precise educational services will promote the development of sustainable smart education.

Predicting the future network traffic through big data analysis technologies has been one of the important preoccupations of network design and management. Combining Markov chains with tensors to implement predictions has received considerable attention in the era of big data. However, when dealing with multi-order Markov models, the existing approaches including the combination of states and Z-eigen decomposition still face some shortcomings. Therefore, this paper focuses on proposing a novel multivariate multi-order Markov transition to realize multi-modal accurate predictions. First, we put forward two new tensor operations including tensor join and unified product (UP). Then a general multivariate multi-order (2M) Markov model with its UP-based state transition is proposed. Afterwards, we develop a multi-step transition tensor for 2M Markov models to implement the multi-step state transition. Furthermore, an UP-based power method is proposed to calculate the stationary joint probability distribution tensor (i.e., stationary joint eigentensor, SJE) and realize SJE based multi-modal accurate predictions. Finally, a series of experiments under various Markov models on real-world network traffic datasets are conducted. Experimental results demonstrate that the proposed SJE based approach can improve the prediction accuracy for network traffic by highest up to 38.47 percentage points compared with the Z-eigen based approach.

This paper presents a new bootstrapping approach to named entity (NE) classification. This approach only requires a few common noun/pronoun seeds that correspond to the concept for the target NE type, e.g. he/she/man/woman for PERSON NE. The entire bootstrapping procedure is implemented as training two successive learners: (i) a decision list is used to learn the parsing-based high precision NE rules; (ii) a Hidden Markov Model is then trained to learn string sequence-based NE patterns. The second learner uses the training corpus automatically tagged by the first learner. The resulting NE system approaches supervised NE performance for some NE types. The system also demonstrates intuitive support for tagging user-defined NE types. The differences of this approach from the co-training-based NE bootstrapping are also discussed.

This paper describes a novel approach to named entity (NE) tagging on degraded documents. NE tagging is the process of identifying salient text strings in unstructured text, corresponding to names of people, places, organizations, times/dates, etc. Although NE tagging is typically part of a larger information extraction process, it has other applications, such as improving search in an information retrieval system, and post-processing the results of an OCR system. We focus on degraded documents, i.e. case insensitive documents that lack orthographic information. Examples include output of speech recognition systems, as well as e-mail. The traditional approach involves retraining an NE tagger on degraded text, a cumbersome operation. This paper describes an approach whereby text is first “restored ” to its implicit case sensitive form, and subsequently processed by the original NE tagger. Results show that this new approach leads to far less precision loss in NE tagging of degraded documents. 1.