Graph-based recommender systems have become prevalent in micro-video recommendation by modeling user-item interactions as a bipartite graph. However, these methods face two inherent limitations: (1) their reliance on a fixed, pre-defined graph structure makes them susceptible to noisy interactions, and (2) the multi-modal representations they learn often contain redundant information that is not discriminative enough for the recommendation task. To overcome these issues, we propose a novel Multi-modal Bipartite Graph Structure Learning network (MBGSL), which leverages the information bottleneck principle for robust micro-video recommendation. Specifically, MBGSL first learns adaptive graph structures from multi-modal content (e.g., visual, acoustic, textual) through dedicated graph learners to mitigate noise. Then, it applies an intra-modality information bottleneck to learn minimal sufficient representations within each modality and an inter-modality information bottleneck to capture distinctive information across modalities, thereby eliminating redundancy. Furthermore, the model incorporates collaborative signals through a contrastive learning objective to guide the graph structure learning process. Extensive experiments on three real-world datasets demonstrate that MBGSL achieves state-of-the-art performance, significantly surpassing existing baselines.

Short-video platforms have gained immense popularity, captivating the interest of millions, if not billions, of users globally. Recently, researchers have highlighted the significance of analyzing the propagation of short-videos, which typically involves discovering commercial values, public opinions, user behaviors, etc. This paper proposes a new Short-video Propagation Influence Rating (SPIR) task and aims to promote SPIR from both the dataset and method perspectives. First, we propose a new Cross-platform Short-Video (XS-Video) dataset, which aims to provide a large-scale and real-world short-video propagation network across various platforms to facilitate research on short-video propagation. Our XS-Video dataset includes 117,720 videos, 381,926 samples, and 535 topics across 5 biggest Chinese platforms, annotated with the propagation influence from level 0 to 9. To the best of our knowledge, this is the first large-scale short-video dataset that contains cross-platform data or provides all of the views, likes, shares, collects, fans, comments, and comment content. Second, we propose a Large Graph Model (LGM) named NetGPT, based on a novel three-stage training mechanism, to bridge heterogeneous graph-structured data with the powerful reasoning ability and knowledge of Large Language Models (LLMs). Our NetGPT can comprehend and analyze the short-video propagation graph, enabling it to predict the long-term propagation influence of short-videos. Comprehensive experimental results evaluated by both classification and regression metrics on our XS-Video dataset indicate the superiority of our method for SPIR. Our dataset and code will be open upon acceptance.

With the rapid development of Internet and the burgeoning scale of social media, Social Event Classification (SEC) has garnered increasing attention. The existing study of SEC focuses on recognizing a fixed set of social events. However, in real-world scenarios, new social events continually emerge on social media, which suggests the necessity for a practical SEC model that can swiftly adapt to the evolving environment with incremental social events. Therefore, in this paper, we study a new yet crucial problem defined as Continual Social Event Classification (C-SEC), where new events continually emerge in the sequentially collected social data. Accordingly, we propose a novel Temporal Event Knowledge Network (TEKNet) to continually learn temporal event knowledge for C-SEC with temporally incremental events. First, we conduct present event knowledge learning to learn the classification of newly emerging events in the presently incoming data. Second, we design past event knowledge replay with self-knowledge distillation to consolidate the learned knowledge of past events and prevent catastrophic forgetting. Finally, we propose future event knowledge pretraining with a modality mixture mechanism to pretrain the classifiers for events that occur in the future. Comprehensive experiments on real-world social event datasets demonstrate the superiority of our proposed TEKNet for C-SEC.

Explainable Fake News Detection (EFND) is a new challenge that aims to verify news authenticity and provide clear explanations for its decisions. Traditional EFND methods often treat the tasks of classification and explanation as separate, ignoring the fact that explanation content can assist in enhancing fake news detection. To overcome this gap, we present a new solution: the End-to-End Explainable Fake News Detection Network ( EExpFND ). Our model includes an evidence-claim variational causal inference component, which not only utilizes explanation content to improve fake news detection but also employs a variational approach to address the distributional bias between the ground truth explanation in the training set and the prediction explanation in the test set. Additionally, we incorporate a masked attention network to detail the nuanced relationships between evidence and claims. Our comprehensive tests across two public datasets show that EExpFND sets a new benchmark in performance. The code is available at https://anonymous.4open.science/r/EExpFND-F5C6 .

With the increasing number of news uploaded to the internet daily, rumor detection has garnered significant attention in recent years. Existing rumor detection methods excel on familiar topics with sufficient training data (high resource) collected from the same domain. However, when facing emergent events or rumors propagated in different languages, the performance of these models is significantly degraded, due to the lack of training data and prior knowledge (low resource). To tackle this challenge, we introduce the Test-Time Training for Rumor Detection (T^3RD) to enhance the performance of rumor detection models on low-resource datasets. Specifically, we introduce self-supervised learning (SSL) as an auxiliary task in the test-time training. It consists of global and local contrastive learning, in which the global contrastive learning focuses on obtaining invariant graph representations and the local one focuses on acquiring invariant node representations. We employ the auxiliary SSL tasks for both the training and test-time training phases to mine the intrinsic traits of test samples and further calibrate the trained model for these test samples. To mitigate the risk of distribution distortion in test-time training, we introduce feature alignment constraints aimed at achieving a balanced synergy between the knowledge derived from the training set and the test samples. The experiments conducted on the two widely used cross-domain datasets demonstrate that the proposed model achieves a new state-of-the-art in performance. Our code is available at https://github.com/social-rumors/T3RD.

Graph Neural Networks (GNNs) have recently been utilized to build Collaborative Filtering (CF) models to predict user preferences based on historical user-item interactions. However, there is relatively little understanding of how GNN-based CF models relate to some traditional Network Representation Learning (NRL) approaches. In this paper, we show the equivalence between some state-of-the-art GNN-based CF models and a traditional 1-layer NRL model based on context encoding. Based on a Markov process that trades off two types of distances, we present Markov Graph Diffusion Collaborative Filtering (MGDCF) to generalize some state-of-the-art GNN-based CF models. Instead of considering the GNN as a trainable black box that propagates learnable user/item vertex embeddings, we treat GNNs as an untrainable Markov process that can construct constant context features of vertices for a traditional NRL model that encodes context features with a fully-connected layer. Such simplification can help us to better understand how GNNs benefit CF models. Especially, it helps us realize that ranking losses play crucial roles in GNN-based CF tasks. With our proposed simple yet powerful ranking loss InfoBPR, the NRL model can still perform well without the context features constructed by GNNs. We conduct experiments to perform detailed analysis on MGDCF.

A Session-Based Recommendation (SBR) seeks to predict users’ future item preferences by analyzing their interactions with previously clicked items. In recent approaches, Graph Neural Networks (GNNs) have been commonly applied to capture item relations within a session to infer user intentions. However, these GNN-based methods typically struggle with feature ambiguity between the sequential session information and the item conversion within an item graph, which may impede the model’s ability to accurately infer user intentions. In this article, we propose a novel Multi-hop Multi-view Memory Transformer (M 3 T) to effectively integrate the sequence-view information and relation conversion (graph-view information) of items in a session. First, we propose a Multi-view Memory Transformer (M 2 T) module to concurrently obtain multi-view information of items. Then, a set of trainable memory matrices are employed to store sharable item features, which mitigates cross-view item feature ambiguity. To comprehensively capture latent user intentions, an M 3 T framework is designed to integrate user intentions across different hops of an item graph. Specifically, a k-order power method is proposed to manage the item graph to alleviate the over-smoothing problem when obtaining high-order relations of items. Extensive experiments conducted on three real-world datasets demonstrate the superiority of our method.

With the rapid development of Internet and the expanding scale of social media, social event classification has attracted increasing attention. The key to social event classification is effectively leveraging the visual and textual semantics for classification. However, most of the existing approaches may suffer from the following limitations: (1) Most of them just simply concatenate the image features and text features to get the multimodal features and ignore the fine-grained semantic relationship between modalities. (2) The majority of them hold the closed-world assumption that all classes in test are already seen in training, while this assumption can be easily broken in real-world applications. In practice, new events on Internet may not belong to any existing/seen class, and therefore cannot be correctly identified by closed-world learning algorithms. To tackle these challenges, we propose an Open-World Social Event Classifier (OWSEC) model in this paper. Firstly, we design a multimodal mask transformer network to capture cross-modal semantic relations and fuse fine-grained multimodal features of social events while masking redundant information. Secondly, we design an open-world classifier and propose a cross-modal event mixture mechanism with a novel open-world classification loss to capture the potential distribution space of the unseen class. Extensive experiments on two public datasets demonstrate the superiority of our proposed OWSEC model for open-world social event classification.

Recently, user cold-start recommendations have attracted a lot of attention from industry and academia. In user cold-start recommendation systems, the user attribute information is often used by existing approaches to learn user preferences due to the unavailability of user action data. However, most existing recommendation methods often ignore the sparsity of user attributes in cold-start recommendation systems. To tackle this limitation, this article proposes a novel Inductive Heterogeneous Graph Neural Network (IHGNN) model, which utilizes the relational information in user cold-start recommendation systems to alleviate the sparsity of user attributes. Our model converts new users, items, and associated multimodal information into a Modality-aware Heterogeneous Graph (M-HG) that preserves the rich and heterogeneous relationship information among them. Specifically, to utilize rich and heterogeneous relational information in an M-HG for enriching the sparse attribute information of new users, we design a strategy based on random walk operations to collect associated neighbors of new users by multiple times sampling operation. Then, a well-designed multiple hierarchical attention aggregation model consisting of the intra- and inter-type attention aggregating module is proposed, focusing on useful connected neighbors and neglecting meaningless and noisy connected neighbors to generate high-quality representations for user cold-start recommendations. Experimental results on three real datasets demonstrate that the IHGNN outperforms the state-of-the-art baselines.

Medicine recommendation systems target to recommend a set of medicines given a set of symptoms which play a crucial role in assisting doctors in their daily clinics. Existing approaches are either rule-based or supervised. However, the former heavily relies on expert labeling, which is time-consuming and costly to collect, and the latter suffers from the data sparse problem. To automate medicine recommendation on sparse data, we propose MedRec, which introduces two graphs in modeling: (1) a knowledge graph connecting diseases, medicines, symptoms, and examinations; (2) an attribute graph connecting medicines via shared attributes and molecular structures. These two graphs enhance the connectivity between symptoms and medicines, which thus alleviate the data sparse problem. By learning the interrelationship between diseases, medicines, symptoms and examinations and the inner relationship within medicine, we can acquire unified embedding representations of symptoms and medicines which can be used in medicine recommendation. The experimental results show that the proposed model outperforms state-of-the-art methods. In addition, we find that these two tasks: learning graph representation and medical recommendation can benefit each other.

The cross-modal image-text retrieval has attracted extensive attention in recent years, which contributes to the development of search engine. Fine-grained features and cross-attention have been widely used in past researches to reach the goal of cross-modal image-text matching. Although cross-related methods have achieved remarkable results, the features must be encoded again in evaluation phase due to the interaction of the two modalities, which is unsuitable for actual scenarios of search engine development. In addition, the aggregated feature does not contain sufficient semantics since it is merely obtained by simple mean pooling. Furthermore, connecting weights of self-attention blocks are target position invariant, which lacks the expected adaptability. To tackle these limitations, in this paper, we propose a novel Global Relation-aware Attention Network (GRAN) for image-text retrieval by designing Global Attention Module (GAM) and Relation-aware Attention Module (RAM) which play an important role in modeling the global feature and the relationships of local fragments. Firstly, we propose Global Attention Module (GAM) followed the fine-grained features to obtain meaningful global feature. Secondly, we use several stacked transformer encoders to further encode features separately. Finally, we propose Relation-aware Attention Module (RAM) to generate a vector which represents the relation information to infer the attention intensity of pairwise fragments. The local features, the global feature, and their relations are considered jointly to conduct an efficient image-text retrieval. Extensive experiments are conducted on the benchmark datasets of Flickr30K and MSCOCO, demonstrating the superiority of our method. On the Flickr30K, compared to the state-of-the-art method TERAN, we improve [email protected](K=1) metric by 5.8% and 4.0 on the image and text retrieval tasks, respectively.

Nowadays, detecting fake news on social media platforms has become a top priority since the widespread dissemination of fake news may mislead readers and have negative effects. To date, many algorithms have been proposed to facilitate the detection of fake news from the hand-crafted feature extraction methods to deep learning approaches. However, these methods may suffer from the following limitations: (1) fail to utilize the multi-modal context information and extract high-order complementary information for each news to enhance the detection of fake news; (2) largely ignore the full hierarchical semantics of textual content to assist in learning a better news representation. To overcome these limitations, this paper proposes a novel hierarchical multi-modal contextual attention network (HMCAN) for fake news detection by jointly modeling the multi-modal context information and the hierarchical semantics of text in a unified deep model. Specifically, we employ BERT and ResNet to learn better representations for text and images, respectively. Then, we feed the obtained representations of images and text into a multi-modal contextual attention network to fuse both inter-modality and intra-modality relationships. Finally, we design a hierarchical encoding network to capture the rich hierarchical semantics for fake news detection. Extensive experiments on three public real datasets demonstrate that our proposed HMCAN achieves state-of-the-art performance.

Knowledge graph embedding (KGE) encodes components of a KG including entities and relations into continuous low vector space. Most existing methods focus on treating entities and relations in triples independently and thus failing to capture the complex and hidden information that is inherently implicit inside the local neighborhood surrounding a triple. In this paper, we present a new approach for knowledge graph completion called GAEAT (Graph Auto-encoder Attention Network Embedding), which can encapsulate both entity and relation features. Specifically, we construct a triple-level auto-encoder by extending graph attention mechanisms to obtain latent representations of entities and relations simultaneously. To justify our proposed model, we evaluate GAEAT on two real-world datasets. The experimental results demonstrate that GAEAT can outperform state-of-the-art KGE models in knowledge graph completion task, which validates the effectiveness of GAEAT. The source code of this paper can be obtained from https://github.com/TomersHan/GAEAT.

Nowadays, with the rapid development of social media, there is a great deal of news produced every day. How to detect fake news automatically from a large of multimedia posts has become very important for people, the government and news recommendation sites. However, most of the existing approaches either extract features from the text of the post which is a single modality or simply concatenate the visual features and textual features of a post to get a multimodal feature and detect fake news. Most of them ignore the background knowledge hidden in the text content of the post which facilitates fake news detection. To address these issues, we propose a novel Knowledge-driven Multimodal Graph Convolutional Network (KMGCN) to model the semantic representations by jointly modeling the textual information, knowledge concepts and visual information into a unified framework for fake news detection. Instead of viewing text content as word sequences normally, we convert them into a graph, which can model non-consecutive phrases for better obtaining the composition of semantics. Besides, we not only convert visual information as nodes of graphs but also retrieve external knowledge from real-world knowledge graph as nodes of graphs to provide complementary semantics information to improve fake news detection. We utilize a well-designed graph convolutional network to extract the semantic representation of these graphs. Extensive experiments on two public real-world datasets illustrate the validation of our approach.

Hashing-based cross-modal search which aims to map multiple modality features into binary codes has attracted increasingly attention due to its storage and search efficiency especially in large-scale database retrieval. Recent unsupervised deep cross-modal hashing methods have shown promising results. However, existing approaches typically suffer from two limitations: (1) They usually learn cross-modal similarity information separately or in a redundant fusion manner, which may fail to capture semantic correlations among instances from different modalities sufficiently and effectively. (2) They seldom consider the sampling and weighting schemes for unsupervised cross-modal hashing, resulting in the lack of satisfactory discriminative ability in hash codes.

Natural language generation has become a fundamental task in dialogue systems. RNN-based natural response generation methods encode the dialogue context and decode it into a response. However, they tend to generate dull and simple responses. In this article, we propose a novel framework, called KAWA-DRG (Knowledge-aware Attentive Wasserstein Adversarial Dialogue Response Generation) to model conversation-specific external knowledge and the importance variances of dialogue context in a unified adversarial encoder-decoder learning framework. In KAWA-DRG, a co-attention mechanism attends to important parts within and among context utterances with word-utterance-level attention. Prior knowledge is integrated into the conditional Wasserstein auto-encoder for learning the latent variable space. The posterior and prior distribution of latent variables are generated and trained through adversarial learning. We evaluate our model on Switchboard, DailyDialog, In-Car Assistant, and Ubuntu Dialogue Corpus. Experimental results show that KAWA-DRG outperforms the existing methods.

Social event is something that occurs at specific place and time associated with some specific actions, and it consists of many stories over time. With the explosion of Web 2.0 platforms, a popular social event that is happening around us and around the world can spread very fast. As a result, social event analysis becomes more and more important for users to understand the whole evolutionary trend of social event over time. However, it is very challenging to do social event analysis because social event data from different social media sites have multi-modal, multi-domain, and large-scale properties. The goal of our research is to design advanced multimedia techniques to deal with the above issues and establish an effective and robust social event analysis framework for social event representation, detection, tracking and evolution analysis. (1) For social event representation, we propose a novel cross-domain collaborative learning algorithm based on non-parametric Bayesian dictionary learning model. It can make use of the shared domain priors and modality priors to collaboratively learn the data's representations by considering the domain discrepancy and the multi-modal property.(2) For social event detection, we propose a boosted multi-modal supervised Latent Dirichlet Allocation model. It can effectively exploit multi-modality information and utilize boosting weighted sampling strategy for large-scale data processing. (3) For social event tracking, we propose a novel multi-modal event topic model, which can effectively model the correlations between textual and visual modalities, and obtain their topics over time. (4) For social event evolution analysis, we propose a novel multi-modal multi-view topic-opinion mining model to conduct fined-grained topic and opinion analysis for social events from multiple social media sites collaboratively. It can discover multi-modal topics and the corresponding opinions over time to understand the evolutionary processes of social event. Extensive experimental results show that the proposed algorithms perform favorably against state-of-the-art methods for social event analysis.