Graph-based recommender systems have become prevalent in micro-video recommendation by modeling user-item interactions as a bipartite graph. However, these methods face two inherent limitations: (1) their reliance on a fixed, pre-defined graph structure makes them susceptible to noisy interactions, and (2) the multi-modal representations they learn often contain redundant information that is not discriminative enough for the recommendation task. To overcome these issues, we propose a novel Multi-modal Bipartite Graph Structure Learning network (MBGSL), which leverages the information bottleneck principle for robust micro-video recommendation. Specifically, MBGSL first learns adaptive graph structures from multi-modal content (e.g., visual, acoustic, textual) through dedicated graph learners to mitigate noise. Then, it applies an intra-modality information bottleneck to learn minimal sufficient representations within each modality and an inter-modality information bottleneck to capture distinctive information across modalities, thereby eliminating redundancy. Furthermore, the model incorporates collaborative signals through a contrastive learning objective to guide the graph structure learning process. Extensive experiments on three real-world datasets demonstrate that MBGSL achieves state-of-the-art performance, significantly surpassing existing baselines.

Formulating statements that support diverse or controversial stances on specific topics is vital for platforms that enable user expression, reshape political discourse, and drive social critique and information dissemination. With the rise of Large Language Models (LLMs), controllable text generation towards specific stances has become a promising research area with applications in shaping public opinion and commercial marketing. However, existing datasets focus solely on unimodal textual inputs and are primarily designed for stance classification rather than controllable generation. In this work, we introduce the task of stance-driven multimodal response generation, where the goal is to produce a stance-aligned comment given a post on social media that includes both textual and visual information. To support this task, we present StanceGen2024, a new dataset comprising tweet-image/video pairs and stance-annotated user comments from the 2024 U.S. presidential election. The dataset captures rich multimodal interactions and includes fine-grained labels for stance and style, enabling a nuanced study of how multimodal political content shapes stance expression. In addition, we propose SDMG (Stance-Driven Multimodal Generation), a baseline framework that integrates weighted fusion of multimodal features and stance guidance to improve semantic consistency and stance control. The code are available at: https://github.com/blingbq/StanceGen.

Graph Neural Networks (GNNs) have recently been utilized to build Collaborative Filtering (CF) models to predict user preferences based on historical user-item interactions. However, there is relatively little understanding of how GNN-based CF models relate to some traditional Network Representation Learning (NRL) approaches. In this paper, we show the equivalence between some state-of-the-art GNN-based CF models and a traditional 1-layer NRL model based on context encoding. Based on a Markov process that trades off two types of distances, we present Markov Graph Diffusion Collaborative Filtering (MGDCF) to generalize some state-of-the-art GNN-based CF models. Instead of considering the GNN as a trainable black box that propagates learnable user/item vertex embeddings, we treat GNNs as an untrainable Markov process that can construct constant context features of vertices for a traditional NRL model that encodes context features with a fully-connected layer. Such simplification can help us to better understand how GNNs benefit CF models. Especially, it helps us realize that ranking losses play crucial roles in GNN-based CF tasks. With our proposed simple yet powerful ranking loss InfoBPR, the NRL model can still perform well without the context features constructed by GNNs. We conduct experiments to perform detailed analysis on MGDCF.

With the rapid development of Internet and the expanding scale of social media, social event classification has attracted increasing attention. The key to social event classification is effectively leveraging the visual and textual semantics for classification. However, most of the existing approaches may suffer from the following limitations: (1) Most of them just simply concatenate the image features and text features to get the multimodal features and ignore the fine-grained semantic relationship between modalities. (2) The majority of them hold the closed-world assumption that all classes in test are already seen in training, while this assumption can be easily broken in real-world applications. In practice, new events on Internet may not belong to any existing/seen class, and therefore cannot be correctly identified by closed-world learning algorithms. To tackle these challenges, we propose an Open-World Social Event Classifier (OWSEC) model in this paper. Firstly, we design a multimodal mask transformer network to capture cross-modal semantic relations and fuse fine-grained multimodal features of social events while masking redundant information. Secondly, we design an open-world classifier and propose a cross-modal event mixture mechanism with a novel open-world classification loss to capture the potential distribution space of the unseen class. Extensive experiments on two public datasets demonstrate the superiority of our proposed OWSEC model for open-world social event classification.

Representation learning of knowledge graphs (KGs) aims to embed both entities and relations as vectors in a continuous low-dimensional space, which has facilitated various applications such as link prediction and entity retrieval. Most existing KG embedding methods focus on modeling the structured fact triples independently and ignore the multi-type relations among triples as well as the variety of data types (e.g., texts and images) associated with entities in KGs, and thus fail to capture the complex and multi-modal information that is inherently inside the entity-relation triples. In this paper, we propose a novel approach for knowledge graph embedding named Contrastive Multi-modal Graph Neural Network (CMGNN), which can encapsulate comprehensive features from multi-modal content descriptions of entities and high-order connectivity structures. Specifically, CMGNN first learns entity embeddings from multi-modal content and then contrasts encodings from multi-relational local neighbors and high-order connectivities to obtain latent representations of entities and relations simultaneously. Experimental results demonstrate that CMGNN can effectively model the multi-modalities and multi-type structures in KGs, and significantly outperforms existing state-of-the-art methods on benchmark datasets for the tasks of link prediction and entity classification.

Recently, user cold-start recommendations have attracted a lot of attention from industry and academia. In user cold-start recommendation systems, the user attribute information is often used by existing approaches to learn user preferences due to the unavailability of user action data. However, most existing recommendation methods often ignore the sparsity of user attributes in cold-start recommendation systems. To tackle this limitation, this article proposes a novel Inductive Heterogeneous Graph Neural Network (IHGNN) model, which utilizes the relational information in user cold-start recommendation systems to alleviate the sparsity of user attributes. Our model converts new users, items, and associated multimodal information into a Modality-aware Heterogeneous Graph (M-HG) that preserves the rich and heterogeneous relationship information among them. Specifically, to utilize rich and heterogeneous relational information in an M-HG for enriching the sparse attribute information of new users, we design a strategy based on random walk operations to collect associated neighbors of new users by multiple times sampling operation. Then, a well-designed multiple hierarchical attention aggregation model consisting of the intra- and inter-type attention aggregating module is proposed, focusing on useful connected neighbors and neglecting meaningless and noisy connected neighbors to generate high-quality representations for user cold-start recommendations. Experimental results on three real datasets demonstrate that the IHGNN outperforms the state-of-the-art baselines.

Medicine recommendation systems target to recommend a set of medicines given a set of symptoms which play a crucial role in assisting doctors in their daily clinics. Existing approaches are either rule-based or supervised. However, the former heavily relies on expert labeling, which is time-consuming and costly to collect, and the latter suffers from the data sparse problem. To automate medicine recommendation on sparse data, we propose MedRec, which introduces two graphs in modeling: (1) a knowledge graph connecting diseases, medicines, symptoms, and examinations; (2) an attribute graph connecting medicines via shared attributes and molecular structures. These two graphs enhance the connectivity between symptoms and medicines, which thus alleviate the data sparse problem. By learning the interrelationship between diseases, medicines, symptoms and examinations and the inner relationship within medicine, we can acquire unified embedding representations of symptoms and medicines which can be used in medicine recommendation. The experimental results show that the proposed model outperforms state-of-the-art methods. In addition, we find that these two tasks: learning graph representation and medical recommendation can benefit each other.

The cross-modal image-text retrieval has attracted extensive attention in recent years, which contributes to the development of search engine. Fine-grained features and cross-attention have been widely used in past researches to reach the goal of cross-modal image-text matching. Although cross-related methods have achieved remarkable results, the features must be encoded again in evaluation phase due to the interaction of the two modalities, which is unsuitable for actual scenarios of search engine development. In addition, the aggregated feature does not contain sufficient semantics since it is merely obtained by simple mean pooling. Furthermore, connecting weights of self-attention blocks are target position invariant, which lacks the expected adaptability. To tackle these limitations, in this paper, we propose a novel Global Relation-aware Attention Network (GRAN) for image-text retrieval by designing Global Attention Module (GAM) and Relation-aware Attention Module (RAM) which play an important role in modeling the global feature and the relationships of local fragments. Firstly, we propose Global Attention Module (GAM) followed the fine-grained features to obtain meaningful global feature. Secondly, we use several stacked transformer encoders to further encode features separately. Finally, we propose Relation-aware Attention Module (RAM) to generate a vector which represents the relation information to infer the attention intensity of pairwise fragments. The local features, the global feature, and their relations are considered jointly to conduct an efficient image-text retrieval. Extensive experiments are conducted on the benchmark datasets of Flickr30K and MSCOCO, demonstrating the superiority of our method. On the Flickr30K, compared to the state-of-the-art method TERAN, we improve [email protected](K=1) metric by 5.8% and 4.0 on the image and text retrieval tasks, respectively.

Nowadays, detecting fake news on social media platforms has become a top priority since the widespread dissemination of fake news may mislead readers and have negative effects. To date, many algorithms have been proposed to facilitate the detection of fake news from the hand-crafted feature extraction methods to deep learning approaches. However, these methods may suffer from the following limitations: (1) fail to utilize the multi-modal context information and extract high-order complementary information for each news to enhance the detection of fake news; (2) largely ignore the full hierarchical semantics of textual content to assist in learning a better news representation. To overcome these limitations, this paper proposes a novel hierarchical multi-modal contextual attention network (HMCAN) for fake news detection by jointly modeling the multi-modal context information and the hierarchical semantics of text in a unified deep model. Specifically, we employ BERT and ResNet to learn better representations for text and images, respectively. Then, we feed the obtained representations of images and text into a multi-modal contextual attention network to fuse both inter-modality and intra-modality relationships. Finally, we design a hierarchical encoding network to capture the rich hierarchical semantics for fake news detection. Extensive experiments on three public real datasets demonstrate that our proposed HMCAN achieves state-of-the-art performance.

Knowledge graph embedding (KGE) encodes components of a KG including entities and relations into continuous low vector space. Most existing methods focus on treating entities and relations in triples independently and thus failing to capture the complex and hidden information that is inherently implicit inside the local neighborhood surrounding a triple. In this paper, we present a new approach for knowledge graph completion called GAEAT (Graph Auto-encoder Attention Network Embedding), which can encapsulate both entity and relation features. Specifically, we construct a triple-level auto-encoder by extending graph attention mechanisms to obtain latent representations of entities and relations simultaneously. To justify our proposed model, we evaluate GAEAT on two real-world datasets. The experimental results demonstrate that GAEAT can outperform state-of-the-art KGE models in knowledge graph completion task, which validates the effectiveness of GAEAT. The source code of this paper can be obtained from https://github.com/TomersHan/GAEAT.

Nowadays, with the rapid development of social media, there is a great deal of news produced every day. How to detect fake news automatically from a large of multimedia posts has become very important for people, the government and news recommendation sites. However, most of the existing approaches either extract features from the text of the post which is a single modality or simply concatenate the visual features and textual features of a post to get a multimodal feature and detect fake news. Most of them ignore the background knowledge hidden in the text content of the post which facilitates fake news detection. To address these issues, we propose a novel Knowledge-driven Multimodal Graph Convolutional Network (KMGCN) to model the semantic representations by jointly modeling the textual information, knowledge concepts and visual information into a unified framework for fake news detection. Instead of viewing text content as word sequences normally, we convert them into a graph, which can model non-consecutive phrases for better obtaining the composition of semantics. Besides, we not only convert visual information as nodes of graphs but also retrieve external knowledge from real-world knowledge graph as nodes of graphs to provide complementary semantics information to improve fake news detection. We utilize a well-designed graph convolutional network to extract the semantic representation of these graphs. Extensive experiments on two public real-world datasets illustrate the validation of our approach.

Natural language generation has become a fundamental task in dialogue systems. RNN-based natural response generation methods encode the dialogue context and decode it into a response. However, they tend to generate dull and simple responses. In this article, we propose a novel framework, called KAWA-DRG (Knowledge-aware Attentive Wasserstein Adversarial Dialogue Response Generation) to model conversation-specific external knowledge and the importance variances of dialogue context in a unified adversarial encoder-decoder learning framework. In KAWA-DRG, a co-attention mechanism attends to important parts within and among context utterances with word-utterance-level attention. Prior knowledge is integrated into the conditional Wasserstein auto-encoder for learning the latent variable space. The posterior and prior distribution of latent variables are generated and trained through adversarial learning. We evaluate our model on Switchboard, DailyDialog, In-Car Assistant, and Ubuntu Dialogue Corpus. Experimental results show that KAWA-DRG outperforms the existing methods.

With a smart phone at hand, it becomes easy now to snap pictures and publish them online with few lines of texts. The GPS coordinates and User-Generated Content (UGC) data embedded in the shared photos provide opportunities to exploit important knowledge to tackle interesting tasks like geographically organizing photos and location visualization. In this work, we propose to organize photos both geographically and semantically, and investigate the problem of location visualization from multiple semantic themes. The novel visualization scheme provides a rich display landscape for geographical exploration from versatile views. A two-level solution is presented, where we first identify the highly photographed places of interest (POI) and discover their focused themes, and then aggregate the lower-level POI themes to generate the higher-level city themes for location visualization. We have conducted experiments on crawled Flickr and Instagram data and exhibited the visualization for the cities of Singapore and Sydney. The experimental results have validated the proposed method and demonstrated the potentials of location visualization from multiple themes.

Newly emerging location-based social media network services (LBSMNS) provide valuable resources to understand users’ behaviors based on their location histories. The location-based behaviors of a user are generally influenced by both user intrinsic interest and the location preference, and moreover are spatial-temporal context dependent. In this article, we propose a spatial-temporal context-aware personalized location recommendation system (STCAPLRS), which offers a particular user a set of location items such as points of interest or venues (e.g., restaurants and shopping malls) within a geospatial range by considering personal interest, local preference, and spatial-temporal context influence. STCAPLRS can make accurate recommendation and facilitate people’s local visiting and new location exploration by exploiting the context information of user behavior, associations between users and location items, and the location and content information of location items. Specifically, STCAPLRS consists of two components: offline modeling and online recommendation. The core module of the offline modeling part is a context-aware regression mixture model that is designed to model the location-based user behaviors in LBSMNS to learn the interest of each individual user, the local preference of each individual location, and the context-aware influence factors. The online recommendation part takes a querying user along with the corresponding querying spatial-temporal context as input and automatically combines the learned interest of the querying user, the local preference of the querying location, and the context-aware influence factor to produce the top- k recommendations. We evaluate the performance of STCAPLRS on two real-world datasets: Dianping and Foursquare. The results demonstrate the superiority of STCAPLRS in recommending location items for users in terms of both effectiveness and efficiency. Moreover, the experimental analysis results also illustrate the excellent interpretability of STCAPLRS.