Text-based visual question answering (TextVQA) aims to answer questions by understanding scene text in images. However, many current methods overly depend on the accuracy of Optical Character Recognition (OCR) systems, while overlooking the significance of visual objects. They tend to perform poorly when the question involves the relationships between visual objects and scene text. To address the above issues, we focus on raising the status of visual objects and innovatively propose a hierarchical semantic reasoning network (CT-HSR) based on the cascade transformer architecture, achieving fine-grained cross-modal reasoning and visual semantic enhancement. Specifically, the visual representations containing rich semantic information of the question modality are obtained through the cross-modal transformer-based vision-language pre-training model firstly. Then, the uni-modal transformer for unified modality encoding module is utilized to capture visual objects that are more semantically related to OCR texts. In addition, we further alleviate the cross-modal noise interference through the feature filtering strategy. Finally, we better align the three modalities by introducing TextVQA pre-training tasks and generate prediction answers through multi-step iterative prediction during fine-tuning. Extensive experiments on the TextVQA, ST-VQA, and OCR-VQA datasets have demonstrated the effectiveness of our proposed model compared to the state-of-the-art methods. The code will be released at https://github.com/FTFWO/CT-HSR .

Recently, the research on fairness in recommendation systems has garnered widespread attention. Moreover, numerous fair recommendation models have been developed for scenarios with limited sensitive information, thereby alleviating the issue of missing sensitive information. However, the performance of these methods still tends to decline significantly when sensitive attributes are extremely scarce. In this paper, we propose FairCL, a novel fair recommendation framework designed to perform effectively under limited sensitive attribute information. FairCL features a contrastive learning-based sensitive attribute encoder that can be integrated with existing fair recommendation algorithms. By leveraging both collaborative information and item side information, we predict unknown sensitive attributes and apply contrastive learning for sensitive attribute modeling. Furthermore, we theoretically demonstrate how FairCL can be integrated with mutual information-based and adversarial learning-based fairness algorithms. Extensive experiments on three real-world datasets show that FairCL significantly enhances fairness, even when only a small portion of users' sensitive attributes are known. The code and data are at: https://anonymous.4open.science/r/CL-for-FairRec-000A/.

In the era of rapid development of social media, social recommendation systems as hybrid recommendation systems have been widely applied. Existing methods capture interest similarity between users to filter out interestirrelevant relations in social networks that inevitably decrease recommendation accuracy, however, limited research has a focus on the mutual influence of semantic information between the social network and the user-item interaction network for further improving social recommendation. To address these issues, we introduce a social recommendation model with robust graph denoising-augmentation fusion and multi-semantic Modeling(Burger). Specifically, we firstly propose to construct a social tensor in order to smooth the training process of the model. Then, a graph convolutional network and a tensor convolutional network are employed to capture user's item preference and social preference, respectively. Considering the different semantic information in the user-item interaction network and the social network, a bi-semantic coordination loss is proposed to model the mutual influence of semantic information. To alleviate the interference of interest-irrelevant relations on multi-semantic modeling, we further use Bayesian posterior probability to mine potential social relations to replace social noise. Finally, the sliding window mechanism is utilized to update the social tensor as the input for the next iteration. Extensive experiments on three real datasets show Burger has a superior performance compared with the state-of-the-art models.

Transformer-based text-image matching model, known as CLIP, has garnered significant attention owing to its exceptional performance in text-image retrieval tasks and downstream applications. However, the interpret-ability of CLIP remains underexplored. Existing interpretation methods for Transformers often struggle with incomplete and unreliable attributions within the image and text modalities, respectively. In this paper, we propose a fine-grained interpretation method, termed Class Activation Values (CAV), to provide lucid and faithful visual explanations for CLIP-based text-image retrievals. Specifically, we systematically perform multi-scale accumulation and fusion of class-specific gradients and activation value features to generate high-definition explanations for the image encoder. Furthermore, we present element-wise gradient-based weights to attribute fine-grained relevance between value features and output similarity within the text encoder. The proposed CAV is capable of simultaneously rendering detailed and credible explanations due to its precise feature attribution. Extensive qualitative and quantitative experiments are conducted on the ImageNet-1k and MS COCO datasets, and the experimental results demonstrate that CAV outperforms state-of-the-art interpretation methods in both faithfulness and localization assessments across image and text modalities.

The continual emergence of new entities and relations drives the dynamic expansion of knowledge graphs (KG). In the face of such growing KG, relearning from scratch wastes acquired knowledge, while learning solely from new snapshots leads to model forgetting of old knowledge. Existing methods focus on lifelong learning in growing KG through transfer and regularize embeddings. However, extensive entity updates to adapt to new snapshots introduce conflicts between old and new knowledge, thereby resulting in the inevitable occurrence of knowledge forgetting. To address these challenges, we propose the Evolutionary Relation Path Passing (ERPP) model for lifelong knowledge graph embedding, aiming to shift from knowledge forgetting to knowledge accumulation, thereby achieving accurate long-term prediction. Specifically, we propose a snapshot conditional relation path passing strategy to generate expressive representations that better adapt to snapshots compared to the transferred embeddings in existing methods. Subsequently, we propose a relation inheritance and evolution mechanism across snapshots and continue relation path passing in next snapshots. This allows ERPP to avoid inevitable catastrophic forgetting from frequent entity embedding updates. ERPP outperforms SOTA models in 35 scenarios, with average improvements of 11.1% in long-term prediction and 12.9% in knowledge transfer. Moreover, ERPP makes a breakthrough in achieving knowledge positive accumulation, in contrast to the negative forgetting of existing models. To the best of our knowledge, ERPP is the first model to realize knowledge accumulation. Our code is available at https://anonymous.4open.science/r/ERPP-6D66.

Multimodal Knowledge Graphs (MMKG) models integrate multimodal contexts to improve link prediction performance. All existing MMKG models follow the transductive setting with a fixed predefined set, meaning that all the entities, relations, and multimodal information in the test graph are observed during training. This hinders their generalization to real-world MMKG with unseen entities and relations. Intuitively, a MMKG model trained on DBpedia cannot infer on Freebase. To address above limitations, we make the first attempt towards inductive learning for MMKG and propose a multimodal Inductive MMKG model (IndMKG) that is universal and transferable to any MMKG. Distinct from existing transductive methods, our model does not rely on specific trained embeddings; instead, IndMKG generates adaptive embeddings conditioned on any new MMKG via multimodal prototypes. Specifically, we construct class-adaptive prototypes to appropriately characterize the multimodal feature distribution of the given graph and equip IndMKG with robust adaptability to multimodal information across MMKGs. In addition, IndMKG learns non-specific structural embeddings based on meta relations. Such strategies tackle the challenge of notable multimodal feature discrepancies in cross-graph induction and allow the pre-trained IndMKG model to effectively zero-shot generalize to any MMKG. The strong performance in both inductive and transductive settings, across more than 20+ different scenarios, confirms the effectiveness and robustness of IndMKG. Our code is released at https://github.com/MMKGer/IndMKG/.

\beginabstract Social network has obtained extensive attention in recommender system. Existing social recommendation models mostly leverage social relations to capture potential interactions between users and items, thereby enhancing recommendation performance. However, these methods ignore the fine-grained bidirectional trust weight and the constraint on the relative positions of entities in social network and user-item interaction network. To this end, in this paper, we propose a social recommendation framework with Trust mining and multi-semantic reGularization (TrustGo). Specifically, we firstly construct a trust network based on the observed social network and establish a high-quality item implicit network. Then, we integrate the trust network, item implicit network, and user-item interaction network into a heterogeneous network. We introduce a meta-path based aggregation in this heterogeneous network to map the users and items into a latent space. And then, by using an ensemble method, we can obtain the final prediction ratings. Considering the users' different behaviors in social network and user-item interaction network, we define two semantic spaces, i.e., the social semantic space and user-item interactional semantic space. And a multi-semantic regularization module is designed to adjust the relative positions of entities in the two kinds of semantic spaces, respectively. Extensive experiments on three real-world datasets demonstrate that our TrustGo model is superior to other state-of-the-art recommendation models. \endabstract

With the development of recommendation algorithms, researchers are paying increasing attention to fairness issues such as user discrimination in recommendations. To address these issues, existing works often filter users’ sensitive information that may cause discrimination during the process of learning user representations. However, these approaches overlook the latent relationship between items’ content attributes and users’ sensitive information. In this article, we propose DALFRec, a fairness-aware recommendation algorithm based on user-side and item-side adversarial learning to mitigate the effects of sensitive information on both sides of the recommendation process. First, we conduct a statistical analysis to demonstrate the latent relationship between items’ information and users’ sensitive attributes. Then, we design a dual-side adversarial learning network that simultaneously filters out users’ sensitive information on the user and item side. Additionally, we propose a new evaluation strategy that leverages the latent relationship between items’ content attributes and users’ sensitive attributes to better assess the algorithm’s ability to reduce discrimination. Our experiments on three real datasets demonstrate the superiority of our proposed algorithm over state-of-the-art methods.

Discrimination against different user groups has received growing attention in the recommendation field. To address this problem, existing works typically remove sensitive attributes that may cause discrimination through adversary learning to achieve fair recommendations. However, these approaches leverage all available interactions for learning user representations and overlook the fact that different interactions have varying relevance to users’ sensitive attributes. Ignoring this issue may weaken the effectiveness of adversary learning in removing sensitive attributes. To tackle this challenge, we propose a novel model called GS-FairRec, which distinguishes between user interactions to achieve better removal of sensitive attributes. The model consists of three modules: graph sampling-based representation learning, pseudo-user representation learning, and adversarial learning. Firstly, the graph sampling-based representation learning module removes some irrelevant neighbors from a user-item bipartite graph and employs a graph convolutional network (GCN) to learn user/item representations. Next, items that are relevant to a user’s sensitive information but do not match their preferences are defined as the user’s pseudo-interest items, which are leveraged to learn the pseudo-user representation. In the adversarial learning module, the user’s two kinds of representations are fused for adversarial learning to remove sensitive information. Additionally, we design a new metric to measure the model’s ability to remove sensitive attributes based on how a generated recommendation list discloses the user’s sensitive attributes. Finally, we conduct experiments on two real-world datasets, and our results demonstrate the superiority of our proposed model in fairness tasks.

Retrieving similar documents from a large-scale text corpus according to a given document is a fundamental technique for many applications. However, most of existing indexing techniques have difficulties to address this problem due to special properties of a document query, e.g. high dimensionality, sparse representation and semantic issue. Towards addressing this problem, we propose a two-level retrieval solution based on a document decomposition idea. A document is decomposed to a compact vector and a few document specific keywords by a dimension reduction approach. The compact vector embodies the major semantics of a document, and the document specific keywords complement the discriminative power lost in dimension reduction process. We adopt locality sensitive hashing (LSH) to index the compact vectors, which guarantees to quickly find a set of related documents according to the vector of a query document. Then we re-rank documents in this set by their document