With the rapid growth of multi-modal content on the Web, robust vision-language models are essential for semantic understanding and classification of web images under diverse and dynamic contexts, supporting Web applications such as multimedia search and recommendation. Prompt learning has proven effective for enhancing vision-language models in semantic image classification tasks. However, previous methods often suffer from poor generalization: the learned prompts tend to overfit the base classes seen during training, leading to poor performance on unseen classes and under distribution shifts. This issue is especially challenging in Web-scale data, where new classes emerge and distributions shift dynamically. To address these limitations, we propose PLIKD, a novel prompt learning method that integrates instance-aware knowledge distillation for robust Web-scale semantic image classification. Specifically, PLIKD introduces an instance-aware knowledge extraction module, which leverages multi-modal large language models through a step-by-step strategy to extract external knowledge for each image instance. To incorporate this extracted knowledge, PLIKD further introduces an instance-aware knowledge distillation module, which consists of two key steps: (1) a dual-teacher strategy for robust and informative knowledge distillation, and (2) fine-grained cross-modal alignment via Smooth and Sparse Optimal Transport. Extensive experiments demonstrate that PLIKD significantly improves generalization to both seen and unseen classes, and remains robust under distribution shifts, outperforming existing state-of-the-art methods on Web-scale semantic image classification.

Relational learning is an essential task in the domain of knowledge representation, particularly in knowledge graph completion (KGC). While relational learning in traditional single-modal settings has been extensively studied, exploring it within a multimodal KGC context presents distinct challenges and opportunities. One of the major challenges is inference on newly discovered relations without any associated training data. This zero-shot relational learning scenario poses unique requirements for multimodal KGC, i.e., utilizing multimodality to facilitate relational learning. However, existing works fail to support the leverage of multimodal information and leave the problem unexplored. In this paper, we propose a novel end-to-end framework, consisting of three components, i.e., multimodal learner, structure consolidator, and relation embedding generator, to integrate diverse multimodal information and knowledge graph structures to facilitate the zero-shot relational learning. Evaluation results on three multimodal knowledge graphs demonstrate the superior performance of our proposed method.

Zero-Shot Story Visualization (ZSV) seeks to depict textual narratives through a sequence of images without relying on pre-existing text-image pairs for training. In this paper, we address the challenge of automated multi-character ZSV, aiming to create distinctive yet compatible character portraits for high-quality story visualization without the need of manual human interventions. Our study is motivated by the limitation of current ZSV approaches that necessitate inefficient manual collection of external images as initial character portraits and suffer from low-quality story visualization, especially with multi-character interactions, when the portraits are not well initiated. To overcome these issues, we develop LeMon, an LLM enhanced Multi-Character Zero-Shot Visualization framework that automates character portrait initialization and supports iterative portrait refinement by exploring the semantic content of the story. In particular, we design an LLM-based portrait generation strategy that matches the story characters with external movie characters, and leverage the matched resources as in-context learning (ICL) samples for LLMs to accurately initialize the character portraits. We then propose a graph-based Text2Image diffusion model that constructs a character interaction graph from the story to iteratively refine the character portraits by maximizing the distinctness of different characters while minimizing their incompatibility in the multi-character story visualization. Our evaluation results show that LeMon outperforms existing ZSV approaches in generating high-quality visualizations for stories across various types with multiple interacted characters. Our code is available at https://github.com/arxrean/LLM-LeMon.

Despite their capacity to convey knowledge, most existing knowledge graphs (KGs) are created for specific domains using low-resource data sources, especially those in non-global languages, and thus unavoidably suffer from the incompleteness problem. The automatic discovery of missing triples for KG completion is thus hindered by the challenging long-tail relations problem in low-resource KGs. Few-shot learning models trained on rich-resource KGs are unable to tackle this challenge due to a lack of generalization. To alleviate the impact of the intractable long-tail problem on low-resource KG completion, in this paper, we propose a novel few-shot learning framework empowered by multi-view task representation generation. The framework consists of four components, i.e., few-shot learner, perturbed few-shot learner, relation knowledge distiller, and pairwise contrastive distiller. The key idea is to utilize the different views of each few-shot task to improve and regulate the training of the few-shot learner. For each few-shot task, instead of augmenting it by complicated task designs, we generate its representation of different views using the relation knowledge distiller and perturbed few-shot learner, which are obtained by distilling knowledge from a KG encoder and perturbing the few-shot learner. Then, the generated representation of different views is utilized by the pairwise contrastive distiller based on a teacher-student framework to distill the knowledge of how to represent relations from different views into the few-shot learner and facilitate few-shot learning. Extensive experiments conducted on several real-world low-resource KGs validate the effectiveness of our proposed method.

Users may demand recommendations with highly personalized requirements involving logical operations, e.g., the intersection of two requirements, where such requirements naturally form structured logical queries on knowledge graphs (KGs). To date, existing recommender systems lack the capability to tackle users' complex logical requirements. In this work, we formulate the problem of recommendation with users' logical requirements (LogicRec) and construct benchmark datasets for LogicRec. Furthermore, we propose an initial solution for LogicRec based on logical requirement retrieval and user preference retrieval, where we face two challenges. First, KGs are incomplete in nature. Therefore, there are always missing true facts, which entails that the answers to logical requirements can not be completely found in KGs. In this case, item selection based on the answers to logical queries is not applicable. We thus resort to logical query embedding (LQE) to jointly infer missing facts and retrieve items based on logical requirements. Second, answer sets are under-exploited. Existing LQE methods can only deal with query-answer pairs, where queries in our case are the intersected user preferences and logical requirements. However, the logical requirements and user preferences have different answer sets, offering us richer knowledge about the requirements and preferences by providing requirement-item and preference-item pairs. Thus, we design a multi-task knowledge-sharing mechanism to exploit these answer sets collectively. Extensive experimental results demonstrate the significance of the LogicRec task and the effectiveness of our proposed method.

Entity synonym discovery (ESD) from text corpus is an essential problem in many entity-leveraging applications, e.g., web search and question answering. This paper aims to address three limitations that widely exist in the current ESD solutions: 1) the lack of effective utilization for synonym set information; 2) the feature extraction of entities from restricted receptive fields; and 3) the incapacity to capture higher-order contextual information. We propose a novel set-aware ESD model that enables a flexible receptive field for ESD by making a breakthrough in using entity synonym set information. The contextual information of entities and entity synonym sets are arranged by a two-level network from which entities and entity synonym sets can be mapped into the same embedding space to facilitate ESD by encoding the high-order contexts from flexible receptive fields. Extensive experimental results on public datasets show that our model consistently outperforms the state-of-the-art with significant improvement.

Pairwise ranking models have been widely used to address recommendation problems. The basic idea is to learn the rank of users' preferred items through separating items into positive samples if user-item interactions exist, and negative samples otherwise. Due to the limited number of observable interactions, pairwise ranking models face serious class-imbalance issues. Our theoretical analysis shows that current sampling-based methods cause the vertex-level imbalance problem, which makes the norm of learned item embeddings towards infinite after a certain training iterations, and consequently results in vanishing gradient and affects the model inference results. We thus propose an efficient Vital Negative Sampler (VINS) to alleviate the class-imbalance issue for pairwise ranking model, in particular for deep learning models optimized by gradient methods. The core of VINS is a bias sampler with reject probability that will tend to accept a negative candidate with a larger degree weight than the given positive item. Evaluation results on several real datasets demonstrate that the proposed sampling method speeds up the training procedure 30% to 50% for ranking models ranging from shallow to deep, while maintaining and even improving the quality of ranking results in top-N item recommendations.

Entity synonym discovery (ESD) from text corpus is an essential problem in many entity-leveraging applications. This paper aims to address three limitations that widely exist in the current ESD solutions: 1) the lack of effective utilization for synonym set information; 2) the feature extraction of entities from restricted receptive fields; and 3) the incapacity to capture higher-order contextual information. We propose a novel set-aware ESD model that enables a flexible receptive field for ESD by using entity synonym set information and constructing a two-level network. Extensive experimental results on public datasets show that our model consistently outperforms the state-of-the-art with significant improvement.

Recent years have witnessed increasing attention on the application of graph alignment to on-Web tasks, such as knowledge graph integration and social network linking. Despite achieving remarkable performance, prevailing graph alignment models still suffer from noisy supervision, yet how to mitigate the impact of noise in labeled data is still under-explored. The negative sampling based noise discrimination model has been a feasible solution to detect the noisy data and filter them out. However, due to its sensitivity to the sampling distribution, the negative sampling based noise discrimination model would lead to an inaccurate decision boundary. Furthermore, it is difficult to find an abiding threshold to separate the potential positive (benign) and negative (noisy) data in the whole training process. To address these important issues, in this paper, we design a non-sampling discrimination model resorting to the unbiased risk estimation of positive-unlabeled learning to circumvent the harmful impact of negative sampling. We also propose to select the appropriate potential positive data at different training stages by an adaptive filtration threshold enabled by curriculum learning, for maximally improving the performance of alignment model and non-sampling discrimination model. Extensive experiments conducted on several real-world datasets validate the effectiveness of our proposed method.

In this paper, we design and implement a generic medical knowledge based system (MKBS) for identifying diseases from several symptoms. In this system, some important aspects like knowledge bases system, knowledge representation, inference engine have been addressed. The system asks users different questions and inference engines will use the certainty factor to prune out low possible solutions. The proposed disease diagnosis system also uses a graphical user interface (GUI) to facilitate users to interact with the expert system. Our expert system is generic and flexible, which can be integrated with any rule bases system in disease diagnosis.

Cross-lingual entity alignment aims at associating semantically similar entities in knowledge graphs with different languages. It has been an essential research problem for knowledge integration and knowledge graph connection, and been studied with supervised or semi-supervised machine learning methods with the assumption of clean labeled data. However, labels from human annotations often include errors, which can largely affect the alignment results. We thus aim to formulate and explore the robust entity alignment problem, which is non-trivial, due to the deficiency of noisy labels. Our proposed method named REA (Robust Entity Alignment) consists of two components: noise detection and noise-aware entity alignment. The noise detection is designed by following the adversarial training principle. The noise-aware entity alignment is devised by leveraging graph neural network based knowledge graph encoder as the core. In order to mutually boost the performance of the two components, we propose a unified reinforced training strategy to combine them. To evaluate our REA method, we conduct extensive experiments on several real-world datasets. The experimental results demonstrate the effectiveness of our proposed method and also show that our model consistently outperforms the state-of-the-art methods with significant improvement on alignment accuracy in the noise-involved scenario.

Entity alignment associates entities in different knowledge graphs if they are semantically same, and has been successfully used in the knowledge graph construction and connection. Most of the recent solutions for entity alignment are based on knowledge graph embedding, which maps knowledge entities in a low-dimension space where entities are connected with the guidance of prior aligned entity pairs. The study in this paper focuses on two important issues that limit the accuracy of current entity alignment solutions: 1) labeled data of priorly aligned entity pairs are difficult and expensive to acquire, whereas abundant of unlabeled data are not used; and 2) knowledge graph embedding is affected by entity's degree difference, which brings challenges to align high frequent and low frequent entities. We propose a semi-supervised entity alignment method (SEA) to leverage both labeled entities and the abundant unlabeled entity information for the alignment. Furthermore, we improve the knowledge graph embedding with awareness of the degree difference by performing the adversarial training. To evaluate our proposed model, we conduct extensive experiments on real-world datasets. The experimental results show that our model consistently outperforms the state-of-the-art methods with significant improvement on alignment accuracy.