Current knowledge distillation methods typically require significant computational resources and time to train task-specific teacher candidates from scratch and identify the optimal teacher. Although self-distillation methods eliminate the dependency on the teacher by allowing the student model to learn independently, they face two challenges: the student learns correct and incorrect knowledge indiscriminately, and the student's learning scope is limited due to the lack of external teacher supervision. Spurred by these deficiencies, this work proposes a CLIP-enhanced Self-Distillation (CLIP-SD) method to overcome these problems, while almost not increasing training time. CLIP-SD comprises two main components: Prediction-oriented Self-Distillation (PSD) and Two-stage Task-guided CLIP Distillation (TTCD). PSD tackles the first challenge by assigning higher and lower weights to correct and incorrect prediction samples, respectively, during self-distillation. This component forces the student to focus on correct knowledge and minimize the impact of incorrect knowledge. Regarding the second challenge, the robust CLIP model is directly introduced into self-distillation. However, CLIP lacks task-specific knowledge and its output is overly smooth during the distillation process, prohibiting the student from learning more effectively. Therefore, TTCD refines CLIP's output through a two-stage process, endowing it with task-specific knowledge to enhance student learning. Experimental results indicate that CLIP-SD significantly improves distillation performance while maintaining training efficiency comparable to self-distillation. Specifically, on the CIFAR-100 dataset, the performance of CLIP-SD reaches 72.48% when trained with ResNet20 as the student model, which is an average improvement of 2.54% and 1.12% over the knowledge distillation and self-distillation methods. Regarding training time, CLIP-SD takes 3.91 hours, an average decrease of 2.73 hours compared to knowledge distillation and an average increase of 0.45 hours compared to self-distillation. Despite the slight increase in training time compared to self-distillation, the overhead is worthwhile and negligible considering its performance improvement.

Knowledge distillation (KD), as an effective compression technology, is used to reduce the resource consumption of graph neural networks (GNNs) and facilitate their deployment on resource-constrained devices. Numerous studies exist on GNN distillation, and however, the impacts of knowledge complexity and differences in learning behavior between teachers and students on distillation efficiency remain underexplored. We propose a KD method for fine-grained learning behavior (FLB), comprising two main components: feature knowledge decoupling (FKD) and teacher learning behavior guidance (TLBG). Specifically, FKD decouples the intermediate-layer features of the student network into two types: teacher-related features (TRFs) and downstream features (DFs), enhancing knowledge comprehension and learning efficiency by guiding the student to simultaneously focus on these features. TLBG maps the teacher model's learning behaviors to provide reliable guidance for correcting deviations in student learning. Extensive experiments across eight datasets and 12 baseline frameworks demonstrate that FLB significantly enhances the performance and robustness of student GNNs within the original framework.

Much effort has been devoted to building multi-modal knowledge graphs by visualizing entities on images, but ignoring the multi-modal information of the relation between entities. Hence, in this paper, we aim to construct a new large-scale multi-modal knowledge graph with triplet facts grounded on images that reflect not only entities but also their relations. To achieve this purpose, we propose a novel pipeline method, including triplet fact filtering, image retrieving, entity-based image filtering, relation-based image filtering, and image clustering. In this way, a multi-modal knowledge graph named ImgFact is constructed, which contains 247,732 triplet facts and 3,730,805 images. In experiments, the manual and automatic evaluations prove the reliable quality of our ImgFact. We further use the obtained images to enhance model performance on two tasks. In particular, the model optimized by our ImgFact achieves an impressive 8.38% and 9.87% improvement over the solutions enhanced by an existing multi-modal knowledge graph and VisualChatGPT on F1 of relation classification. We release ImgFact and its instructions at https://github.com/kleinercubs/ImgFact.

Quantities are distinct and critical components of texts that characterize the magnitude properties of entities, providing a precise perspective for the understanding of natural language, especially for reasoning tasks. In recent years, there has been a flurry of research on reasoning tasks based on large language models (LLMs), most of which solely focus on numerical values, neglecting the dimensional concept of quantities with units despite its importance. We argue that the concept of dimension is essential for precisely understanding quantities and of great significance for LLMs to perform quantitative reasoning. However, the lack of dimension knowledge and quantity-related benchmarks has resulted in low performance of LLMs. Hence, we present a framework to enhance the quantitative reasoning ability of language models based on dimension perception. We first construct a dimensional unit knowledge base (DimUnitKB) to address the knowledge gap in this area. We propose a benchmark DimEval consisting of seven tasks of three categories to probe and enhance the dimension perception skills of LLMs. To evaluate the effectiveness of our methods, we propose a quantitative reasoning task and conduct experiments. The experimental results show that our dimension perception method dramatically improves accuracy (43.55%→50.67%) on quantitative reasoning tasks compared to GPT-4.

The research on group behavior is effective for establishing a good network environment since people in social networks tend to form groups spontaneously. Most studies on group behavior on online social networks assume that all individuals are reduced to one cluster, ignoring the existence of potential clusters and their importance in group opinion dynamics. This article introduces a novel group-gravitational field (GGF) model to investigate the opinion evolution based on group behavior by the following aspects: 1) the GGF model reduces a cluster in the social network into a charge and the whole network into a gravitational field; 2) the GGF model calculates the initial influence of a cluster according to the topology information and further constructs a gravity matrix of the network based on the Coulomb law; and 3) opinion-leader clusters exert the internal field force on common opinion clusters inside the gravitational field. The GGF model simulates the evolution of opinions among clusters in a network and studies the law of group behavior according to the influence between clusters based on Coulomb’s law. Experiments on real social networks verify that the GGF model enhances the speed of opinion evolution significantly. The simulation experiments indicate that the existence of clusters promotes the rapid convergence of opinions, a gathering of followers influences information dissemination in social networks, and the GGF model fits the reality better. This article provides a new approach to network supervision and control.

The recommendation of long-tail items has been a persistent issue in recommender system research. The primary reason for this problem is that the model cannot learn better item features due to the lack of interactive record data of tail items, which leads to a decline in the model's recommendation performance. Existing methods transfer the features of the head items to the tail items, thereby ignoring their differences and failing to produce a satisfactory recommendation effect. To address the issue, we propose a novel recommendation model called MetaGA based on metalearning. The MetaGA model obtains initial parameters from head items through metalearning and fine-tunes model parameters during the learning process of tail item features. Additionally, it employs a graph convolutional network and attention mechanism to enhance tail data and reduce the difference between head and tail data. Through the above two steps, the model utilizes the abundant data of the head items to address the problem of sparse data of the tail items, resulting in improved recommendation performance. We conducted extensive experiments on three real-world datasets, and the results demonstrate that our proposed MetaGA model significantly outperforms other state-of-the-art baselines for tail item recommendation.

Knowledge distillation (KD) is a technique that transfers “dark knowledge” from a deep teacher network (teacher) to a shallow student network (student). Despite significant advances in KD, existing work has not adequately mined two crucial types of knowledge: 1) the knowledge of head categories, which represents the relationship between the target category and its similar categories. Our findings reveal that this highly similar (complex) knowledge is essential for improving student’s performance; and 2) the effectively utilized knowledge of tail categories. Existing studies often treat the non-target categories collectively without sufficiently considering the effectiveness of knowledge from tail categories. To tackle these challenges, we reformulate classical KD (ReKD) into two components: Top-KInter-class Similar Distillation (TISD) and Non-Top-KInter-class Discriminability (NTID). Firstly, TISD captures and imparts the knowledge of head categories to the student. Our experimental results have verified that TISD is particularly effective in transferring the knowledge of head categories, even in fine-grained dataset classification. Secondly, we theoretically show that the weighting coefficient of NTID increases with the probability of Top-K, leading to stronger suppression of knowledge transfer for tail categories. This observation explains why difficult samples are more informative than simple ones. To better utilize both types of knowledge, we optimize both TISD and NTID using different weighting coefficients, thereby enhancing the student’s ability to learn this valuable knowledge from both head and tail categories. Furthermore, our extensive experimental results demonstrate that ReKD achieves state-of-the-art performance on various image classification datasets, including CIFAR-100, Tiny-ImageNet, and ImageNet-1K, as well as object detection and instance segmentation using the MS-COCO dataset.

Knowledge distillation, a widely adopted model compression technique, distils knowledge from a large teacher model to a smaller student model, with the goal of reducing the computational resources required for the student model. However, most existing distillation approaches focus on the types of knowledge and how to distil them, which neglect the student model's neuronal responses to the knowledge. In this article, we demonstrate that the kullback-leibler loss inhibits the neuronal responses in the opposite gradient direction, which injures the student model's potential during distilling. To address this problem, we introduce a principled dual-stage distillation scheme to rejuvenate all inhibited neurons at the neuronal level. In the first stage, we detect all the neurons in the student model during the standard distillation period and divide them into two parts according to their responses. In the second stage, we propose three strategies to resuscitate the neurons differently, which allows us to exploit the full potential of the student model. Through the experiments in various aspects of knowledge distillation, it is verified that the proposed approach outperforms the current state-of-the-art approaches. Our work provides a neuronal perspective for studying the response of the student model to the knowledge from the teacher model.

Similes play an imperative role in creative writing such as story and dialogue generation.Proper evaluation metrics are like a beacon guiding the research of simile generation (SG).However, it remains under-explored as to what criteria should be considered, how to quantify each criterion into metrics, and whether the metrics are effective for comprehensive, efficient, and reliable SG evaluation.To address the issues, we establish HAUSER, a holistic and automatic evaluation system for the SG task, which consists of five criteria from three perspectives and automatic metrics for each criterion.Through extensive experiments, we verify that our metrics are significantly more correlated with human ratings from each perspective compared with prior automatic metrics.Resources of HAUSER are publicly available at https://github.com/Abbey4799/HAUSER.

Humor understanding is an important and challenging research in natural language processing. As the popularity of pre-trained language models (PLMs), some recent work makes preliminary attempts to adopt PLMs for humor recognition and generation. However, these simple attempts do not substantially answer the question: whether PLMs are capable of humor understanding? This paper is the first work that systematically investigates the humor understanding ability of PLMs. For this purpose, a comprehensive framework with three evaluation steps and four evaluation tasks is designed. We also construct a comprehensive Chinese humor dataset, which can fully meet all the data requirements of the proposed evaluation framework. Our empirical study on the Chinese humor dataset yields some valuable observations, which are of great guiding value for future optimization of PLMs in humor understanding and generation.

Convolutional neural networks (CNNs) have demonstrated excellent capability in various visual recognition tasks but impose an excessive computational burden. The latter problem is commonly solved by utilizing lightweight sparse networks. However, such networks have a limited receptive field in a few layers, and the majority of these networks face a severe information barrage due to their sparse structures. Spurred by these deficiencies, this work proposes a Squeeze Convolution block with Information Flow Enhancement (SCIFE), comprising a Divide-and-Squeeze Convolution and an Information Flow Enhancement scheme. The former module constructs a multi-layer structure through multiple squeeze operations to increase the receptive field and reduce computation. The latter replaces the affine transformation with the point convolution and dynamically adjusts the activation function’s threshold, enhancing information flow in both channels and layers. Moreover, we reveal that the original affine transformation may harm the network’s generalization capability. To overcome this issue, we utilize a point convolution with a zero initial mean. SCIFE can serve as a plug-and-play replacement for vanilla convolution blocks in mainstream CNNs, while extensive experimental results demonstrate that CNNs equipped with SCIFE compress benchmark structures without sacrificing performance, outperforming their competitors.

Graph convolutional network (GCN)-based recommendation has recently attracted significant attention in the recommender system community. Although current studies propose various GCNs to improve recommendation performance, existing methods suffer from two main limitations. First, user–item interaction data is generally sparse in practice, highlighting these methods' ineffectiveness in learning user and item feature representations. Second, they usually perform a dot-product operation to model and calculate user preferences on items, leading to inaccurate user preference learning. To address these limitations, this study adopts a design idea that sharply differs from existing works. Specifically, we introduce the knowledge distillation concept into GCN-based recommendation and propose a two-phase knowledge distillation model (TKDM) improving recommendation performance. In Phase I, a self-distillation method on a graph auto-encoder learns the user and item feature representations. This auto-encoder employs a simple two-layer GCN as an encoder and a fully connected layer as a decoder. On this basis, in Phase II, a mutual-distillation method on a fully connected layer is introduced to learn user preferences on items with triple-based Bayesian personalized ranking. Extensive experiments on three real-world data sets demonstrate that TKDM outperforms classic and state-of-the-art methods related to GCN-based recommendation problems.

Intelligent video surveillance is an important computer vision application in natural environments. Since detected objects under surveillance are usually low-resolution and noisy, their accurate recognition represents a huge challenge. Knowledge distillation is an effective method to deal with it, but existing related work usually focuses on reducing the channel count of a student network, not feature map size. As a result, they cannot transfer "privilege information" hidden in feature maps of a wide and deep teacher network into a thin and shallow student one, leading to the latter's poor performance. To address this issue, we propose a Feature Map Distillation (FMD) framework under which the feature map size of teacher and student networks is different. FMD consists of two main components: Feature Decoder Distillation (FDD) and Feature Map Consistency-enforcement (FMC). FDD reconstructs the shallow texture features of a thin student network to approximate the corresponding samples in a teacher network, which allows the high-resolution ones to directly guide the learning of the shallow features of the student network. FMC makes the size and direction of each deep feature map consistent between student and teacher networks, which constrains each pair of feature maps to produce the same feature distribution. FDD and FMC allow a thin student network to learn rich "privilege information" in feature maps of a wide teacher network. The overall performance of FMD is verified in multiple recognition tasks by comparing it with state-of-the-art knowledge distillation methods on low-resolution and noisy objects.

Group recommendation has recently drawn a lot of attention to the recommender system community. Currently, several deep learning-based approaches are leveraged to learn preferences of groups for items and predict next items in which groups may be interested. Yet, their recommendation performance is still unsatisfactory due to sparse group–item interactions. To address this challenge, this study presents a novel model, called group recommendation model with two-stage deep learning (GRMTDL), which encompasses two sequential stages: 1) group representation learning (GRL) and 2) group preference learning (GPL). In GRL, we first construct an undirected tripartite graph over group–user–item interactions, and then employ it to accurately learn group semantic features through a spatial-based variational graph autoencoder network. While in GPL, we first introduce a dual PL-network that contains two structure-sharing subnetworks: 1) group PL-network employed for GPL and 2) user PL-network utilized for user preference learning. Then, we design a novel layered transfer learning (LTL) method to learn group preferences by alternately optimizing these two subnetworks. In particular, it can effectively absorb knowledge of user preferences into the process of GPL. Furthermore, extensive experiments on four real-world datasets demonstrate that the proposed GRMTDL model outperforms the state-of-the-art baselines for group recommendation.

Lexical taxonomies, a special kind of knowledge graph, are essential for natural language understanding. This paper studies the problem of lexical taxonomy embedding. Most existing graph embedding methods are difficult to apply to lexical taxonomies since 1) they ignore implicit but important information, namely, sibling relations, which are not explicitly mentioned in lexical taxonomies and 2) there are lots of polysemous terms in lexical taxonomies. In this paper, we propose a novel method for lexical taxonomy embedding. This method optimizes an objective function that models both hyponym-hypernym relations and sibling relations. A term-level attention mechanism and a random walk based metric are then proposed to assist the modeling of these two kinds of relations, respectively. Finally, a novel training method based on curriculum learning is proposed. We conduct extensive experiments on two tasks to show that our approach outperforms other embedding methods and we use the learned term embeddings to enhance the performance of the state-of-the-art models that are based on BERT and RoBERTa on text classification.

Location-based recommendation has recently received a lot of attention in the communities of information service and mobile application. Its task is to provide personalized recommendations of points of interest (POIs) to users at a certain time and location. However, existing location-based recommendation models have at least two main drawbacks: first they cannot adequately capture semantic features of POIs and users, which may lead to unsatisfactory recommendations and second they cannot effectively address the cold-start problem. To address the above drawbacks, in this article, we first propose a novel deep representation learning-based model (DRLM) for improving the recommendation accuracy. In DRLM, we mainly focus on learning to accurately represent semantic features of POIs and users. Specifically, four co-occurrence matrices are constructed to produce four different original features for each POI, and a principal component analysis (PCA) algorithm is utilized to generate a semantic feature of each POI from its four original features. On the other hand, a three-modal simple recurrent unit (TMSRU) network is given to constructed semantic features of users using semantic features of POIs, times, and locations. We further propose minimum description length (MDL)-based and skyline-based strategies to address the cold-start issues for new users and new POIs, respectively. Through experiments on two real-world data sets, we show that compared with the state-of-the-art approaches, the proposed model DRLM can achieve the superior performance in terms of high recommendation accuracy and effectiveness in handling the cold-start problem.

In recent years, group recommendation has become a research hotspot and focus in online social network community. Currently, several deep-learning-based approaches are leveraged to learn preferences of groups for items and predict the next items in which groups may be interested. Yet, their recommendation performance is still unsatisfactory due to the sparse group-item interactions. In order to address this problem, in this article, we introduce an effective model, namely Social Group Recommendation model with TrAdaBoost (SGRTAB), to raise the performance of group recommendation in online social networks. The SGRTAB model includes two stages: data preprocessing (DP) and model optimization (MO). In DP, SGRTAB produces inputs for MO and implements three related tasks: extracting individual features, handling group data via GloVe, and utilizing user contribute ratings to their own groups, whereas in MO, SGRTAB implements group preference learning with the assistance of user preference learning based on the TrAdaBoost algorithm. Specifically, SGRTAB can effectively absorb the knowledge of user preferences into the process of group preference learning through the idea of transferring-ensemble learning. Moreover, extensive experiments on four real-world data sets indicate that the proposed SGRTAB model significantly outperforms the state-of-the-art baselines for social group recommendation.

Using large-scale GPS trajectory data to improve taxi services has recently attracted much attention in Internet of Things and smart city communities. In this paper, we use a large-scale GPS trajectory dataset generated by over 12 000 taxis in a period of three months in Shanghai, China, and present an efficient passenger-hunting recommendation framework with the multitask deep learning paradigm. This framework contains two modules: 1) offline training of passenger-hunting recommendation model (OT-PHRM) and 2) online application of passenger-hunting recommendation model (OA-PHRM). The module OT-PHRM mainly includes two deep convolutional neural networks (DCNNs) and uses the multitask learning strategy. The first DCNN realizes the region prediction for picking up passengers, while the second DCNN uses the weight-sharing structure to predict the levels of road congestion and earnings of carrying passengers. In particular, for the input of two DCNNs, we not only consider contextual features of taxi driving, region features and valuable statistical features, but also combine individual features into meaningful ones. In the module OA-PHRM, we propose DL-PHRec, which calculates three prediction values using two trained DCNNs in OT-PHRM in real time, and then recommends a personal ranking-list of regions to each taxi driver according to their scores. The experimental results show the feasibility and effectiveness of our recommendation framework.