Kinship verification aims to determine whether two individuals share a familial relationship based on facial information. Cross-gender relationships (i.e., Father-Daughter and Mother-Son) continue to face formidable challenges due to the diversity and uncertainty of genetic inheritance. Existing studies primarily focus on extracting robust features and measuring similarity, with limited attention given to the fuzziness of gender differences. To address this issue, this paper proposes a kinship verification framework based on a fuzzy neural network, which adaptively extracts gender-independent kinship features and handles relationship fuzziness to improve cross-gender verification performance. Specifically, the Swin Transformer, which has demonstrated excellent performance in facial analysis, is employed to extract initial features. A fuzzy neural network is then designed to disentangle gender and kinship features, with a gender recognition task introduced to further enhance this disentanglement and improve the gender independence of kinship features. Subsequently, a multi-metric fuzzy reasoning module is adopted to integrate kinship features, extract latent kinship cues, and leverage a contrastive loss function to effectively mine potential negative sample information, thereby significantly enhancing the model's robustness. Experimental results on three publicly available datasets demonstrate that the proposed method achieves state-of-the-art performance.

Federated domain generalization (FedDG) aims to improve the global model’s generalization ability in unseen domains by addressing data heterogeneity under privacy-preserving constraints. A common strategy in existing FedDG studies involves sharing domain-specific knowledge among clients, such as spectrum information, class prototypes, and data styles. However, this knowledge is extracted directly from local client samples, and sharing such sensitive information poses a potential risk of data leakage, which might not fully meet the FedDG requirements. In this paper, we introduce prompt learning to adapt pretrained vision-language models (VLMs) in the FedDG scenario, and leverage locally learned prompts as a more secure bridge to facilitate knowledge transfer among clients. Specifically, we propose a novel FedDG framework through Prompt Learning and AggregatioN (PLAN), which comprises two training stages to collaboratively generate local prompts and global prompts at each federated round. First, each client performs both text and visual prompt learning using their own data, with local prompts indirectly synchronized by regarding the global prompts as a common reference. Second, all domain-specific local prompts are exchanged among clients and selectively aggregated into global prompts using lightweight attention-based aggregators. The global prompts are finally applied to adapt the VLMs to unseen target domains. As our PLAN framework requires training only a limited number of prompts and lightweight aggregators, it offers notable advantages in terms of computational and communication efficiency for FedDG. Extensive experiments demonstrate the superior generalization ability of PLAN across four benchmark datasets. We have released our code at https://github.com/GongShuai8210/PLAN.

Federated Domain Generalization (FedDG) aims to train a globally generalizable model on data from decentralized, heterogeneous clients. While recent work has adapted vision-language models for FedDG using prompt learning, the prevailing "one-prompt-fits-all" paradigm struggles with sample diversity, causing a marked performance decline on personalized samples. The Mixture of Experts (MoE) architecture offers a promising solution for specialization. However, existing MoE-based prompt learning methods suffer from two key limitations: coarse image-level expert assignment and high communication costs from parameterized routers. To address these limitations, we propose TRIP, a Token-level pRompt mIxture with Parameter-free routing framework for FedDG. TRIP treats prompts as multiple experts, and assigns individual tokens within an image to distinct experts, facilitating the capture of fine-grained visual patterns. To ensure communication efficiency, TRIP introduces a parameter-free routing mechanism based on capacity-aware clustering and Optimal Transport (OT). First, tokens are grouped into capacity-aware clusters to ensure balanced workloads. These clusters are then assigned to experts via OT, stabilized by mapping cluster centroids to static, non-learnable keys. The final instance-specific prompt is synthesized by aggregating experts, weighted by the number of tokens assigned to each. Extensive experiments across four benchmarks demonstrate that TRIP achieves optimal generalization results, with communicating as few as 1K parameters. Our code is available at https://github.com/GongShuai8210/TRIP.

Test-time prompt tuning (TPT) aims to adjust the vision-language models (e.g., CLIP) with learnable prompts during the inference phase. However, previous works overlooked that pre-trained models as a service (MaaS) have become a noticeable trend due to their commercial usage and potential risk of misuse. In the context of MaaS, users can only design prompts in inputs and query the black-box vision-language models through inference APIs, rendering the previous paradigm of utilizing gradient for prompt tuning is infeasible. In this paper, we propose black-box test-time prompt tuning (B²TPT), a novel framework that addresses the challenge of optimizing prompts without gradients in an unsupervised manner. Specifically, B²TPT designs a consistent or confident (CoC) pseudo-labeling strategy to generate high-quality pseudo-labels from the outputs. Subsequently, we propose to optimize low-dimensional intrinsic prompts using a derivative-free evolution algorithm and to project them onto the original text and vision prompts. This strategy addresses the gradient-free challenge while reducing complexity. Extensive experiments across 15 datasets demonstrate the superiority of B²TPT. The results show that B²TPT not only outperforms CLIP's zero-shot inference at test time, but also surpasses other gradient-based TPT methods.

Cross-topic automated essay scoring (AES) aims to develop a transferable model capable of effectively evaluating essays on a target topic. A significant challenge in this domain arises from the inherent discrepancies between topics. While existing methods predominantly focus on extracting topic-shared features through distribution alignment of source and target topics, they often neglect topic-specific features, limiting their ability to assess critical traits such as topic adherence. To address this limitation, we propose an Adversarial TOpic-aware Prompt-tuning (ATOP), a novel method that jointly learns topic-shared and topic-specific features to improve cross-topic AES. ATOP achieves this by optimizing a learnable topic-aware prompt—comprising both shared and specific components—to elicit relevant knowledge from pre-trained language models (PLMs). To enhance the robustness of topic-shared prompt learning and mitigate feature scale sensitivity introduced by topic alignment, we incorporate adversarial training within a unified regression and classification framework. In addition, we employ a neighbor-based classifier to model the local structure of essay representations and generate pseudo-labels for target-topic essays. These pseudo-labels are then used to guide the supervised learning of topic-specific prompts tailored to the target topic. Extensive experiments on the publicly available ASAP++ dataset demonstrate that ATOP significantly outperforms existing state-of-the-art methods in both holistic and multi-trait essay scoring. The implementation of our method is publicly available at: https://github.com/zhaohy777/ATOP.

The rapid increase in encrypted network traffic has made detecting malicious activities a critical challenge in network management, attracting significant research attention. However, most existing methods focus primarily on flow-based features, often neglecting the inherent structural heterogeneity and dynamic temporal variations in encrypted traffic, which limits their effectiveness in capturing the evolving nature of malicious activities. To address the aforementioned issue, we present a novel Dynamic Heterogeneous Information Network framework, DMT-DHIN, for Detecting Malicious Encrypted Traffic. DMT-DHIN constructs dynamic heterogeneous graphs by partitioning traffic into time slices, with nodes representing network entities (e.g., packets, protocols, IPs, and ports) and edges capturing their interactions. A heterogeneous graph attention mechanism effectively captures spatial dependencies among different node types. At the same time, the Transformer captures the temporal evolution of node features, enabling DMT-DHIN to identify dynamic patterns in encrypted traffic. We validate the effectiveness of DMT-DHIN by conducting evaluations on three datasets: CICIDS2017, USTC-TFC2016, and CICIoT2023. The results demonstrate that DMT-DHIN surpasses existing state-of-the-art methods, delivering marked enhancements in detection accuracy and F1-score, highlighting its superior ability to identify malicious encrypted traffic.

Unsupervised domain adaptation (UDA) aims to adapt models learned from a well-annotated source domain to a target domain, where only unlabeled samples are available. To this end, adversarial training is widely used in conventional UDA methods to reduce the discrepancy between source and target domains. Recently, prompt tuning has emerged as an efficient way to adapt large pre-trained vision-language models like CLIP to a variety of downstream tasks. In this paper, we present a novel method named Adversarial DuAl Prompt Tuning (ADAPT) for UDA, which employs text prompts and visual prompts to guide CLIP simultaneously. Rather than simply performing a joint optimization of text prompts and visual prompts, we integrate text prompt tuning and visual prompt tuning into a collaborative framework where they engage in an adversarial game: text prompt tuning focuses on distinguishing between source and target images, whereas visual prompt tuning seeks to align source and target domains. Unlike most existing adversarial training-based UDA approaches, ADAPT does not require explicit domain discriminators for domain alignment. Instead, the objective is effectively achieved at both global and category levels through modeling the joint probability distribution of images on domains and categories. Extensive experiments on four benchmark datasets demonstrate the effectiveness of our ADAPT method for UDA. We have released our code at https://github.com/Liuziyi1999/ADAPT.

Knowledge tracing (KT) aims to monitor students' evolving knowledge states through their learning interactions with concept-related questions, and can be indirectly evaluated by predicting how students will perform on future questions. In this paper, we observe that there is a common phenomenon of answer bias, i.e., a highly unbalanced distribution of correct and incorrect answers for each question. Existing models tend to memorize the answer bias as a shortcut for achieving high prediction performance in KT, thereby failing to fully understand students' knowledge states. To address this issue, we approach the KT task from a causality perspective. A causal graph of KT is first established, from which we identify that the impact of answer bias lies in the direct causal effect of questions on students' responses. A novel COunterfactual REasoning (CORE) framework for KT is further proposed, which separately captures the total causal effect and direct causal effect during training, and mitigates answer bias by subtracting the latter from the former in testing. The CORE framework is applicable to various existing KT models, and we implement it based on the prevailing DKT, DKVMN, and AKT models, respectively. Extensive experiments on three benchmark datasets demonstrate the effectiveness of CORE in making the debiased inference for KT. We have released our code at https://github.com/lucky7-code/CORE.

Unsupervised domain adaptation aims to transfer the knowledge learned from a labeled source domain to an unlabeled target domain with different data distributions. However, in practice, source samples are not always available due to privacy protection and storage resource limitations. To address this concern, Source-Free Domain Adaptation (SFDA) has recently attracted growing research attention, as it only needs a pre-trained source model without direct access to source data. In this paper, we propose a novel Adversarial SOurce GEneration (ASOGE) method for SFDA, which introduces an additional generative module to produce synthetic labeled source samples and uses them to facilitate cross-domain adaptation. Unlike early studies that train the generator independently and perform the adaptation only after the generator is finished, ASOGE integrates the generation and adaptation stages within a collaborative framework by making them play an adversarial game. In the generation stage, the labeled source samples are not produced blindly; instead, they are hard-to-align samples that provide knowledge more worth learning for the adaptation stage. To achieve a fine-grained domain alignment, a class-aware discrepancy between source and target domains is measured via contrastive learning. Extensive experiments on benchmark datasets demonstrate the effectiveness of ASOGE compared to the state-of-the-art methods.

Academic performance prediction aims to leverage student-related information to predict their future academic outcomes, which is beneficial to numerous educational applications, such as personalized teaching and academic early warning. In this article, we reveal the students' behavior trajectories by mining campus smartcard records, and capture the characteristics inherent in trajectories for academic performance prediction. Particularly, we carefully design a tri-branch convolutional neural network (CNN) architecture, which is equipped with rowwise, columnwise, and depthwise convolutions and attention operations, to effectively capture the persistence, regularity, and temporal distribution of student behavior in an end-to-end manner, respectively. However, different from existing works mainly targeting at improving the prediction performance for the whole students, we propose to cast academic performance prediction as a top-k ranking problem, and introduce a top-k focused loss to ensure the accuracy of identifying academically at-risk students. Extensive experiments were carried out on a large-scale real-world dataset, and we show that our approach substantially outperforms recently proposed methods for academic performance prediction. For the sake of reproducibility, our codes have been released at https://github.com/ZongJ1111/Academic-Performance-Prediction.

Knowledge tracing aims to trace students’ evolving knowledge states by predicting their future performance on concept-related exercises. Recently, some graph-based models have been developed to incorporate the relationships between exercises to improve knowledge tracing, but only a single type of relationship information is generally explored. In this article, we present a novel Dual Graph Ensemble learning method for Knowledge Tracing (DGEKT), which establishes a dual graph structure of students’ learning interactions to capture the heterogeneous exercise–concept associations and interaction transitions by hypergraph modeling and directed graph modeling, respectively. To combine the dual graph models, we introduce the technique of online knowledge distillation. This choice arises from the observation that, while the knowledge tracing model is designed to predict students’ responses to the exercises related to different concepts, it is optimized merely with respect to the prediction accuracy on a single exercise at each step. With online knowledge distillation, the dual graph models are adaptively combined to form a stronger ensemble teacher model, which provides its predictions on all exercises as extra supervision for better modeling ability. In the experiments, we compare DGEKT against eight knowledge tracing baselines on three benchmark datasets, and the results demonstrate that DGEKT achieves state-of-the-art performance.

Stock portfolio selection is an important financial planning task that dynamically re-allocates the investments to stock assets to achieve the goals such as maximal profits and minimal risks. In this paper, we propose a hypergraph-based reinforcement learning method for stock portfolio selection, in which the fundamental issue is to learn a policy function generating appropriate trading actions given the current environments. The historical time-series patterns of stocks are firstly captured. Then, different from prior works ignoring or implicitly modeling stock pairwise correlations, we present a HyperGraph Attention Module (HGAM) in the portfolio policy learning, which utilizes the hypergraph structure to explicitly model the group-wise industry-belonging relationships among stocks. The attention mechanism is also introduced in HGAM that quantifies the importance of different neighbors regarding the target node to aggregate the information on the stock hypergraph adaptively. Extensive experiments on the real-world dataset collected from China’s A-share market demonstrate the significant superiority of our method, compared with state-of-the-art methods in portfolio selection, including both online learning-based methods and reinforcement learning-based methods. The data and codes of our work have been released at https://github.com/lixiaojieff/stock-portfolio.

Multi-task learning in Convolutional Neural Networks (CNNs) has led to remarkable success in a variety of applications of computer vision. Towards effective multi-task CNN architectures, recent studies automatically learn the optimal combinations of task-specific features at single network layers. However, they generally learn an unchanged operation of feature combination after training, regardless of the characteristic changes of task-specific features across different inputs. In this paper, we propose a novel Adaptive Feature Aggregation (AFA) layer for multi-task CNNs, in which a dynamic aggregation mechanism is designed to allow each task adaptively determines the degree to which the knowledge sharing or preserving between tasks is needed based on the characteristics of inputs. We introduce two types of aggregation modules to the AFA layer, which realize the adaptive feature aggregation by capturing the feature dependencies of different tasks along the channel and spatial axes, respectively. The AFA layer is a plug-and-play component with low parameter and computation overheads, and can be trained end-to-end along with backbone networks. For both pixel-level and image-level tasks, we empirically show that our approach substantially outperforms the previous state-of-the-art methods of multi-task CNNs. The code and models are available athttps://github.com/zhenshen-mla/AFANet.

Convolutional Neural Network (CNN) based multi-task learning methods have been widely used in a variety of applications of computer vision. Towards effective multi-task CNN architectures, recent studies automatically learn the optimal combinations of task-specific features at single network layers. However, they generally construct an unchanged operation of feature aggregation after training, regardless of the characteristics of input features. In this paper, we propose a novel Adaptive Feature Aggregation (AFA) layer for multi-task CNNs, in which a dynamic aggregation mechanism is designed to allow each task to adaptively determine the degree to which the feature aggregation of different tasks is needed according to the feature dependencies. On both pixel-level and image-level tasks, we demonstrate that our approach significantly outperforms the previous state-of-the-art methods of multi-task CNNs.

Student performance prediction aims to leverage student-related information to predict their future academic outcomes, which may be beneficial to numerous educational applications, such as personalized teaching and academic early warning. In this paper, we seek to address the problem by analyzing students' daily studying and living behavior, which is comprehensively recorded via campus smart cards. Different from previous studies, we propose an end-to-end student performance prediction model, namely Tri-branch CNN, which is equipped with three types of convolutional filters, i.e., the row-wise convolution, column-wise convolution, and group-wise convolution, to effectively capture the duration, periodicity, and location-aware characteristic of student behavior, respectively. We also introduce the attention mechanism and cost-sensitive learning strategy to further improve the accuracy of our approach. Extensive experiments on a large-scale real-world dataset demonstrate the potential of our approach for student performance prediction.

People used to take a series of nearly identical photos about the same subject, but it is usually a tedious chore to select the reversed ones from them. Despite the remarkable progress, most existing studies on image aesthetics assessment fail to fulfill the task of series photo selection. In this paper, we develop a novel deep CNN architecture that aggregates multi-scale features from different network layers, in order to capture the subtle differences between series photos. To reduce the risk of redundant or even interfering features, we introduce the spatial-channel self-attention mechanism to adaptively recalibrate the features at each layer, so that informative features can be selectively emphasized and less useful ones suppressed. Extensive experiments on a benchmark dataset well demonstrate the potential of our approach for series photo selection.

In non-stationary environments, learning machines usually confront the domain adaptation scenario where the data distribution does change over time. Previous domain adaptation works have achieved great success in theory and practice. However, they always lose robustness in noisy environments where the labels and features of examples from the source domain become corrupted. In this paper, we report our attempt towards achieving accurate noise-robust domain adaptation. We first give a theoretical analysis that reveals how harmful noises influence unsupervised domain adaptation. To eliminate the effect of label noise, we propose an offline curriculum learning for minimizing a newly-defined empirical source risk. To reduce the impact of feature noise, we propose a proxy distribution based margin discrepancy. We seamlessly transform our methods into an adversarial network that performs efficient joint optimization for them, successfully mitigating the negative influence from both data corruption and distribution shift. A series of empirical studies show that our algorithm remarkably outperforms state of the art, over 10% accuracy improvements in some domain adaptation tasks under noisy environments.

Multi-view hashing can well support large-scale near-duplicate video retrieval, due to its desirable advantages of mutual reinforcement of multiple features, low storage cost, and fast retrieval speed. However, there are still two limitations that impede its performance. First, existing methods only consider local structures in multiple features. They ignore the global structure that is important for near-duplicate video retrieval, and cannot fully exploit the dependence and complementarity of multiple features. Second, existing works always learn hashing functions bit by bit, which unfortunately increases the time complexity of hash function learning. In this paper, we propose a supervised hashing scheme, termed as joint multi-view hashing (JMVH), to address the aforementioned problems. It jointly preserves the global and local structures of multiple features while learning hashing functions efficiently. Specially, JMVH considers features of video as items, based on which an underlying Hamming space is learned by simultaneously preserving their local and global structures. In addition, a simple but efficient multi-bit hash function learning based on generalized eigenvalue decomposition is devised to learn multiple hash functions within a single step. It can significantly reduce the time complexity of conventional hash function learning processes that sequentially learn multiple hash functions bit by bit. The proposed JMVH is evaluated on two public databases: CC_WEB_VIDEO and UQ_VIDEO. Experimental results demonstrate that the proposed JMVH achieves more than a 5 percent improvement compared to several state-of-the-art methods which indicates the superior performance of JMVH.

Image aesthetics assessment aims to endow computers with the ability to judge the aesthetic values of images, and its potential has been recognized in a variety of applications. Most previous studies perform aesthetics assessment purely based on image content. However, given the fact that aesthetic perceiving is a human cognitive activity, it is necessary to consider users’ perception of an image when judging its aesthetic quality. In this article, we regard users’ social behavior as the reflection of their perception of images and harness these additional clues to improve image aesthetics assessment. Specifically, we first merge the raw social interactions between users and images into clusters as the social labels of images, so the collective social behavioral information associated with an image can be well represented over a structured and compact space. Then, we develop a novel deep multi-task network to jointly learn social labels in different modalities from social images and apply it to common web images. In this manner, our approach is readily generalized to web images without social behavioral information. Finally, we introduce a high-level fusion sub-network to the aesthetics model, in which the social and visual representations of images are well balanced for aesthetics assessment. Experimental results on two benchmark datasets well verify the effectiveness of our approach and highlight the benefits of different types of social behavioral information for image aesthetics assessment.

Aesthetics assessment and emotion recognition are two fundamental problems in user perception understanding. While the two tasks are correlated and mutually beneficial, they are usually solved separately in existing studies. In this paper, we resort to multi-task learning to deal with aesthetics assessment and emotion recognition for images in a unified framework. Towards this goal, we extend a large scale emotion dataset by further manually rating the aesthetic qualities of images. To our best knowledge, the new dataset is the first collection of images that are associated with both aesthetic and emotional labels. Besides, we present a novel Aesthetics-Emotion hybrid Network (AENet) for multi-task learning on aesthetics assessment and emotion recognition. Task-specific and shared features have been explicitly separated by different network streams, and effectively fused at multiple network levels. Experiments on our new and benchmark datasets verify the effectiveness of our approach for unified aesthetics and emotion prediction.

In the past few years, micro-videos have become the dominant trend in the social media era. Meanwhile, as the number of microvideos increases, users are frequently overwhelmed by their uninterested ones. Despite the success of existing recommendation systems developed for various communities, they cannot be applied to routing micro-videos, since users in micro-video platforms have their unique characteristics: diverse and dynamic interest, multilevel interest, as well as true negative samples. To address these problems, we present a temporal graph-guided recommendation system. In particular, we first design a novel graph-based sequential network to simultaneously model users' dynamic and diverse interest.Similarly, uninterested information can be captured from users'true negative samples. Beyond that, we introduce users' multi-level interest into our recommendation model via a user matrix that is able to learn the enhanced representation of users' interest. Finally, the system can make accurate recommendation by considering the above characteristics. Experimental results on two public datasets verify the effectiveness of our proposed model.

Image aesthetics assessment has emerged as a hot topic in recent years due to its potential in numerous high-level vision applications. In this paper, distinguished from existing studies relying on a single label, we propose quantifying image aesthetics by a distribution over multiple quality levels. The distribution-based representation characterizes the disagreement among users' aesthetic preferences regarding the same image, and is also compatible with the traditional task of aesthetic label prediction. Our framework is developed based on fully convolutional networks and enables inputs of varying sizes. In this way, we circumvent the fixed-size constraint of prevalent convolutional neural networks, and avoid the risk of impairing the intrinsic aesthetic appeal of images. Moreover, given the fact that aesthetic perceiving is coupled with semantic understanding, we present a novel semantic-aware hybrid NEtwork (SANE), which harvests the information from object categorization and scene recognition to enhance image aesthetics assessment. Experiments on two benchmark datasets have well verified the effectiveness of our approach in both scenarios of aesthetic distribution prediction and aesthetic label prediction, and highlighted the benefits of input preserving as well as semantic understanding for images.

Hashing-based methods have made great advancements in cross-modal retrieval in both computational efficiency and storage. Learning a common space from different modalities is the common strategy of hashing-based methods, however, relational and structural information between samples in each modality, namely, a modality-specific structure, is always discarded during learning. In addition, cross-modality samples sometimes suffer from inter-class ambiguity and intra-class variability because of the uncertainty of manual labeling. To address these issues, we propose a novel method named Modality-specific structure Preserving Hashing (MsPH), which learns hashes by preserving the local structure and relations between samples in each modality. Moreover, label enhancement is utilized in MsPH to address label ambiguity and variability. Extensive experiments conducted on three benchmark datasets demonstrate the superiority of MsPH under various cross-modal scenarios.

Hashing has recently received great attention in cross-modal retrieval. Cross-modal retrieval aims at retrieving information across heterogeneous modalities (e.g., texts vs. images). Cross-modal hashing compresses heterogeneous high-dimensional data into compact binary codes with similarity preserving, which provides efficiency and facility in both retrieval and storage. In this study, we propose a novel fast discrete cross-modal hashing (FDCH) method with regressing from semantic labels to take advantage of supervised labels to improve retrieval performance. In contrast to existing methods that learn the projection from hash codes to semantic labels, the proposed FDCH regresses the semantic labels of training examples to the corresponding hash codes with a drift. It not only accelerates the hash learning process, but also helps generate stable hash codes. Furthermore, the drift can adjust the regression and enhance the discriminative capability of hash codes. Especially in the case of training efficiency, FDCH is much faster than existing methods. Comparisons with several state-of-the-art techniques on three benchmark datasets have demonstrated the superiority of FDCH under various cross-modal retrieval scenarios.

Local feature points have been widely employed in robust image fingerprinting. One of their intrinsic advantages is their invariance under geometric transforms. However, their robustness against certain attacks that modify the positions of points, such as additive noising and blurring, is limited. In addition, local-feature-point-based approaches ignore the distribution of the feature points. In this paper, we harness feature point relationships, including local structures and global relevance, to overcome these limitations. In the relationship mining strategy, Delaunay triangulation is first applied to the feature points to capture their geometric structures. Subsequently, local structures are represented by searching for an independent set in the mapping graph constructed via Delaunay triangulation, whereas the global relevance is represented by the Laplacian of the graph. Finally, the local structures and global relevance are used as input to the quantization process of the image fingerprinting system. In the process of quantization, we propose an unsupervised quantization strategy called between-cluster distance-based quantization to preserve the neighborhood structure between the binary fingerprint space and the original feature space. Experimental results show that the proposed method achieves effective performance under common modifications.

Automatically assessing image quality from an aesthetic perspective is of great interest to the high-level vision research community. Existing methods are typically non-personalized and quantify image aesthetics with a universal label. However, given the fact that aesthetics is a subjective perception, how to understand user aesthetic perceptions poses a formidable challenge to image aesthetics assessment. In this paper, we propose to model user aesthetic perceptions using a set of exemplar images from social media platforms, and realize personalized aesthetics assessment by transferring this knowledge to adapt the results of the trained generic model. In this way, image aesthetics is measured from both aspects of visual quality and user tastes. Extensive experiments on two benchmark datasets well verified the potential of our approach for personalized image aesthetics assessment.

Nature Language Processing has been paid more attention recently. Traditional approaches for language model primarily rely on elaborately designed features and complicated natural language processing tools, which take a large amount of human effort and are prone to error propagation and data sparse problem. Deep neural network method has been shown to be able to learn implicit semantics of text without extra knowledge. To better learn deep underlying semantics of sentences, most deepneuralnetworklanguagemodelsutilizemulti-gramstrategy. However, the current multi-gram strategies in CNN framework are mostly realized by concatenating trained multi-gram vectors to form the sentence vector, which can increase the number of parameters to be learned and is prone to over fitting. To alleviate the problem mentioned above, we propose a novel adaptive sentence representation learning model based on multigram CNN framework. It learns adaptive importance weights of different n-gram features and forms sentence representation by using weighted sum operation on extracted n-gram features, which can largely reduce parameters to be learned and alleviate the threat of over fitting. Experimental results show that the proposed method can improve performances when be used in sentiment and relation classification tasks.

Aesthetics has become increasingly prominent for image search to enhance user satisfaction. Therefore, image aesthetics assessment is emerging as a promising research topic in recent years. In this paper, distinguished from existing studies relying on a single label, we propose to quantify the image aesthetics by a distribution over quality levels. The distribution representation can effectively characterize the disagreement among the aesthetic perceptions of users regarding the same image. Our framework is developed on the foundation of label distribution learning, in which the reliability of training examples and the correlations between quality levels are fully taken into account. Extensive experiments on two benchmark datasets well verified the potential of our approach for aesthetics assessment. The role of aesthetics in image search was also rigorously investigated.

Online identities (OIDs) refer to accounts that Internet users create on Web platforms. In investigations of OID-involved criminal cases, investigators often encounter multiple OIDs used by the same criminal among a number of suspect OIDs. Recognizing such OIDs is referred to as OID linkage (OL) and is a critical task, because it helps investigators consolidate information from different sources, identify new investigation leads, perform further analysis by connecting seemingly unrelated cases, and finally ideate the real criminal. However, few existing OL techniques can achieve satisfactory performance in criminal investigation scenarios due to information asymmetry, information unreliability, and a lack of training data. We propose an unsupervised spectral co-clustering-based OL framework that takes OID access trajectories as linkage evidence. By leveraging a spectral co-clustering algorithm, we integrate access location consistency and access time consistency as heuristics to direct the OL process. Experiments performed using actual investigation data demonstrate the feasibility and promise of the proposed framework.

As mobile device penetration increases, it has become pervasive for images to be associated with locations in the form of geotags. Geotags bridge the gap between the physical world and the cyberspace, giving rise to new opportunities to extract further insights into user preferences and behaviors. In this article, we aim to exploit geotagged photos from online photo-sharing sites for the purpose of personalized Point-of-Interest (POI) recommendation. Owing to the fact that most users have only very limited travel experiences, data sparseness poses a formidable challenge to personalized POI recommendation. To alleviate data sparseness, we propose to augment current collaborative filtering algorithms along from multiple perspectives. Specifically, hybrid preference cues comprising user-uploaded and user-favored photos are harvested to study users’ tastes. Moreover, heterogeneous high-order relationship information is jointly captured from user social networks and POI multimodal contents with hypergraph models. We also build upon the matrix factorization algorithm to integrate the disparate sources of preference and relationship information, and apply our approach to directly optimize user preference rankings. Extensive experiments on a large and publicly accessible dataset well verified the potential of our approach for addressing data sparseness and offering quality recommendations to users, especially for those who have only limited travel experiences.

Content-based near-duplicate video detection (NDVD) is essential for effective search and retrieval, and robust video fingerprinting is a good solution for NDVD. Most existing video fingerprinting methods use a single feature or concatenate different features to generate video fingerprints, and show good performance under single-mode modifications such as noise addition and blurring. However, when they suffer combined modifications, the performance is degraded to a certain extent because such features cannot characterize the video content completely. By contrast, the assistance and consensus among different features can improve the performance of video fingerprinting. Therefore, in the present study, we mine the assistance and consensus among different features based on a tensor model, and we present a new comprehensive feature to fully use them in the proposed video fingerprinting framework. We also analyze what the comprehensive feature really is for representing the original video. In this framework, the video is initially set as a high-order tensor that consists of different features, and the video tensor is decomposed via the Tucker model with a solution that determines the number of components. Subsequently, the comprehensive feature is generated by the low-order tensor obtained from tensor decomposition. Finally, the video fingerprint is computed using this feature. A matching strategy used for narrowing the search is also proposed based on the core tensor. The robust video fingerprinting framework is resistant not only to single-mode modifications but also to their combination.

User-generated tags associated with social images are frequently imprecise and incomplete. Therefore, a fundamental challenge in tag-based applications is the problem of tag relevance estimation, which concerns how to interpret and quantify the relevance of a tag with respect to the contents of an image. In this paper, we address the key problem from a new perspective of learning to rank, and develop a novel approach to facilitate tag relevance estimation to directly optimize the ranking performance of tag-based image search. A supervision step is introduced into the neighbour voting scheme, in which tag relevance is estimated by accumulating votes from visual neighbours. Through explicitly modelling the neighbour weights and tag correlations, the risk of making heuristic assumptions is effectively avoided for conventional methods. Extensive experiments on a benchmark dataset in comparison with the state-of-the-art methods demonstrate the promise of our approach.

Automatic image annotation plays a critical role in modern keyword‐based image retrieval systems. For this task, the nearest‐neighbor–based scheme works in two phases: first, it finds the most similar neighbors of a new image from the set of labeled images; then, it propagates the keywords associated with the neighbors to the new image. In this article, we propose a novel approach for image annotation, which simultaneously improves both phases of the nearest‐neighbor–based scheme. In the phase of neighbor search, different from existing work discovering the nearest neighbors with the predicted distance, we introduce a ranking‐oriented neighbor search mechanism (RNSM), where the ordering of labeled images is optimized directly without going through the intermediate step of distance prediction. In the phase of keyword propagation, different from existing work using simple heuristic rules to select the propagated keywords, we present a learning‐based keyword propagation strategy (LKPS), where a scoring function is learned to evaluate the relevance of keywords based on their multiple relations with the nearest neighbors. Extensive experiments on the Corel 5K data set and the MIR Flickr data set demonstrate the effectiveness of our approach.

Automatic image annotation plays a critical role in keyword-based image retrieval systems. Recently, the nearest-neighbor based scheme has been proposed and achieved good performance for image annotation. Given a new image, the scheme is to first find its most similar neighbors from labeled images, and then propagate the keywords associated with the neighbors to it. Many studies focused on designing a suitable distance metric between images so that all labeled images can be ranked by their distance to the given image. However, higher accuracy in distance prediction does not necessarily lead to better ordering of labeled images. In this paper, we propose a ranking-oriented neighbor search mechanism to rank labeled images directly without going through the intermediate step of distance prediction. In particular, a new learning to rank algorithm is developed, which exploits the implicit preference information of labeled images and underlines the accuracy of the top-ranked results. Experiments on two benchmark datasets demonstrate the effectiveness of our approach for image annotation.

Automatic image annotation plays an important role in modern keyword-based image retrieval systems. Recently, many neighbor-based methods have been proposed and achieved good performance for image annotation. However, existing work mainly focused on exploring a distance metric learning algorithm to determine the neighbors of an image, and neglected the subsequent keyword propagation process. They usually used some simple heuristic propagation rules, and propagated each keyword independently without considering the inherent semantic coherence among keywords. In this paper, we propose a novel learning-based keyword propagation strategy and incorporate it into the neighbor-based method framework. In particular, we employ the structural SVM to learn a scoring function which can evaluate different candidate keyword sets for a test image. Moreover, we explicitly enforce the semantic coherence constraint for the propagated keywords in our approach. The annotation of the test image is propagated as a whole rather than separate keywords. Experiments on two benchmark data sets demonstrate the effectiveness of our approach for image annotation and ranked retrieval.

Most of the state-of-the-art systems for object localization rely on supervised machine learning techniques, and are thus limited by the lack of labeled training data. In this paper, our motivation is to provide training dataset for object localization effectively and efficiently. We argue that the notes data in Flickr can be exploited as a novel source for object modeling. At first, we apply a text mining method to gather semantically related images for a specific class. Then a handful of images are selected manually as seed images or initial training set. At last, the training set is expanded by an incremental active learning framework. Our approach requires significantly less manual supervision compared to standard methods. The experimental results on the PASCAL VOC 2007 and NUS-WIDE datasets show that the training data acquired by our approach can complement or even substitute conventional training data for object localization.

The huge amount of knowledge in web forums has motivated great research interests in recent years. However, tracking semantic dependencies in each thread in web forums has posed a challenging problem for researchers. In this paper, we explore an unsupervised topic model to burst through this issue by simultaneously modeling the semantics and the reply relationship in a thread. The proposed model is a dynamic extension of Latent Dirichlet Allocation (LDA) for the structure of web forum threads, where each post is considered as a mixture of topics that vary along the asynchronous conversation. The experimental results on two different forum data sets show encouraging performance of our proposed PPM in ranking the influence of posts.

Publication Time (P-time for short) of Web pages is often required in many application areas. In this paper, we address the issue of P-time detection and its application for page rank. We first propose an approach to extract P-time for a page with explicit P-time displayed on its body. We then present a method to infer P-time for a page without P-time. We further introduce a temporal sensitive page rank model using P-time. Experiments demonstrate that our methods outperform the baseline methods significantly.