Sequential recommendation has become indispensable in modern digital services. Prevalent recommendation techniques formulate the recommendation task with a language instruction fed into large language models (LLMs) to generate recommendations. However, the implicit interaction scenario of recommendation task cannot provide explicit reasoning supervision to activate LLM's multi-step reasoning capability. Besides, the manner of reasoning for enhancing recommendation is still underexplored. Therefore, we investigate activating multi-step reasoning with users' interactions and propose a multi-step reasoning-enhanced LLM (MSR-Rec), which tightly integrates reasoning with recommendation from designing reasoning chain to reasoning-based recommendation. A task-decomposed reasoning chain is elaborately designed to imitate users' thinking process, seamlessly involving reasoning into recommendation. Following the reasoning chain, MSR-Rec synthesizes reasoning supervision and fine-tunes LLM to adapt for task-specific reasoning. In inference, bidirectional reasoning is implemented from user and item sides, performing a closed-loop reasoning for recommendation. Comprehensive experiments demonstrate that MSR-Rec achieves the state-of-the-art performance in both recommendation quality and reasoning interpretability, advancing the integration of reasoning and recommendation in LLM-based systems.

Beyond behavioral interaction records, multimedia recommendation scenarios possess abundant semantic signals, which provide excellent data support for user interest mining. Recently, the multimodal enhanced interaction graph has been actively explored and has achieved great progress. However, these methods overlook the capability disparity of various modalities in learning users' interests and lack the ability to explore the hierarchical relationships of interests in modality, resulting in suboptimal recommendation performance. Therefore, this work investigates intra-modality hierarchical learning and inter-modality guidance, proposing a hyperbolic self-distillation (HSD) model for multimedia recommendation. In each modality space, HSD introduces a hyperbolic propagation to filter users' hierarchical interests from the interaction graph effectively. Inter-modality interests are aligned further by a two-level self-distillation strategy to designate multimodal interactions to teach single-modal learning, aiming at teaching and learning to promote each other. Extensive experiments on four public datasets demonstrate that the proposed HSD outperforms leading baselines for multimedia recommendation, verifying the effectiveness of hierarchical propagation and two-level self-distillation in mining users' hierarchical interests.

To address the interaction sparsity, recent recommendation techniques introduce multimodal semantics to enrich collaborative signals for modeling users' interests. However, these models neglect to depict interaction behaviors and omit to involve diverse behavioral patterns, which are the core characters of the recommendation scenario. Therefore, we propose an intent-augmented multimodal graph embedding (IMGE) model for multimodal recommendation, which constructs an interaction graph with multimodal interactions and augments the graph with behavioral interactions to promote recommendation. Unlike the conventional interaction graph, edges are encoded with multimodal semantics, and intent auxiliary nodes are injected into the neighbors of user/item nodes. IMGE innovatively modifies graph convolution by explicitly involving edge signals, which aggregates collaborative signals from the intent-augmented multimodal graph to embed both semantic and behavioral collaborative signals. Experiments on three public datasets demonstrate the superiority of the proposed IMGE, especially on the Baby dataset achieving 40.78% improvement by NDCG, verifying the effectiveness of encoding interactions with semantic and behavioral signals for recommendation.

Video captioning aims to generate natural language descriptions of video content. Recent methods extract temporal and spatial information separately and use dataset-specific prior knowledge to enhance caption quality. However, they may be inadequate in joint spatiotemporal modeling and lack the utilization of commonsense knowledge, making it difficult to fully understand the video. To address these issues, this paper proposes a dual-stream state-space model (DSSM-KG) based on cross-modal knowledge injection. Specifically, by integrating the heterogeneous Mamba with the Transformer in both parallel and sequential manners, we construct the spatially enhanced dual-stream state-space module (S-DSSM) and the temporally enhanced dual-stream state-space module (T-DSSM) to strengthen joint spatiotemporal modeling. Additionally, a knowledge graph that integrates both commonsense and dataset-specific information is constructed and adaptively injected into the decoder to furnish the model with extensive video-related knowledge. Experimental results indicate that the structural designs of DSSM-KG, together with the knowledge injection mechanism, demonstrate significant efficacy, yielding competitive performance on mainstream video captioning datasets such as MSVD and MSR-VTT.

Most of existing weakly supervised GAR methods are typically bottom-up, automatically mining key areas by the attention mechanism. Due to the lack of a semantic connection to individual actions, some regions associated with these actions may be omitted, potentially impacting performance. In fact, a group activity is a combination of multiple individual actions, and the prototype of a specific action can be obtained from visual representations of individuals performing it, denoted as visual conceptual knowledge. In this paper, we propose a Visual Conceptual Knowledge Guided Action Map framework. It uses prototypes to produce individual action maps that indicate the likelihood of actions occurring at different locations. In some scenarios, the spatial distribution of actions shows strong regularity, which we compile as A-A Maps to enhance individual action maps. The action maps are integrated with action semantic representations for group activity recognition. Extensive experiments on two public benchmarks, the Volleyball and the NBA datasets, demonstrate the effectiveness of our proposed method, even in cases of limited training data.

In this paper, we propose a Knowledge-guided Sample Selection for Prompt Generation (KGSS) approach to reduce model hallucination by focusing on the logical relevance of knowledge key points. We argue that a task-specific context should be derived from knowledge key points rather than relying solely on semantic similarity, particularly in complex scenarios such as college entrance exams. KGSS first constructs a knowledge graph of knowledge key points within a subject. Then, it employs a two-stage sample selection strategy that integrates knowledge key points filtering and knowledge-level feature extraction. Initially, the system filters out irrelevant knowledge key points based on the relevance of the input, ensuring that the candidate samples are contextually appropriate. Then, a multilevel sample selection process is conducted to balance similarity and diversity across knowledge key points. Experimental results on the Gaokao-Bench dataset, derived from Chinese college entrance exam questions, demonstrate that KGSS significantly outperforms existing methods in improving model accuracy and robustness across various disciplines. Our findings highlight the importance of logical relevance and knowledge diversity in improving LLM performance, particularly in tasks with high logical complexity and abstraction.

Recent Face Anti-Spoofing (FAS) methods have improved generalization to unseen domains by leveraging domain generalization techniques. However, they overlooked the semantic relationships between local features, resulting in suboptimal feature alignment and limited performance. To this end, pixel-wise supervision has been introduced to offer contextual guidance for better feature alignment. Unfortunately, the semantic ambiguity in coarsely designed pixel-wise supervision often leads to misalignment. This paper proposes a novel Dual Consistency Regularization Network (DCRN). It promotes the fine-grained alignment of local features with dense semantic correspondence for FAS. Specifically, a Dual Consistency Learning module (DCL) is devised to capture the inter- and intra-similarity between each region of sample pairs. In this module, a dual consistency regularization learning objective enhances the semantic consistency of local features by minimizing both the variance of inter-similarity and the distance between inter- and intra-similarity. Further, a weight matrix is estimated based on the inter-similarity, representing the possibility that each region belongs to the living class. Based on this weight matrix, WMSE loss is designed to guide the model in avoiding mapping the live regions to the spoofing class, thus alleviating semantic ambiguity in pixel-wise supervision. Extensive experiments on four widely used datasets clearly demonstrate the superiority and high generalization of the proposed DCRN.

Palmprint recognition has recently garnered attention due to its high accuracy, strong robustness, and high security. Existing deep learning-based palmprint recognition methods usually require large amounts of data for centralized training, facing the challenge of privacy disclosure. In addition, the non-independent and identically distributed (non-IID) issue in the multi-spectral palmprint images generally leads to the degradation of recognition performance. To tackle these problems, this paper proposes a dynamic personalized federated learning model for cross-spectral palmprint recognition, called DPFed-Palm. Specifically, for each client's local training, we present a new combination of loss functions to enforce the constraints of local models and effectively enhance the feature representation capability of models. Subsequently, DPFed-Palm aggregates the above-trained local models by using the combined aggregation strategies of the Federated Averaging (FedAvg) and Personalized Federated Learning (PFL) to obtain the best personalized global model of each client. For the selection of the best personalized global model, we develop a dynamic weight selection strategy to obtain the optimal weights of the local and global models by cross-spectral (cross-client) testing. Extensive experimental results on three public PolyU multispectral, IITD, and CASIA datasets show that the proposed method outperforms the existing techniques in privacy-preserving and recognition performance.

The existing attention-based label-free weakly supervised group activity recognition methods can automatically learn tokens related to the actors. And they have difficulties generating sufficiently diverse token embeddings. To address these issues, we automatically obtain the grayscale motion mask of all the moving objects based on the motion direction not the motion amplitude. A Motion-Guided Mask Generator module (MGMG) is proposed to estimate the attention region mask under the supervision of the grayscale motion mask. MGMG involves four parts. A correlation layer measures the relative displacement between two adjacent feature maps. A cosine attention mechanism is designed to reduce the module's sensitivity to feature amplitude changes. A mask generator is built to generate the attention region mask. And a specifically designed activation function is used to refine the attention region mask and to enhance its focus on actor motion regions. We also customize a normalized relative error loss function for MGMG module. This loss can address the value range mismatch problem for the estimated attention mask as well as the grayscale motion mask. Furthermore, a Motion Attention-Guided Relational Reasoning (MAGRR) framework is presented for the weakly supervised condition. It uses the MGMG module to estimate the attention region automatically, and a Spatial-temporal Aggregation Stack (SAS) module to activate the attention regions of the features at the spatial level, then transform them into multiple tokens, which are further captured by the attention mechanism for their temporal dependencies and interrelationships. MAGRR is experimented on the Collective Activity dataset and the Collective Activity Extension dataset, achieving state-of-the-art performance and competitive performance on the Volleyball and the NBA datasets.

In the domain of recommendations, previous works often retrieve items through sampling strategies from the database to gather negative signals for exploring implicit feedback. However, because of extremely sparse records, the existing items used as negative samples may not sufficiently support the interacted items in depicting the diverse interests of users. Consequently, the generation of negative samples needs to be explored in recommendation systems. In this study, we propose an interest-disentangled contrastive sample generation (IDCG) model to enhance interest modeling by contrasting interacted items with the generated samples for recommendation. Specifically, we decouple the interacted items of users into positively relevant and irrelevant factors of interest, providing a valuable clue to learn negatively relevant factors in personalized interests. Then, negative samples are generated by merging the learned negatively relevant factors and irrelevant factors. At this point, a two-level contrast is constructed between positive and negative samples and between the relevant factors of positives and negatives, providing auxiliary collaborative signals to debias and alleviate the interaction sparsity issue. Extensive experiments on three real datasets demonstrate the effectiveness of IDCG in generating targeted and meaningful negative samples from the perspective of disentangling relevant factors to promote interest modeling for recommendation.

To address the data sparsity issue, conventional graph-based models leverage structural signals from the interaction graph to embed users' interests. However, these models learn a uniform representation for interest modeling, which blends users' diverse intents and inevitably biases interest learning, hindering recommendations. Although the fine-grained paradigm can learn the intents of interactions separately to alleviate learning bias, the relationships among intents and the disentangled manner require elaborate design. Existing fine-grained models emphasize intent diversity and employ additional data splitting for disentanglement, which ignores the hierarchical relationship, exacerbates data sparsity, and increases the computational burden. To address these issues, we explore hierarchical intents and adaptive intent learning, proposing a hierarchical intent-based interest disentanglement (HIID) model for personalized recommendation. HIID introduces learnable intent queries to guide interest disentanglement from global interactions in a split-free manner. It raises a hierarchical intent hypothesis to involve hierarchical CF signals for interest modeling, where intents within the same level appear relatively diverse, and the in-depth intents are abstracted from the superficial ones. Both adaptive intent learning and hierarchical hypothesis help extract significant CF signals to promote personalized recommendation. Extensive experiments on public datasets show that the proposed HIID outperforms the state-of-the-art CF models for recommendation. Furthermore, HIID implements adaptive interest disentanglement in a split-free manner, improving the training efficiency of the recommender model compared to the existing fine-grained interest models.

The prevalent recommendation techniques explore the graph structure of interactions to alleviate the interaction sparsity issue for inferring users’ interests. These graph models focus on extracting local structural signals to model users’ interests, introducing grid-like distortion and ignoring the hierarchical tree-like structure when learning from the interaction graph. The learned interests lack significant hierarchical signals, resulting in suboptimal recommendation performance. In this article, we investigate geometric-augmented graph learning with hyperbolic and Euclidean geometries to delve into local structural and hierarchical knowledge from the interaction graph. A self-teaching network called geometric-augmented self-distillation (GASD) is proposed to transfer hierarchical knowledge from hyperbolic to Euclidean space. The transfer learning enables shrinking of the network into a primary student to implement effective and efficient inference in Euclidean space, preventing computational burden in hyperbolic space. Experiments on publicly available datasets demonstrate that the proposed GASD outperforms the state-of-the-art models, verifying the effectiveness and efficiency of knowledge transfer by self-distillation to aggregate knowledge adaptively for personalized recommendation.

Social recommendation utilizes social relations to extract auxiliary collaborative signals, effectively mitigating data sparsity issues. However, existing approaches predominantly focus on static influence from social friends while neglecting two critical aspects: the dynamic contextual patterns in user behaviors and the potential of collaborative users. To address these limitations and further alleviate data sparsity, we propose a context-aware dual graph attention network (CDGA) that simultaneously captures users’ static and dynamic interests through social relations and interaction records. The proposed CDGA model introduces a dynamic activation mechanism to simulate contextual influences, generating dynamic embeddings for users and items. Furthermore, we develop an adaptive fusion mechanism that integrates interaction channels across static and dynamic embeddings for interaction prediction. Extensive experiments on three benchmark datasets demonstrate that CDGA consistently outperforms state-of-the-art social recommendation methods, confirming its effectiveness.

Face Anti-Spoofing is critical to secure face recognition systems from presentation attacks. Existing methods often suffer from performance degradation due to image quality issues, such as blurring, overexposure, or varied background, which cause distribution deviations of face images in the quality space, and hinder the learning of effective liveness features. In this paper, we propose a novel method that interactively co-reinforces the liveness and Face Quality representations for Face Anti-Spoofing (FQ-FAS). Specifically, to enhance the discrimination of face quality representation, FQ-FAS first designs a face quality learning module that naturally mitigates the interference from background. Subsequently, a quality-spoofing feature interaction module is devised to co-reinforce both liveness and face quality representations. Meanwhile, we propose a quality aware triplet loss to align the distribution of face images from two aspects: one is to pull the homogeneous face images with different quality together, while the other is to push the inhomogeneous samples with similar quality away in the feature space. In this way, FQ-FAS can learn reliable and discriminative representations for face anti-spoofing. Extensive intra-dataset and cross-dataset experiments clearly demonstrate that our method obtains better performance than previous state-of-the-art methods.

Group activity recognition is a challenging task that involves multiple moving actors within a cluttered scene. Existing methods often rely on object detector to avoid individual bounding box labeling during testing, but are prone to false detections due to factors such as occlusion and background clutter. In addition, existing detector-free method based on Transformer attends to attention map that is too sparse, resulting in the loss of some important foreground information. In this paper, we introduce foreground-background contrast loss (FB-Loss) to help accurately seek discriminative cues in the foreground and eliminate noise interference in the background. Neither ground-truth bounding boxes nor object detectors are required during both training and testing. Experimental results on public datasets show that our proposed method achieves the state-of-the-art performance.

In E-Commence applications, online shoppers increasingly rely on reviews to find answers for their queries on products. Recently, there has been significant research on product question answering using reviews. While many neural models have been proposed to generate answers from reviews to answer product questions, the evaluation of review-based product question answering (RPQA) models is still largely using lexicon-based text similarity metrics against reference answers. In this paper, we propose a distant supervised contrastive learning network based on modern transformer-based language models by leveraging distant ground-truth community question answers to train a model for RPQA evaluation. At deployment, the learned metric does not require reference answers to evaluate the generated answers for product queries. We conducted extensive experiments on publicly available AmazonQA datasets to evaluate RPQA models. Experiment results show that our learned metric correlates well with human judgements, and outperforms existing lexicon-based or embedding-based metrics.

Image sentiment analysis is a domain fraught with the dual challenges of interpreting complex visual content and discerning the subtle emotional undertones it may convey. Despite the notable successes of existing visual language pretraining (VLP) models in a variety of visual tasks, they fall short in the nuanced realm of sentiment analysis. This shortfall is primarily due to their inadequate processing of sentiment-specific cues—most notably, the oversight of localized sentioment cues within images and the intricate interplay of these signals. Furthermore, these models inadequately harness the rich sentiment cues often embedded in accompanying text. In response to these shortcomings, we introduce the Specific Sentiment Mask Auto-encoder (S2MA) model, which is expressly designed to integrate sentiment information during the pretraining process. S2MA is meticulously engineered to focuse on both intermodal and intramodal sentiment cue, thereby augmenting the model’s proficiency in anlysising the sentiment knowledge within visual content. Rigorous comparative evaluations of S2MA against the CLIP model, across a spectrum of downstream datasets in zero-shot and supervised learning scenarios, have validated the superiority of our approach. The empirical outcomes affirm S2MA’s capacity to significantly enhance the analytical landscape of image sentiment analysis.

Image emotion classification (IEC) aims to extract the abstract emotions evoked in images. Recently, language-supervised methods such as contrastive language-image pretraining (CLIP) have demonstrated superior performance in image understanding. However, the underexplored task of IEC presents three major challenges: a tremendous training objective gap between pretraining and IEC, shared suboptimal prompts, and invariant prompts for all instances. In this study, we propose a general framework that effectively exploits the language-supervised CLIP method for the IEC task. First, a prompt-tuning method that mimics the pretraining objective of CLIP is introduced, to exploit the rich image and text semantics associated with CLIP. Subsequently, instance-specific prompts are automatically composed, conditioning them on the categories and image content of instances, diversifying the prompts, and thus avoiding suboptimal problems. Evaluations on six widely used affective datasets show that the proposed method significantly outperforms state-of-the-art methods (up to 9.29% accuracy gain on the EmotionROI dataset) on IEC tasks with only a few trained parameters. The code is publicly available at https://github.com/dsn0w/PT-DPC/for research purposes .

Previous recommendation models build interest embeddings heavily relying on the observed interactions and optimize the embeddings with a contrast between the interactions and randomly sampled negative instances. To our knowledge, the negative interest signals remain unexplored in interest encoding, which merely serves losses for backpropagation. Besides, the sparse undifferentiated interactions inherently bring implicit bias in revealing users’ interests, leading to suboptimal interest prediction. The negative interest signals would be a piece of promising evidence to support detailed interest modeling. In this work, we propose a perturbed graph contrastive learning with negative propagation (PCNP) for recommendation, which introduces negative interest to assist interest modeling in a contrastive learning (CL) architecture. An auxiliary channel of negative interest learning generates a contrastive graph by negative sampling and propagates complementary embeddings of users and items to encode negative signals. The proposed PCNP contrasts positive and negative embeddings to promote interest modeling for recommendation. Extensive experiments demonstrate the capability of PCNP using two-level CL to alleviate interaction sparsity and bias issues for recommendation.

Group activity recognition is a challenging task because it involves diverse individual actions and complex relations. Most existing methods enhance individual representation by introducing relation inference using appearance features. Some methods utilize extra knowledge, such as action labels, to enhance relation inference and refine the individual representation, but the knowledge they explored is simple and insufficient. In this paper, we propose a novel idea of knowledge concretization and further develop a Knowledge Augmented Relation Inference framework (KARI) for group activity recognition. Specifically, we first concretize knowledge from training data, and then represent them as Class-Class co-occurrence Map (C-C Map) and Class-Position distribution Map (C-P Map). On top of them, KARI explores concretized knowledge to integrate visual and semantic representation in a unified architecture for group activity recognition. Experimental results on two public datasets show that the proposed framework performs favorably compared with state-of-the-art approaches.

Over the last decades, finger vein biometric recognition has generated increasing attention because of its high security, accuracy, and natural anti-counterfeiting. However, most of the existing finger vein recognition approaches rely on image enhancement or require much prior knowledge, which limits their generalization ability to different databases and different scenarios. Additionally, these methods rarely take into account the interference of noise elements in feature representation, which is detrimental to the final recognition results. To tackle these problems, we propose a novel jointly embedding model, called Joint Discriminative Analysis with Low-Rank Projection (JDA-LRP), to simultaneously extract noise component and salient information from the raw image pixels. Specifically, JDA-LRP decomposes the input image into noise and clean components via low-rank representation and transforms the clean data into a subspace to adaptively learn salient features. To further extract the most representative features, the proposed JDA-LRP enforces the discriminative class-induced constraint of the training samples as well as the sparse constraint of the embedding matrix to aggregate the embedded data of each class in their respective subspace. In this way, the discriminant ability of the jointly embedding model is greatly improved, such that JDA-LRP can be adapted to multiple scenarios. Comprehensive experiments conducted on three commonly used finger vein databases and four palm-based biometric databases illustrate the superiority of our proposed model in recognition accuracy, computational efficiency, and domain adaptation.

Due to their high reliability, security, and anti-counterfeiting, finger-based biometrics (such as finger vein and finger knuckle print) have recently received considerable attention. Despite recent advances in finger-based biometrics, most of these approaches leverage much prior information and are non-robust for different modalities or different scenarios. To address this problem, we propose a structured Robust and Sparse Least Square Regression (RSLSR) framework to adaptively learn discriminative features for personal identification. To achieve the powerful representation capacity of the input data, RSLSR synchronously integrates robust projection learning, noise decomposition, and discriminant sparse representation into a unified learning framework. Specifically, RSLSR jointly learns the most discriminative information from the original pixels of the finger images by introducing the$l_{2,1}$norm. A sparse transformation matrix and reconstruction error are simultaneously enforced to enhance its robustness to noise, thus making RSLSR adaptable to multi-scenarios. Extensive experiments on five contact-based and contactless-based finger databases demonstrate the clear superiority of the proposed RSLSR in terms of recognition accuracy and computational efficiency.

Recently, recommender systems have witnessed the fast evolution of Internet services. However, it suffers hugely from inherent bias and sparsity issues in interactions. The conventional uniform embedding learning policies fail to utilize the imbalanced interaction clue and produce suboptimal representations to users and items for recommendation. Towards the issue, this work is dedicated to bias-aware embedding learning in a decomposed manner and proposes a counterfactual graph convolutional learning (CGCL) model for personalized recommendation. Instead of debiasing with uniform interaction sampling, we follow the natural interaction bias to model users’ interests with a counterfactual hypothesis. CGCL introduces bias-aware counterfactual masking on interactions to distinguish the effects between majority and minority causes on the counterfactual gap. It forms multiple counterfactual worlds to extract users’ interests in minority causes compared to the factual world. Concretely, users and items are represented with a causal decomposed embedding of majority and minority interests for recommendation. Experiments show that the proposed CGCL is superior to the state-of-the-art baselines. The performance illustrates the rationality of the counterfactual hypothesis in bias-aware embedding learning for personalized recommendation.

The information era brings both opportunities and challenges to information services. Confronting information overload, recommendation technology is dedicated to filtering personalized content to meet users’ requirements. The extremely sparse interaction records and their imbalanced distribution become a big obstacle to building a high-quality recommendation model. In this article, we propose a swarm self-supervised hypergraph embedding (SHE) model to predict users’ interests by hypergraph convolution and self-supervised discrimination. SHE builds a hypergraph with multiple interest clues to alleviate the interaction sparsity issue and performs interest propagation to embed CF signals in hybrid learning on the hypergraph. It follows an auxiliary local view by similar hypergraph construction and interest propagation to restrain unnecessary propagation between user swarms. Besides, interest contrast further inserts self-discrimination to deal with long-tail bias issue and enhance interest modeling, which aid recommendation by a multi-task learning optimization. Experiments on public datasets show that the proposed SHE outperforms the state-of-the-art models demonstrating the effectiveness of hypergraph-based interest propagation and swarm-aware interest contrast to enhance embedding for recommendation.

Weakly supervised group activity recognition deals with the dependence on individual-level annotations during understanding scenes involving multiple individuals, which is a challenging task. Existing methods either take the trained detectors to extract individual features or utilize the attention mechanisms for partial context encoding, followed by integration to form the final group-level representations. However, the detectors require individual-level annotations during the training phase and have a mis-detection issue, and the partial contexts extracted immediately from the whole complex scene are too ambiguous without the guidance of concrete semantics. In this paper, we investigate the hierarchical structure inherent in group-level labels to extract the fine-grained semantics without using detectors for weakly supervised group activity recognition. A multi-hot encoding strategy combined with a semantic encoder is first adopted to get the label semantics embeddings. The semantic and visual scene information are then fused through a semantic decoder to obtain activity-specific features. Lastly, we employ the multi-label classification and integrate the scores of hierarchical activity labels. Experimental results show that our proposed method achieves the state-of-the-art performance on three benchmarks, and the accuracy on the Volleyball dataset exceeds the second-best method by 2%.

Previous works build interest learning via mining deeply on interactions. However, the interactions come incomplete and insufficient to support interest modeling, even bringing severe bias into recommendations. To address the interaction sparsity and the consequent bias challenges, we propose a graph contrastive learning on complementary embedding (GCCE), which introduces negative interests to assist positive interests of interactions for interest modeling. To embed interest, we design a perturbed graph convolution by preventing embedding distribution from bias. Since negative samples are not available in the general scenario of implicit feedback, we elaborate a complementary embedding generation to depict users’ negative interests. Finally, we develop a new contrastive task to contrastively learn from the positive and negative interests to promote recommendation. We validate the effectiveness of GCCE on two real datasets, where it outperforms the state-of-the-art models for recommendation.

Image emotion classification is an important computer vision task to extract emotions from images. The methods for image emotion classification (IEC) are primarily based on label or distribution as a supervision signal, which neither has enough accessibility nor diversity, limiting the development of IEC research. Inspired by psychology research and the recent booming of large-scale pretrained language models. We figure out a language-supervised paradigm, which can cleverly combine the features of language and visual emotion to drive the visual model to gain stronger emotional discernment with language prompts. To practice the paradigm, we present a conceptually simple while empirically powerful framework for image emotion classification, SimEmotion. That we propose a prompt-based fine-tuning strategy to learn task-specific representations by composing a template with the emotion-level concept and entity-level information. Evaluations on four widely-used affective datasets, namely, Flickr and Instagram (FI), EmotionROI, Twitter I, and Twitter II, demonstrate that the proposed algorithm outperforms the state-of-the-art methods with a large margin (i.e.,$8.42\%$absolute accuracy gain on EmotionROI) on image emotion classification tasks. Our codes will be publicly available for research purposes.

Group activity recognition aims to recognize behaviors characterized by multiple individuals within a scene. Existing schemes rely on individual relation inference and usually take the individuals as tokens. Essentially they select the most relevant region of the group activity from the entire image while filtering out irrelevant background noises. However, these schemes require individual bounding box labeling in both training and testing stages. Since individuals have usually been presented at one scale, multi-scale individuals cannot be combined in an effective way. In this paper, we present a novel end-to-end hierarchical relation inference framework based on active spatial positions for group activity recognition. This framework is designed to locate active spatial positions and use them as visual tokens to infer the relations for token embeddings. It requires individual bounding box labeling only in the training stage while automatically eliminating the background after locating active spatial positions from the entire scene. The hierarchical relations can be naturally inferred based on the visual tokens at different scales, contributing to further performance improvement. Experimental results demonstrate that the proposed framework is competitive against existing schemes that require more laboring and computation to generate labels in both the training and testing stage.

Face anti-spoofing (FAS) plays a vital role in securing face recognition systems. Previous approaches usually learn spoofing features from a single perspective, in which only universal cues shared by all attack types are explored. However, such single-perspective-based approaches ignore the differences among various attacks and commonness between certain attacks and bona fides, thus tending to neglect some non-universal cues that contain strong discernibility against certain types. As a result, when dealing with multiple types of attacks, the above approaches may suffer from the uncomprehensive representation of bona fides and spoof faces. In this work, we propose a novel Advanced Multi-Perspective Feature Learning network (AMPFL), in which multiple perspectives are adopted to learn discriminative features, to improve the performance of FAS. Specifically, the proposed network first learns universal cues and several perspective-specific cues from multiple perspectives, then aggregates the above features and further enhances them to perform face anti-spoofing. In this way, AMPFL obtains features that are difficult to be captured by single-perspective-based methods and provides more comprehensive information on bona fides and spoof faces, thus achieving better performance for FAS. Experimental results show that our AMPFL achieves promising results in public databases, and it effectively solves the issues of single-perspective-based approaches.

Matchmaking systems are vital for creating fair matches in online multiplayer games, which directly affects players' satisfactions and game experience. Most of the matchmaking systems largely rely on precise estimation of players' game skills to construct equitable games. However, the skill rating of a novice is usually inaccurate, as current matchmaking rating algorithms require considerable amount of games for learning the true skill of a new player. Using these unreliable skill scores at early stages for matchmaking usually leads to disparities in terms of team performance, which causes negative game experience. This is known as the "cold-start" problem for matchmaking rating algorithms.

Deep supervised hashing for Hamming space retrieval has recently attracted increasing attention because it enables large-scale image retrieval with constant-time cost. However, the existing Hamming space retrieval methods cannot effectively focus on different pairs simultaneously inside and outside the Hamming ball, making it difficult to push dissimilar pairs outside or pull similar pairs inside the Hamming ball. We propose a novel Boundary-Guided Probability Hashing (BGPH) method that introduces a boundary to guide probability distribution. It makes the probability of similar pairs within the Hamming ball greater than dissimilar pairs and vice versa, which fits the purpose of Hamming space retrieval well. Moreover, we propose a threshold weighting method to indicate when optimization should be stopped to avoid the problem that dissimilar data are pulled into the ball caused by over-optimization in multi-label retrieval scenarios. Comprehensive experiments on three benchmark datasets demonstrate that BGPH yields state-of-the-art retrieval performance.

Scale variation is one of the key challenges in the object detection. Most previous object detectors remedy this by using dilated convolution to enlarge the receptive fields of the vanilla convolutional layers. However, these methods focus on either the spatial information of small objects or the semantics of middle and large objects, which still fail to effectively adapt the scale variance of different objects, resulting in a sub-optimal performance for the object detection. In this paper, we propose a novel Multi-Sequence Dilated Network (MSDN) that stacks different dilated convolutions with different orders in parallel for improving the performance of the object detection. Concretely, MSDN contains a sequential dilated module and a dilated attention module. The former aims to generate scale-specific feature maps with fine-spatial and semantic information of objects at different scales, while the latter further selects more powerful information to adaptively enlarge the receptive fields of object features at different scales. Facilitated with these modules, MSDN well obtains the fine-spatial and semantic information of objects at different scales, thus solving the problem of the scale variation. Comprehensive experimental results over two public object detection benchmarks clearly demonstrate the effectiveness of our proposed MSDN. Particularly, on the COCO dataset, the mAP value of MSDN is 48.7%, outperforming existing state-of-the-art methods in a single model manner.

Group activity recognition is a subject with broad applications, and its main challenge is to model the interactions between individuals. Existing algorithms mostly model the interactions merely based on holistic features of persons, which completely ignore the local details and local interactions that could be significant for recognition. In this paper, we propose a novel part based interaction learning algorithm for group activity recognition. Our proposed algorithm introduces both the physical structural information and fine-grained contextual information into representations, through exploring the intraand inter-actor part interactions. Specifically, a dual-branch framework is adopted to extract the appearance and motion features respectively. For each branch, we utilize the key point detection technique for proper part division and then extract the part features. The part features are further enhanced by the transformers for intra- and inter-actor part interactions, and are lastly used for group activity recognition. Comparison with the state-of-the-arts on two public datasets demonstrate the effectiveness of our proposed algorithm.

Manufacturing speed is one of the biggest challenges in 3D printing. Continuous stereolithography printing can effectively improve the printing speed. However, the model it can print is limited to the hollow out structure or flake structure. In the real application scenario, the models are the composition of multiple kinds of structures such as solid structure, hollow out structure, or flake structure. The continuous stereolithography printing scheme will not work for such models. We first propose a concept of maximum fillable distance (MFD) for a set of resin material and printing settings. And for a specific kind of printing setting, the MFD of resin material at different moving speeds is estimated by experiments. Furthermore, the max-min distance of each slice of the model is computed. And a printing control scheme to combining the continuous and layer-wise printing is generated automatically by comparing the max-min distance and MFD. Using the printing control scheme, two real models are successfully printed.

Face anti-spoofing plays a vital role in face recognition systems. The existed deep learning approaches have effectively improved the performance of presentation attack detection (PAD). However, they learn a uniform feature for different types of presentation attacks, which ignore the diversity of the inherent cues presented in different spoofing types. As a result, they can not effectively represent the intrinsic difference between different spoof faces and live faces, and the performance drops on the cross-domain databases. In this paper, we introduce the inherent cues of different spoofing types by non-uniform learning as complements to uniform features. Two lightweight sub-networks are designed to learn inherent motion patterns from photo attacks and the inherent texture cues from video attacks. Furthermore, an element-wise weighting fusion strategy is proposed to integrate the non-uniform inherent cues and uniform features. Extensive experiments on four public databases demonstrate that our approach outperforms the state-of-the-art methods and achieves a superior performance of 3.7% ACER in the cross-domain Protocol 4 of the Oulu-NPU database. Code is available at https://github.com/BJUT-VIP/Non-uniform-cues.

In this work, we study the problem of separating the global camera motion and the local dynamic motion from an optical flow. Previous methods either estimate global motions by a parametric model, such as a homography, or estimate both of them by an optical flow field. However, none of these methods can directly estimate global and local motions through an end-to-end manner. In addition, separating the two motions accurately from a hybrid flow field is challenging. Because one motion can easily confuse the estimate of the other one when they are compounded together. To this end, we propose an end-to-end global and local motion estimation network GLM-Net. We design two encoder-decoder structures for the motion separation in the optical flow based on different task orientations. One structure adopts a mask autoencoder to extract the global motion, while the other one uses attention U-net for the local motion refinement. We further designed two effective training methods to overcome the problem of lacking supervisions. We apply our method on the action recognition datasets NCAA and UCF-101 to verify the accuracy of the local motion, and the homography estimation dataset DHE for the accuracy of the global motion. Experimental results show that our method can achieve competitive performance in both tasks at the same time, validating the effectiveness of the motion separation.

Abstract Motion information has been widely exploited for group activity recognition in sports video. However, in order to model and extract the various motion information between the adjacent frames, existing algorithms only use the coarse video‐level labels as supervision cues. This may lead to the ambiguity of extracted features and the omission of changing rules of motion patterns that are also important sports video recognition. In this paper, a latent label mining strategy for group activity recognition in basketball videos is proposed. The authors' novel strategy allows them to obtain the latent labels set for marking different frames in an unsupervised way, and build the frame‐level and video‐level representations with two separate levels of supervision signal. Firstly, the latent labels of motion patterns are digged using the unsupervised hierarchical clustering technique. The generated latent labels are then taken as the frame‐level supervision signal to train a deep CNN for the frame‐level features extraction. Lastly, the frame‐level features are fed into an LSTM network to build the spatio‐temporal representation for group activity recognition. Experimental results on the public NCAA dataset demonstrate that the proposed algorithm achieves state‐of‐the‐art performance.

Abstract Underwater image enhancement is an important low‐level vision task with much attention of community. Clear underwater images are helpful for underwater operations. However, raw underwater images often suffer from different types of distortions caused by the underwater environment. To solve these problems, this paper proposes an end‐to‐end dual generative adversarial network (DuGAN) for underwater image enhancement. The images processed by existing methods are taken as training samples for reference, and they are segmented into clear parts and unclear parts. Two discriminators are used to complete adversarial training toward different areas of images with different training strategies, respectively. The proposed method is able to output more pleasing images than reference images benefit by this framework. Meanwhile, to ensure the authenticity of the enhanced images, content loss, adversarial loss, and style loss are combined as loss function of our framework. This framework is easy to use, and the subjective and objective experiments show that excellent results are achieved compared to those methods mentioned in the literature.

Recent years have witnessed the spread of electronic devices especially the mobile phones, which have become almost the necessities in people's daily lives. An effective and efficient technique for blindly assessing the quality of display products is greatly helpful to improve the experiences of users, such as displaying the pictures or texts in a more comfortable manner. In this paper, we put forward a novel no-reference (NR) quality metric of display products, dubbed as NQMDP. First, we have established a new subjective photo quality database, in which 50 photos shown on three different types of display products were captured to constitute a total of 150 photos and then scored by more than 40 inexperienced observers. Second, 19 effective image features are extracted by using six different influencing factors (including complexity, contrast, sharpness, brightness, colorfulness and naturalness) on the quality of display products and then were learned with the support vector regressor (SVR) to estimate the objective quality score of each photo. Results of experiments show that our proposed method has obtained better performance than the state-of-the-art algorithms.

In multi-person videos, especially team sport videos, a semantic event is usually represented as a confrontation between two teams of players, which can be represented as collective motion. In broadcast basketball videos, specific camera motions are used to present specific events. Therefore, a semantic event in broadcast basketball videos is closely related to both the global motion (camera motion) and the collective motion. A semantic event in basketball videos can be generally divided into three stages: pre-event, event occurrence (event-occ), and post-event. By analyzing the influence of different stages of video segments to semantic events discrimination, it is observed that the pre-event and event-occ segments are effective for classification, while the post-events are effective for event success/failure classification. In this paper, we propose an ontology-based global and collective motion pattern (On_GCMP) algorithm for the basketball event classification. First, a two-stage GCMP-based event classification scheme is proposed. The GCMP is extracted using the optical flow. The two-stage scheme progressively combines a five-class event classification algorithm on event-occs and a two-class event classification algorithm on pre-events. Both algorithms utilize the sequential convolutional neural networks (CNNs) and the long short-term memory (LSTM) networks to extract the spatial and temporal features of GCMP for event classification. Second, we utilize the post-event segments to predict success/failure using deep features of images in the video frames (RGB_DF_VF)-based algorithms. Finally, the event classification results and success/failure classification results are integrated to obtain the final results. To evaluate the proposed scheme, we collected a new dataset called NCAA+, which is automatically obtained from the NCAA dataset by extending the fixed length of video clips forward and backward of the corresponding semantic events. The experimental results demonstrate that the proposed scheme achieves the mean average precision of 58.10% on NCAA+. It is higher by 6.50% than the state of the art on NCAA.

Free viewpoint video (FVV) has received considerable attention owing to its widespread applications in several areas such as immersive entertainment, remote surveillance and distanced education. Since FVV images are synthesized via a depth image-based rendering (DIBR) procedure in the "blind" environment (without reference images), a real-time and reliable blind quality assessment metric is urgently required. However, the existing image quality assessment metrics are insensitive to the geometric distortions engendered by DIBR. In this research, a novel blind method of DIBR-synthesized images is proposed based on measuring geometric distortion, global sharpness and image complexity. First, a DIBR-synthesized image is decomposed into wavelet subbands by using discrete wavelet transform. Then, the Canny operator is employed to detect the edges of the binarized low-frequency subband and high-frequency subbands. The edge similarities between the binarized low-frequency subband and high-frequency subbands are further computed to quantify geometric distortions in DIBR-synthesized images. Second, the log-energies of wavelet subbands are calculated to evaluate global sharpness in DIBR-synthesized images. Third, a hybrid filter combining the autoregressive and bilateral filters is adopted to compute image complexity. Finally, the overall quality score is derived to normalize geometric distortion and global sharpness by the image complexity. Experiments show that our proposed quality method is superior to the competing reference-free state-of-the-art DIBR-synthesized image quality models.

With the rapid evolution of social networks, the increasing user intention gap and visual semantic gap both bring great challenge for users to access satisfied contents. It becomes promising to investigate users' customized multimedia recommendation. In this paper, we propose cross-modal collaborative manifold propagation (CMP) for image recommendation. CMP leverages users' interest distribution to propagate images' user records, which lets users know the trend from others and produces interest-aware image candidates upon users' interests. Visual distribution is investigated simultaneously to propagate users' visual records along dense semantic visual manifold. Visual manifold propagation helps to estimate semantic accurate user-image correlations for the candidate images in recommendation ranking. Experimental performance demonstrate the collaborative user-image inferring ability of CMP with effective user interest manifold propagation and semantic visual manifold propagation in personalized image recommendation.

In this paper, we propose a novel visual tracking algorithm by combining the structure-aware network (SA-Net) and spatial-temporal regression model. We first use SA-Net to obtain the initial location proposal, and the deep features are extracted using a fine-tuned convolutional neural network model. Finally, both the location proposal and deep features, including historical information, are input into the long short-term memory (LSTM) for end-to-end spatial temporal regression to adjust the initial location proposal from SA-Net. The experimental results on the challenging OTB dataset demonstrate that the proposed scheme is robust to missing tracking caused by occlusion or object deformation. Additionally, the compared experiments show that the proposed scheme is more competitive than state-of-the-art algorithms.

Deep learning-based visual sentiment analysis requires a large dataset for training. Dataset from social networks is popular but noisy because some images collected in this manner are mislabeled. Therefore, it is necessary to refine the dataset. Based on observations to such datasets, we propose a refinement algorithm based on the sentiments of adjective-noun pairs (ANPs) and tags. We first determine the unreliably labeled images through the sentiment contradiction between the ANPs and tags. These images are removed if the numbers of tags with positive and negative sentiments are equal. The remaining images are labeled again based on the majority vote of the tags' sentiments. Furthermore, we improve the traditional deep learning model by combining the softmax and Euclidean loss functions. Additionally, the improved model is trained using the refined dataset. Experiments demonstrate that both the dataset refinement algorithm and improved deep learning model are beneficial. The proposed algorithms outperform the benchmark results.

Hashing methods play an important role in large scale image retrieval. Traditional hashing methods use hand-crafted features to learn hash functions, which can not capture the high level semantic information. Deep hashing algorithms use deep neural networks to learn feature representation and hash functions simultaneously. Most of these algorithms exploit supervised information to train the deep network. However, supervised information is expensive to obtain. In this paper, we propose a pseudo label based unsupervised deep discriminative hashing algorithm. First, we cluster images via K-means and the cluster labels are treated as pseudo labels. Then we train a deep hashing network with pseudo labels by minimizing the classification loss and quantization loss. Experiments on two datasets demonstrate that our unsupervised deep discriminative hashing method outperforms the state-of-art unsupervised hashing methods.

No single retargeting approach performs well on all images and all target sizes; therefore, hybrid algorithms are often considered as promising alternatives. However, most hybrid schemes are time consuming. In this paper, we propose a fast hybrid framework in which the Fast Content-Aware Image Distance (FCAID) is used to connect fast seam carving with neighboring probability constraints (FSc_Neip) and scaling. FCAID is used to measure the image distance between the resized image given by FSc_Neip and the original image. This fast technique is embedded within the FSc_Neip framework. Our hybrid scheme is locally applied in strip regions. This makes the retargeting scheme globally non-homogeneous. Experimental results demonstrate that our approach comprehensively outperforms other state-of-the-art techniques in terms of image quality and computational complexity.

Pose variation is one of challenging problems in face recognition. Complete Pose Binary SIFT(CPBS) [1]has been proposed to extract binary SIFT from face images of five poses. It could resolve the problem of large pose variation. However, in CPBS, each feature is represented as a bit, it possibly brings about information loss. In this paper, we propose maximum bit average entropy algorithm (MBAE) for binarization of CPBS features. And these features are used for face recognition with pose variation. It preserve at most information with the fewer quantization bits. Firstly, the CPBS features are extracted from the gallery images. Secondly, the CPBS features are binarized according to the maximum bit average entropy adaptively. Finally, the distance between a pair of features is computed based on the weighted hamming distance. The compared experimental results on the CMU-PIE and FERET face databases show that our approach is much better than state-of-the-art algorithms.

Photo recommendation in photo-sharing social networks like Flickr is an important problem. Collaborative filtering is very popular, which assumes each item has the same weight for recommendation. In practice some items are representatives for a class of items and therefore are more important for recommendation. In this paper, we model the importance for items by examining sentiment from the general public towards items. Specifically we propose a model using the temporal dynamic user `favor' information to infer photo importance on Flickr. It is further combined with local community user ratings to improve the Probabilistic Matrix Factorization (PMF) framework for photo recommendation. Experiment results show the effectiveness of the proposed approach.

Seam carving is an effective way of image retargeting. However, existing seam carving schemes often come with unacceptable artifacts and are quite time consuming. In this paper, we propose a fast seam carving scheme with strip partition and neighboring probability constraints to resolve these two problems simultaneously. Firstly, we split the original image into several strips of equal space and we estimate the importance of each strip by its average saliency values. This partition results that more seams are removed from the strips consisting of more unimportant regions while fewer seams are removed from that of more important regions. Then, we establish the adjacency relationship by maximum correlation [8]. The neighboring probability is obtained to describe the neighboring relationship between the seams. Finally, by combining the neighboring probability and their accumulated energy, least important seams are removed. The neighboring probability constraint ensures that the seam removal is distributed to avoid abrupt changes in the scene. This leads to an improved quality in the resized image. The experimental results show that the proposed approach performs better than the state-of-the-art seam carving schemes.

SUMMARY: ODORactor is an open access web server aimed at providing a platform for identifying odorant receptors (ORs) for small molecules and for browsing existing OR-ligand pairs. It enables the prediction of ORs from the molecular structures of arbitrary chemicals by integrating two individual functionalities: odorant verification and OR recognition. The prediction of the ORs for several odorants was experimentally validated in the study. In addition, ODORactor features a comprehensive repertoire of olfactory information that has been manually curated from literature. Therefore, ODORactor may provide an effective way to decipher olfactory coding and could be a useful server tool for both basic olfaction research in academia and for odorant discovery in industry. AVAILABILITY: Freely available at http://mdl.shsmu.edu.cn/ODORactor CONTACT: [email protected] SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Vision based human motion tracking has drawn considerable interests recently because of its extensive applications. In this paper, we propose an approach to tracking the body motion of human balancing on each foot. The ability to balance properly is an important indication of neurological condition. Comparing with many other human motion tracking, there is much less occlusion in human balancing tracking. This less constrained problem allows us to combine a 2D model of human body with image analysis techniques to develop an efficient motion tracking algorithm. First we define a hierarchical 2D model consisting of six components including head, body and four limbs. Each of the four limbs involves primary component (upper arms and legs) and secondary component (lower arms and legs) respectively. In this model, we assume each of the components can be represented by quadrangles and every component is connected to one of others by a joint. By making use of inherent correlation between different components, we design a top-down updating framework and an adaptive algorithm with constraints of foreground regions for robust and efficient tracking. The approach has been tested using the balancing movement in HumanEva-I/II dataset. The average tracking time is under one second, which is much shorter than most of current schemes.