Continual Text-to-Video Retrieval (CTVR) is a challenging multimodal continual learning setting, where models must incrementally learn new semantic categories while maintaining accurate text-video alignment for previously learned ones, thus making it particularly prone to catastrophic forgetting. A key challenge in CTVR is feature drift, which manifests in two forms: intra-modal feature drift caused by continual learning within each modality, and non-cooperative feature drift across modalities that leads to modality misalignment. To mitigate these issues, we propose StructAlign, a structured cross-modal alignment method for CTVR. First, StructAlign introduces a simplex Equiangular Tight Frame (ETF) geometry as a unified geometric prior to mitigate modality misalignment. Building upon this geometric prior, we design a cross-modal ETF alignment loss that aligns text and video features with category-level ETF prototypes, encouraging the learned representations to form an approximate simplex ETF geometry. In addition, to suppress intra-modal feature drift, we design a Cross-modal Relation Preserving loss, which leverages complementary modalities to preserve cross-modal similarity relations, providing stable relational supervision for feature updates. By jointly addressing non-cooperative feature drift across modalities and intra-modal feature drift, StructAlign effectively alleviates catastrophic forgetting in CTVR. Extensive experiments on benchmark datasets demonstrate that our method shows competitive advantages over state-of-the-art continual retrieval approaches.

Unlike conventional visual question answering, video-grounded dialog requires a deep understanding of both the dialog history and the video content to generate accurate responses. Although existing methods have achieved promising results, they still struggle with progressively comprehending complex dialog history and effectively integrating video information. To address these challenges, we propose an iterative search and reasoning framework composed of a textual encoder, a visual encoder, and a generator. Specifically, the textual encoder adopts a path search and aggregation strategy to identify key cues in the dialog history that are essential for understanding the current question. Meanwhile, the visual encoder employs an iterative reasoning network to extract and highlight critical visual evidence from the video, thereby enabling more comprehensive visual understanding. Finally, we use a pretrained GPT-2 model as the answer generator to transform the discovered latent cues into coherent and contextually appropriate responses. Extensive experiments on three public datasets demonstrate the effectiveness and generalizability of the proposed framework.

Current knowledge distillation methods for semantic segmentation are primarily designed for knowledge transfer within homogeneous networks, and are less effective for heterogeneous networks. The feature information output by heterogeneous networks faces several challenges, including differences in feature scales and varying capabilities to represent local and global contextual information. To address these issues, we propose a novel Heterogeneous Model Knowledge Distillation (HMKD) framework using a dual alignment method to improve distillation performance between CNN-based and Transformer-based semantic segmentation models. Specifically, we introduce the Patch-based Self-attention Alignment Module (PSAM), which computes and aligns patch-level self-attention across distinct feature map spaces, enabling the transfer of local or global contextual information between heterogeneous models. Additionally, the Heterogeneous Scale Alignment Module (HSAM) is designed to ensure consistency across heterogeneous feature scales and enrich the semantic content. We have also conducted extensive experiments on two benchmark datasets (Cityscapes and CamVid) to validate the effectiveness and superiority of our approach compared with several recent state-of-the-art (SOTA) methods. Our code is deposited at https://github.com/xumingzhu989/HMKD-ICMR.

With the explosive growth of multimedia contents, multimedia retrieval is facing unprecedented challenges on both storage cost and retrieval speed. Hashing technique can project the high-dimensional data into compact binary hash codes. With it, the most time-consuming semantic similarity computation during the multimedia retrieval process can be significantly accelerated with fast Hamming distance computation, and meanwhile the storage cost can be reduced greatly by the binary embedding. In the light of this, multi-modal hashing has recently received considerable attention to support large-scale multimedia retrieval. Different from uni-modal hashing, the multi-modal hashing focuses on modeling the multi-modal semantics and further preserving them into binary hash codes with hash learning. In this paper, we first systematically review the existing learning to hash methods for efficient multimedia retrieval, categorizing them according to the multimedia retrieval tasks, the specific multi-modal semantic modeling techniques, and hash learning strategies. Thereafter, we present the performance comparison results. We ultimately discuss the challenges and potential research directions that may require further investigation in multi-modal hash learning.

Unsupervised Non-overlapping Cross-domain Recommendation (UNCR) is the task that recommends source domain items to the target domain users, which is more challenging as the users are non-overlapped, and its learning process is unsupervised. Unsupervised Non-overlapping Cross-domain Recommendation UNCR is still unsolved due to the following: (1) Previous studies need extra auxiliary information to learn transferable features when aligning two domains, which is unrealistic and hard to obtain due to privacy concerns. (2) Since the adoption of the shared network, existing works cannot well eliminate the domain-specific features in the common feature space, which may incorporate domain noise and harm the cross-domain recommendation. In this work, we propose a domain adaption-based method, namely DA-DAN, to address the above challenges. Specifically, to let DA-DAN be free of auxiliary information, we learn users’ preferences by only exploring their sequential patterns, and propose an improved self-attention layer to model them. To well eliminate the domain-specific features from the common feature space, we resort to a dual generative adversarial network with a multi-target adversarial loss, where two generators and discriminators are leveraged to model each domain separately. Experimental results on three real-world datasets demonstrate the advantage of DA-DAN compared with the state-of-the-art recommendation baselines. Moreover, our source codes have been publicly released. 1

Abstract - Finding tampered regions in images is a common research topic in machine learning and computer vision. Although many image manipulation location algorithms have been proposed, most of them only focus on RGB images with different color spaces, and the frequency information that contains the potential tampering clues is often ignored. Moreover, among the manipulation operations, splicing and copy-move are two frequently used methods, but as their characteristics are quite different, specific methods have been individually designed for detecting the operations of either splicing or copy-move, and it is very difficult to widely apply these methods in practice. To solve these issues, in this work, a novel end-to-end two-stream boundary-aware network (abbreviated as TBNet) is proposed for generic image manipulation localization where the RGB stream, the frequency stream, and the boundary artifact location are explored in a unified framework. Specifically, we first design an adaptive frequency selection module (AFS) to adaptively select the appropriate frequency to mine inconsistent statistics and eliminate the interference of redundant statistics. Then, an adaptive cross-attention fusion module (ACF) is proposed to adaptively fuse the RGB feature and the frequency feature. Finally, the boundary artifact location network (BAL) is designed to locate the boundary artifacts for which the parameters are jointly updated by the outputs of the ACF, and its results are further fed into the decoder. Thus, the parameters of the RGB stream, the frequency stream, and the boundary artifact location network are jointly optimized, and their latent complementary relationships are fully mined. The results of the extensive experiments performed on six public benchmarks of the image manipulation localization task, namely, CASIA1.0, COVER, Carvalho, In-The-Wild, NIST-16, and IMD-2020, demonstrate that the proposed TBNet can substantially outperform state-of-the-art generic image manipulation localization methods in terms of MCC, F1, and AUC while maintaining robustness with respect to various attacks. Compared with DeepLabV3+ on the CASIA1.0, COVER, Carvalho, In-The-Wild, and NIST-16 datasets, the improvements in MCC/F1 reach 11%/11.1%, 8.2%/10.3%, 10.2%/11.6%, 8.9%/6.2%, and 13.3%/16.0%, respectively. Moreover, when IMD2020 is utilized, its AUC improvement can achieve 14.7%.

Session-based recommendation (SBR) has drawn increasingly research attention in recent years, due to its great practical value by only exploiting the limited user behavior history in the current session. The key of SBR is to accurately infer the anonymous user purpose in a session which is typically represented as session embedding, and then match it with the item embeddings for the next item prediction. Existing methods typically learn the session embedding at the item level, namely, aggregating the embeddings of items with or without assigned attention weights to items. However, they ignore the fact that a user's intent on adopting an item is driven by certain factors of the item (e.g., theleading actorsof an movie). In other words, they have not explored finer-granularity interests of users at the factor level to generate the session embedding, leading to sub-optimal performance. To address the problem, we propose a novel method called Disentangled Graph Neural Network (Disen-GNN) to capture the session purpose with the consideration of factor-level attention on each item. Specifically, we first employ the disentangled learning technique to cast item embeddings into the embeddings of multiple factors, and then use the gated graph neural network (GGNN) to learn the embedding factor-wisely based on the item adjacent similarity matrix computed for each factor. Moreover, the distance correlation is adopted to enhance the independence between each pair of factors. After representing each item with independent factors, an attention mechanism is designed to learn user intent to different factors of each item in the session. The session embedding is then generated by aggregating the item embeddings with attention weights of each item's factors. To this end, our model takes user intents at the factor level into account to infer the user purpose in a session. Extensive experiments on three benchmark datasets demonstrate the superiority of our method over existing methods.

Cross-media hashing, which encodes data points from different modalities into a common Hamming space, has been successfully applied to solve large-scale multimedia retrieval issue due to storage efficiency and search effectiveness. Recently, matrix factorization based hashing methods have drawn considerable attention for their promising search accuracy. However, pioneer methods mainly focus on learning consensus hash codes for different modalities, but neglect the potential inconsistency among different modalities, \emph{e.g.,} the diversities of different modalities and noises, which may undermine the retrieval accuracy. To address this problem, we propose a novel unsupervised hashing model, namely, Discrete Robust Matrix Factorization Hashing (DRMFH), which simultaneously formulates the consistency and inconsistency across different modalities into a matrix factorization based model. Specifically, a homogenous space composed of a consistent Hamming space and an inconsistent diversity part, are generated by matrix factorization for each modality. Therefore, the consensus information across different modalities can be well captured in the learnt hash codes, leading to improved retrieval performance. Moreover, we design an effective optimization algorithm which is able to obtain an approximate discrete code matrix with linear time complexity. Comprehensive experimental results on three public multimedia retrieval datasets show that the proposed DRMFH outperforms several state-of-the-art methods.

Graphs are widely used to model various practical applications. In recent years, graph convolution networks (GCNs) have attracted increasing attention due to the extension of convolution operation from traditional grid data to graph one. However, the representation ability of current GCNs is undoubtedly limited because existing work fails to consider feature interactions. Toward this end, we propose a Dual Feature Interaction-based GCN. Specifically, it models feature interaction in the aspects of 1) node features where we use Newton's identity to extract different-order cross features implicit in the original features and design an attention mechanism to fuse them; and 2) graph convolution where we capture the pairwise interactions among nodes in the neighborhood to expand a weighted sum operation. We evaluate the proposed model with graph data from different fields, and the experimental results on semi-supervised node classification and link prediction demonstrate the effectiveness of the proposed GCN. The data and source codes of this work are available athttps://github.com/ZZY-GraphMiningLab/DFI-GCN.

Unsupervised cross-modal hashing has attracted considerable attention to support large-scale cross-modal retrieval. Although promising progresses have been made so far, existing methods still suffer from limited capability on excavating and preserving the intrinsic multi-modal semantics. In this paper, we propose a Correlation-Identity Reconstruction Hashing (CIRH) method to alleviate this challenging problem. We develop a new unsupervised deep cross-modal hash learning framework to model and preserve the heterogeneous multi-modal correlation semantics into both hash codes and functions, and simultaneously, we involve both the hash codes and functions with the descriptive identity semantics. Specifically, we construct a multi-modal collaborated graph to model the heterogeneous multi-modal correlations, and jointly perform the intra-modal and cross-modal semantic aggregation on homogeneous and heterogeneous graph networks to generate a multi-modal complementary representation with correlation reconstruction. Furthermore, an identity semantic reconstruction process is designed to involve the generated representation with identity semantics by reconstructing the input modality representations. Finally, we propose a correlation-identity consistent hash function learning strategy to transfer the modelled multi-modal semantics into the neural networks of modality-specific deep hash functions. Experiments demonstrate the superior performance of the proposed method on both retrieval accuracy and efficiency. We provide our source codes and experimental datasets athttps://github.com/XizeWu/CIRH

Using hashing to learn the binary codes of users and items significantly improves the efficiency and reduces the space consumption of the recommender system. However, existing hashing-based recommender systems remain black boxes without any explainable outputs that illustrate why the system recommends the items. In this paper, we present a new end-to-end discrete recommendation framework based on the multi-task learning to simultaneously perform explainable and efficient recommendation. Toward this goal, an Explainable Discrete Collaborative Filtering (EDCF) method is proposed to preserve the user-item interaction features and semantic text features into binary hash codes by adaptively exploiting the correlations between the preference prediction task and the explanation generation task. At the online recommendation stage, EDCF makes efficient top-K recommendation by calculating the Hamming distances between the feature hash codes, and simultaneously generates natural language explanations for recommendation results through the explanation generation module. To obtain the hash codes directly from the end-to-end neural network, we introduce an attentive TextCNN and an Adaptive Tanh layer in the preference prediction task. For explanation generation, Long Short-Term Memory is employed to generate the explanations for recommendation results from the binary hash codes of user and item. Experiments demonstrate the superiority of the proposed method.

Online recommendation requires handling rapidly changing user preferences. Deep reinforcement learning (DRL) is an effective means of capturing users' dynamic interest during interactions with recommender systems. Generally, it is challenging to train a DRL agent in online recommender systems because of the sparse rewards caused by the large action space (e.g., candidate item space) and comparatively fewer user interactions. Leveraging experience replay (ER) has been extensively studied to conquer the issue of sparse rewards. However, they adapt poorly to the complex environment of online recommender systems and are inefficient in learning an optimal strategy from past experience. As a step to filling this gap, we propose a novel state-aware experience replay model, in which the agent selectively discovers the most relevant and salient experiences and is guided to find the optimal policy for online recommendations. In particular, a locality-sensitive hashing method is proposed to selectively retain the most meaningful experience at scale and a prioritized reward-driven strategy is designed to replay more valuable experiences with higher chance. We formally show that the proposed method guarantees the upper and lower bound on experience replay and optimizes the space complexity, as well as empirically demonstrate our model's superiority to several existing experience replay methods over three benchmark simulation platforms.

Fashion Compatibility Modeling (FCM) is a new yet challenging task, which aims to automatically access the matching degree among a set of complementary items. Most of existing methods evaluate the fashion compatibility from the common perspective, but overlook the user's personal preference. Inspired by this, a few pioneers study the Personalized Fashion Compatibility Modeling (PFCM). Despite their significance, these PFCM methods mainly concentrate on the user and item entities, as well as their interactions, but ignore the attribute entities, which contain rich semantics. To address this problem, we propose to fully explore the related entities and their relations involved in PFCM to boost the PFCM performance. This is, however, non-trivial due to the heterogeneous contents of different entities, embeddings for new users, and various high-order relations. Towards these ends, we present a novel metapath-guided personalized fashion compatibility modeling, dubbed as MG-PFCM. In particular, we creatively build a heterogeneous graph to unify the three types of entities (i.e., users, items, and attributes) and their relations (i.e., user-item interactions, item-item matching relations, and item-attribute association relations). Thereafter, we design a multi-modal content-oriented user embedding module to learn user representations by inheriting the contents of their interacted items. Meanwhile, we define the user-oriented and item-oriented metapaths, and perform the metapath-guided heterogeneous graph learning to enhance the user and item embeddings. In addition, we introduce the contrastive regularization to improve the model performance. We conduct extensive experiments on the real-world benchmark dataset, which verifies the superiority of our proposed scheme over several cutting-edge baselines. As a byproduct, we have released our source codes to benefit other researchers.

Multi-modal hashing learns binary hash codes with extremely low storage cost and high retrieval speed. It can support efficient multi-modal retrieval well. However, most existing methods still suffer from three important problems: 1) Limited semantic representation capability with shallow learning. 2) Mandatory feature-level multi-modal fusion ignores heterogeneous multi-modal semantic gaps. 3) Direct coarse pairwise semantic preserving cannot effectively capture the fine-grained semantic correlations. For solving these problems, in this paper, we propose a Bit-aware Semantic Transformer Hashing (BSTH) framework to excavate bit-wise semantic concepts and simultaneously align the heterogeneous modalities for multi-modal hash learning on the concept-level. Specifically, the bit-wise implicit semantic concepts are learned with the transformer in a self-attention manner, which can achieve implicit semantic alignment on the fine-grained concept-level and reduce the heterogeneous modality gaps. Then, the concept-level multi-modal fusion is performed to enhance the semantic representation capability of each implicit concept and the fused concept representations are further encoded to the corresponding hash bits via bit-wise hash functions. Further, to supervise the bit-aware transformer module, a label prototype learning module is developed to learn prototype embeddings for all categories that capture the explicit semantic correlations on the category-level by considering the co-occurrence priors. Experiments on three widely tested multi-modal retrieval datasets demonstrate the superiority of the proposed method from various aspects.

View-based 3D model retrieval has become an important task in both computer vision and machine learning domains. Although deep learning methods have achieved excellent performances on view-based 3D model retrieval, the intrinsic correlation and the degree of view discrimination among multiple views in a 3D model have not been effectively exploited. To obtain a more efficient feature descriptor for 3D model retrieval, in this work, we propose the pairwise view weighted graph network (abbreviated PVWGN) for view-based 3D model retrieval where non-local graph layers are embedded into the network architecture to automatically mine the intrinsic relationship among multiple views of a 3D model. Furthermore, the view weighted layer is employed in the PVWGN to adaptively assign the weight to each view according to its aggregation information. In addition, the pairwise discrimination loss function is designed to improve the feature discrimination of the 3D model. Most importantly, these three issues are integrated into a unified framework. Extensive experimental results on the ModelNet40 and ModelNet10 3D model retrieval datasets show that PVWGN can outperform all state-of-the-art methods on the 3D model retrieval task with mAPs of 93.2% and 96.2%, respectively.