Tuning-free face personalization methods have developed along two distinct paradigms: text embedding approaches that map facial features into the text embedding space, and adapter-based methods that inject features through auxiliary cross-attention layers. While both paradigms have shown promise, existing methods struggle to simultaneously achieve high identity fidelity and flexible text controllability. We introduce UniID, a unified tuning-free framework that synergistically integrates both paradigms. Our key insight is that when merging these approaches, they should mutually reinforce only identity-relevant information while preserving the original diffusion prior for non-identity attributes. We realize this through a principled training-inference strategy: during training, we employ an identity-focused learning scheme that guides both branches to capture identity features exclusively; at inference, we introduce a normalized rescaling mechanism that recovers the text controllability of the base diffusion model while enabling complementary identity signals to enhance each other. This principled design enables UniID to achieve high-fidelity face personalization with flexible text controllability. Extensive experiments against six state-of-the-art methods demonstrate that UniID achieves superior performance in both identity preservation and text controllability.

3D Visual Question Answering (3D-VQA) aims to understand the positional relationship, object attributes and layout in 3D scenes. A challenging issue is to align the representations between the entities associated with questions and the relevant 3D objects while diminishing the fine-grained vision-language relations. To this end, we propose a Scene-guided Attention Network for Spatial Understanding, denoted as SceSU, to perceive spatial information and attribute information based on different types of questions in 3D scenes. More specifically, a Scene-Driven Spatial Understanding (SDSU) mechanism is designed to obtain the crucial entities relevant to the question, and construct the fine-grained scene description via a large language model. Furthermore, a 3D Perception Attention (3D-PA) module is devised to fuse natural language and 3D features while understanding the detailed relationship between them by employing a dual-branch attention network. Finally, SceSU utilizes the 3D-PA to comprehend the fine-grained scene description generated by the SDSU mechanism, bridging the gap between natural language and 3D domains. Extensive experiments conducted on SQA3D and ScanQA datasets demonstrate the effectiveness of the SceSU for 3D-VQA.

The remarkable capability of generative methods to synthesize public-domain speeches that are difficult to differentiate from natural ones has raised concerns about the proliferation of disinformation. To this end, we contribute a fake speech detection dataset, denoted as the SynSpeech dataset, which includes fully fake and partially fake speeches in multiple domains covering both GANs-based and diffusion-based generative methods, serving as a prerequisite to support fake speech detectors. For the partially fake speeches, we utilize a large language model to replace interest segments like individual names, locations, adjectives, and verbs. Additionally, these segments will be substituted with fabricated clips synthesized by generative methods to create sentences with completely opposite meanings as partially fake speech. More specifically, a total of 164,924 fully fake speeches from six distinct generators, 44,070 real speeches from 108 individuals, and 99,496 partially fake speeches have been annotated as three kinds of labels, i.e., fully fake, real, and partially fake, respectively. In the experiments, we investigate quite a few fake speech detection models and customize them for two fake speech detection tasks for comparisons.

Knowledge-based visual question answering (KB-VQA) requires reasoning about the visual grounding relations between the images and questions by incorporating external knowledge. Existing works typically retrieve knowledge from knowledge graphs by leveraging global multimodal representations of image–text pairs for graph convolution, which neglect contextual clues at hop granularity, resulting in suboptimal spreading and leveraging of contextual information. To this end, we propose a multi-hop graph reasoning network (MGRN) for KB-VQA, which consists of a knowledge graph constructor (KGC) module, a semantic-instructed graph reasoning (SGR) module, and an answering module. MGRN exploits multimodal semantics from given images and questions as instructions for graph reasoning to obtain the knowledge representation from either the scene graph or knowledge base. Specifically, KGC fuses the scene graph with triplets from ConceptNet and Comet to construct a contextual knowledge graph for retrieving knowledge representation. Furthermore, SGR conducts multi-hop graph reasoning to select top- K knowledge items for answering by passing and filtering interplay messages on contextual knowledge graphs under the guidance of multimodal semantic representation. Extensive experiments conducted on two public datasets show the effectiveness and outperformance of our method.

Multimodal news event detection aims to identify and categorize significant events across media platforms using multimodal data. Previous work was limited to a single platform and assumed complete multimodal data. In this paper, we explore a novel task of cross-platform multimodal news event detection to enhance model generalization for cross-platform scenarios. We propose a Self-Supervised Modality Complementation (SSMC) method to tackle the challenges of incomplete modalities and platform heterogeneity presented in this task. Specifically, a Missing Data Complementation (MDC) module is designed to overcome the limitations caused by incomplete modalities. It employs a separation mechanism that distinguishes between modality-specific and modality-shared features across all modalities, allowing for the augmentation of missing modalities with information extracted from common features. Meanwhile, a Multimodal Self-Learning (MSL) module addresses platform heterogeneity by extracting pseudo labels from the target platform's multimodal views and incorporating a self-penalization mechanism to reduce reliance on low-confidence labels. Additionally, we collect a comprehensive cross-platform news event detection (CNED) dataset encompassing 37,711 multimodal samples from Twitter, Flickr, and online news media, covering 40 public news events verified by Wikipedia. Extensive experiments on the CNED dataset demonstrate the superior performance of our proposed method. The dataset is available athttps://github.com/RetrainIt/CNED.

In this paper, we propose a two-stage network with modality-specific and -shared contrastive learning (MMCL) for multimodal sentiment analysis. MMCL comprises a category-aware modality-specific contrastive (CMC) module and a self-decoupled modality-shared contrastive (SMC) module. In the first stage, the CMC module guides the encoders to extract modality-specific representations by constructing positive-negative pairs according to sample categories. In the second stage, the SMC module guides the encoders to extract modality-shared representations by constructing positive-negative pairs based on modalities and decoupling the self-contrast of all modalities. In the aforementioned modules, we leverage self-modulation factors to focus more on hard positive pairs through assigning different loss weights to positive pairs depending on their distance. In particular, we introduce a dynamic routing algorithm to cluster the inputs of the contrastive modules during training, where a gradient stopping strategy is utilized to isolate the backpropagation process of the CMC and SMC modules. Extensive experiments on the CMU-MOSI and CMU-MOSEI datasets show that MMCL achieves the state-of-the-art performance.

The cross-attention mechanism has been widely employed in the multimodal rumor detection task, which is computation-intensive and suffers from the restricted modal receptive field. In this paper, we propose a multimodal rumor detection model (MKV), which maps multimodal key semantics with discrimination into feature vectors for rumor detection. More specifically, MKV extracts high-dimensional features for each modality separately by the Multimodal Feature Extractor (MFE). The mapping mechanism learns low-dimensional mapping scheme (Map) and key semantics (Key) with discrimination from the different modal features respectively. Subsequently, the Map and Key jointly construct a state matrix (State) containing all possible permutations of modalities. In particular, a max pooling operation is performed on State and products a feature vector (Vector). The mapping mechanism is able to incrementally learn the discriminative semantics by stacking manner. Vectors from the stacking process are leveraged in the Rumor Detection module (RD). Extensive experiments on two public datasets show that the MKV achieves the state-of-the-art performance.

Social media serves as a real-time collecting and disseminating center of users’ ideas, opinions, and experiences. The deliberate disinformation and rumors propagate rapidly online due to their exaggerated facts, controversial opinions, divisive perspectives, and stunning expressions. Rumor detection approaches typically use social media posts with rumor or non-rumor labels for training and testing without disclosing the rationale behind decision-makings. On one hand, collecting evidence data to verify claims relies on expert efforts. On the other hand, verifying the truthfulness of confusing claims with distracting and lengthy evidences is still challenging. In this paper, we contribute a rumor detection dataset with multi-granularity evidences, denoted as the RD-E dataset, which includes response, fact-check, article, sourcing data and generated evidence by large language models, supporting models to verify the truthfulness of claims on social media. A number of 32,892 claims from 4,525 public individuals and organizations are annotated to 6 kinds of labels, including true, mostly true, half true, mostly false, false, pants on fire, covering a wide range of topics, e.g., politics, economy, society, technology, and health. In the experiments, seven rumor detection models have been investigated and customized on four predefined subtasks for comparisons.

Social media allows the public to access information conveniently, in which the false messages that are eye-catching may spread fast. In this paper, we propose a two-stage confidence-guided boundary adaption (CBA) network, consisting of a feature preprocessing (FP) module, a biased ambiguity learning (BA) module and a confidence-guided boundary adaptation (CG) module. In the first stage, the FP module obtains the textual and visual features, which are fused by conducting the visual-to-textual and textual-to-visual correlation coefficients with attention mechanism. Furthermore, BA evaluates the distribution distance between fused features and single modalities to determine the weights between modalities, capturing the semantics of key modality. In the second stage, CG leverages samples from the low-confidence interval to generate new instances using a mixup of augmentation techniques, aiming to occupy the decision space and optimize the decision boundary of the classifier. Extensive experiments on two public datasets show that our CBA model is 1.6% and 2.6% higher than the state-of-the-art methods.

Visual question answering (VQA) is a challenging task that reasons over questions on images with knowledge. A prerequisite for VQA is the availability of annotated datasets, while the available datasets have several limitations. 1) The diversity of questions and answers are limited to a few question categories and certain concepts (e.g., objects, relations, actions.) with somewhat mechanical answers. 2) The availability of background knowledge or context information has been disregarded with just images, questions and answers being provided. 3) The timeliness of knowledge has not been examined, though some works may introduce factual or commonsense knowledge bases, e.g., ConceptNet, DBPedia. In this paper, we provide an Event-oriented Visual Question Answering (E-VQA) dataset including free-form questions and answers for real-world event concepts, which provides context information of events as domain knowledge in addition to images. E-VQA consists of 2,690 social media images, 9,088 questions, 5,479 answers, and 1,157 news media articles for references being annotated to 182 real-world events, covering a wide range of topics, such as armed conflicts and attacks, disasters and accidents, law and crime. For comparisons, we investigate 10 state-of-the-art VQA methods as benchmarks.

In this paper, we propose a multi-branch semantic learning network (MSLN) to generate image according to textual description by taking into account global and local textual semantics, which consists of two stages. The first stage generates a coarse-grained image based on the sentence features. In the second stage, a multi-branch fine-grained generation model is constructed to inject the sentence-level and word-level semantics into two coarse-grained images by global and local attention modules, which generate global and local fine-grained image textures, respectively. In particular, we devise a channel fusion module (CFM) to fuse the global and local fine-grained features in the multi-branch fine-grained stage and generate the output image. Extensive experiments conducted on the CUB-200 dataset and Oxford-102 dataset demonstrate the superior performance of the proposed method. (e.g., FID is reduced from 16.09 to 14.43 on CUB-200).

In this paper, a novel Deep Semantic Space learning model with Intra-class Low-rank constraint (DSSIL) is proposed for cross-modal retrieval, which is composed of two subnetworks for modality-specific representation learning, followed by projection layers for common space mapping. In particular, DSSIL takes into account semantic consistency to fuse the cross-modal data in a high-level common space, and constrains the common representation matrix within the same class to be low-rank, in order to induce the intra-class representations more relevant. More formally, two regularization terms are devised for the two aspects, which have been incorporated into the objective of DSSIL. To optimize the modality-specific subnetworks and the projection layers simultaneously by exploiting the gradient decent directly, we approximate the nonconvex low-rank constraint by minimizing a few smallest singular values of the intra-class matrix with theoretical analysis. Extensive experiments conducted on three public datasets demonstrate the competitive superiority of DSSIL for cross-modal retrieval compared with the state-of-the-art methods.

Scholars' homepages are important places to show personal research interest and academic achievement through the Web. However, according to our observation, only a small portion of scholars update their publications and related events on their homepages in time. In this paper, we propose a homepage augmentation technique, which automatically shows the newest academic events related to a scholar on his/her homepage. Specifically, we model the relations between homepages and the events collected from the Web as a complex heterogenous network, and propose an Embedding-based Heterogenous random Walk algorithm, namely EHWalk, to predict the links between homepages and events. Compared with existing embedding-based link prediction algorithms, EHWalk supports more efficient modeling of complex heterogenous relations in a dynamically changing network, which helps link the massive new updated events to homepages precisely and efficiently. Comprehensive experiments on a real-world dataset are conducted and the results show that our algorithm can achieve both good effectiveness and efficiency for real-world deployment.