Multi-task test-time adaptation (MT-TTA) aims to adapt pre-trained models to dynamic environments during multi-task inference by leveraging unlabeled test data. This task is particularly challenging as different tasks respond divergently to distribution shifts, and mixed input streams containing both in-distribution (ID) and out-of-distribution (OOD) samples make the models after test-time adaptation prone to catastrophic forgetting of ID knowledge. Although the existing methods like M-TENT extend the classic test entropy minimization (TENT) by minimizing multi-task entropies and employing task-average gradient to adapt a model, it suffers from two key limitations: 1) the average gradient strategy proposed by M-TENT may exacerbate multi-task test-time optimization conflicts, harming individual tasks when gradients are directionally non-consensual; 2) aggressive updates on mixed ID/OOD data cause severe forgetting of ID knowledge. In this paper, we theoretically establish a formal connection between multi-task loss differences and test-time performance under the first-order Taylor analysis, demonstrating that consensual multi-task entropy reductions are likely to increase the performance, while non-consensual ones might decrease the performance. To this end, we propose Consensus-driven Constrained Multi-Task Test-Time Adaptation (CoCo-MT-TTA), consisting of 1) multi-task gradient consensus adaptation, which aligns cross-task gradient directions to seek a consensus gradient; 2) multi-task plasticity-constraint adaptation, which constrains parameter updates using second-moment statistics to preserve ID knowledge. Extensive experiments on benchmark datasets, including CelebA and PlantData, demonstrate that our method achieves an absolute improvement of up to 16.02% in mean ID/OOD F1-score (Mean I&O) under domain shifts over non-adapted models, outperforming the recent baselines.

Tuning-free face personalization methods have developed along two distinct paradigms: text embedding approaches that map facial features into the text embedding space, and adapter-based methods that inject features through auxiliary cross-attention layers. While both paradigms have shown promise, existing methods struggle to simultaneously achieve high identity fidelity and flexible text controllability. We introduce UniID, a unified tuning-free framework that synergistically integrates both paradigms. Our key insight is that when merging these approaches, they should mutually reinforce only identity-relevant information while preserving the original diffusion prior for non-identity attributes. We realize this through a principled training-inference strategy: during training, we employ an identity-focused learning scheme that guides both branches to capture identity features exclusively; at inference, we introduce a normalized rescaling mechanism that recovers the text controllability of the base diffusion model while enabling complementary identity signals to enhance each other. This principled design enables UniID to achieve high-fidelity face personalization with flexible text controllability. Extensive experiments against six state-of-the-art methods demonstrate that UniID achieves superior performance in both identity preservation and text controllability.

The prevalence of generative models in image and video generation has raised extensive concerns about potential harm and misuse. To identify the truthfulness of generated images, most of the existing methods typically apply Fast Fourier Transform (FFT) for frequency extraction. An existing problem is that the frequency-domain representations extracted by FFT are not comprehensive for AI-generated image detection. In this paper, we propose a Multi-perspective Frequency Domain Learning (MFDL) framework, which aims to learn both generalized and discriminative frequency representations via DWT and FFT. Specifically, we design a Frequency Representation Enhancement (FRE) module using the Discrete Wavelet Transform (DWT) and incorporating a multi-granularity enhancement strategy that amplifies all subbands across high frequency to improve discriminability. Additionally, we introduce a Frequency Representation Consistency (FRC) module, which employs complex convolution to capture and preserve forgery patterns in the real and imaginary components derived from FFT. By integrating complementary frequency representations from the DWT and FFT domains obtained through the FRE and FRC modules, MFDL achieves a comprehensive understanding of forgery traces in the frequency domain. This enhances the model’s generalization capability for detecting generated content. Extensive experiments conducted on 32 distinct datasets, covering both GAN-generated and Diffusion-based images, demonstrate the effectiveness of our proposed MFDL framework. These experiments validate the effectiveness of multi-perspective frequency domain learning and show that MFDL outperforms existing detection methods, confirming its strong generalization ability across diverse generative models.

External knowledge provides critical clues for knowledge-based visual question answering (KB-VQA), while the implicit knowledge in images is difficult to capture in order to construct effective queries for knowledge bases. To this end, we propose a visual entity-centric prompting for knowledge retrieval (VEPR) to bridge the gap between the implicit and explicit knowledge driven by visual entities via large language models. More specifically, a visual entity question answering (EQ) module is devised to localize the critical entities from the given images and questions to generate entity-centric questions via a large language model. In particular, EQ obtains several entity-centric question-answer pairs via a visual language model. Furthermore, a question-answer-enhanced retrieval (ER) module is devised to construct a query by summarizing the text including question-answer pairs, captions and questions, in order to require explicit knowledge items. Finally, a multi-branch reader (MR) module is designed to encode the given questions, visual content and retrieved knowledge items, which are decoded to make answer predictions. Extensive experiments conducted on two public datasets demonstrate the effectiveness of the VEPR.

In this paper, we propose a Modality-aware Diffusion Distillation Network (MDDN), consisting of Diffusion-based Modality Imputation (DMI) and Margin-aware Distillation (MAD) modules, for multimodal sentiment analysis under uncertain missing modalities. More specifically, the DMI module incrementally adds Gaussian noise into the modality-specific distribution space of the available data and recovers missing modalities while adhering to their original distributions, enabling effective imputation of missing data for the network. Furthermore, the MAD module introduces the classification uncertainty of generated samples and available data to reweigh their contributions to the loss function. Consequently, the missing modalities can be reconstructed by MDDN, and be fused dynamically with the existing ones for sentiment analysis. Extensive experiments on the MOSI and MOSEI datasets demonstrate the effectiveness and robustness of the proposed network for dealing with missing modalities in sentiment analysis.

Multimodal Aspect-based Sentiment Analysis aims at identifying specific aspects like celebrities, locations, and organizations, along with their corresponding sentiments. Typically, existing methods comprehend each aspect within images and their relationship without external knowledge. To this end, we propose an Aspect-Attentioned Prompting Framework (AAPF) with Vision-language Prompting (VP) and Attention Aggregation (AA) modules to leverage the fine-grained visual semantics corresponding to the conditional sentences. More specifically, the VP constructs aspect-oriented questions from the perspectives of objects, relations, and sentence-oriented questions, leveraging chain-of-thought prompting to guide the reasoning process. Given the questions, sentences, and images, a visual language model is exploited to perceive the aspect-sensitive intricate semantics, which are further filtered by CLIP scores and condensed into an aspect-relevant summarization. Furthermore, the AA module conducts complementary cross-attention on the summarizations and the conditional sentences to model their interactions by achieving aspect-attentioned features, which can be used to make sentiment predictions. Experiments conducted on two public datasets indicate that our approach achieves superior performance against the state-of-the-art methods.

Multimodal sentiment analysis uses text, visual, and audio data to assess user sentiment, while both the discrimination power of modalities and sample distributions over categories remain imbalanced in practice. To address these challenges, we propose a Multi-grained Perception Model with Perceived Contrastive Focal loss, denoted MGSA1. More specifically, we design a Multi-grained Cross-modal Attention Perception (MCP) module, which employs coarse-grained and fine-grained cross-modal attention to deeply explore the complementary semantics between modalities, thereby modeling sentiment polarity and intensity by fusing text-video and text-audio data, respectively. Modeling sentiment polarity and intensity helps alleviate feature interference between modalities due to their differing discriminative power. Furthermore, the Perceived Contrastive Focal (PCF) loss is designed to address the challenge of unbalanced samples. We enhance the focal loss by incorporating inverse document frequency to dynamically weight samples within each class. Furthermore, information noise contrastive estimation is introduced to replace the class probability predictions in the enhanced focal loss, thereby more effective differentiation between positive and negative samples. Experiments on the MOSI and MOSEI datasets demonstrate that MGSA outperforms all baselines across a range of metrics.

3D Visual Question Answering (3D-VQA) aims to understand the positional relationship, object attributes and layout in 3D scenes. A challenging issue is to align the representations between the entities associated with questions and the relevant 3D objects while diminishing the fine-grained vision-language relations. To this end, we propose a Scene-guided Attention Network for Spatial Understanding, denoted as SceSU, to perceive spatial information and attribute information based on different types of questions in 3D scenes. More specifically, a Scene-Driven Spatial Understanding (SDSU) mechanism is designed to obtain the crucial entities relevant to the question, and construct the fine-grained scene description via a large language model. Furthermore, a 3D Perception Attention (3D-PA) module is devised to fuse natural language and 3D features while understanding the detailed relationship between them by employing a dual-branch attention network. Finally, SceSU utilizes the 3D-PA to comprehend the fine-grained scene description generated by the SDSU mechanism, bridging the gap between natural language and 3D domains. Extensive experiments conducted on SQA3D and ScanQA datasets demonstrate the effectiveness of the SceSU for 3D-VQA.

Identity preservation is a critical capability in video generation and one of the core requirements for high-quality video synthesis. Existing approaches typically extract facial features from reference images as conditional inputs and inject them into the generation pipeline to maintain subject identity. However, in the IPVG Challenge 2025, state-of-the-art models such as ConcatID still fall short of delivering satisfactory identity preservation. To address this limitation, we propose a simple yet highly effective multi-branch video generation framework based on entity routing. Concretely, we integrate several fine-tuned dedicated models to compensate for the base model's weaknesses in identity preservation, dynamically selecting the appropriate branch according to each prompt. In addition, we employ enhanced prompts to further steer the generation process. Remarkably, using just a single NVIDIA RTX 3090 GPU for 120 hours of training, we boost the baseline's cur_score from 0.242 to 0.313.

The remarkable capability of generative methods to synthesize public-domain speeches that are difficult to differentiate from natural ones has raised concerns about the proliferation of disinformation. To this end, we contribute a fake speech detection dataset, denoted as the SynSpeech dataset, which includes fully fake and partially fake speeches in multiple domains covering both GANs-based and diffusion-based generative methods, serving as a prerequisite to support fake speech detectors. For the partially fake speeches, we utilize a large language model to replace interest segments like individual names, locations, adjectives, and verbs. Additionally, these segments will be substituted with fabricated clips synthesized by generative methods to create sentences with completely opposite meanings as partially fake speech. More specifically, a total of 164,924 fully fake speeches from six distinct generators, 44,070 real speeches from 108 individuals, and 99,496 partially fake speeches have been annotated as three kinds of labels, i.e., fully fake, real, and partially fake, respectively. In the experiments, we investigate quite a few fake speech detection models and customize them for two fake speech detection tasks for comparisons.

Recent advancements in text-guided image editing have achieved notable success by leveraging natural language prompts for fine-grained semantic control. However, certain editing semantics are challenging to specify precisely using textual descriptions alone. A practical alternative involves learning editing semantics from paired source-target examples. Existing exemplar-based editing methods still rely on text prompts describing the change within paired examples or learning implicit text-based editing instructions. In this paper, we introduce PairEdit, a novel visual editing method designed to effectively learn complex editing semantics from a limited number of image pairs or even a single image pair, without using any textual guidance. We propose a target noise prediction that explicitly models semantic variations within paired images through a guidance direction term. Moreover, we introduce a content-preserving noise schedule to facilitate more effective semantic learning. We also propose optimizing distinct LoRAs to disentangle the learning of semantic variations from content. Extensive qualitative and quantitative evaluations demonstrate that PairEdit successfully learns intricate semantics while significantly improving content consistency compared to baseline methods. Code is available at https://github.com/xudonmao/PairEdit.

Addressing the bias problem in multimodal zero-shot learning tasks is challenging due to the domain shift between seen and unseen classes, as well as the semantic gap across different modalities. To tackle these challenges, we propose a multimodal disentangled fusion network (MDFN) that unifies the class embedding space for multimodal zero-shot learning. MDFN exploits feature disentangled variational autoencoder (FD-VAE) in two branches to distangle unimodal features into modality-specific representations that are semantically consistent and unrelated, where semantics are shared within classes. In particular, semantically consistent representations and unimodal features are integrated to retain the semantics of the original features in the form of residuals. Furthermore, multimodal conditional VAE (MC-VAE) in two branches is adopted to learn cross-modal interactions with modality-specific conditions. Finally, the complementary multimodal representations achieved by MC-VAE are encoded into a fusion network (FN) with a self-adaptive margin center loss (SAMC-loss) to predict target class labels in embedding forms. By learning the distance among domain samples, SAMC-loss promotes intraclass compactness and interclass separability. Experiments on zero-shot and news event datasets demonstrate the superior performance of MDFN, with the harmonic mean improved by 27.2% on the MMED dataset and 5.1% on the SUN dataset.

Knowledge-based visual question answering (KB-VQA) requires reasoning about the visual grounding relations between the images and questions by incorporating external knowledge. Existing works typically retrieve knowledge from knowledge graphs by leveraging global multimodal representations of image–text pairs for graph convolution, which neglect contextual clues at hop granularity, resulting in suboptimal spreading and leveraging of contextual information. To this end, we propose a multi-hop graph reasoning network (MGRN) for KB-VQA, which consists of a knowledge graph constructor (KGC) module, a semantic-instructed graph reasoning (SGR) module, and an answering module. MGRN exploits multimodal semantics from given images and questions as instructions for graph reasoning to obtain the knowledge representation from either the scene graph or knowledge base. Specifically, KGC fuses the scene graph with triplets from ConceptNet and Comet to construct a contextual knowledge graph for retrieving knowledge representation. Furthermore, SGR conducts multi-hop graph reasoning to select top- K knowledge items for answering by passing and filtering interplay messages on contextual knowledge graphs under the guidance of multimodal semantic representation. Extensive experiments conducted on two public datasets show the effectiveness and outperformance of our method.

Multimodal news event detection aims to identify and categorize significant events across media platforms using multimodal data. Previous work was limited to a single platform and assumed complete multimodal data. In this paper, we explore a novel task of cross-platform multimodal news event detection to enhance model generalization for cross-platform scenarios. We propose a Self-Supervised Modality Complementation (SSMC) method to tackle the challenges of incomplete modalities and platform heterogeneity presented in this task. Specifically, a Missing Data Complementation (MDC) module is designed to overcome the limitations caused by incomplete modalities. It employs a separation mechanism that distinguishes between modality-specific and modality-shared features across all modalities, allowing for the augmentation of missing modalities with information extracted from common features. Meanwhile, a Multimodal Self-Learning (MSL) module addresses platform heterogeneity by extracting pseudo labels from the target platform's multimodal views and incorporating a self-penalization mechanism to reduce reliance on low-confidence labels. Additionally, we collect a comprehensive cross-platform news event detection (CNED) dataset encompassing 37,711 multimodal samples from Twitter, Flickr, and online news media, covering 40 public news events verified by Wikipedia. Extensive experiments on the CNED dataset demonstrate the superior performance of our proposed method. The dataset is available athttps://github.com/RetrainIt/CNED.

Text-based CAPTCHA remains a widely employed scheme for distinguishing between human users and machine attackers during logging-in on systems. In this paper, we propose a reinforced perturbation generation (RPG) framework to automatically construct effective perturbation factors with reinforcement learning, and achieve a perturbed CAPTCHA that is user-friendly but challenging for machine attackers. More specifically, RPG exploits a perturbation initialization (PI) component to provide a preliminary perturbation factor. Furthermore, a perturbation reinforcement (PR) component is devised to optimize the combinations of multiple perturbation factors by a number of perturbation generation methods, which is achieved by reducing the gap between estimated cumulative rewards and real cumulative rewards. In particular, an attack model is introduced to produce the reward based on whether it can correctly recognize the perturbation CAPTCHA. The multiple perturbation factors are fused to be combined with the original CAPTCHA to against machine attackers. Extensive experiments conducted on eight real-world CAPTCHA datasets show outstanding performance against the CAPTCHA attack models.

In this paper, we propose a multimodal contrastive network with unbiased distillation (MCUD) for knowledge-based VQA, which consists of contrastive sample construction (CSC), unbiased contrastive distillation (UCD), and hierarchical reasoning (HR) modules. Specifically, CSC constructs contrastive samples by transforming the knowledge corpus and adopting entropy-adjusted answer frequencies to identify the unbiased samples. Additionally, UCD employs a dual-branch feature extractor architecture to separately encode the knowledge corpus and image-question pairs into a shared embedding space. Subsequently, the knowledge-driven contrastive learning is supposed to bridge the modality gap between the knowledge corpus and the cross-modality of image-question pairs. Furthermore, the teacher model in UCD utilizes different distillation strategies for biased and unbiased samples, guiding the student model to establish a generalized unbiased representation. Finally, the HR module conducts chain-of-thought outputs, sequentially locates the contextual sentences in the knowledge corpus, generates the rationale, and infers the answer. Extensive experiments with two datasets, E-VQA and ScienceQA, demonstrate the effectiveness and outperformance of our method.

In this paper, we propose a two-stage network with modality-specific and -shared contrastive learning (MMCL) for multimodal sentiment analysis. MMCL comprises a category-aware modality-specific contrastive (CMC) module and a self-decoupled modality-shared contrastive (SMC) module. In the first stage, the CMC module guides the encoders to extract modality-specific representations by constructing positive-negative pairs according to sample categories. In the second stage, the SMC module guides the encoders to extract modality-shared representations by constructing positive-negative pairs based on modalities and decoupling the self-contrast of all modalities. In the aforementioned modules, we leverage self-modulation factors to focus more on hard positive pairs through assigning different loss weights to positive pairs depending on their distance. In particular, we introduce a dynamic routing algorithm to cluster the inputs of the contrastive modules during training, where a gradient stopping strategy is utilized to isolate the backpropagation process of the CMC and SMC modules. Extensive experiments on the CMU-MOSI and CMU-MOSEI datasets show that MMCL achieves the state-of-the-art performance.

News event discovery refers to the identification and detection of news events using multimodal data on social media. Currently, most works assume that the test set consists of known events. However, in real life, the emergence of new events is more frequent, which invalidates this assumption. In this paper, we propose a Dynamic Augmentation and Entropy Optimization (DAEO) model to address the scenario of generalized news event discovery, which requires the model to not only identify known events but also distinguish various new events. Specifically, we first introduce a multimodal augmentation module, which utilizes adversarial learning to enhance the multimodal representation capability. Secondly, we design an adaptive entropy optimization strategy combined with a self-distillation method, which uses multi-view pseudo-label consistency to improve the model's performance on both known and new events. In addition, we collect a multimodal news event discovery (MNED) dataset of 161,350 samples annotated with 66 real-world events. Extensive experimental results on the MNED dataset demonstrate the effectiveness of our proposed method. Our dataset is available on https://github.com/RetrainIt/MNED.

The cross-attention mechanism has been widely employed in the multimodal rumor detection task, which is computation-intensive and suffers from the restricted modal receptive field. In this paper, we propose a multimodal rumor detection model (MKV), which maps multimodal key semantics with discrimination into feature vectors for rumor detection. More specifically, MKV extracts high-dimensional features for each modality separately by the Multimodal Feature Extractor (MFE). The mapping mechanism learns low-dimensional mapping scheme (Map) and key semantics (Key) with discrimination from the different modal features respectively. Subsequently, the Map and Key jointly construct a state matrix (State) containing all possible permutations of modalities. In particular, a max pooling operation is performed on State and products a feature vector (Vector). The mapping mechanism is able to incrementally learn the discriminative semantics by stacking manner. Vectors from the stacking process are leveraged in the Rumor Detection module (RD). Extensive experiments on two public datasets show that the MKV achieves the state-of-the-art performance.

Misinformation detection in short videos on social media has become a pressing issue due to its popularity. However, datasets for misinformation detection are limited in terms of modality and sources, hindering the development of effective detection methods. In this article, we introduce a novel dataset denoted the multiplatform multimodal misinformation (3M) dataset. Our dataset is collected specifically to investigate and address misinformation in a multimodal context. A total of 17 352 videos were collected from two prominent social media platforms, namely TikTok and Weibo. The 3M dataset covers 30 different topics, such as sports, health, news, and art, providing a diverse range of content for analysis. We propose a novel approach named cross-modal attention misinformation detection (CAMD) for effectively detecting and addressing multimodal misinformation. CAMD leverages the cross-modal attention module to facilitate effective information exchange and fusion between modalities by learning the correlations and weights among them. The cross-modal attention module is capable of learning multilevel modality correlations, focuses primarily on the interaction between multimodal sequences across different time steps, and simultaneously adjusts the information from the source modality based on the information of the target modality. Extensive experiments on the 3M dataset show that the proposed method achieves state-of- the-art performance. Specifically, CAMD achieves accuracy, F1-score, precision, and recall values of 76.86%, 58.05%, 87.86%, and 58.70%, respectively, on the 3M dataset.

Social media serves as a real-time collecting and disseminating center of users’ ideas, opinions, and experiences. The deliberate disinformation and rumors propagate rapidly online due to their exaggerated facts, controversial opinions, divisive perspectives, and stunning expressions. Rumor detection approaches typically use social media posts with rumor or non-rumor labels for training and testing without disclosing the rationale behind decision-makings. On one hand, collecting evidence data to verify claims relies on expert efforts. On the other hand, verifying the truthfulness of confusing claims with distracting and lengthy evidences is still challenging. In this paper, we contribute a rumor detection dataset with multi-granularity evidences, denoted as the RD-E dataset, which includes response, fact-check, article, sourcing data and generated evidence by large language models, supporting models to verify the truthfulness of claims on social media. A number of 32,892 claims from 4,525 public individuals and organizations are annotated to 6 kinds of labels, including true, mostly true, half true, mostly false, false, pants on fire, covering a wide range of topics, e.g., politics, economy, society, technology, and health. In the experiments, seven rumor detection models have been investigated and customized on four predefined subtasks for comparisons.

The availability of datasets annotated with verified events by the public is a necessary prerequisite for unleashing the potential of multimodal deep learning for news event detection. Publicly available datasets are either incompletely annotated due to expensive cost, or ignore the verifiability of event labels, which are susceptible to bias and errors introduced by a limited number of annotators. In this article, we provide a YouTube dataset labelled by real-world news events that can be verified by Wikipedia-like crowd sourcing platforms, with the target of advancing temporal event detection. The events in our dataset cover a wide range of event topics including public security, natural disasters, elections, sports, and entertainment events, etc. In the dataset, each sample is labelled with real-world event that is verifiable by the public. We extensively evaluate the performance of 13 state-of-the-art algorithms on our dataset in a temporal manner, involving the multiple relationships between training and testing event labels, and provide a thorough analysis of the findings.

It is challenging to eliminate the domain shift between seen and unseen classes in multimodal zero-shot learning tasks due to the underlying disparity between the data distributions in the seen and unseen domains. In this paper, we propose a progressive placeholder learning network with mixup hallucination and an alternating mixer, denoted as MHAM, to maintain embedding spaces for unseen classes. Utilizing mixup hallucination (MH) on the visual and textual features obtained by BERT and a vision transformer, MHAM generates visual and textual hallucinated representations with pseudo class embeddings as placeholders for the unseen classes. Furthermore, a number of alternating mixer (AM) blocks are stacked to obtain modality-shared representations for the seen classes and hallucinated representations of progressive placeholders for the unseen classes. In particular, modality-shared representations are obtained by a mixer in an AM block by reversing the dimensionality of the modality-specific and raw representations to model intermodal interactions. MHAM exploits a freezing strategy by fixing the weights over the unseen classes in the last fully connected layer; this step acts as a projection from the raw and modality-shared representations to the embedding space of the seen and unseen classes. Experiments conducted on zero-shot datasets and news event datasets demonstrate the superior performance of the proposed MHAM method.

Existing social networks, such as Twitter and Facebook, are rife with inaccurate and damaging information that is bad for society. Most existing solutions usually use deep learning models for disinformation detection in addition to artificial recognition. However, the result is easily tampered with by people. At the same time, if we strictly manage public opinions, freedom of speech will also cause controversy. In order to solve the above problems and maintain a good social network environment, we propose a new reputation mechanism based on blockchain and deep learning. To assess the reputation of message senders, our proposed mechanism utilizes smart contracts that automate programs without human intervention. Our approach avoids unduly restricting users’ freedom of expression and instead employs deep learning models for rumor detection and sentiment analysis to identify and label messages. By controlling the dissemination of messages based on labels of messages and the sender’s reputation, we aim to balance freedom of speech with social stability. Finally, we analyze the usability and performance of our proposed system.

With the spread of information on the Internet and the explosive growth of intellectual property information, the traditional intellectual property management model relying on third-party institutions cannot meet the demand for intellectual property protection, which has a cumbersome process, low efficiency, and insufficient evidence of rights protection. To address the issues of information falsification and leakage, we present an anonymous and traceable intellectual property management system based on blockchain, namely ATIPM, which employs the non-interactive zero knowledge proof to realize user unlinkability and anonymous transactions to protect the users’ intellectual property information. To avoid a single point of accountability, the ATIPM introduces a threshold ramp secret sharing scheme to achieve the traceability of intellectual property for all users and greatly improve the users’ privacy security and autonomy by preventing information leakage from malicious third-party institutions. Furthermore, the ATIPM can improve the management efficiency of intellectual property by utilizing smart contracts to realize efficient retrieval and verification of intellectual property. The evaluation results demonstrate the effectiveness of our proposed system.

In this paper, we propose a Retriever-Reader framework with Visual Entity Linking (RR-VEL) for knowledge-based visual question answering. Given images and original questions, the visual entity linking (VEL) module extracts key entities in images to replace the question referents for semantic disambiguation, achieving entity-oriented queries with explicit entities. Furthermore, the Retriever encodes the queries and knowledge items by Bert with a feed-forward layer, and obtains a set of knowledge candidates. The Reader encodes the questions with image captions and knowledge candidates in two branches, which avoids their interference during self-attentive encoding. Finally, the decoder of Reader fuses the encoded features to generate answers. Extensive experiments conducted on the two public datasets show that our method significantly outperforms the existing baselines.

Social media allows the public to access information conveniently, in which the false messages that are eye-catching may spread fast. In this paper, we propose a two-stage confidence-guided boundary adaption (CBA) network, consisting of a feature preprocessing (FP) module, a biased ambiguity learning (BA) module and a confidence-guided boundary adaptation (CG) module. In the first stage, the FP module obtains the textual and visual features, which are fused by conducting the visual-to-textual and textual-to-visual correlation coefficients with attention mechanism. Furthermore, BA evaluates the distribution distance between fused features and single modalities to determine the weights between modalities, capturing the semantics of key modality. In the second stage, CG leverages samples from the low-confidence interval to generate new instances using a mixup of augmentation techniques, aiming to occupy the decision space and optimize the decision boundary of the classifier. Extensive experiments on two public datasets show that our CBA model is 1.6% and 2.6% higher than the state-of-the-art methods.

Visual question answering (VQA) is a challenging task that reasons over questions on images with knowledge. A prerequisite for VQA is the availability of annotated datasets, while the available datasets have several limitations. 1) The diversity of questions and answers are limited to a few question categories and certain concepts (e.g., objects, relations, actions.) with somewhat mechanical answers. 2) The availability of background knowledge or context information has been disregarded with just images, questions and answers being provided. 3) The timeliness of knowledge has not been examined, though some works may introduce factual or commonsense knowledge bases, e.g., ConceptNet, DBPedia. In this paper, we provide an Event-oriented Visual Question Answering (E-VQA) dataset including free-form questions and answers for real-world event concepts, which provides context information of events as domain knowledge in addition to images. E-VQA consists of 2,690 social media images, 9,088 questions, 5,479 answers, and 1,157 news media articles for references being annotated to 182 real-world events, covering a wide range of topics, such as armed conflicts and attacks, disasters and accidents, law and crime. For comparisons, we investigate 10 state-of-the-art VQA methods as benchmarks.

This paper proposes a dense fusion transformer (DFT) framework to integrate textual, acoustic, and visual information for multimodal affective computing. DFT exploits a modality-shared transformer (MT) module to extract the modality-shared features by modelling unimodal, bimodal, and trimodal interactions jointly. MT constructs a series of dense fusion blocks to fuse utterance-level sequential features of the multiple modalities from the perspectives of low-level and high-level semantics. In particular, MT adopts local and global transformers to learn modality-shared representations by modelling inter- and intra-modality interactions. Furthermore, we devise a modality-specific representation (MR) module with a soft orthogonality constraint to penalize the distance between modality-specific and modality-shared representations, which are fused by a transformer to make affective predictions. Extensive experiments conducted on five public benchmark datasets show that DFT outperforms the state-of-the-art baselines.

In this paper, we present a new dataset with the target of advancing temporal real-world event detection from video-sharing social media. The collected temporal event dataset (TED) has certain characteristics. 1) Certainty of event labels that can be recognized as real-world happening. 2) Wide range of event categories covering public security, natural disasters, elections, sports and entertainment events, etc. 3) High overlap among event topics ensuring the difficulties in distinguishing the labels. 4) Multiple data modalities involving textual, acoustic, and visual information, etc. More specifically, two scenarios are defined based on the close or open domains, i.e., temporal event detection without/with new events, denoted as TED-W and TED-N, respectively. For comparisons, a few benchmarks are investigated on the two scenarios with the dataset.

GAN inversion aims to invert an input image into the latent space of a pre-trained GAN. Despite the recent advances in GAN inversion, there remain challenges to mitigate the tradeoff between distortion and editability, i.e. reconstructing the input image accurately and editing the inverted image with a small visual quality drop. The recently proposed pivotal tuning model makes significant progress towards reconstruction and editability, by using a two-step approach that first inverts the input image into a latent code, called pivot code, and then alters the generator so that the input image can be accurately mapped into the pivot code. Here, we show that both reconstruction and editability can be improved by a proper design of the pivot code. We present a simple yet effective method, named cycle encoding, for a high-quality pivot code. The key idea of our method is to progressively train an encoder in varying spaces according to a cycle scheme: W->W+->W. This training methodology preserves the properties of both W and W+ spaces, i.e. high editability of W and low distortion of W+. To further decrease the distortion, we also propose to refine the pivot code with an optimization-based method, where a regularization term is introduced to reduce the degradation in editability. Qualitative and quantitative comparisons to several state-of-the-art methods demonstrate the superiority of our approach.

In this paper, we propose a multimodal fusion network (termed as MFN) to integrate the text and image data from social media for rumor detection. Given the multimodal features, MFN exploits self-attentive fusion (SAF) mechanism to conduct feature-level fusion by assigning corresponding weights to the complementary modalities. In particular, the textual features are combined with the fused features in a skip-connection manner, as textual features tend to be more distinguishable compared with visual features. Furthermore, MFN introduces latent topic memory (LTM) to store the semantic information about rumor and non-rumor events, benefiting to the identification of the upcoming posts. Extensive experiments conducted on two public datasets show that the proposed MFN outperforms the state-of-the-art approaches.

In this paper, we propose a deep dense fusion network with multimodal residual (DFMR) to integrate multimodal information including language, acoustic speeches, and visual images for sentiment analysis. DFMR exploits a dense fusion (DF) block to fuse the multimodal features obtained by modality-specific sequence networks, which is achieved by modelling their unimodal, bimodal and trimodal interactions jointly. Instead of concatenating the multimodal features directly, DF block conducts fusion for any two paired modalities firstly, and the fused information will be integrated with the other modalities subsequently. Furthermore, DFMR stacks multiple DF blocks to capture high-level semantic information conveyed by the multimodal representations. In particular, DFMR adopts a multimodal residual (MR) block to integrate the modality-specific features and fused features in each DF blocks, to avoid forgetting the multi-aspect information and alleviate gradient vanishing during stacking. Extensive experiments conducted on four public benchmark datasets show that DFMR outperforms eleven state-of-the-art baselines.

In this paper, we propose a temporal graph convolutional network (TGCN) to recognize the sentiments from language (textual), acoustic, and visual (facial expressions) modalities. TGCN constructs a modality-specific graph whose nodes are the aligned segments in the multimodal utterances and edges are weighted according to the distances between their features, in order to learn node embeddings with sequential semantics underlying the utterances. In particular, we use positional encoding by interleaving sine and cosine embedding to encode the positions of the segments in the utterances into their features. Given the modality-specific embeddings of the segments in utterances, we create an attention mechanism corresponding to the segments to capture the sentiment-related ones and obtain the unified embeddings of utterances. Furthermore, we fuse the attended embeddings of the multimodel utterances and conduct the attention to capture their interaction. Finally, the fused embeddings together with their raw features are concatenated together for sentiment predictions. Extensive experiments on three publicly available datasets show that TGCN outperforms the state-of-the-art methods.

Transparent objects with unique visual properties often make depth cameras fail to scan their reflective and refractive surfaces. Recent studies on depth completion of transparent objects have leveraged a linear system based on the geometric constraints to predict the missing depth, which is hard to be employed in an end-to-end framework and achieve joint optimization. In this paper, we propose DepthGrasp - a deep learning approach for depth completion of transparent objects from a raw RGB-D image. More specifically, we use a generative adversarial network, which utilizes the generator to complete the depth maps by predicting the missing or inaccurate depth values, and use discriminator to guide the completed depth maps against the groundtruth. In the generator, we devise spectral residual blocks (SRB) with spectral normalization for network stability, and residual block to pass the attention map in order to capture the structure information and distinguish the geometric shape of transparent objects. In the discriminator, we use a patch-based convolutional network to adapt the data distributions of the predicted depth maps according to groundtruth. Extensive experiments conducted on ClearGrasp dataset show the effectiveness and generalization of the DepthGrasp for depth completion, and the deployed robotic picking system makes significant improvement on the performance of grasping on transparent objects.

Text-guided image inpainting aims to complete the corrupted patches coherent with both visual and textual context. On one hand, existing works focus on surrounding pixels of the corrupted patches without considering the objects in the image, resulting in the characteristics of objects described in text being painted on non-object regions. On the other hand, the redundant information in text may distract the generation of objects of interest in the restored image. In this paper, we propose an adversarial learning framework with mask reconstruction (ALMR) for image inpainting with textual guidance, which consists of a two-stage generator and dual discriminators. The two-stage generator aims to restore coarse-grained and fine-grained images, respectively. In particular, we devise a dual-attention module (DAM) to incorporate the word-level and sentence-level textual features as guidance on generating the coarse-grained and fine-grained details in the two stages. Furthermore, we design a mask reconstruction module (MRM) to penalize the restoration of the objects of interest with the given textual descriptions about the objects. For adversarial training, we exploit global and local discriminators for the whole image and corrupted patches, respectively. Extensive experiments conducted on CUB-200-2011, Oxford-102 and CelebA-HQ show the outperformance of the proposed ALMR (e.g., FID value is reduced from 29.69 to 14.69 compared with the state-of-the-art approach on CUB-200-2011). Codes are available at https://github.com/GaranWu/ALMR

In this paper, we propose a multi-branch semantic learning network (MSLN) to generate image according to textual description by taking into account global and local textual semantics, which consists of two stages. The first stage generates a coarse-grained image based on the sentence features. In the second stage, a multi-branch fine-grained generation model is constructed to inject the sentence-level and word-level semantics into two coarse-grained images by global and local attention modules, which generate global and local fine-grained image textures, respectively. In particular, we devise a channel fusion module (CFM) to fuse the global and local fine-grained features in the multi-branch fine-grained stage and generate the output image. Extensive experiments conducted on the CUB-200 dataset and Oxford-102 dataset demonstrate the superior performance of the proposed method. (e.g., FID is reduced from 16.09 to 14.43 on CUB-200).

Multi-domain event detection (MED) task aims to discover real-world events from multi-domain and multi-view data sources. Most of the existing methods process multiple data modalities alone or concatenate single modality for the MED task, ignoring diverse intrinsic structures and underlying relations between each modality. In this paper, we address the heterogeneity of modalities' structure and data views' relations, and present a novel heterogeneous graph embedding method HGE2MED to learn the heterogeneous underlying relations and robust representation of each node. In particular, we construct heterogeneous graph for multi-domain and multi-view data, and obtain heterogeneous walking sequences by heterogeneous random walk. Using our heterogeneous graph embedding method to sample the triplets from above sequences, we are able to obtain robust embedding of each node by optimizing the network. Finally, taking the embedding of news as input, the event labels are predicted by a simple linear classifier. Comprehensive experiments are conducted to demonstrate the effectiveness of our proposed approach over state-of-the-art event detection algorithms on a real-world dataset.

Internet platforms provide new ways for people to share experiences, generating massive amounts of data related to various real-world concepts. In this paper, we present an event detection framework to discover real-world events from multiple data domains, including online news media and social media. As multi-domain data possess multiple data views that are heterogeneous, initial dictionaries consisting of labeled data samples are exploited to align the multi-view data. Furthermore, a shared multi-view data representation (SMDR) model is devised, which learns underlying and intrinsic structures shared among the data views by considering the structures underlying the data, data variations, and informativeness of dictionaries. SMDR incorpvarious constraints in the objective function, including shared representation, low-rank, local invariance, reconstruction error, and dictionary independence constraints. Given the data representations achieved by SMDR, class-wise residual models are designed to discover the events underlying the data based on the reconstruction residuals. Extensive experiments conducted on two real-world event detection datasets, i.e., Multi-domain and Multi-modality Event Detection dataset, and MediaEval Social Event Detection 2014 dataset, indicating the effectiveness of the proposed approaches.

In this article, we propose to learn shared semantic space with correlation alignment ( S 3 CA ) for multimodal data representations, which aligns nonlinear correlations of multimodal data distributions in deep neural networks designed for heterogeneous data. In the context of cross-modal (event) retrieval, we design a neural network with convolutional layers and fully connected layers to extract features for images, including images on Flickr-like social media. Simultaneously, we exploit a fully connected neural network to extract semantic features for text documents, including news articles from news media. In particular, nonlinear correlations of layer activations in the two neural networks are aligned with correlation alignment during the joint training of the networks. Furthermore, we project the multimodal data into a shared semantic space for cross-modal (event) retrieval, where the distances between heterogeneous data samples can be measured directly. In addition, we contribute a Wiki-Flickr Event dataset, where the multimodal data samples are not describing each other in pairs like the existing paired datasets, but all of them are describing semantic events. Extensive experiments conducted on both paired and unpaired datasets manifest the effectiveness of S 3 CA , outperforming the state-of-the-art methods.

In this paper, a novel Deep Semantic Space learning model with Intra-class Low-rank constraint (DSSIL) is proposed for cross-modal retrieval, which is composed of two subnetworks for modality-specific representation learning, followed by projection layers for common space mapping. In particular, DSSIL takes into account semantic consistency to fuse the cross-modal data in a high-level common space, and constrains the common representation matrix within the same class to be low-rank, in order to induce the intra-class representations more relevant. More formally, two regularization terms are devised for the two aspects, which have been incorporated into the objective of DSSIL. To optimize the modality-specific subnetworks and the projection layers simultaneously by exploiting the gradient decent directly, we approximate the nonconvex low-rank constraint by minimizing a few smallest singular values of the intra-class matrix with theoretical analysis. Extensive experiments conducted on three public datasets demonstrate the competitive superiority of DSSIL for cross-modal retrieval compared with the state-of-the-art methods.

In this paper, we propose to address the Social Media Prediction (SMP) Challenge by using regression model with multiple features extracted from various aspects of posts. More specifically, we extract textual features, numeric features, and construct user-related features to this end. For textual features, the rich texts possessed by the posts are integrated to build a corpus, based on which we train a language model to learn the vector representation of semantic information. For numeric features, we construct several new features, including the length and the word numbers of title. For the user-related features, we design a "user id count" based on the number of times each user posted in the entire dataset to show the activity of the user. Finally, the multiple features are feed into LightGBM to predict popularity scores. Extensive experiments conducted on the Social Media Prediction Dataset show the superiority of our method. Our approach achieves the 3rd place in the SMP Challenge.

Cross-domain image retrieval is always encountering insufficient labelled data in real world. In this paper, we propose unsupervised embedding learning (UEL) for cross-domain beauty and personal care product retrieval to finetune the convolutional neural network (CNN). More specifically, UEL utilizes the non-parametric softmax to train the CNN model as instance-level classification, which reduces the influence of some inevitable problems (e.g., shape variations). In order to obtain better performance, we integrate a few existing retrieval methods trained on different datasets. Furthermore, a query expansion strategy (i.e., diffusion) is adopted to improve the performance. Extensive experiments conducted on a dataset including half million images of beauty and personal product items (Perfect-500K) manifest the effectiveness of our proposed method. Our approach achieves the 2nd place in the leader board of the Grand Challenge of AI Meets Beauty in ACM Multimedia 2019. Our code is available at: https://github.com/RetrainIt/Perfect-Half-Million-Beauty-Product-Image-Recognition-Challenge-2019.

Scholars' homepages are important places to show personal research interest and academic achievement through the Web. However, according to our observation, only a small portion of scholars update their publications and related events on their homepages in time. In this paper, we propose a homepage augmentation technique, which automatically shows the newest academic events related to a scholar on his/her homepage. Specifically, we model the relations between homepages and the events collected from the Web as a complex heterogenous network, and propose an Embedding-based Heterogenous random Walk algorithm, namely EHWalk, to predict the links between homepages and events. Compared with existing embedding-based link prediction algorithms, EHWalk supports more efficient modeling of complex heterogenous relations in a dynamically changing network, which helps link the massive new updated events to homepages precisely and efficiently. Comprehensive experiments on a real-world dataset are conducted and the results show that our algorithm can achieve both good effectiveness and efficiency for real-world deployment.

In this paper, we propose to extract features from URLs and webpage links to detect phishing websites and their targets. In addition to the basic features of a given URL, such as length, suspicious characters, number of dots, a feature matrix is also constructed from these basic features of the links in the given URL's webpage. Furthermore, certain statistical features are extracted from each column of the feature matrix, such as mean, median, and variance. Lexical features are also extracted from the given URL, the links and content in its webpage, such as title and textual content. A number of machine learning models have been investigated for phishing detection, among which Deep Forest model shows competitive performance, achieving a true positive rate of 98.3% and a false alarm rate of 2.6%. In particular, we design an effective strategy based on search operator via search engines to find the phishing targets, which achieves an accuracy of 93.98%.

Social media headline prediction (SMHP) is a thriving application scenario, which aims to predict the popularity of the post data shared on social media. In this paper, we propose to use multi-aspect features combined with the random forest (RF) model for popularity predictions. Firstly, we extract features by combining both metadata of the posts and users' features. More specifically, we adopt the binary coding strategy for dimensionality reduction and deal with the missing values by using some strategies, i.e., estimating the missing geographic information according to the information of users, and filling the missing features with median. Furthermore, regression models can be used directly to make predictions. In particular, a random forest (RF) model is adopted since it does not require much effort in tuning hyper-parameters and performs effectively. Extensive experiments conducted on the SMHP dataset consisting of 340K image posts shared by 80K users manifest the effectiveness of our method. Our approach achieves the 4nd place in the leader board of the Grand Challenge of SMHP in ACM Multimedia 2018.

Cross-domain beauty and personal care product image retrieval is a challenging problem due to data variations (e.g., brightness, viewpoint, and scale), and the rich types of items. In this paper, we present a regional maximum activations of convolutions with attention (RA-MAC) descriptor to extract image features for retrieval. RA-MAC improves the regional maximum activations of convolutions (R-MAC) descriptor considering the influence of background in cross-domain images (i.e., shopper domain and seller domain). More specifically, RA-MAC utilizes the characteristics of the convolutional layer to find the attention of an image, and reduces the influence of the unimportant regions in an unsupervised manner. Furthermore, a few strategies have been exploited to improve the performance, such as multiple features fusion, query expansion, and database augmentation. Extensive experiments conducted on a dataset consisting of half a million images of beauty care products (Perfect-500K) manifest the effectiveness of RA-MAC. Our approach achieves the 2nd place in the leader board of the Grand Challenge of AI Meets Beauty in ACM Multimedia 2018. Our code is available at: https://github.com/RetrainIt/Perfect-Half-Million-Beauty-Product-Image-Recognition-Challenge.

A hybrid model for social media popularity prediction is proposed by combining Convolutional Neural Network (CNN) with XGBoost. The CNN model is exploited to learn high-level representations from the social cues of the data. These high-level representations are used in XGBoost to predict the popularity of the social posts. We evaluate our approach on a real-world Social Media Prediction (SMP) dataset, which consists of 432K Flickr images. The experimental results show that the proposed approach is effective, achieving the following performance: Spearman's Rho: 0.7406, MSE: 2.7293, MAE: 1.2475.

In this work, a three-stage social event detection (SED) framework is proposed to discover events from Flickr-like data. First, multiple bipartite graphs are constructed for the heterogeneous feature modalities to achieve fused features. Furthermore, considering the geometrical structures of dictionary and data, a dual structure constrained multimodal feature coding model is designed to learn discriminative feature codes by incorporating corresponding regularization terms into the objective. Finally, clustering models utilizing density or label knowledge and data recovery residual models are devised to discover real-world events. The proposed SED approach achieves the highest performance on the MediaEval 2014 SED dataset.

A theory of multiparty sessions is presented in this paper, where the semantic analysis of global types ensures that the projection onto each participant preserves a strong fairness assumption, i.e., no participant of multiparty sessions will wait for messages that are never sent or will send messages that no other participant will receive. In addition, once a global type is agreed upon by all participants involved, a local protocol from each local viewpoint will be generated as a projection onto each participant using a top-down approach. Furthermore, the introduction of asynchronous communication sub typing allows permutation of actions to raise efficiency, where each local type can be locally refined using our action asynchronous sub typing rules for both dependent and independent messages, which gives a more optimized protocol.