Addressing the bias problem in multimodal zero-shot learning tasks is challenging due to the domain shift between seen and unseen classes, as well as the semantic gap across different modalities. To tackle these challenges, we propose a multimodal disentangled fusion network (MDFN) that unifies the class embedding space for multimodal zero-shot learning. MDFN exploits feature disentangled variational autoencoder (FD-VAE) in two branches to distangle unimodal features into modality-specific representations that are semantically consistent and unrelated, where semantics are shared within classes. In particular, semantically consistent representations and unimodal features are integrated to retain the semantics of the original features in the form of residuals. Furthermore, multimodal conditional VAE (MC-VAE) in two branches is adopted to learn cross-modal interactions with modality-specific conditions. Finally, the complementary multimodal representations achieved by MC-VAE are encoded into a fusion network (FN) with a self-adaptive margin center loss (SAMC-loss) to predict target class labels in embedding forms. By learning the distance among domain samples, SAMC-loss promotes intraclass compactness and interclass separability. Experiments on zero-shot and news event datasets demonstrate the superior performance of MDFN, with the harmonic mean improved by 27.2% on the MMED dataset and 5.1% on the SUN dataset.

Text-based CAPTCHA remains a widely employed scheme for distinguishing between human users and machine attackers during logging-in on systems. In this paper, we propose a reinforced perturbation generation (RPG) framework to automatically construct effective perturbation factors with reinforcement learning, and achieve a perturbed CAPTCHA that is user-friendly but challenging for machine attackers. More specifically, RPG exploits a perturbation initialization (PI) component to provide a preliminary perturbation factor. Furthermore, a perturbation reinforcement (PR) component is devised to optimize the combinations of multiple perturbation factors by a number of perturbation generation methods, which is achieved by reducing the gap between estimated cumulative rewards and real cumulative rewards. In particular, an attack model is introduced to produce the reward based on whether it can correctly recognize the perturbation CAPTCHA. The multiple perturbation factors are fused to be combined with the original CAPTCHA to against machine attackers. Extensive experiments conducted on eight real-world CAPTCHA datasets show outstanding performance against the CAPTCHA attack models.

The availability of datasets annotated with verified events by the public is a necessary prerequisite for unleashing the potential of multimodal deep learning for news event detection. Publicly available datasets are either incompletely annotated due to expensive cost, or ignore the verifiability of event labels, which are susceptible to bias and errors introduced by a limited number of annotators. In this article, we provide a YouTube dataset labelled by real-world news events that can be verified by Wikipedia-like crowd sourcing platforms, with the target of advancing temporal event detection. The events in our dataset cover a wide range of event topics including public security, natural disasters, elections, sports, and entertainment events, etc. In the dataset, each sample is labelled with real-world event that is verifiable by the public. We extensively evaluate the performance of 13 state-of-the-art algorithms on our dataset in a temporal manner, involving the multiple relationships between training and testing event labels, and provide a thorough analysis of the findings.

It is challenging to eliminate the domain shift between seen and unseen classes in multimodal zero-shot learning tasks due to the underlying disparity between the data distributions in the seen and unseen domains. In this paper, we propose a progressive placeholder learning network with mixup hallucination and an alternating mixer, denoted as MHAM, to maintain embedding spaces for unseen classes. Utilizing mixup hallucination (MH) on the visual and textual features obtained by BERT and a vision transformer, MHAM generates visual and textual hallucinated representations with pseudo class embeddings as placeholders for the unseen classes. Furthermore, a number of alternating mixer (AM) blocks are stacked to obtain modality-shared representations for the seen classes and hallucinated representations of progressive placeholders for the unseen classes. In particular, modality-shared representations are obtained by a mixer in an AM block by reversing the dimensionality of the modality-specific and raw representations to model intermodal interactions. MHAM exploits a freezing strategy by fixing the weights over the unseen classes in the last fully connected layer; this step acts as a projection from the raw and modality-shared representations to the embedding space of the seen and unseen classes. Experiments conducted on zero-shot datasets and news event datasets demonstrate the superior performance of the proposed MHAM method.

Social media allows the public to access information conveniently, in which the false messages that are eye-catching may spread fast. In this paper, we propose a two-stage confidence-guided boundary adaption (CBA) network, consisting of a feature preprocessing (FP) module, a biased ambiguity learning (BA) module and a confidence-guided boundary adaptation (CG) module. In the first stage, the FP module obtains the textual and visual features, which are fused by conducting the visual-to-textual and textual-to-visual correlation coefficients with attention mechanism. Furthermore, BA evaluates the distribution distance between fused features and single modalities to determine the weights between modalities, capturing the semantics of key modality. In the second stage, CG leverages samples from the low-confidence interval to generate new instances using a mixup of augmentation techniques, aiming to occupy the decision space and optimize the decision boundary of the classifier. Extensive experiments on two public datasets show that our CBA model is 1.6% and 2.6% higher than the state-of-the-art methods.