Recent advancements in multimodal large language models (MLLMs) have shown remarkable progress in video understanding. However, video MLLMs (VideoMLLMs) still suffer from hallucinations, generating nonsensical or irrelevant content. This issue partly stems from over-reliance on pre-trained knowledge, sometimes neglecting the rich visual information present in the video. Additionally, many existing methods rely on uniform frame sampling, which can overlook critical visual cues. To address these challenges, we present EchoBat, a novel approach that leverages audio information as well as video temporal and logical consistency to improve preference data construction and keyframe extraction. Our method integrates Direct Preference Optimization (DPO) to mitigate hallucinations by leveraging high-quality, contextually rich preference feedback. Specifically, we use GPT-4o to generate high-quality video descriptions and integrate visually relevant segments from Whisper-derived transcripts to construct preference responses. Correspondingly, we use the reference model itself to describe the reversed video, and use GPT-4o to flashback the text and fill in the hallucination to produce non-preferred responses. This strategy enhances the model’s ability to better understand visual content and temporal, logical relationships within videos. Furthermore, we propose an echo-layered sampling strategy for keyframe extraction from videos, which can provide more precise visual supervision compared to uniform sampling. Experimental results on the three latest video hallucination benchmarks demonstrate the effectiveness of our approach.

Mobile motion sensors such as accelerometers and gyroscopes are now ubiquitously accessible by third-party apps via standard APIs. While enabling rich functionalities like activity recognition and step counting, this openness has also enabled unregulated inference of sensitive user traits, such as gender, age, and even identity, without user consent. Existing privacy-preserving techniques, such as GAN-based obfuscation or differential privacy, typically require access to the full input sequence, introducing latency that is incompatible with real-time scenarios. Worse, they tend to distort temporal and semantic patterns, degrading the utility of the data for benign tasks like activity recognition. To address these limitations, we propose the Predictive Adversarial Transformation Network (PATN), a real-time privacy-preserving framework that leverages historical signals to generate adversarial perturbations proactively. The perturbations are applied immediately upon data acquisition, enabling continuous protection without disrupting application functionality. Experiments on two datasets demonstrate that PATN substantially degrades the performance of privacy inference models, achieving Attack Success Rate (ASR) of 40.11% and 44.65% (reducing inference accuracy to near-random) and increasing the Equal Error Rate (EER) from 8.30% and 7.56% to 41.65% and 46.22%. On ASR, PATN outperforms baseline methods by 16.16% and 31.96%, respectively.

Deep neural networks are known to be susceptible to imperceptible adversarial perturbations. Many studies aim to interpret adversarial examples in the frequency domain. However, existing research often relies on a limited number of datasets, models, and adversarial attacks, leading to incomplete conclusions. Moreover, a quantitative interpretation of adversarial examples remains lacking. This paper proposes a quantitative frequency-based framework to comprehensively investigate adversarial examples, where six kinds of attacks against naturally and adversarially trained models across three datasets are adopted. Initially, our framework visualizes the distributions of successful adversarial perturbations in the frequency domain to locate their target regions. Subsequently, we characterize the importance of perturbations contained in different frequency bands and define adversarially effective frequency bands (AEFBs). Furthermore, we leverage the identified AEFBs to enhance two query-based black-box adversarial attacks. Our experimental results uncover the varying characteristics of adversarial perturbations, which are analyzed from dataset-level, model-level, and attack-level perspectives. After reordering frequency bands and identifying AEFBs, we further demonstrate that adversarial attacks guided by AEFBs can achieve superior performance, verifying their effectiveness and generalization. These significant findings contribute to a deeper understanding of adversarial examples and provide valuable insights for future research.

While multi-label learning has shown excellent performance in False Data Injection Attack (FDIA) locational detection, it has also exposed some potential security risks and vulnerabilities. However, unlike the image domain, the vulnerabilities of multi-label learning in the field of power grid have just received attention and urgently need to be explored and addressed. In this paper, to achieve a better understanding for the security risks of deep learning-based multi-label FDIA detectors, we propose two Alternating Direction Method of Multipliers (ADMM) based adversarial attacks, which are applicable to two different scenarios. The proposed two ADMM-based attacks aim to reduce additional attack costs while seeking suitable adversarial perturbations, making the attacks more realistic and feasible. The experimental results verify the effectiveness of the proposed ADMM-based attacks, making noteworthy strides in fostering a profound comprehension of the vulnerabilities in the unique field of deep multi-label learning for power systems.

The rapid advancement of generative models necessitates detection methods that generalize to synthetic images containing diverse generator and semantic artifacts. Recent research has leveraged pre-trained vision-language models, such as CLIP, to extract forensic features that distinguish real and fake images, illustrating their promising performance in synthetic image detection. However, a systematic investigation into the embedding space of CLIP to guide its principled utilization for synthetic image detection remains largely unexplored. This paper addresses this gap by first analyzing the multi-stage CLIP image embedding space to uncover its relationship with cross-artifact forensic patterns. Our findings reveal that the mid-level stages primarily encode forensic and generator artifact features, while the high-level stages primarily encode semantic artifact features. Building upon these insights, we propose the CLIP-guided Dual-level Augmentation and Forensic Distribution Adaptation (CLIP-ADA) framework to perform artifact-invariant generalizable detection. Specifically, dual-level augmentation diversifies fake embeddings and suppresses artifact encoding during training to mitigate detectors from excessively relying on artifact features. Moreover, forensic distribution adaptation reformulates synthetic image detection as identifying distributional deviations from the CLIP encoded real embeddings and thereby designing adapters to extract cross-artifact forensic features in a detection scenario-adaptive manner. Extensive evaluations on both the conventional single-generator and continual learning-based multi-generator training settings demonstrate the effectiveness of our method, both suppressing the state-of-the-art methods by over 6% of average accuracy on unseen data from more than 10 generators.

Pretrained Language Models (PLMs) and Graph Neural Networks (GNNs) have emerged as promising approaches for software vulnerability detection. However, existing methods still face limitations, including the absence of fine-grained cross-modal interaction and the impact of data noise. Approaches integrating PLMs and GNNs fail to fully leverage their complementary strengths, while unreliable labels hinder generalization, further degrading real-world detection performance. To over-come these limitations, we propose Vul-CTG, a multimodal integration framework for software vulnerability detection that combines Code Text, and program Graph representations. Vul-CTG constructs enriched code graph representations by integrating statement-level source code graphs and abstract code property graphs, enabling more effective alignment between structural and semantic information. To enhance robustness against noisy labels and improve cross-modal consistency, the model incorporates contrastive learning and pre-training techniques. Central to Vul-CTG is CTG-Former, a novel alignment architecture that projects both code text and graph modalities into a unified latent space, allowing the model to capture complex structural and semantic patterns for more accurate vulnerability detection. Experimental results on recent function-level datasets demonstrate the effectiveness of Vul-CTG, showing an approximate 3% improvement in F1-score over state-of-the-art methods. Our code is available at https://github.com/ryxFry/Vul-CTG.

Vision-Language Pre-training Models (VLPMs) are known to be vulnerable to adversarial attacks. Recent transferable attacks on VLPMs have followed a common pipeline with complicated loss functions or multi-stage text/image attacks. However, in this paper, we demonstrate that such a sophisticated attack pipeline can be simpler yet more successful. Specifically, we identify three previously overlooked issues caused by inappropriate cross-modal interactions and excessive operations. To address them, we propose the Simple Vision-Language Attack (SimVLA) pipeline, which observably improves transferability and efficiency. Experiments on four datasets and three downstream tasks validate the superiority of our pipeline. For instance, on Flickr30k text-image retrieval dataset, our SimVLA outperforms the SOTA baseline in R@1 transferability by 8.01\%-14.71\%, while consuming only about 35.73\% of the time and 46.26\% of the max VRAM. Overall, the superiority of our SimVLA highlights the importance of leveraging domain knowledge (e.g., our proposed cross-modal word identification), while blindly pursuing intricate operations (e.g, complex loss functions and redundant multi-stage designs) may even be harmful. We hope our SimVLA can serve as a simple yet effective backbone for future extensions. Code is available at https://github.com/RYC-98/SimVLA.

Jamming attacks pose a significant threat to the security of air-ground communications, where the challenge becomes more severe when involving multiple unmanned aerial vehicles (UAVs) incurring complex interference. To address this issue, this paper proposes a graph attention-based reinforcement learning strategy for anti-jamming UAV communications. Specifically, we consider the multi-UAV transmission and deployment in the presence of jamming attacks. Then, we formulate a zero-sum game with the legitimate side and adversary to maximize and minimize the overall transmission rate, respectively. Given the complicated structure of the game, we decompose it into two layers, tackled in a hierarchical learning framework. Particularly, the inner layer addresses the legitimate beamforming, for which we establish the graph attention network (GAT) to track the complicated interference and jamming relationship based on the graph representation of the UAV network. The outer layer address the legitimate UAV deployment and adversarial jamming policy, which is reinterpreted in a multi-agent deep reinforcement learning framework to obtain the strategies of both sides. The inner GAT is then nested within the outer multi-agent learning framework in a hierarchical manner to approximate the equilibrium of the original game model. Simulation results demonstrate the convergence and the performance superiority of the proposed learning scheme in terms of anti-jamming transmission rate. Also, the results exhibit significant generalization capability to cover different network configurations and parameters with reliable communication performance.

Transferable adversarial examples are known to cause threats in practical, black-box attack scenarios. A notable approach to improving transferability is using integrated gradients (IG), originally developed for model interpretability. In this paper, we find that existing IG-based attacks have limited transferability due to their naive adoption of IG in model interpretability. To address this limitation, we focus on the IG integration path and refine it in three aspects: multiplicity, monotonicity, and diversity, supported by theoretical analyses. We propose the Multiple Monotonic Diversified Integrated Gradients (MuMoDIG) attack, which can generate highly transferable adversarial examples on different CNN and ViT models and defenses. Experiments validate that MuMoDIG outperforms the latest IG-based attack by up to 37.3% and other state-of-the-art attacks by 8.4%. In general, our study reveals that migrating established techniques to improve transferability may require non-trivial efforts.

Vision Transformers (ViTs) have been widely applied in various computer vision and vision-language tasks. To gain insights into their robustness in practical scenarios, transferable adversarial examples on ViTs have been extensively studied. A typical approach to improving adversarial transferability is by refining the surrogate model. However, existing work on ViTs has restricted their surrogate refinement to backward propagation. In this work, we instead focus on Forward Propagation Refinement (FPR) and specifically refine two key modules of ViTs: attention maps and token embeddings. For attention maps, we propose Attention Map Diversification (AMD), which diversifies certain attention maps and also implicitly imposes beneficial gradient vanishing during backward propagation. For token embeddings, we propose Momentum Token Embedding (MTE), which accumulates historical token embeddings to stabilize the forward updates in both the Attention and MLP blocks. We conduct extensive experiments with adversarial examples transferred from ViTs to various CNNs and ViTs, demonstrating that our FPR outperforms the current best (backward) surrogate refinement by up to 7.0% on average. We also validate its superiority against popular defenses and its compatibility with other transfer methods. Codes and appendix are available at https://github.com/RYC-98/FPR.

Recent studies have shown that large vision-language models (LVLMs) often suffer from the issue of object hallucinations (OH). To mitigate this issue, we introduce an efficient method that edits the model weights based on an unsafe subspace, which we call HalluSpace in this paper. With truthful and hallucinated text prompts accompanying the visual content as inputs, the HalluSpace can be identified by extracting the hallucinated embedding features and removing the truthful representations in LVLMs. By orthog-onalizing the model weights, input features will be projected into the Null space of the HalluSpace to reduce OH, based on which we name our method Nullu. We reveal that Hal-luSpaces generally contain prior information in the large language models (LLMs) applied to build LVLMs, which have been shown as essential causes of OH in previous studies. Therefore, null space projection suppresses the LLMs’ priors to filter out the hallucinated features, resulting in contextually accurate outputs. Experiments show that our method can effectively mitigate OH across different LVLM families without extra inference costs and also show strong performance in general LVLM benchmarks. Code is released at https://github.com/Ziwei-Zheng/Nullu.

While virtual try-on for clothes and shoes with diffusion models has gained attraction, virtual try-on for ornaments, such as bracelets, rings, earrings, and necklaces, remains largely unexplored. Due to the intricate tiny patterns and repeated geometric sub-structures in most ornaments, it is much more difficult to guarantee identity and appearance consistency under large pose and scale variances between ornaments and models. This paper proposes the task of virtual try-on for ornaments and presents a method to improve the geometric and appearance preservation of ornament virtual try-ons. Specifically, we estimate an accurate wearing mask to improve the alignments between ornaments and models in an iterative scheme alongside the denoising process. To preserve structure details, we further regularize attention layers to map the reference ornament mask to the wearing mask in an implicit way. Experimental results demonstrate that our method successfully wears ornaments from reference images onto target models, handling substantial differences in scale and pose while preserving identity and achieving realistic visual effects.

Recent advances in generative modeling have enabled the generation of high-quality synthetic data that is applicable in a variety of domains, including face recognition. Here, state-of-the-art generative models typically rely on conditioning and fine-tuning of powerful pretrained diffusion models to facilitate the synthesis of realistic images of a desired identity. Yet, these models often do not consider the identity of subjects during training, leading to poor consistency between generated and intended identities. In contrast, methods that employ identity-based training objectives tend to overfit on various aspects of the identity, and in turn, lower the diversity of images that can be generated. To address these issues, we present in this paper a novel generative diffusion-based framework, called ID-Booth. ID-Booth consists of a denoising network responsible for data generation, a variational auto-encoder for mapping images to and from a lower-dimensional latent space and a text encoder that allows for prompt-based control over the generation procedure. The framework utilizes a novel triplet identity training objective and enables identity-consistent image generation while retaining the synthesis capabilities of pretrained diffusion models. Experiments with a state-of-the-art latent diffusion model and diverse prompts reveal that our method facilitates better intra-identity consistency and inter-identity separability than competing methods, while achieving higher image diversity. In turn, the produced data allows for effective augmentation of small-scale datasets and training of betterperforming recognition models in a privacy-preserving manner. The source code for the ID-Booth framework is publicly available at https://github.com/dariant/ID-Booth.

Controllable image generation has emerged as a cutting-edge subject of interest. Current interactive point-based image editing frameworks, such as DragGAN, achieve impressive results in fine-grained and controllable image editing. However, relying solely on point-based manipulations can lead to unintended outcomes due to the inherent lack of the users’ semantic intent. To address this issue, we introduce Text-Guided Drag (TGDrag), a novel approach to adding semantic control into point-based image editing by using text prompts to guide the manipulation of handle and target points. Specifically, we design a channel correlation calculator that adaptively selects channels for the text and points to mitigate the potential influence of semantic control on point control. Furthermore, we introduce a text loss function to minimize the discrepancy between the generated images and the text prompts. Experimental results demonstrate that TGDrag achieves the expected function of semantic control while maintaining effectiveness regarding point control.

Face avatar generation has gained significant attention recently. With the help of the Neural Radiance Field (NeRF), existing 3D methods alleviate facial distortion in 2D methods under large pose changes. However, the state-of-the-art 3D methods still require additional optimization for generation on each given portrait, even in a one-shot manner. To address this research gap, we propose a novel one-shot approach, which achieves effective face avatar generation in only a single forward pass. This is made possible by introducing an inversion encoder trained on a large-scale dataset for accurate latent code estimation and an expression animator for accurate expression control. Our approach is also designed for better preservation of the face identity by training an additional 3D feature refiner based on cross-attention. Experimental results demonstrate the superiority of our approach in terms of 3D consistency, identity similarity, and image quality.

Developing reliable defenses against patch attacks on object detectors has attracted increasing interest. However, we identify that existing defense evaluations lack a unified and comprehensive framework, resulting in inconsistent and incomplete assessments of current methods. To address this issue, we revisit 11 representative defenses and present the first patch defense benchmark, involving 2 attack goals, 13 patch attacks, 11 object detectors, and 4 diverse metrics. This leads to the large-scale adversarial patch dataset with 94 types of patches and 94,000 images. Our comprehensive analyses reveal new insights: (1) The difficulty in defending against naturalistic patches lies in the data distribution, rather than the commonly believed high frequencies. Our new dataset with diverse patch distributions can be used to improve existing defenses by 15.09% [email protected]. (2) The average precision of the attacked object, rather than the commonly pursued patch detection accuracy, shows high consistency with defense performance. (3) Adaptive attacks can substantially bypass existing defenses, and defenses with complex/stochastic models or universal patch properties are relatively robust. We hope that our analyses will serve as guidance on properly evaluating patch attacks/defenses and advancing their design. Code and dataset are available at https://github.com/Gandolfczjh/APDE, where we will keep integrating new attacks/defenses.

Deepfake detection can prevent the misuse of deep generative techniques but is known to be vulnerable to adversarial attacks. However, most existing attacks introduce noticeable noise, resulting in an unsatisfactory trade-off between attack effectiveness and imperceptibility. In this paper, we propose a new generative attack based on adversarially Degrading and Restoring (DR) fake images, eliminating the use of noisy perturbations. Specifically, degradation works by removing high-frequency deepfake artifacts with the guidance of adversarial loss from the detector, and the subsequent restoration aims to maintain the image quality by restoring high-frequency details of natural images. Our analysis confirms that combining degradation and restoration effectively aligns the distribution of adversarial (fake) images and real images in both frequency and pixel domains. Our experimental results across eight popular detectors and three popular deep-fake datasets prove the effectiveness of our method compared with several state-of-the-art methods. Our code is available at https://github.com/fanoflck/DR_attack.

Fine-tuning is an essential process to improve the performance of Large Language Models (LLMs) in specific domains, with Parameter-Efficient Fine-Tuning (PEFT) gaining popularity due to its capacity to reduce computational demands through the integration of low-rank adapters. These lightweight adapters, such as LoRA, can be shared and utilized on open-source platforms. However, adversaries could exploit this mechanism to inject backdoors into these adapters, resulting in malicious behaviors like incorrect or harmful outputs, which pose serious security risks to the community. Unfortunately, few current efforts concentrate on analyzing the backdoor patterns or detecting the backdoors in the adapters. To fill this gap, we first construct and release PADBench, a comprehensive benchmark that contains 13, 300 benign and backdoored adapters fine-tuned with various datasets, attack strategies, PEFT methods, and LLMs. Moreover, we propose PEFTGuard, the first backdoor detection framework against PEFT-based adapters. Extensive evaluation upon PADBench shows that PEFTGuard outperforms existing detection methods, achieving nearly perfect detection accuracy (100%) in most cases. Notably, PEFTGuard exhibits zero-shot transferability on three aspects, including different attacks, PEFT methods, and adapter ranks. In addition, we consider various adaptive attacks to demonstrate the high robustness of PEFTGuard. We further explore several possible backdoor mitigation defenses, finding fine-mixing to be the most effective method. We envision that our benchmark and method can shed light on future LLM backdoor detection research.11Our code and dataset are available at: https://github.com/Vincent-HKUSTGZ/PEFTGuard.

Abstract Artificial intelligence (AI) is revolutionizing both industries and reshaping the global economy. However, the rapid advancement of AI technologies brings significant security and privacy challenges. Recent incidents highlight vulnerabilities in AI systems, such as data leakage and malicious code injection, leading to severe financial losses and privacy breaches. Although existing studies have discussed specific security threats, they often lack detailed granularity and cover a limited scope. In this survey, we fill this gap by systematically categorizing and analyzing the threats and countermeasures in AI systems, which span both the training and inference stages, encompass centralized and distributed settings, and address both conventional and foundation AI models. By reviewing existing literature, we aim to provide AI researchers and practitioners with a thorough understanding of system vulnerabilities and current countermeasures. We hope to inspire further research into robust solutions, ultimately contributing to the development of resilient AI technologies.

Dynamic searchable symmetric encryption (DSSE) schemes empower data owners to outsource their encrypted data to clouds while retaining the ability to update or search on it. Despite a lot of efforts devoted in recent years, there are still several challenges that have not been well addressed. First, the confidentiality of data might be compromised if forward privacy and backward privacy cannot be ensured. Second, only the traditional single keyword-file search has attracted tremendous attention, while other popular queries like Boolean queries and range queries are not fully investigated. Lastly, how to solve these problems on untrusted servers that may deviate from pre-defined protocols is also challenging. In this article, aiming to tackle the above problems, we propose a novel DSSE scheme named${\sf HeX}$based on Trusted Execution Environment (TEE) that supports rich queries on untrusted servers while guaranteeing forward and backward privacy. We achieve strong forward and backward security by designing a deferred obfuscating read-write technique atop the bitmap index. We further extend the basic scheme to realize Boolean queries and range queries by reducing them to basic keyword queries. Strict theoretical analysis is conducted to prove the security of${\sf HeX}$, and extensive evaluations illustrate its efficiency and practicality.

With the widespread adoption of machine learning models, especially deep neural networks (DNNs), as an integral part of new intelligent software, the new tools to effectively support the model engineering and debugging process have received extensive attention. However, the existing tools only provide limited support for the training process. They are either post-training tools that fail to detect problems timely, resulting in wasting time and resources on training buggy models, or merely collecting the training data and still require manual analysis. In this paper, we proposeAutoTrainer, an automated monitoring and repairing system for DNN training, which provides real-time monitoring for the model training process and automatically repairs eight commonly seen training problems.AutoTrainermonitors the training process and detects potential training problems. For any detected problem,AutoTrainertries to fix it with the built-in state-of-the-art solutions. Our experiments on six datasets and 701 models show that the problem detection accuracy ofAutoTrainerreaches 100% without false positives. Moreover, it fixes 98.42% of all detected problems and improves the model accuracy by 36.42% on average.

Federated Learning (FL) enables geographically distributed clients to collaboratively train machine learning models by exchanging local model parameters while preserving data privacy. In practice, FL faces two critical challenges. First, it is vulnerable to security issues as malicious clients would artificially harm the functionality of FL by launching poisoning attacks. Second, the inherent data heterogeneity among clients (termed Non-IID data in FL) naturally arises from distributed data ownership and significantly degrades model convergence and accuracy. However, with studies separately devoted to these two research lines, the interplay between data heterogeneity and security remains poorly understood. In this paper, we systematically investigate the relationship between data heterogeneity and adversarial robustness in FL. Specifically, we propose novel data partitioning algorithms that simulate Label-Conditional Non-IID and Feature-Conditional Non-IID with quantifiable heterogeneity levels. Further, we conduct extensive experiments to evaluate classical defense methods in the practical FL environment under state-of-the-art untargeted attacks. With results in various settings, we separately analyze the connection between Non-IID to defenses and attacks. Regarding attacks, with similar effects on models, Non-IID impacts the training in a different way compared with attacks. The interaction between attacks and Non-IID provides an opportunity to cause severe damage to FL. Regarding defenses, Non-IID induces heterogeneity in model distribution among clients which raises the difficulty of maintaining fidelity and robustness for defense methods.

Adversarial training (AT) is widely considered the state-of-the-art technique for improving the robustness of deep neural networks (DNNs) against adversarial examples (AEs). Nevertheless, recent studies have revealed that adversarially trained models are prone to unfairness problems. Recent works in this field usually apply class-wise regularization methods to enhance the fairness of AT. However, this paper discovers that these paradigms can be sub-optimal in improving robust fairness. Specifically, we empirically observe that the AEs that are already robust (referred to as “easy AEs” in this paper) are useless and even harmful in improving robust fairness. To this end, we propose the hard adversarial example mining (HAM) technique which concentrates on mining hard AEs while discarding the easy AEs in AT. Specifically, HAM identifies the easy AEs and hard AEs with a fast adversarial attack method. By discarding the easy AEs and reweighting the hard AEs, the robust fairness of the model can be efficiently and effectively improved. Extensive experimental results on four image classification datasets demonstrate the improvement of HAM in robust fairness and training efficiency compared to several state-of-the-art fair adversarial training methods. Our code is available athttps://github.com/yyl-github-1896/HAM.

Trojans (backdoors) are known to raise critical security concerns for deep neural networks in machine learning (ML) systems. Despite the extensive backdoor methods and benchmarks, existing research overlooks the perspective of the ML lifecycle (i.e., the entire process from system design to data collection to model deployment). To address this gap, this paper introduces De2Trojan, a Deployable Trojan Analysis Tool via Decoupling, which establishes a standardized pipeline to investigate backdoor attacks and defenses within the ML lifecycle. De2Trojan decouples the attack surface from the general ML process through a stage-first hijacking approach, using an abstract interface for ML lifecycle stages to enhance the deployability to the ML lifecycle. Besides, its benefits are two-fold: (1) Facilitating the systematic analyses of multi-stage attacks/defenses and their combinations, shedding light on how to improve attack and defense strategies. For example, we find that current attacks (defenses) are not effective in continuous scenarios, and combining attacks (defenses) at different stages improves their effectiveness from 30.11% (8.63%), the worst cases, to 90.27% (68.73%). (2) Making it possible to identify potentially vulnerable stages, especially when iteratively updating the model in ML lifecycle. For example, we identify that backdoor attacks in the data collection stage are more vulnerable than expected, and it is more difficult to remove them from the ML lifecycle. To eliminate the impact of such attacks, it is most effective to apply backdoor defense during the deployment stage, in addition to cleaning the data before training. Overall, we present a comprehensive benchmark of backdoors within the ML lifecycle, involving 20 representative attacks and defenses, as well as their combinations, using 11 evaluation metrics.

Transfer-based adversarial attacks pose a severe threat to real-world deep learning systems since they do not require access to target models. Adversarial training (AT), which is recognized as the most effective defense against white-box attacks, also ensures high robustness against (black-box) transfer-based attacks. However, AT suffers from significant computational overhead because it repeatedly generates adversarial examples (AEs) throughout the entire training process. In this paper, we demonstrate that such repeated generation is unnecessary to achieve robustness against transfer-based attacks. Instead, pre-generating AEs all at once before training is sufficient, as proposed in our new defense paradigm called Data-Centric Robust Training (DCRT). DCRT employs clean data augmentation and adversarial data augmentation techniques to enhance the dataset before training. Our experimental results show that DCRT outperforms widely-used AT techniques (e.g., PGD-AT, TRADES, EAT, and FAT) in terms of transfer-based black-box robustness and even surpasses the top-1 defense on RobustBench when combined with common model-centric techniques. We also highlight additional benefits of DCRT, such as improved training efficiency and class-wise fairness.Our code will be available on GitHub.

Transferable adversarial examples cause practical security risks since they can mislead a target model without knowing its internal knowledge. A conventional recipe for maximizing transferability is to keep only the optimal adversarial example from all those obtained in the optimization pipeline. In this paper, for the first time, we revisit this convention and demonstrate that those discarded, sub-optimal adversarial examples can be reused to boost transferability. Specifically, we propose “Adversarial Example Soups” (AES), with AES-tune for averaging discarded adversarial examples in hyperparameter tuning and AES-rand for stability testing. In addition, our AES is inspired by “model soups”, which averages weights of multiple fine-tuned models for improved accuracy without increasing inference time. Extensive experiments validate the global effectiveness of our AES, boosting 10 state-of-the-art transfer attacks and their combinations by up to 13% against 10 diverse (defensive) target models. We also show the possibility of generalizing AES to other types, e.g., directly averaging multiple in-the-wild adversarial examples that yield comparable success. A promising byproduct of AES is the improved stealthiness of adversarial examples since the perturbation variances are naturally reduced.

In complex traffic environments, the driving performance of the leader vehicle in a platoon can be greatly impacted by sudden and unexpected changes in vehicle acceleration rates. This phenomenon is known as unknown jerk dynamics (JDs), and it can lead to more extreme car-following behaviors (CFBs) in platoon tracking control, which may raise safety and traffic capacity issues. To tackle these concerns, this work studies cooperative platoon tracking control and intermittent optimization problems for connected autonomous vehicles (CAVs) with unknown reaction time delays (RTDs) using a nonlinear car following model (NCFM). In a free-design but directed communication network, we assume that the leader CAV’s external inputs have unknown but bounded parameters both for the JDs and RTDs, while only a small number of nearby follower CAVs are aware of the leader CAV’s acceleration signals. To solve these issues, we consider that each follower CAV implements a distributed observer law, which provides a reference signal stated as an estimated JD of the leader CAV. Then, a distributed platoon tracking control protocol is proposed to construct cooperative tracking controllers with identical inter-vehicle constraints (ICs). This maintains the desired safety distance between the CAVs and allows each follower CAV to track its leader CAV only through local information exchange. In addition, we present a robust intermittent optimization design and a novel intermittent sampling condition that can guarantee optimally scheduled feedback gains for the cooperative platoon tracking controllers to minimize the control cost in the presence of unknown JDs and RTDs under non-identical ICs. Simulation case studies are conducted to demonstrate the effectiveness of the proposed approaches. We also demonstrate the efficient development of such a distributed cooperative car-following model for the platoon’s motion (or as an intelligent speed advising system for automated or human-driven vehicles), resulting in a trip that is safe, comfortable, and energy efficient.

Blockchain has become a popular paradigm for secure and immutable data storage. Despite its numerous applications across various fields, concerns regarding the user privacy and result integrity during data queries persist. Additionally, the need for rich query functionalities to harness the full potential of blockchain data remains an area ripe for exploration. In order to address these challenges, our paper first utilizes a framework based on the Trusted Execution Environment (TEE) and oblivious RAM technique to achieve both privacy and data integrity. To enhance the query efficiency over the entire blockchain, we then devise a two-level learned indexing methodology named TELEX within the TEE for both integer and string keys. We also propose different query processing algorithms for versatile query types, including exact queries, aggregate queries, Boolean queries, and range queries. By implementing the prototype and conducting extensive evaluation, we demonstrate the feasibility and remarkable improvement in efficiency compared to existing solutions.

In recent few years, DeepFakes are posing serve threats and concerns to both individuals and celebrities, as realistic DeepFakes facilitate the spread of disinformation. Model attribution techniques aim at attributing the adopted forgery models of DeepFakes for provenance purposes and providing explainable results to DeepFake forensics. However, the existing model attribution techniques rely on the trace left in the DeepFake creation, which can become futile if such traces were disrupted. Motivated by our observation that certain traces served for model attribution appeared in both the high-frequency and low-frequency domains and play a divergent role in model attribution. In this work, for the first time, we propose a novel training-free evasion attack, TraceEvader, in the most practical non-box setting. Specifically, TraceEvader injects a universal imitated traces learned from wild DeepFakes into the high-frequency component and introduces adversarial blur into the domain of the low-frequency component, where the added distortion confuses the extraction of certain traces for model attribution. The comprehensive evaluation on 4 state-of-the-art (SOTA) model attribution techniques and fake images generated by 8 generative models including generative adversarial networks (GANs) and diffusion models (DMs) demonstrates the effectiveness of our method. Overall, our TraceEvader achieves the highest average attack success rate of 79% and is robust against image transformations and dedicated denoising techniques as well where the average attack success rate is still around 75%. Our TraceEvader confirms the limitations of current model attribution techniques and calls the attention of DeepFake researchers and practitioners for more robust-purpose model attribution techniques.

Deep learning-based monocular depth estimation (MDE), extensively applied in autonomous driving, is known to be vulnerable to adversarial attacks. Previous physical attacks against MDE models rely on 2D adversarial patches, so they only affect a small, localized region in the MDE map but fail under various viewpoints. To address these limitations, we propose 3D Depth Fool (3D2Fool), the first 3D texture-based adversarial attack against MDE models. 3D2Fool is specifically optimized to generate 3D adversarial textures agnostic to model types of vehicles and to have improved robustness in bad weather conditions, such as rain and fog. Experimental results validate the superior performance of our 3D2Fool across various scenarios, including vehicles, MDE models, weather conditions, and viewpoints. Real-world experiments with printed 3D textures on physical vehicle models further demonstrate that our 3D2Fool can cause an MDE error of over 10 meters. The code is available at https://github.com/GandolfczjhI3D2Fool.

Adversarial training has achieved substantial performance in defending image retrieval against adversarial examples. However, existing studies in deep metric learning (DML) still suffer from two major limitations: weak adversary and model collapse. In this paper, we address these two limitations by proposing Collapse-Aware TRIplet DEcoupling (CA-TRIDE). Specifically, TRIDE yields a stronger adversary by spatially decoupling the perturbation targets into the anchor and the other candidates. Furthermore, CA prevents the consequential model collapse, based on a novel metric, collapseness, which is incorporated into the optimization of perturbation. We also identify two drawbacks of the existing robustness metric in image retrieval and propose a new metric for a more reasonable robustness evaluation. Extensive experiments on three datasets demonstrate that CA-TRIDE outperforms existing defense methods in both conventional and new metrics. Codes are available at https://github.com/michaeltian108/CA-TRIDE.

With the continuous evolution of AI-generated images, the generalized detection of them has become a crucial aspect of AI security. Existing detectors have focused on cross-generator generalization, while it remains unexplored whether these detectors can generalize across different image scenes, e.g., images from different datasets with different semantics. In this paper, we reveal that existing detectors suffer from substantial Accuracy drops in such cross-scene generalization. In particular, we attribute their failures to ''semantic artifacts'' in both real and generated images, to which detectors may overfit. To break such ''semantic artifacts'', we propose a simple yet effective approach based on conducting an image patch shuffle and then training an end-to-end patch-based classifier. We conduct a comprehensive open-world evaluation on 31 test sets, covering 7 Generative Adversarial Networks, 18 (variants of) Diffusion Models, and another 6 CNN-based generative models. The results demonstrate that our approach outperforms previous approaches by 2.08\% (absolute) on average regarding cross-scene detection Accuracy. We also notice the superiority of our approach in open-world generalization, with an average Accuracy improvement of 10.59\% (absolute) across all test sets. Our code is available at *https://github.com/Zig-HS/FakeImageDetection*.

Deepfake detection automatically recognizes the manipulated media by analyzing whether it contains forgeries generated through deep learning. It is natural to ask which among the existing deepfake detection approaches stand out as top performers. This question is pivotal for identifying promising research directions and offering practical guidance. Unfortunately, conducting a sound benchmark comparison of popular detection approaches based on literature results is challenging due to inconsistent evaluation conditions across studies. In this paper, our objective is to achieve a sound comparison between detection approaches by establishing a comprehensive and consistent benchmark, developing a repeatable evaluation procedure, and performing extensive performance evaluation. Accordingly, a challenging dataset consisting of the manipulated samples generated by more than 12 different methods is collected. Subsequently, we implement and evaluate 13 prominent detection approaches (comprising 11 algorithms) from existing literature, utilizing five fair-minded and practical evaluation metrics. Finally, we provide up to 882 comprehensive evaluations by training 117 detection models. The results, along with the shared data and evaluation methodology, constitute a benchmark for comparing deepfake detection approaches and measuring progress.

In recent works, robust networks have consistently exhibited more discriminative saliency map that proves to indicate sufficient adversarial robustness. In existed safe training paradigms e.g., adversarial training, however, the progressive saliency information regarding on what input semantic feature model prediction relies, have not yet been fully-explored. Due to this, we consider the incorporation of posterior saliency properties of robust model in training, as an efficient supervision signal on robust learning. It thus provides an alternative direction to enhance robustness, from the saliency interpretability perspective. In this article, to harden model we propose to optimize the discrimination of intermediate gradient-based saliency and maintain its consensus in training, which encourage model to behave according to task-relevant feature from the salient region such as object edges in image. Then, we introduce Adversarially Gradient-based Saliency Consensus Training method, dubbedAdv-GSCT. Within it, we preserve the similarity between the learned model saliency and the target one as label, approximated in the most offending case representing the least but essential information scenario. Meanwhile, a constructed pseudo-input coupled with feature importance, is feed into model to ensure the discrimination of estimated target saliency. Besides providing a novel insight into adversarial defense,Adv-GSCTdiffers from the current most effective adversarial training and does not need multiple iterative generations of adversarial perturbation whose computational cost and sensitivity direction of prediction concern. Finally, extensive performance evaluations on MNIST, CIFAR-10 and ImageNet datasets demonstrate the superiority of our proposed method.

Smartphone addiction among child users is becoming a severe global social problem. Uncontrolled and unsupervised use of smartphones by children has posed a significant threat to the health and property of both children and their parents. Automatic identification of child users on smartphones can be an effective way to alleviate this problem. Unfortunately, existing works usually require additional input devices like cameras for face biometrics or additional applications for users’ specific touch-interaction behavior to identify child users, leading to poor user experience and privacy concerns. This paper develops a novel, implicit and continuous system, named ChildShield, for child identification on smartphones. Specifically, our system providing a built-in data acquisition service can automatically and real-timely collect users’ behavioral data in a non-conscious and privacy-preserving manner. We build a large-scale database by collecting users’ operations in 5 complex and popular mobile game applications on 12 different models of smartphones from 1875 subjects. Based on the feature extracted from multi-finger interaction data in realistic and complex usage scenarios, ChildShield can learn the discriminative behavioral patterns for accurate child identification using the specifically designed deep learning-based classifiers. Then when a child user is identified, the pre-setting subsequent operation like Enable Kids Mode in ChildShield can be executed to provide a protective shield for children. The effectiveness of ChildShield is validated on the created database. Our approach significantly outperforms existing methods, achieves an EER of 4.38% for child identification, and performs an even lower EER of 2.12% for the younger age group.

Deep learning methods can not only detect false data injection attacks (FDIA) but also locate attacks of FDIA. Although adversarial false data injection attacks (AFDIA) based on deep learning vulnerabilities have been studied in the field of single-label FDIA detection, the adversarial attack and defense against multi-label FDIA locational detection are still not involved. To bridge this gap, this paper first explores the multi-label adversarial example attacks against multi-label FDIA locational detectors and proposes a general multi-label adversarial attack framework, namely muLti-labEl adverSarial falSe data injectiON attack (LESSON). The proposed LESSON attack framework includes three key designs, namely Perturbing State Variables, Tailored Loss Function Design, and Change of Variables, which can help find suitable multi-label adversarial perturbations within the physical constraints to circumvent both Bad Data Detection (BDD) and Neural Attack Location (NAL). Four typical LESSON attacks based on the proposed framework and two dimensions of attack objectives are examined, and the experimental results demonstrate the effectiveness of the proposed attack framework, posing serious and pressing security concerns in smart grids.

Adversarial Defense of deep neural networks have gained significant attention and there have been active research efforts on model vulnerabilities for attacking such as gradient-based attack and pre-defined semantic manipulation. However, they often lack clear adversarial pattern connecting model extracted notion and are restricted to fixed constraint, making the gradual inability to proposed robust defense. In this paper, we propose to utilize the learned semantics of model, possibly not be the true one for the correct prediction, as inspiring clue in adversarial example construction. And we propose a new attention-based semantic oriented adversarial attack without any prior constraint about semantic preservation, dubbed Attention-SA from the learned task-related decision factors perspective. Specifically, to capture the learned factor, we introduce a post-hoc soft attention with a gradient-sensitivity activation consistency to probe the information of latent representation that bridge the input and prediction. With the attention guidance, we perturb the separated and semantic units, then back-propagate the variation onto input to discover expanded adversarial examples. Finally, extensive performance evaluations on CIFAR-10 and ImageNet datasets demonstrate the superiority of our proposed method. And we verify the effectiveness of our method on various robust defenses.

The emergence of advanced Deepfake technologies has gradually raised concerns in society, prompting significant attention to Deepfake detection. However, in real-world scenarios, Deepfakes often involve multiple faces. Despite this, most existing detection methods still detect these faces individually, overlooking the informative correlation between them and the relationship between the global information of the image and the local information of the faces. In this paper, we address this limitation by proposing FILTER, a novel framework for multi-face forgery detection that explicitly captures underlying correlations. FILTER consists of two main modules: Multi-face Relationship Learning (MRL) and Global Feature Aggregation (GFA). Specifically, MRL learns the correlation of local facial features in multi-face images, and GFA constructs the relationship between image-level labels and individual facial features to enhance performance from a global perspective. In particular, a contrastive learning loss function is used to better discriminate between real and fake faces. Extensive experiments on two publicly available multi-face forgery datasets demonstrate the state-of-the-art performance of FILTER in multi-face forgery detection. For example, on Openforensics Test-Challenge dataset, FILTER outperforms the previous state-of-the-art methods with a higher AUC score (0.980) and higher detection accuracy (92.04%).

State-of-the-art source code classification models exhibit excellent task transferability, in which the source code encoders are first pre-trained on a source domain dataset in a self-supervised manner and then fine-tuned on a supervised downstream dataset. Recent studies reveal that source code models are vulnerable to adversarial examples, which are crafted by applying semantic-preserving transformations that can mislead the prediction of the victim model. While existing research has introduced practical black-box adversarial attacks, these are often designed for transfer-based or query-based scenarios, necessitating access to the victim domain dataset or the query feedback of the victim system. These attack resources are very challenging or expensive to obtain in real-world situations. This paper proposes the cross-domain attack threat model against the transfer learning of source code where the adversary has only access to an open-sourced pre-trained code encoder. To achieve such realistic attacks, this paper designs the Code Transfer learning Adversarial Example (CodeTAE) method. CodeTAE applies various semantic-preserving transformations and utilizes a genetic algorithm to generate powerful identifiers, thereby enhancing the transferability of the generated adversarial examples. Experimental results on three code classification tasks show that the CodeTAE attack can achieve 30%$\sim ~80$% attack success rates under the cross-domain cross-architecture setting. Besides, the generated CodeTAE adversarial examples can be used in adversarial fine-tuning to enhance both the clean accuracy and the robustness of the code model. Our code is available athttps://github.com/yyl-github-1896/CodeTAE/.

Quantized neural networks (QNNs) have received increasing attention in resource-constrained scenarios due to their exceptional generalizability. However, their robustness against realistic black-box adversarial attacks has not been extensively studied. In this scenario, adversarial transferability is pursued across QNNs with different quantization bitwidths, which particularly involve unknown architectures and defense methods. Previous studies claim that transferability is difficult to achieve across QNNs with different bitwidths on the condition that they share the same architecture. However, we discover that under different architectures, transferability can be largely improved by using a QNN quantized with an extremely low bitwidth as the substitute model. We further improve the attack transferability by proposingquantization aware attack(QAA), which fine-tunes a QNN substitute model with a multiple-bitwidth training objective. In particular, we demonstrate that QAA addresses the two issues that are commonly known to hinder transferability: 1) quantization shifts and 2) gradient misalignments. Extensive experimental results validate the high transferability of the QAA to diverse target models. For instance, when adopting the ResNet-34 substitute model on ImageNet, QAA outperforms the current best attack in attacking standardly trained DNNs, adversarially trained DNNs, and QNNs with varied bitwidths by 4.6% ~ 20.9%, 8.8% ~ 13.4%, and 2.6% ~ 11.8% (absolute), respectively. In addition, QAA is efficient since it only takes one epoch for fine-tuning. In the end, we empirically explain the effectiveness of QAA from the view of the loss landscape. Our code is available at https://github.com/yyl-github-1896/QAA/.

The utilization of web social media for depression detection has been proven effective in recent years since the multimedia signal on web can reflect users’ emotions, feelings, and personality traits in advance. However, most earlier studies simply used users’ submitted words or user profiles to predict depression risk. The implicit information accessible in users’ posted images, which can be effective in depression detection, still remains unexplored. In this paper, an implicit and explicit multi-modal feature fusion (IEMFF) model is proposed for depression detection. We successfully make the implicit information inherent in users’ posted images explicit and further incorporate such explicit features with the textual features directly extracted from user-posted texts. A multi-modal feature fusion approach is applied for depression detection. Extensive experiments have been conducted on public Twitter datasets. Experimental results show that our approach has achieved state-of-the-art performance for depression detection.

The approaches based on deep learning have achieved remarkable success in diabetic retinopathy detection. Due to the accountability in medical diagnosis, the interpretability of computer-aided diagnosis has recently been investigated. However, few existing approaches make full use of the explainable evidence to improve the diagnosis accuracy. In this paper, we propose a Model-free Lesion Generation and Learning (MLGL) framework to study the interpretability of diabetic retinopathy detection. We first generate visual explanations for diabetic retinopathy diagnosis using the proposed Gated Multi-layer Saliency Map (GMSM) module, which locates the accurate region of lesions by combining multi-layer heatmaps. Then we use the GMSM to extract the lesion patches and conduct the adaptive lesion transfer, iteratively generating new retinal fundus images with lesions. Especially, in this process, no additional generative models are trained. Finally, we merge the generated and original retinal fundus images for the model's training to learn robust lesion features. Overall, our method provides accurate explainable evidence and further addresses the data imbalance problem in diabetic retinopathy detection. The experimental results on four public datasets demonstrate the efficiency of our approach.

Owing to the boom of smartphone industries, the expansion of phone users has also been significant. Besides adults, children and elders have also begun to join the population of daily smartphone users. Such an expansion indeed facilitates the further exploration of the versatility and flexibility of digitization. However, these new users may also be susceptible to issues such as addiction, fraud, and insufficient accessibility. To fully utilize the capability of mobile devices without breaching personal privacy, we build the first corpus for age group recognition on smartphones with more than 1,445,087 unrestricted actions from 2,100 subjects. Then a series of heuristically-selected and neurally-guided features are proposed to increase the separability of the above dataset. Finally, we develop AgeCare, the first implicit and continuous system incorporated with bottom-to-top functionality without any restriction on user-phone interaction scenarios, for accurate age group recognition and age-tailored assistance on smartphones. Our system performs impressively well on this dataset and significantly surpasses the state-of-the-art methods.

Deep Neural Networks are well known to be vulnerable to adversarial attacks and backdoor attacks, where minor modifications on the input are able to mislead the models to give wrong results. Although defenses against adversarial attacks have been widely studied, investigation on mitigating backdoor attacks is still at an early stage. It is unknown whether there are any connections and common characteristics between the defenses against these two attacks. We conduct comprehensive studies on the connections between adversarial examples and backdoor examples of Deep Neural Networks to seek to answer the question: can we detect backdoor using adversarial detection methods. Our insights are based on the observation that both adversarial examples and backdoor examples have anomalies during the inference process, highly distinguishable from benign samples. As a result, we revise four existing adversarial defense methods for detecting backdoor examples. Extensive evaluations indicate that these approaches provide reliable protection against backdoor attacks, with a higher accuracy than detecting adversarial examples. These solutions also reveal the relations of adversarial examples, backdoor examples and normal samples in model sensitivity, activation space and feature space. This is able to enhance our understanding about the inherent features of these two attacks and the defense opportunities.

Behavioral biometrics has been widely investigated and deployed in real world scenarios for human authentication. However, there has been almost nil attempt to identify behavior patterns in a cross-scenario setting, which is common in practice and needs urgent attention. This paper defines and investigates cross-scenario behavioral biometrics authentication using keystroke dynamics. A novel system called CrossBehaAuth is presented for extending keystroke dynamics-based behavior authentication to new scenarios and extensive problems. We design a temporal-aware learning mechanism based deep neural network for cross-scenario keystroke dynamics authentication. This mechanism selectively learns and encodes temporal information for efficient behavioral pattern transfer in cross-scenario settings. A local Gaussian data augmentation approach is proposed to increase the diversity of behavioral data and therefore, further improve the performance. We evaluate the proposed approach on two publicly available datasets. The extensive experimental results confirm the efficacy of our CrossBehaAuth for cross-scenario keystroke dynamics authentication. Our approach significantly improves the authentication accuracy in cross-scenario settings and even achieves comparable performance on single-scenario authentication tasks. In addition, our approach shows its generalizability and advantages in both single and cross scenario keystroke dynamics authentication.

Despite regularizing the Jacobians of neural networks to enhance model robustness has directly theoretical correlation with model prediction stability, a large defense performance gap exists when compared to the empirically perturbation-based adversarial training e.g. PGD-based, which enjoys nice discriminative saliency maps as well. To mitigate this issue, in this paper we first analyze the dilemma that the gradient map of its resulting model has no content hierarchy to mark out salient profile of input, as a negative signal of the obstructive for effective adversarial defense. Based on this, we argue that incorporating robust gradient-based saliency properties into regularized training may be helpful to reduce the performance gap. Specifically, we propose a simple method called Saliency-aware Gradient Regularization (SAGR), where a biased weight distribution strategy is introduced on positive gradient to structure and increase the impact of class-gradient components inside the Jacobian of model. The strategy maintains the dominant role of saliency-critical true-class gradient in learning process and differentiates diverse importance of gradient sensitivities that would localize input salient areas. Herein we interpret the sharpness of true-class sensitivity as robust recognition of more learning-relevant features e.g., regions containing dominant object in image for classification. Instead, false-class parts are considered as recognition-irrelevant nuisance factors e.g. the backgrounds, which are thus depressed with more strength. Experimental results demonstrate the efficacy of the proposed method and validate that distinguishment of sensitivities could further yield more robustness gain and sharper gradient saliency map.

In interconnected industrial control networks like smart grids, topology attacks on physical grids can lead to severe cascading failures and large-scale blackouts. Effective defense on vulnerable devices can significantly reduce the risk of cascading failures and improve overall system robustness. In this paper, we investigate the vulnerability analysis problem from a graph theoretical classification perspective. By calculating a node vulnerability vector composed of features based on complex network theory, node embedding, extended betweenness and power flow distribution, we propose a node vulnerability analysis method based on XGBoost classifier. A cascading failure simulation model based on DC power flow is used to simulate the smart grid behaviours under topology attacks and create the dataset for the XGBoost classifier. The effectiveness of the proposed XGBoost-based method with newly-introduced features is demonstrated by case studies.

Weakly supervised object detection (WSOD) using only image-level annotations has attracted growing attention over the past few years. Existing approaches using multiple instance learning easily fall into local optima, because such mechanism tends to learn from the most discriminative object in an image for each category. Therefore, these methods suffer from missing object instances which degrade the performance of WSOD. To address this problem, this paper introduces an end-to-end object instance mining (OIM) framework for weakly supervised object detection. OIM attempts to detect all possible object instances existing in each image by introducing information propagation on the spatial and appearance graphs, without any additional annotations. During the iterative learning process, the less discriminative object instances from the same class can be gradually detected and utilized for training. In addition, we design an object instance reweighted loss to learn larger portion of each object instance to further improve the performance. The experimental results on two publicly available databases, VOC 2007 and 2012, demonstrate the efficacy of proposed approach.

The use of social media runs through our lives, and users' emotions are also affected by it. Previous studies have reported social organizations and psychologists using social media to find depressed patients. However, due to the variety of content published by users, it isn't effortless for the system to consider the text, image, and even the hidden information behind the image. To address this problem, we proposed a new system for social media screening of depressed patients named BlueMemo. We collected real-time posts from Twitter. Based on the posts, learned text features, image features, and visual attributes were extracted as three modalities and were fed into a multi-modal fusion and classification model to implement our system. The proposed BlueMemo has the power to help physicians and clinicians quickly and accurately identify users at potential risk for depression.

More than 300 million people have been affected by depression all over the world. Due to the medical equipment and knowledge limitations, most of them are not diagnosed at the early stages. Recent work attempts to use social media to detect depression since the patterns of opinions and thoughts expression of the posted text and images, can reflect users' mental state to some extent. In this work, we design a system dubbed SenseMood to demonstrate that the users with depression can be efficiently detected and analyzed by using proposed system. A deep visual-textual multimodal learning approach has been proposed to reveal the psychological state of the users on social networks. The posted images and tweets data from users with/without depression on Twitter have been collected and used for depression detection. CNN-based classifier and Bert are applied to extract the deep features from the pictures and text posted by users respectively. Then visual and textual features are combined to reflect the emotional expression of users. Finally our system classifies the users with depression and normal users through a neural network and the analysis report is generated automatically.

Accurate comparison of contactless 2-D fingerprint images with contact-based fingerprints is critical for the success of emerging contactless 2-D fingerprint technologies, which offer more hygienic and deformation-free acquisition of fingerprint features. Convolutional neural networks (CNNs) have shown remarkable capabilities in biometrics recognition. However, there has been almost nil attempt to match fingerprint images using CNN-based approaches. This paper develops a CNN-based framework to accurately match contactless and contact-based fingerprint images. Our framework first trains a multi-Siamese CNN using fingerprint minutiae, respective ridge map and specific region of ridge map. This network is used to generate deep fingerprint representation using a distance-aware loss function. Deep fingerprint representations generated in such multi-Siamese network are concatenated for more accurate cross comparison. The proposed approach for cross-fingerprint comparison is evaluated on two publicly available databases containing contactless 2-D fingerprints and respective contact-based fingerprints. Our experiments presented in this paper consistently achieve outperforming results over several popular deep learning architectures and over contactless to contact-based fingerprints comparison methods in the literature.

Emerging 3D fingerprint recognition technologies have attracted growing attention in addressing the limitations from contact-based fingerprint acquisition and improve recognition accuracy. However, the complex 3D imaging setups employed in these systems typically require structured lighting with scanners or multiple cameras which are bulky with higher cost. This paper presents a more accurate and efficient 3D fingerprint identification approach using a single 2D camera with multiple colored LED illumination. A 3D minutiae tetrahedron based algorithm is developed to more efficiently match recovered minutiae features in 3D space and address the limitations of 3D minutiae matching approach in the literature. This algorithm significantly improves the matching time to about 15 times than the state-of-art in the reference. A hierarchical tetrahedron matching scheme is also developed to further improve the matching accuracy with faster speed. The 2D images acquired to reconstruct the 3D fingerprints are also used to recover 2D minutiae and further improve matching performance for 3D fingerprints. A new two-session database acquiring from 300 different clients consists of 2760 3D fingerprints reconstructed from 5520 colored 2D fingerprints is also developed and shared in public domain to further advance much needed research in this area. Extensive experimental results presented in this paper validate our approach and demonstrate the effectiveness of proposed algorithms.

Vast databases of billions of contact-based fingerprints have been developed to protect national borders and support e-governance programs. Emerging contactless fingerprint sensors offer better hygiene, security and accuracy. However the adoption/success of such contactless fingerprint technologies largely depends on advanced capability to match contactless 2D fingerprints with legacy contact-based fingerprint databases. This paper investigates such problem and develops a new approach to accurately match such fingerprint images. Robust thin-plate spline (RTPS) is developed to more accurately model elastic fingerprint deformations using splines. In order to correct such deformations on the contact-based fingerprints, RTPS based generalized fingerprint deformation correction model (DCM) is proposed. The usage of DCM results in accurate alignment of key minutiae features observed on the contactless and contactbased fingerprints. Further improvement in such cross-matching performance is investigated by incorporating minutiae related ridges. We also develop a new database of 1800 contactless 2D fingerprints and the corresponding contact-based fingerprints acquired from 300 clients which is made publicly accessible for further research. The experimental results presented in this paper, using two publicly available databases, validate our approach and achieve outperforming results for matching contactless 2D and contact-based fingerprint images.

Contactless 2D fingerprint identification is more hygienic, and enables deformation free imaging for higher accuracy. Success of such emerging contactless fingerprint technologies requires advanced capabilities to accurately match such fingerprint images with the conventional fingerprint databases which have been developed and deployed in last two decades. Convolutional neural networks have shown remarkable success for the face recognition problem. However, there has been very few attempts to develop CNN-based methods to address challenges in fingerprint identification problems. This paper proposes a multi-Siamese CNN architecture for accurately matching contactless and contact-based fingerprint images. In addition to the fingerprint images, hand-crafted fingerprint features, e.g. minutiae and core point, are also incorporated into the proposed architecture. This multi-Siamese CNN is trained using the fingerprint images and extracted features. Therefore, a more robust deep fingerprint representation is formed from the concatenation of deep feature vectors generated from multi-networks. In order to demonstrate the effectiveness of the proposed approach, a publicly available database consisting of contact-based and respective contactless finger-prints is utilized. The experimental evaluations presented in this paper achieve outperforming results, over other CNN-based methods and the traditional fingerprint cross matching methods, and validate our approach.

Improving accuracy of matching fingerprint images acquired from two different fingerprint sensors is an important research problem with several promising studies in the literature. Most of these studies focus on sensor interoperability using fingerprints acquired from different kinds of contact-based sensors. However emerging contactless fingerprint technologies have shown its benefits. This paper investigates fingerprint sensor interoperability problem using fingerprints acquired from contact-based and contactless sensor. We propose a generalized contact-based fingerprint deformation correction model (DCM) to improve the matching accuracy. This model is trained by estimating the deformation between contact-based fingerprint and corresponding contactless fingerprint (ground truth). We present a method to estimate contact-based fingerprint impression type and intensity. As a result, minutiae features from contact-based and contactless fingerprint can be better aligned using the proposed model. A database of 1200 2D contactless fingerprints and respective contact-based fingerprints from 200 clients is used for the experiments. The experimental results presented in this paper validate our approach and illustrate promising improvement in performance using the proposed model.