The rapid development of AI-generated content (AIGC) technology has led to the misuse of highly realistic AI-generated images (AIGI) in spreading misinformation, posing a threat to public information security. Although existing AIGI detection techniques are generally effective, they face two issues: 1) a lack of human-verifiable explanations, and 2) a lack of generalization in the latest generation technology. To address these issues, we introduce a large-scale and comprehensive dataset, Holmes-Set, which includes the Holmes-SFTSet, an instruction-tuning dataset with explanations on whether images are AI-generated, and the Holmes-DPOSet, a human-aligned preference dataset. Our work introduces an efficient data annotation method called the Multi-Expert Jury, enhancing data generation through structured MLLM explanations and quality control via cross-model evaluation, expert defect filtering, and human preference modification. In addition, we propose Holmes Pipeline, a meticulously designed three-stage training framework comprising visual expert pre-training, supervised fine-tuning, and direct preference optimization. Holmes Pipeline adapts multimodal large language models (MLLMs) for AIGI detection while generating human-verifiable and human-aligned explanations, ultimately yielding our model AIGI-Holmes. During the inference stage, we introduce a collaborative decoding strategy that integrates the model perception of the visual expert with the semantic reasoning of MLLMs, further enhancing the generalization capabilities. Extensive experiments on three benchmarks validate the effectiveness of our AIGI-Holmes.

Despite the great success, most models in image captioning (IC) are still stuck in the dilemma of generating simple and non-discriminative captions. In this paper, we study this problem from the perspective of data augmentation and propose a novel method called Multi-modal Mixup ( M 3 ixup). Compared with the original Mixup strategy designed for image classification , the proposed M 3 ixup has three novel designs to mix IC samples from the aspects of visual features, sentence embeddings and loss values, respectively. In practice, M 3 ixup can not only enrich the diversity of IC training data , but also enforce the model to focus more on visual information for captioning, thereby alleviating the negative effect of dataset bias and addressing the issue of simple captioning. To validate M 3 ixup, we apply it to three baseline models and conduct extensive experiments on MS COCO. The experimental results demonstrate that our proposed M 3 ixup can not only improve the discriminability and quality of generated captions, but also help the baseline models obtain obvious performance gains, i.e. , improving the CIDEr scores of the state-of-the-art model from 133.8 to 135.3 on off-line testing and 135.4 to 137.1 on online testing.

The evolution of Diffusion Models has dramatically improved image generation quality, making it increasingly difficult to differentiate between real and generated images. This development, while impressive, also raises significant privacy and security concerns. In response to this, we propose a novel Latent REconstruction error guided feature REfinement method (LaRE2) for detecting the diffusion-generated images. We come up with the Latent Reconstruction Error (LaRE), the first reconstruction-error based feature in the latent space for generated image detection. LaRE surpasses existing methods in terms of feature extraction efficiency while preserving crucial cues required to differentiate between the real and the fake. To exploit LaRE, we propose an Error-Guided feature REfinement module (EGRE), which can refine the image feature guided by LaRE to enhance the discriminativeness of the feature. Our EGRE utilizes an align-then-refine mechanism, which effectively refines the image feature for generated-image detection from both spatial and channel perspectives. Extensive experiments on the large-scale GenImage benchmark demonstrate the superiority of our LaRE2, which surpasses the best SoTA method by up to 11.9%/12.1% average ACC/AP across 8 different image generators. LaRE also surpasses existing methods in terms of feature extraction cost, delivering an impressive speed enhancement of 8 times.

The rapid advancement of edge and cloud computing platforms, vehicular ad-hoc networks, and machine learning techniques have brought both opportunities and challenges for next-generation connected and automated vehicles (CAVs). On the one hand, these technologies can enable vehicles to leverage more computing power from edge and cloud servers and to share information with each other and surrounding infrastructures for better situation awareness and more intelligent decision making. On the other hand, the more distributed computing process and the wireless nature of V2X (vehicle-to-everything) communication expose vulnerabilities to various disturbances and attacks. In this paper, we discuss the security and safety challenges for edge- and cloud-enabled CAVs, particularly when they are under environment interferences, execution errors, and malicious attacks, and we will introduce our recent work and future directions in developing system-driven, end-to-end methodologies and tools to address these challenges and ensure system resiliency under uncertainties.

In autonomous driving (AD), accurate perception is indispensable to achieving safe and secure driving. Due to its safety-criticality, the security of AD perception has been widely studied. Among different attacks on AD perception, the physical adversarial object evasion attacks are especially severe. However, we find that all existing literature only evaluates their attack effect at the targeted AI component level but not at the system level, i.e., with the entire system semantics and context such as the full AD pipeline. Thereby, this raises a critical research question: can these existing researches effectively achieve system-level attack effects (e.g., traffic rule violations) in the real-world AD context? In this work, we conduct the first measurement study on whether and how effectively the existing designs can lead to system-level effects, especially for the STOP sign-evasion attacks due to their popularity and severity. Our evaluation results show that all the representative prior works cannot achieve any system-level effects. We observe two design limitations in the prior works: 1) physical model-inconsistent object size distribution in pixel sampling and 2) lack of vehicle plant model and AD system model consideration. Then, we propose SysAdv, a novel system-driven attack design in the AD context and our evaluation results show that the system-level effects can be significantly improved, i.e., the violation rate increases by around 70%.

Localization in high-level Autonomous Driving (AD) systems is highly security critical. Recently, researchers found that state-of-the-art Multi-Sensor Fusion (MSF) based localization is vulnerable to GPS spoofing, which can cause road hazards such as driving off road or onto the wrong way. In this work, we perform the first exploration of using Lane Detection (LD) to detect and correct deviations caused by such attacks and design a novel LD-based system-level defense, LD3. We evaluate LD3 on real-world sensor traces and find that it can achieve effective and timely detection against the state-of-the-art attack with 100% true positive rates and 0% false positive rates. Results show that LD3 can be highly effective at steering the AD vehicle to safely stop within the current traffic lane. We implement LD3 on 2 open-source AD systems and validate its end-to-end defense capability using an industry-grade AD simulator and also in the physical world with a real vehicle-sized AD R&D vehicle.

Connected Vehicle (CV) technologies are under rapid deployment across the globe and will soon reshape our transportation systems, bringing benefits to mobility, safety, environment, etc. Meanwhile, such technologies also attract attention from cyberattacks. Recent work shows that CV-based Intelligent Traffic Signal Control Systems are vulnerable to data spoofing attacks, which can cause severe congestion effects in intersections. In this work, we explore a general detection strategy for infrastructure-side CV applications by estimating the trustworthiness of CVs based on readily-available infrastructure-side sensors. We implement our detector for the CV-based traffic signal control and evaluate it against two representative congestion attacks. Our evaluation in the industrial-grade traffic simulator shows that the detector can detect attacks with at least 95% true positive rates while keeping false positive rate below 7% and is robust to sensor noises.

In Autonomous Driving (AD) systems, perception is both security and safety-critical. Among different attacks on AD perception, object-hiding adversarial attack is one of the most critical ones due to the direct impact on safety-critical driving decisions such as collision avoidance. However, all of the prior works on physical object-hiding adversarial attacks only study the security of the AI component alone rather than with the entire AD system pipeline with closed-loop control. This thus inevitably raises a critical research question: can these prior works actually achieve system-level effects (e.g., vehicle collisions, traffic rule violation) under real-world AD settings with closed-loop control?

Transformer-based architectures have shown great success in image captioning, where object regions are encoded and then attended into the vectorial representations to guide the caption decoding. However, such vectorial representations only contain region-level information without considering the global information reflecting the entire image, which fails to expand the capability of complex multi-modal reasoning in image captioning. In this paper, we introduce a Global Enhanced Transformer (termed GET) to enable the extraction of a more comprehensive global representation, and then adaptively guide the decoder to generate high-quality captions. In GET, a Global Enhanced Encoder is designed for the embedding of the global feature, and a Global Adaptive Decoder are designed for the guidance of the caption generation. The former models intra- and inter-layer global representation by taking advantage of the proposed Global Enhanced Attention and a layer-wise fusion module. The latter contains a Global Adaptive Controller that can adaptively fuse the global information into the decoder to guide the caption generation. Extensive experiments on MS COCO dataset demonstrate the superiority of our GET over many state-of-the-arts.

Descriptive region features extracted by object detection networks have played an important role in the recent advancements of image captioning. However, they are still criticized for the lack of contextual information and fine-grained details, which in contrast are the merits of traditional grid features. In this paper, we introduce a novel Dual-Level Collaborative Transformer (DLCT) network to realize the complementary advantages of the two features. Concretely, in DLCT, these two features are first processed by a novel Dual-way Self Attenion (DWSA) to mine their intrinsic properties, where a Comprehensive Relation Attention component is also introduced to embed the geometric information. In addition, we propose a Locality-Constrained Cross Attention module to address the semantic noises caused by the direct fusion of these two features, where a geometric alignment graph is constructed to accurately align and reinforce region and grid features. To validate our model, we conduct extensive experiments on the highly competitive MS-COCO dataset, and achieve new state-of-the-art performance on both local and online test sets, i.e., 133.8% CIDEr on Karpathy split and 135.4% CIDEr on the official split.

Recent progress on visual question answering has explored the merits of grid features for vision language tasks. Meanwhile, transformer-based models have shown remarkable performance in various sequence prediction problems. However, the spatial information loss of grid features caused by flattening operation, as well as the defect of the transformer model in distinguishing visual words and non visual words, are still left unexplored. In this paper, we first propose Grid-Augmented (GA) module, in which relative geometry features between grids are incorporated to enhance visual representations. Then, we build a BERT-based language model to extract language context and propose Adaptive-Attention (AA) module on top of a transformer decoder to adaptively measure the contribution of visual and language cues before making decisions for word prediction. To prove the generality of our proposals, we apply the two modules to the vanilla transformer model to build our Relationship-Sensitive Transformer (RSTNet) for image captioning task. The proposed model is tested on the MSCOCO benchmark, where it achieves new state-of-art results on both the Karpathy test split and the online test server. Source code is available at GitHub1.