Large Language Models (LLMs) demonstrate promising capabilities in natural language understanding and reasoning with enormous parameter spaces and vast amounts of training data. These attributes have facilitated their deployment into diverse application domains. However, the underlying parameters implicitly assume decision-making boundaries, resulting in a significant number of decision spaces not covered by training data. This makes them susceptible to adversarial manipulations through carefully crafted inputs. To illuminate the vulnerabilities of LLMs, we propose a unified token-level jailbreaking attack that makes victim models generate responses for potentially harmful queries. Specifically, we propose an evolutionary algorithm to evolve perturbation sets, utilizing gradient-based and crossover-based operators to enhance performance under multiple scenarios. Furthermore, we develop a repository for reusing past perturbations and conduct an analysis of token sensitivity within LLMs, facilitating zero-shot attacks with convergence. Extensive benchmark experiments validate the effectiveness of our method on three different models, achieving increases of 62.36%, 57.89%, and 64.81% in attack success rates compared to baseline methods under white-box scenarios. In addition, the evaluation experiments demonstrate our method is effective for multiple scenarios and different size of models. This research reveals vulnerabilities of LLMs, provides theoretical foundations for developing more robust defense strategies, and contributes to building more reliable AI systems.

Diffusion models (DMs) have demonstrated great potential in the field of adversarial robustness, where DM-based defense methods can achieve superior defense capability without adversarial training. However, they all require huge computational costs due to the usage of large-scale pre-trained DMs, making it difficult to conduct full evaluation under strong attacks and compare with traditional CNN-based methods. Simply reducing the network size and timesteps in DMs could significantly harm the image generation quality, which invalidates previous frameworks. To alleviate this issue, we redesign the diffusion framework from generating high-quality images to predicting distinguishable image labels. Specifically, we employ an image translation framework to learn many-to-one mapping from input samples to designed orthogonal image labels. Based on this framework, we introduce an efficient Image-to-Image diffusion classifier with a pruned U-Net structure and reduced diffusion timesteps. Besides the framework, we redesign the optimization objective of DMs to fit the target of image classification, where a new classification loss is incorporated in the DM-based image translation framework to distinguish the generated label from those of other classes. We conduct sufficient evaluations of the proposed classifier under various attacks on popular benchmarks. Extensive experiments show that our method achieves better adversarial robustness with fewer computational costs than DM-based and CNN-based methods.

Vision Language Pre-training Models have shown significant potential in various domains, but there are few attempts to introduce it in the field of continual learning for video action recognition. We propose Video Class-Incremental Learner with CLIP based Transformer (VCIL-CT), which uses CLIP based vision transformer to train action recognition task by class-incremental learning pipeline. To specifically address the issue of catastrophic forgetting in transformer, we introduce Attention Distillation which distilling the attention feature from each transformer decoder. In the process of incremental learning of classes, there may be a problem of high bias towards new classes, we incorporate Class Balance Module to prevent bias on new task. Furthermore, we adopt Exemplar Augment strategy to improve exemplar quality on data replay step. We evaluate our proposed method based on the incremental action recognition benchmark presented by TCD, using UCF101, HMDB51, and UESTC-MMEA-CL datasets, and demonstrate the effectiveness of our algorithm compared to existing state-of-the-art continuous learning methods for action recognition.

Few-shot object detection aims at detecting previously unseen objects given only a few annotated samples. Most existing approaches treat the model obtained from the base training stage with abundant data as a container of prior knowledge that can be transferred to novel objects. Knowledge with similar properties is also contained in Contrastive Language-Image Pretraining (CLIP). In this paper, we utilize this external prior knowledge to generate proposal-level classification scores to improve the detection results. We notice that these scores can hardly reflect the quality of proposal localization, so we combine them with the ones from a conventional detector to obtain the ability to distinguish the background. Moreover, we propose a new score fusion module with regularization to alleviate the ambiguity of detection results generated by a trivial element-wise multiplication fusion method. To further improve the quality of classification scores in our proposed branch, we add learnable prompts to mitigate the inaccurate classification problem we observe. We conduct extensive experiments on the PASCAL VOC dataset and demonstrate the effectiveness of our approach.

In aerial scenes, oriented object detection is sensitive to the orientation of objects, which makes the formulation of orientation-aware object representation become a critical problem. Existing methods mostly adopt rectangle anchor or discrete points as object representation, which may lead to the feature aliasing between overlapping objects and ignore the orientation information of objects. To solve these issues, we propose a novel anchor-free oriented object detection network named DRDet, which adopts Dual-angle Rotated Lines (DRL) as object representation. Different from other object representations, DRL can adaptively rotate and extend to the boundary of the object according to its orientation and shape, which explicitly introduces the orientation information into the formulation of object representation. And it can adaptively cope with the geometric deformation of objects. Based on the dual-angle rotated lines, we design an Orientation-guided Feature Encoder (OFE) to encode discriminant object feature along each rotated line, respectively. Instead of encoding rectangle feature, the OFE module adopts line features for orientation-guided feature encoding, which can alleviate the feature aliasing between neighboring objects or background. To further enhance the flexibility of dual-angle rotated lines, we design a Dual-angle Decoder (DD) that predicts two angle offsets according to the orientation-guided feature and converts the angle offsets and regression offsets into dual-angle rotated line representation, which can help to guide the adaptive rotation of each rotated line, respectively. Our proposed method achieves consistent improvement on both DOTA and HRSC2016 datasets. Extensive experimental results verify the effectiveness of our method in oriented object detection.

Object detection aims to localize and classify objects. Suitable object representation plays an important role in accurate detection. Because a complete crossline inevitably passes through the noise of backgrounds or other objects, object features directly extracted by the whole crossline are often confused. In this paper, we present a new feature extraction method, DE-Crossline, which can enhance the original crossline representation to capture more accurate object information. Specifically, we divide the crossline into several segments, each of which extracts the maximum activation key point respectively to reduce the impact of noise mentioned above. Furthermore, considering various shapes and sizes of objects, we design a Deformable Width Extension Module to learn a suitable width of each crossline, so as to capture richer object information. Extensive experiments prove the effectiveness of our proposed method. The total performance of our proposed detector can reach 49.0% AP, using ResNet-101 as backbone on the MS-COCO dataset.