As Large Language Models (LLMs) become more advanced, the security risks they pose also increase. Ensuring that LLM behavior aligns with human values, particularly in mitigating jailbreak attacks with elusive and implicit intentions, has become a significant challenge. To address this issue, we propose a jailbreak defense method called Real Intentions Defense (RID), which involves two phases: soft extraction and hard deletion. In the soft extraction phase, LLMs are leveraged to extract unbiased, genuine intentions, while in the hard deletion phase, a greedy gradient-based algorithm is used to remove the least important parts of a sentence, based on the insight that words with smaller gradients have less impact on its meaning. We conduct extensive experiments on Vicuna and Llama2 models using eight state-of-the-art jailbreak attacks and six benchmark datasets. Our results show a significant reduction in both Attack Success Rate (ASR) and Harmful Score of jailbreak attacks, while maintaining overall model performance. Further analysis sheds light on the underlying mechanisms of our approach.

Open-vocabulary object detection aims to detect novel categories that are independent from the base categories used during training. Most modern methods adhere to the paradigm of learning vision-language space from a large-scale multi-modal corpus and subsequently transferring the acquired knowledge to off-the-shelf detectors like Faster-RCNN. However, information attenuation or destruction may occur during the process of knowledge transfer due to the domain gap, hampering the generalization ability on novel categories. To mitigate this predicament, in this paper, we present a novel framework named BIND, standing for Bulit-IN Detector, to eliminate the need for module replacement or knowledge transfer to off-the-shelf detectors. Specifically, we design a two-stage training framework with an Encoder-Decoder structure. In the first stage, an image-text dual encoder is trained to learn region-word alignment from a corpus of image-text pairs. In the second stage, a DETR-style decoder is trained to perform detection on annotated object detection datasets. In contrast to conventional manually designed non-adaptive anchors, which generate numerous redundant proposals, we develop an anchor proposal network that generates anchor proposals with high likelihood based on candidates adaptively, thereby substantially improving detection efficiency. Experimental results on two public benchmarks, COCO and LVIS, demonstrate that our method stands as a state-of-the-art approach for open-vocabulary object detection.

Image captioning is one of the fundamental problems of computer vision and has drawn great attention over the years. However, most existing methods in image captioning focus on improving the quality of the image captions, while ignoring the ability of controlling the caption style. In this work, we aim to improve the controllability of image captioning methods, especially, by choosing to describe the image either roughly or in detail. We find this can be achieved by adding a simple length level embedding into existing models, which enables them to generate length-controllable captions describing the image at a specified level of detail, and further improve the diversity. On top of it, we propose to learn a length-level reranking transformer that captures the correlation between the semantic complexities of the image and text modalities, which can be used to select the most suitable length level for each image to make the captions informative while not being redundant. Moreover, when the length of the generated captions grows, existing methods usually suffer from a linearly increased computational complexity due to their autoregressive (AR) nature. To aid this, we devise a non-autoregressive (NAR) approach that generates captions in a length-irrelevant complexity. Besides, we propose a refinement-enhanced sequence training scheme and also adopt a sequence-level knowledge distillation technique for the training of our NAR model to bridge its performance gap with the AR models. In the experiments, our length-controllable models not only achieve SOTA performance in terms of caption quality on the MS COCO dataset but more importantly, generate controllable and diverse image captions. Specifically, our NAR model outperforms the AR baselines in terms of controllability and diversity, and also significantly improves the decoding efficiency for longer captions. By further applying the sequence-level training schemes, the caption quality of our NAR model improves clearly and is competitive with the state-of-the-art AR baselines.

Face clustering is a method for unlabeled image annotation and has attracted increasing attention. Existing methods have made significant breakthroughs by introducing Graph Convolutional Networks (GCNs) on the affinity graph. However, such graphs will contain many vertex pairs with inconsistent similarities and labels, thus degrading the model’s performance. There are already relevant efforts for this problem, but the information about features needs to be mined further. In this paper, we define a new concept called confidence edge and guide the construction of graphs. Furthermore, a novel confidence-GCN is proposed to cluster face images by deriving more confidence edges. Firstly, Local Information Fusion is advanced to obtain a more accurate similarity metric by considering the neighbors of vertices. Then Unsupervised Neighbor Determination is used to discard low-quality edges based on similarity differences. Moreover, we elaborate that the remaining edges retain the most beneficial information to demonstrate the validity. At last, the confidence-GCN takes the graph as the input and fully uses the confidence edges to complete the clustering. Experiments show that our method outperforms existing methods on the face and person datasets to achieve state-of-the-art. At the same time, comparable results are obtained on the fashion dataset.

Semantic retrieval, which retrieves semantically matched items given a textual query, has been an essential component to enhance system effectiveness in e-commerce search. In this paper, we study the multimodal retrieval problem, where the visual information (e.g, image) of item is leveraged as supplementary of textual information to enrich item representation and further improve retrieval performance. Though learning from cross-modality data has been studied extensively in tasks such as visual question answering or media summarization, multimodal retrieval remains a non-trivial and unsolved problem especially in the asymmetric scenario where the query is unimodal while the item is multimodal. In this paper, we propose a novel model named SMAR, which stands for Semantic-enhanced Modality-Asymmetric Retrieval, to tackle the problem of modality fusion and alignment in this kind of asymmetric scenario. Extensive experimental results on an industrial dataset show that the proposed model outperforms baseline models significantly in retrieval accuracy. We have open sourced our industrial dataset for the sake of reproducibility and future research works.

The evaluation of Network-On-Chip (NoC) architectures is an up to date problem in the design of System-on-Chip. Emulation on FPGA (Field Programmable Gate Array) is used to cover all possible NoC solutions in a reduced exploration time. Emulation requires multi-FPGA platform as the resources for large NoC is important and cannot be handling by one FPGA. In the same time, SoC community is exploring 3D technology for the next generation of large SoC with 3D NoC, making emulation more complex. This paper presents a case study of the deployment of the 2D NoC structure to 3D. A design flow is proposed for the automatic generation of a NoC targeting 3D on multi-FPGAs. The flow integrates emulation blocks used for the validation and exploration on the NoC. With this automatic aided tool, the designer can evaluate and explore the NoC architecture and extract performances of the NoC regardless of the multi-component platform. One may expect a communication performance improvement using an adapted partitioning of the NoC, as highlighted by the results given in this paper.