Pre-trained large deep learning models are now serving as the dominant component for downstream middleware users and have revolutionized the learning paradigm, replacing the traditional approach of training from scratch locally. To reduce development costs, developers often integrate third-party pre-trained deep neural networks (DNNs) into their intelligent software systems. However, utilizing untrusted DNNs presents significant security risks, as these models may contain intentional backdoor defects resulting from the black-box training process. These backdoor defects can be activated by hidden triggers, allowing attackers to maliciously control the model and compromise the overall reliability of the intelligent software. To ensure the safe adoption of DNNs in critical software systems, it is crucial to establish a backdoor defect database for localization studies. This paper addresses this research gap by introducing BDefects4NN, the first backdoor defect database, which provides labeled backdoor-defected DNNs at the neuron granularity and enables controlled localization studies of defect root causes. In BDefects$4 N N$, we define three defect injection rules and employ four representative backdoor attacks across four popular network architectures and three widely adopted datasets, yielding a comprehensive database of$\mathbf{1, 6 5 4}$backdoor-defected DNNs with four defect quantities and varying infected neurons. Based on BDefects4NN, we conduct extensive experiments on evaluating six fault localization criteria and two defect repair techniques, which show limited effectiveness for backdoor defects. Additionally, we investigate backdoor-defected models in practical scenarios, specifically in lane detection for autonomous driving and large language models (LLMs), revealing potential threats and highlighting current limitations in precise defect localization. This paper aims to raise awareness of the threats brought by backdoor defects in our community and inspire future advancements in fault localization methods.

Abstract Person image generation aims to synthesize realistic person images that follow the same distribution as the given dataset. Previous attempts can be generally categorized into two classes: conditional GAN and unconditional GAN. The former usually uses pose information as condition to make pose transfer using GAN. The generated person have the same identity as the source person. The latter generates person images from scratch, and the real person images are only used as references for the discriminator. While conditional GAN is widely studied, unconditional GAN is also worth exploring because it can synthesize person image with new identity, which is a useful manner of data augmentation. These two types of generating methods have their different advantages and disadvantages, and sometimes they are complementary. This paper proposes a CoGAN to cooperatively train two types of GANs in an end‐to‐end framework. The two GANs serve different purposes, and can learn from each other during the cooperative learning procedure. The experimental results on public datasets show that the proposed CoGAN improves the performance of both baseline methods, and achieves competitive results compared with state‐of‐the‐art methods.

Vehicle Re-Identification (ReID) aims to retrieve images of vehicles with the same identity from different scenarios. It is a challenging task due to the large intra-identity discrepancy caused by viewpoint variations and the subtle inter-identity difference produced by similar appearances. In this paper, we propose a Viewpoint-Aware Loss (VAL) function to deal with these challenges. Specifically, we propose partition and reunion operations in VAL, which significantly shrinks the intra-identity distance and acquires viewpoint-invariant representations. In addition, we embed a multi-decision boundary mechanism in VAL. It contributes to enlarging the inter-identity distance. A comprehensive evaluation on two benchmarks shows the superiority of our method in contrast to a series of existing state-of-the-arts.

Multi-object tracking (MOT) has become a hot task in multi-media analysis. It not only locates the objects but also maintains their unique identities. However, previous methods encounter tracking failures in complex scenes, since they lose most of the unique attributes of each target. In this paper, we formulate the MOT problem as Tracking Game and propose a Self-adaptative Agent Tracker (SAT) framework to solve this problem. The roles in Tracking Game are divided into two classes including the agent player and the game organizer. The organizer controls the game and optimizes the agents' actions from a global perspective. The agent encodes the attributes of targets and selects action dynamically. For these purposes, we design the State Transition Net to update the agent state and the Action Decision Net to implement the flexible tracking strategy for each agent. Finally, we present the organizer-agent coordination tracking algorithm to leverage both global and individual information. The experiments show that the proposed SAT achieves the state-of-the-art performance on both MOT17 and MOT20 benchmarks.

Vehicle re-identification aims to associate images or videos of the same vehicle collected from different cameras. Many existing methods address the vehicle re-identification problem by explicitly learning distinguishable global features. However, vehicle attributes, i.e., logo category and orientation, play an indispensable role in identifying vehicles. In this paper, we first propose deep models to recognize vehicle attributes. Then, based on these attributes, we adopt a High Confidence Attribute Network (HCANet) to extract weighted global features. A comprehensive evaluation on the VehicleID dataset shows that our approach achieves competitive results.

Vehicle reidentification, aiming at identifying vehicles across images, has drawn a lot of attention and has made significant achievements in recent years. However, vehicle reidentification remains a challenging task caused by severe appearance changes due to different orientations. In practice, the result of reidentification is greatly influenced by the pose of vehicles, and we call this influence as a pose barrier problem. One way to address the pose barrier problem is to train a feature representation that is invariant for various vehicle poses. To this end, we present pose robust features (PRFs) that contains two components: 1) pose-invariant features (PIFs) and 2) pose discriminative features (PDFs). On the one hand, PIF is the expert in exploring the overall characteristic of vehicles. When training PIF, we adopt an identity classifier as well as an orientation classifier. In addition, an adversarial loss is deployed in the PIF network. On the other hand, we design a PDF network, which has a similar architecture to the PIF network but can distinguish the difference between local details. The difference between PDF and PIF is that the network of training PDF does not apply the adversarial loss. Finally, by combining PIF and PDF, PRF has the advantages of the two features and can alleviate the influence of the pose barrier problem. Experiments are conducted on the VeRi-776 and VehicleID data sets. We show that PIF and PDF are complementary and that PRF produces competitive performance compared with state-of-the-art approaches.

Deep salient object detection (DSOD), which leverages the popular deep learning techniques, is a promising new branch of salient object detection (SOD). By training on large-scale public datasets, DSOD methods showed significant performance improvement while avoiding the involvement of manually designed visual features and prior knowledge of specific datasets. This paper proposes a novel superpixel-based DSOD method based on fuzzy superpixel extraction (FSE), a neural network-based differentiable superpixel extraction method, and controlled filter convolution (CFC), a modified convolution operation that accepts two input feature maps and can balance their influences without hand-picked coefficients. Different from other superpixel-based methods, by using FSE, the proposed method is able to include superpixel extraction in the training process, which optimizes the superpixel representations according to the datasets. Then, the CFC layers combine two different parts of the information possessed by the superpixels, which are intrasuperpixel features and intersuperpixel features, to generate a unified feature map. In the experiments conducted on 5 widely used public datasets, the proposed method significantly outperformed state-of-the-art models, which proved its effectiveness and generalization ability.

Text matching is a core issue for question answering (QA), information retrieval (IR) and many other fields. We propose to reformulate the original text, i.e., generating a new text that is semantically equivalent to original text, to improve text matching degree. Intuitively, the generated text improves mutual information between two text sequences. We employ the generative adversarial network as the reformulation model where there is a discriminator to guide the text generating process. In this work, we focus on matching question and answers. The task is to rank answers based on QA matching degree. We first reformulate the original question without changing the asker's intent, then compute a relevance score for each answer. To evaluate the method, we collected questions and answers from Zhihu. In addition, we also conduct substantial experiments on public data such as SemEval and WikiQA to compare our method with existing methods. Experimental results demonstrate that after adding the reformulated question, the ranking performance across different matching models can be improved consistently, indicating that the reformulated question has enhanced mutual information and effectively bridged the semantic gap between QA.

Training deep neural networks is difficult for the pathological curvature problem. Re-parameterization is an effective way to relieve the problem by learning the curvature approximately or constraining the solutions of weights with good properties for optimization. This paper proposes to reparameterize the input weight of each neuron in deep neural networks by normalizing it with zero-mean and unit-norm, followed by a learnable scalar parameter to adjust the norm of the weight. This technique effectively stabilizes the distribution implicitly. Besides, it improves the conditioning of the optimization problem and thus accelerates the training of deep neural networks. It can be wrapped as a linear module in practice and plugged in any architecture to replace the standard linear module. We highlight the benefits of our method on both multi-layer perceptrons and convolutional neural networks, and demonstrate its scalability and efficiency on SVHN, CIFAR-10, CIFAR-100 and ImageNet datasets.

Region-based Image Retrieval (RBIR), which bases itself on image segmentation rather than global features or key-point-based local features, is a branch of Content-based Image Retrieval. This paper proposes a novel RBIR-oriented image segmentation algorithm named Edge Integrated Minimum Spanning Tree (EI-MST). The difference between EI-MST and the traditional MST-based methods is that EI-MST generates MSTs over edge-maps rather than the original images, which achieved high retrieval performance cooperating with state-of-the-art matching strategies. In addition, by limiting the nodes in every MST with adaptive scale selection, EI-MST is efficient especially when processing high resolution images. The experiments on four popular public datasets proved that, EI-MST is capable of achieving higher retrieval accuracy over four widely used segmentation methods while only consuming moderate amount of time in both online and offline parts of RBIR systems.

In this paper, we propose a novel unsupervised color spatial pyramid matching (UCSPM) approach for person re-identification. It is well motivated by our study on spatial pyramid to build effective structural object representation for person re-identification. Through the combination of illumination invariance color feature, UCSPM can well cope with the variations of viewpoint, illumination and pose. First, local superpixel regions are divided to accurately represent the color feature. Second, human body are divided into increasing fine vertical sub-regions to construct the spatial pyramid matching scheme. Third, the color feature and its spatial distribution information are used in a pyramid match kernel for calculating the similarity between person and person. The effectiveness of our approach is validated on the VIPeR dataset and CUHK campus dataset. Comparing with other approaches, our UCSPM improves the best unsupervised rank-1 matching rate on the VIPeR dataset by 3.08% with only one kind of feature—color.

Semi-supervised learning and active learning are important techniques to build more accurate model while labeled data are scarce. The objective of this paper is combining both to effectively relieve user labor for multi-class annotation. We propose a novel graph-based active semi-supervised learning framework which aim at efficiently learning a multi-class model with minimal human labor. In particular, we propose Minimize Expected Global Uncertainty algorithm to actively select examples (for labels), which naturally integrates with the probabilistic results of graph-based semi-supervised learning. Meanwhile, we update the model incrementally by decomposed formulation while the new example are incorporated for training, which only has the time complexity of O(n), compared to the original re-training of O(n3). Extensive evaluations over three real-world datasets demonstrate that our proposed method has the superior performance comparing with the baselines and the capability to efficiently build more accurate model with fractional human labor.