In crowded scenes, it is difficult to extract discriminating human features due to occlusion. Some human detectors have improved this issue by introducing head detection. However, a complex problem still exists in associating full-body detection with its corresponding head detection. Instead of learning the association, we propose a Head-dominant Enhancement Module (HDEM) that uses the full-body proposal to regress the head bounding box. To embed the head information into the target human feature, we further propose a Consistent Weighted (CW) loss. Additionally, existing Non-Maximum Suppression (NMS) algorithms do not consider density changes with the selection of detection boxes, which leads to false and missed detection. Similar to human visual habits in occluded scenarios, we propose a Count-aware Dynamic Threshold Module (CADTM) that utilizes head context information to predict local count, which is associated with crowd density. CADTM can solve the inherent defects of Greedy-NMS in crowded scenes by adjusting the Intersection over Union (IoU) threshold dynamically. Ultimately, through the combination of HDEM and CADTM, we achieve state-of-the-art performance on CrowdHuman with a small computational cost. Our method achieves 4.6% AP gains, 2.2% MR-2 gains, and 3.2% JI gains over a Cascade R-CNN baseline. Furthermore, the proposed method is flexible and can be used with most proposal-based detection frameworks and various IoU-based NMS. Note to Practitioners—The motivation for this study arises from a prevalent issue encountered in human detection applications, particularly in densely populated areas such as shopping malls, streets, and subway stations, where occlusion poses a significant challenge. Employing a generic object detector results in numerous missed detections, thereby significantly compromising the overall performance of the detector. This research proposes a convenient plugin that achieves significant performance improvements at a very small cost. To validate its efficacy, our proposed method is thoroughly evaluated on proposal-based detection frameworks. The experimental results demonstrate the robustness of our approach and its ability to adapt to diverse crowded scenarios.

Accurate data association plays a crucial role in Multi-Object Tracking (MOT) as it helps reduce confusion such as identity switches and assignment errors. However, many existing advanced methods often overlook the diversity among trajectories and the ambiguity and conflicts present in various types of cues. Consequently, when performing simple global data association, confusion arises between detections, trajectories, and associations. To address this problem, we propose a simple, versatile, and highly interpretable Deconfused Data Association Framework (DDAF). DDAF decomposes the traditional association problem into multiple sub-problems using a series of non-learnable modules, and selectively resolves confusion in each sub-problem by strategically utilizing new cues. Building upon DDAF, we design a powerful multi-object tracker named DfTrack, which specifically targets confusion in MOT. Furthermore, we discuss different specific implementations of DDAF to tackle challenging environments characterized by low frame rate, camera motion, and cross-domain scenarios. Correspondingly, we also develop several variants of DfTrack, demonstrating the remarkable scalability and adaptability of DDAF. Extensive experiments conducted on the MOT17, MOT20, and DanceTrack datasets demonstrate that DDAF significantly outperforms simple global association methods, and its variants can adapt to various challenging environments. Furthermore, DfTrack achieves state-of-the-art performance on multiple datasets, with HOTA of 65.2%, 63.9%, and 64.4% on MOT17, MOT20, and DanceTrack, respectively. The DfTrack-Hybrid variant further improves the performance on this basis. These results validate that our DDAF can effectively decompose and resolve various confusion in global association without any learning cost.

Accurate data association is crucial in reducing confusion, such as ID switches and assignment errors, in multi-object tracking (MOT). However, existing advanced methods often overlook the diversity among trajectories and the am-biguity and conflicts present in motion and appearance cues, leading to confusion among detections, trajectories, and associations when performing simple global data association. To address this issue, we propose a simple, versatile, and highly interpretable data association approach called De-composed Data Association (DDA). DDA decomposes the traditional association problem into multiple sub-problems using a series of non-learning-based modules and selectively addresses the confusion in each sub-problem by incorporating targeted exploitation of new cues. Additionally, we introduce Occlusion-aware Non-Maximum Suppression (ONMS) to retain more occluded detections, thereby increasing op-portunities for association with trajectories and indirectly reducing the confusion caused by missed detections. Finally, based on DDA and ONMS, we design a powerful multi-object tracker named DeconfuseTrack, specifically focused on resolving confusion in MOT. Extensive experiments conducted on the MOT17 and MOT20 datasets demonstrate that our proposed DDA and ONMS significantly enhance the performance of several popular trackers. Moreover, De-confuseTrack achieves state-of-the-art performance on the MOT17 and MOT20 test sets, significantly outperforms the baseline tracker ByteTrack in metrics such as HOTA, IDF1, AssA. This validates that our tracking design effectively reduces confusion caused by simple global association.

Although existing multi-object tracking (MOT) algorithms have obtained competitive performance on various benchmarks, almost all of them train and validate models on the same domain. The domain generalization problem of MOT is hardly studied. To bridge this gap, we first draw the observation that the high-level information contained in natural language is domain invariant to different tracking domains. Based on this observation, we propose to introduce natural language representation into visual MOT models for boosting the domain generalization ability. However, it is infeasible to label every tracking target with a textual description. To tackle this problem, we design two modules, namely visual context prompting (VCP) and visual-language mixing (VLM). Specifically, VCP generates visual prompts based on the input frames. VLM joints the information in the generated visual prompts and the textual prompts from a pre-defined Trackbook to obtain instance-level pseudo textual description, which is domain invariant to different tracking scenes. Through training models on MOT17 and validating them on MOT20, we observe that the pseudo textual descriptions generated by our proposed modules improve the generalization performance of query-based trackers by large margins.

Discriminative representation is essential to keep a unique identifier for each target in Multiple object tracking (MOT). Some recent MOT methods extract features of the bounding box region or the center point as identity embeddings. However, when targets are occluded, these coarse-grained global representations become unreliable. To this end, we propose exploring diverse fine-grained representation, which describes appearance comprehensively from global and local perspectives. This fine-grained represen-tation requires high feature resolution and precise semantic information. To effectively alleviate the semantic misalignment caused by indiscriminate contextual information aggregation, Flow Alignment FPN (FAFPN) is proposed for multi-scale feature alignment aggregation. It generates semantic flow among feature maps from different resolutions to transform their pixel positions. Furthermore, we present a Multi-head Part Mask Generator (MPMG) to extract fine-grained representation based on the aligned feature maps. Multiple parallel branches of MPMG allow it to focus on different parts of targets to generate local masks without label supervision. The diverse details in target masks facilitate fine-grained representation. Eventually, benefiting from a Shuffle-Group Sampling (SGS) training strategy with positive and negative samples balanced, we achieve state-of-the-art performance on MOT17 and MOT20 test sets. Even on DanceTrack, where the appearance of targets is extremely similar, our method significantly outperforms Byte-Track by 5.0% on HOTA and 5.6% on IDF1. Extensive experiments have proved that diverse fine- grained representation makes Re-ID great again in MOT.

Existing online multiple object tracking (MOT) algorithms often consist of two subtasks, detection and re-identification (ReID). In order to enhance the inference speed and reduce the complexity, current methods commonly integrate these double subtasks into a unified framework. Nevertheless, detection and ReID demand diverse features. This issue results in an optimization contradiction during the training procedure. With the target of alleviating this contradiction, we devise a module named Global Context Disentangling (GCD) that decouples the learned representation into detection-specific and ReID-specific embeddings. As such, this module provides an implicit manner to balance the different requirements of these two subtasks. Moreover, we observe that preceding MOT methods typically leverage local information to associate the detected targets and neglect to consider the global semantic relation. To resolve this limitation, we develop a module, referred to as Guided Transformer Encoder (GTE), by combining the powerful reasoning ability of Transformer encoder and deformable attention. Unlike previous works, GTE avoids analyzing all the pixels and only attends to capture the relation between query nodes and a few self-adaptively selected key samples. Therefore, it is computationally efficient. Extensive experiments have been conducted on the MOT16, MOT17 and MOT20 benchmarks to demonstrate the superiority of the proposed MOT framework, namely RelationTrack. The experimental results indicate that RelationTrack has surpassed preceding methods significantly and established a new state-of-the-art performance, e.g., IDF1 of 70.5% and MOTA of 67.2% on MOT20.

Discriminative representation is crucial for the association step in multi-object tracking. Recent work mainly utilizes features in single or neighboring frames for constructing metric loss and empowering networks to extract representation of targets. Although this strategy is effective, it fails to fully exploit the information contained in a whole trajectory. To this end, we propose a strategy, namely multi-view trajectory contrastive learning, in which each trajectory is represented as a center vector. By maintaining all the vectors in a dynamically updated memory bank, a trajectory-level contrastive loss is devised to explore the inter-frame information in the whole trajectories. Besides, in this strategy, each target is represented as multiple adaptively selected keypoints rather than a pre-defined anchor or center. This design allows the network to generate richer representation from multiple views of the same target, which can better characterize occluded objects. Additionally, in the inference stage, a similarity-guided feature fusion strategy is developed for further boosting the quality of the trajectory representation. Extensive experiments have been conducted on MOTChallenge to verify the effectiveness of the proposed techniques. The experimental results indicate that our method has surpassed preceding trackers and established new state-of-the-art performance.

Intelligent Internet of Things (IoT) systems based on deep neural networks (DNNs) have been widely deployed in the real world. However, DNNs are found to be vulnerable to adversarial examples, which raises people’s concerns about intelligent IoT systems’ reliability and security. Testing and evaluating the robustness of IoT systems become necessary and essential. Recently, various attacks and strategies have been proposed, but the efficiency problem remains unsolved properly. Existing methods are either computationally extensive or time consuming, which is not applicable in practice. In this article, we propose a novel framework, called attack-inspired generative adversarial networks (AI-GAN) to generate adversarial examples conditionally. Once trained, it can generate adversarial perturbations efficiently given input images and target classes. We apply AI-GAN on different data sets in white-box settings, black-box settings, and targeted models protected by state-of-the-art defenses. Through extensive experiments, AI-GAN achieves high attack success rates, outperforming existing methods, and reduces generation time significantly. Moreover, for the first time, AI-GAN successfully scales to complex data sets, e.g., CIFAR-100 and ImageNet, with about 90% success rates among all classes.

Deep neural networks (DNNs) are vulnerable to adversarial examples, which are crafted by adding imperceptible perturbations to inputs. Recently different attacks and strategies have been proposed, but how to generate adversarial examples perceptually realistic and more efficiently remains unsolved. This paper proposes a novel framework called Attack-Inspired GAN (AI-GAN), where a generator, a discriminator, and an attacker are trained jointly. Once trained, it can generate adversarial perturbations efficiently given input images and target classes. Through extensive experiments on several popular datasets e.g., MNIST and CFAR-10, AI-GAN achieves high attack success rates and reduces generation time significantly in various settings. Moreover, for the first time, AI-GAN successfully scales to complicated datasets e.g., CFAR-100 with around 90% success rates among all classes.

In this paper, we propose an interactive color natural image segmentation method. The method integrates color feature with multiscale nonlinear structure tensor texture (MSNST) feature and then uses GrabCut method to obtain the segmentations. The MSNST feature is used to describe the texture feature of an image and integrated into GrabCut framework to overcome the problem of the scale difference of textured images. In addition, we extend the Gaussian Mixture Model (GMM) to MSNST feature and GMM based on MSNST is constructed to describe the energy function so that the texture feature can be suitably integrated into GrabCut framework and fused with the color feature to achieve the more superior image segmentation performance than the original GrabCut method. For easier implementation and more efficient computation, the symmetric KL divergence is chosen to produce the estimates of the tensor statistics instead of the Riemannian structure of the space of tensor. The Conjugate norm was employed using Locality Preserving Projections (LPP) technique as the distance measure in the color space for more discriminating power. An adaptive fusing strategy is presented to effectively adjust the mixing factor so that the color and MSNST texture features are efficiently integrated to achieve more robust segmentation performance. Last, an iteration convergence criterion is proposed to reduce the time of the iteration of GrabCut algorithm dramatically with satisfied segmentation accuracy. Experiments using synthesis texture images and real natural scene images demonstrate the superior performance of our proposed method.