The substantial successes achieved by diffusion probabilistic models have prompted the study of their employment in resource-limited scenarios. Pruning methods have been proven effective in compressing discriminative models relying on the correlation between training losses and model performances. However, diffusion models employ an iterative process for generating high-quality images, leading to a breakdown of such connections. To address this challenge, we propose a simple yet effective method, named NiCI-Pruning (Noise in Clean Image Pruning), for the compression of diffusion models. NiCI-Pruning capitalizes the noise predicted by the model based on clean image inputs, favoring it as a feature for establishing reconstruction losses. Accordingly, Taylor expansion is employed for the proposed reconstruction loss to evaluate the parameter importance effectively. Moreover, we propose an interval sampling strategy that incorporates a timestep-weighted schema, alleviating the risk of misleading information obtained at later timesteps. We provide comprehensive experimental results to affirm the superiority of our proposed approach. Notably, our method achieves a remarkable average reduction of 30.4% in FID score increase across five different datasets compared to the state-of-the-art diffusion pruning method at equivalent pruning rates. Our code and models have been made available at https://github.com/NUST-Machine-Intelligence-Laboratory/NiCI-Pruning.

Token compression is essential for reducing the computational and memory requirements of transformer models, enabling their deployment in resource-constrained environments. In this work, we propose an efficient and hardware-compatible token compression method called Prune and Merge. Our approach integrates token pruning and merging operations within transformer models to achieve layer-wise token compression. By introducing trainable merge and reconstruct matrices and utilizing shortcut connections, we efficiently merge tokens while preserving important information and enabling the restoration of pruned tokens. Additionally, we introduce a novel gradient-weighted attention scoring mechanism that computes token importance scores during the training phase, eliminating the need for separate computations during inference and enhancing compression efficiency. We also leverage gradient information to capture the global impact of tokens and automatically identify optimal compression structures. Extensive experiments on the ImageNet-1 k and ADE20 K datasets validate the effectiveness of our approach, achieving significant speed-ups with minimal accuracy degradation compared to state-of-the-art methods. For instance, on DeiT-Small, we achieve a 1.64× speed-up with only a 0.2% drop in accuracy on ImageNet-1k. Moreover, by compressing segmenter models and comparing with existing methods, we demonstrate the superior performance of our approach in terms of efficiency and effectiveness.

For autonomous vehicles, free-space detection is an essential part of visual perception. With the development of multi-modal convolutional neural networks (CNNs) in recent years, the performance of driving scene semantic segmentation algorithms has been dramatically improved. Therefore most free-space detection algorithms are developed based on multiple sensors. However, multi-modal CNNs have high data throughput and contain a large number of computationally intensive convolution calculations, limiting their feasibility for real-time applications. Field Programmable Gate Arrays (FPGAs) provide a unique combination of flexibility, performance, and low power for these problems to accommodate multi-modal data and the computational acceleration of different compression algorithms. Network lightweight methods offer great assurance for facilitating the deployment of CNNs on such resource-constrained devices. In this paper, we propose a network lightweight method for a multi-modal free-space detection algorithm. We first propose an FPGA-friendly multi-modal free-space detection lightweight network. It comprises operators that FPGA prefers and achieves a 95.54% MaxF score on the test set of KITTI-Road free-space detection tasks and 81 ms runtime when running on 700 W GPU devices. Then we present a pruning approach for this network to reduce the number of parameters in case the complete model exceeds the FPGA chip memory. The pruning is in two parts. For the feature extractors, we propose a data-dependent filter pruner according to the principle that the low-rank feature map contains less information. To not compromise the integrity of the multi-modal information, the pruner is independent for each modality. For the segmentation decoder, we apply a channel pruning approach to remove redundant parameters. Finally, we implement our designs on an FPGA board using 8-bit quantisation, and the accelerator achieves outstanding performance. A real-time application of scene segmentation on KITTI-Road is used to evaluate our algorithm, and the model achieves a 94.39% MaxF score and minimum 14 ms runtime on 20W FPGA devices.

Due to its significant capability of modeling long-range dependencies, vision transformer (ViT) has achieved promising success in both holistic and occluded person re-identification (Re-ID) tasks. However, the inherent problems of transformers such as the huge computational cost and memory footprint are still two unsolved issues that will block the deployment of ViT based person Re-ID models on resource-limited edge devices. Our goal is to reduce both the inference complexity and model size without sacrificing the comparable accuracy on person Re-ID, especially for tasks with occlusion. To this end, we propose a novel attention map guided (AMG) transformer pruning method, which removes both redundant tokens and heads with the guidance of the attention map in a hardware-friendly way. We first calculate the entropy in the key dimension and sum it up for the whole map, and the corresponding head parameters of maps with high entropy will be removed for model size reduction. Then we combine the similarity and first-order gradients of key tokens along the query dimension for token importance estimation and remove redundant key and value tokens to further reduce the inference complexity. Comprehensive experiments on Occluded DukeMTMC and Market-1501 demonstrate the effectiveness of our proposals. For example, our proposed pruning strategy on ViT-Base enjoys29.4%FLOPssavings with0.2%drop on Rank-1 and0.4%improvement on mAP, respectively. Code and models have been made available athttps://github.com/NUST-Machine-Intelligence-Laboratory/AMG.