Large language models (LLMs) have achieved outstanding performance in natural language processing, but enormous model sizes and high computational costs limit their practical deployment. Structured pruning can effectively reduce the resource demands for deployment by removing redundant model parameters. However, the randomly selected calibration data and fixed single importance estimation metrics in existing structured pruning methods lead to degraded performance of pruned models. This study introduces AdaPruner, a sample-aware adaptive structured pruning framework for LLMs, aiming to optimize the calibration data and importance estimation metrics in the structured pruning process. Specifically, AdaPruner effectively removes redundant parameters from LLMs by constructing a structured pruning solution space and then employing Bayesian optimization to adaptively search for the optimal calibration data and importance estimation metrics. Experimental results show that the AdaPruner outperforms existing structured pruning methods on a family of LLMs with varying pruning ratios, demonstrating its applicability and robustness. Remarkably, at a 20% pruning ratio, the model pruned with AdaPruner maintains 97% of the performance of the unpruned model.

Chain-of-thought (CoT) reasoning research has predominantly focused on language modality, neglecting the intricate interaction of multiple modalities crucial for real-world reasoning, such as visual question answering. Current methods primarily concentrate on modal conversion and feature fusion to enable language models to utilize CoT in a multimodal environment. However, these methods have inherent hallucination limitations. They tend to excessively rely on the prior knowledge of language unimodal and lack guidance in learning visual differences, often resulting in content incongruent with the images. This study introduces a novel approach, visual self-contrastive distillation (VSCD). The proposed method equalizes the status of language and vision modalities by freezing both encoders, enabling more balanced learning. Furthermore, we use contrastive decoding to enable the model to learn image differences during distillation, enhancing its understanding of visual nuances. The comprehensive experiment on the ScienceQA dataset demonstrates the superiority of the proposed VSCD method across various categories of multimodal CoT. Code and data are released at https://github.com/zgMin/VSCD.

Conditional computation algorithms, e.g., the early exiting (EE) strategy, can accelerate the inference of pretrained language models (PLMs) by exiting shallow layers without calculating the entire model. In addition to the adaptive inference of EE prediction for downstream tasks, self-distillation (SD) can encourage EE classifiers to mimic the behavior of the final classifier to enhance their representation capacity. However, the gradients from different tasks of EE classifiers will conflict and cancel one another out. The parameters of the backbone and some EE classifiers will be implicitly prevented from updating. Moreover, if the semantic gap between the final classifier and EE classifiers is significant, the EE classifier's performance will decrease. That is, the final classifier would not be the best choice to enhance the performance of the EE classifier. This study proposed an early exiting strategy with adaptive ensemble self-distillation and consistent gradients for the inference acceleration of PLMs. To mitigate gradient conflicts, we orthogonally projected the distillation loss's backpropagated gradient onto the classification loss's normal plane. Instead of directly using the final classifier as a single-teacher for self-distillation, we dynamically assemble different adaptive teachers for different EE classifiers according to the learning abilities of the EE classifiers. The accumulative decision was drawn for adaptive inference to make accurate and reliable predictions. Experimental results show that the proposed model outperforms existing models with the same speed-up ratio and effectively balances model performance and inference time.

The computational cost of the vision and language pretrained models (VL-PTMs) limits their deployment in resource-constrained devices that require low latency. One existing solution is to apply the early exiting (EE) strategy to accelerate the inference. This technique can force model prediction using only a few former transformer layers. However, these former layers behave differently with the final classifier, inevitably resulting in performance decline. To counter such limitation, self-distillation has been commonly introduced to enhance the representation abilities of the EE classifiers. This results in a semantic gap since EE classifiers are directly trained to mimic the outputs of the final classifier without access to the modality-specific behaviors. This study proposes a multimodality self-distillation method for the fast inference of VL-PTMs. To fill the semantic gap between modalities, we split the multimodalities into separate modalities and added them as extra inputs to encourage the effective distillation of each modality. Furthermore, the mean squared error (MSE) is introduced to minimize the distance of feature maps and further enhance the representation ability of the EE classifiers. Experiments show that the proposed method outperforms the previous EE strategies with the same inference time, and performs competitively even if the model exited very early.

Conditional computation algorithms, such as the early exiting (EE) algorithm, can be applied to accelerate the inference of pretrained language models (PLMs) while maintaining competitive performance on resource-constrained devices. However, this approach is only applied to the vertical architecture to decide which layers should be used for inference. Conversely, the operation of the horizontal perspective is ignored, and the determination of which tokens in each layer should participate in the computation fails, leading to a high redundancy for adaptive inference. To address this limitation, a unified horizontal and vertical multi-perspective early exiting (MPEE) framework is proposed in this study to accelerate the inference of transformer-based models. Specifically, the vertical architecture uses recycling EE classifier memory and weighted self-distillation to enhance the performance of the EE classifiers. Then, the horizontal perspective uses recycling class attention memory to emphasize the informative tokens. Conversely, the tokens with less information are truncated by weighted fusion and isolated from the following computation. Based on this, both horizontal and vertical EE are unified to obtain a better tradeoff between performance and efficiency. Extensive experimental results show that MPEE can achieve higher acceleration inference with competent performance than existing competitive methods.