Abstract Traditional knowledge distillation relies on high‐capacity teacher models to supervise the training of compact student networks. To avoid the computational resource costs associated with pretraining high‐capacity teacher models, teacher‐free online knowledge distillation methods have achieved satisfactory performance. Among these methods, feature fusion methods have effectively alleviated the limitations of training without the strong guidance of a powerful teacher model. However, existing feature fusion methods often focus primarily on end‐layer features, overlooking the efficient utilization of holistic knowledge loops and high‐level information within the network. In this article, we propose a new feature fusion‐based mutual learning method called Diversify Feature Enhancement and Fusion for Online Knowledge Distillation (DFEF). First, we enhance advanced semantic information by mapping multiple end‐of‐network features to obtain richer feature representations. Next, we design a self‐distillation module to strengthen knowledge interactions between the deep and shallow network layers. Additionally, we employ attention mechanisms to provide deeper and more diversified enhancements to the input feature maps of the self‐distillation module, allowing the entire network architecture to acquire a broader range of knowledge. Finally, we employ feature fusion to merge the enhanced features and generate a high‐performance virtual teacher to guide the training of the student model. Extensive evaluations on the CIFAR‐10, CIFAR‐100, and CINIC‐10 datasets demonstrate that our proposed method can significantly enhance performance compared to state‐of‐the‐art feature fusion‐based online knowledge distillation methods. Our code can be found at https://github.com/JSJ515-Group/DFEF-Liu .

The manifold-based learning methods have recently drawn more and more attention in dimension reduction. In this paper, a novel manifold-based learning method named enhanced parameter-free diversity discriminant preserving projections (EPFDDPP) is presented, which effectively avoids the neighborhood parameter selection and characterizes the manifold structure well. EPFDDPP redefines the weighted matrices, the discriminating similarity matrix and the discriminating diversity matrix, respectively. The weighted matrices are computed by the cosine angle distance between two data points and take special consideration of both the local information and the class label information, which are parameterless and favorable for face recognition. After characterizing the discriminating similarity scatter matrix and the discriminating diversity scatter matrix, the novel feature extraction criterion is derived based on maximum margin criterion. Experimental results on the Wine data set, Olivetti Research Laboratory (ORL); AR (face database created by Aleix Martinez and Robert Benavente); and Pose, Illumination, and Expression (PIE) face databases show the effectiveness of the proposed method.