Conventionally, during the knowledge distillation process (e.g., feature distillation), an additional projector is often required to perform feature transformation due to the dimension mismatch between the teacher and the student networks. Interestingly, we discovered that even if the student and the teacher have the same feature dimensions, adding a projector still helps to improve the distillation performance. In addition, projectors even improve logit distillation if we add them to the architecture too. Inspired by these surprising findings and the general lack of understanding of the projectors in the knowledge distillation process from existing literature, this paper investigates the implicit role that projectors play, but so far been overlooked. Our empirical study shows that the student with a projector 1) obtains a better trade-off between the training accuracy and the testing accuracy compared to the student without a projector when it has the same feature dimensions as the teacher, 2) better preserves its similarity to the teacher beyond shallow and numeric resemblance, from the view of Centered Kernel Alignment (CKA) (Kornblith et al., 2019), and 3) avoids being over-confident (Guo et al., 2017) as the teacher does at the testing phase. Motivated by the positive effects of projectors, we propose a projector ensemble-based feature distillation method to further improve distillation performance. Despite the simplicity of the proposed strategy, empirical results from the evaluation of classification tasks on benchmark datasets demonstrate the superior classification performance of our method on a broad range of teacher-student pairs and verify, from the aspects of CKA and model calibration that the student's features are of improved quality with the projector ensemble design.

Coreset selection methods aim to identify a representative subset of training data that preserves competitive performance. However, mainstream coreset selection approaches are model-specific and assume they already have full information about the target model when the coreset is selected. This largely restricts the usefulness of coreset selection in practice. This work aims to fill that gap by formulating and investigating the problem of cross-architecture generalizability of coresets: we develop a unified theoretical framework that analyzes the upper bound of coreset selection objective functions, extend it to scenarios involving multiple downstream architectures, and provide an empirical analysis on cross-architecture coreset performance. Based on our findings, we propose a novel ensemble scoring method that aggregates multi-source knowledge to enhance cross-architecture generalizability. Our extensive experiments across thirteen architectures and six selection ratios provide comprehensive verification of our theoretical analysis. The source code is available at https://github.com/diqichen91/CACS.git .

Coreset selection for classification often relies on assessing individual sample difficulty or importance, leading to sample-wise or range-based selection, but this can overlook the collective impact on model decision boundaries. Realizing that the representative power a coreset possesses is tightly associated with the decision boundaries a model can form on it, we propose a novel approach that directly optimizes the Decision Boundary (DB) formed by the selected coreset. Specifically, we ask: How can we collectively select samples to create a DB that is globally smoothed yet locally detailed, ensuring maximum generalizability and noise-resilience to the original dataset? To address this, we define two key objectives: (1) Global shape retention – The selected coreset should form a smoothed version of the original DB, preserving its overall structure and preventing overfitting; (2) Local detail preservation – While smoothing prevents overfitting, excessive smoothing risks losing critical nuances. Thus, the selection must also retain key points near the original DB to capture local complexities. We formulate these objectives as a convex quadratic optimization problem with linear constraints and solve it efficiently. Extensive evaluations demonstrate the consistent and substantial advantages of our method over the state-of-the-art coreset selection strategies. The source code is available at https://github.com/diqichen91/DBCore.git .

This paper tackles a new problem of dataset pruning for Knowledge Distillation (KD), from a fresh perspective of Decision Boundary (DB) preservation and drifts. Existing dataset pruning methods generally assume that the post-pruning DB formed by the selected samples can be well-captured by future networks that use those samples for training. Therefore, they tend to preserve hard samples since hard samples are closer to the DB and better characterize the nuances in the distribution of the entire dataset. However, in KD, the limited learning capacity from the student network leads to imperfect preservation of the teacher's feature distribution, resulting in the drift of DB in the student space. Specifically, hard samples worsen such drifts as they are difficult for the student to learn, creating a situation where the student's DB can drift deeper into other classes and make incorrect classifications. Motivated by these findings, our method selects medium-difficulty samples for KD-based dataset pruning. We show that these samples constitute a smoothed version of the teacher's DB and are easier for the student to learn, obtaining a general feature distribution preservation for a class of samples and reasonable DB between different classes for the student. In addition, to reduce the distributional shift due to dataset pruning, we leverage the class-wise distributional information of the teacher's outputs to reshape the logits of the preserved samples. Experiments show that the proposed static pruning method can even perform better than the state-of-the-art dynamic pruning method which needs access to the entire dataset. In addition, our method halves the training times of KD and improves the student's accuracy by 0.4% on ImageNet with a 50% keep ratio. When the ratio further increases to 70%, our method achieves higher accuracy over the vanilla KD while reducing the training times by 30%. Code is available at https://github.com/chenyd7/MDSLR.

Monitoring of physiology and behavior of marine animals living undisturbed in their natural habitats can provide valuable information about their well-being and response to environmental stressors. We focus on detecting the feeding behavior in predatory fish using implantable biologgers that record and analyze electrocardiogram (ECG) signals. We propose a novel processing pipeline for resource-constrained embedded systems that can infer higher-level information, such as heart-rate and feeding events, from the ECG signals in situ. Our main contributions are in proposing efficient event detection algorithms that can reliably detect fish feeding events from noisy heart-rate data based on the unique statistical properties of feeding-induced changes in the heart-rate. We evaluate our approaches using an in-house biologger that we surgically implant in twelve coral trout fish and use to collect data during an experiment for a period of ten weeks and show that our signal processing pipeline performs well with noisy ECG signals overall. Specifically, our heart-rate estimation algorithm achieves errors of less than one beat per minute even in scenarios where popular algorithms used by domain specialists perform poorly. Furthermore, our feeding detection algorithms offer improved accuracy compared with the state-of-the-art algorithms while requiring significantly reduced computational and energy resources. We implement the proposed heart-rate estimation and feeding detection algorithms on the biologger and evaluate the associated system overhead. The results show that our proposed heart-rate estimation and feeding detection algorithms can run in-situ on the biologger as they demand rather small computational and energy resources that can conveniently be provisioned. This work is an important first step towards developing effective tools for long-term monitoring of high-level parameters pertaining to the health and behavior of marine animals in the wild.

In knowledge distillation, previous feature distillation methods mainly focus on the design of loss functions and the selection of the distilled layers, while the effect of the feature projector between the student and the teacher remains under-explored. In this paper, we first discuss a plausible mechanism of the projector with empirical evidence and then propose a new feature distillation method based on a projector ensemble for further performance improvement. We observe that the student network benefits from a projector even if the feature dimensions of the student and the teacher are the same. Training a student backbone without a projector can be considered as a multi-task learning process, namely achieving discriminative feature extraction for classification and feature matching between the student and the teacher for distillation at the same time. We hypothesize and empirically verify that without a projector, the student network tends to overfit the teacher's feature distributions despite having different architecture and weights initialization. This leads to degradation on the quality of the student's deep features that are eventually used in classification. Adding a projector, on the other hand, disentangles the two learning tasks and helps the student network to focus better on the main feature extraction task while still being able to utilize teacher features as a guidance through the projector. Motivated by the positive effect of the projector in feature distillation, we propose an ensemble of projectors to further improve the quality of student features. Experimental results on different datasets with a series of teacher-student pairs illustrate the effectiveness of the proposed method. Code is available at https://github.com/chenyd7/PEFD.

Monitoring of physiology and behavior of marine animals living undisturbed in their natural habitats can provide valuable data on their well-being and response to environmental stressors. We focus on detection of feeding of predatory fish using implantable biologgers that record electrocardiogram (ECG) signals. We propose a novel processing pipeline for resource-constrained embedded systems that can infer higher-level information, such as heart-rate and feeding events, from the ECG signals. Our main contribution is a lightweight change-detection algorithm, that can reliably detect fish feeding in noisy heart-rate data based on unique statistical properties of feeding-induced changes in heart-rate. We evaluate our approach using an in-house biologger that we surgically implant in twelve coral trouts over a period of ten weeks. We show that our signal processing pipeline performs well with noisy ECG signals overall. Specifically, our heart-rate estimation algorithm achieves errors of less than one beat per minute even in scenarios where popular algorithms used by domain scientists perform poorly. Furthermore, our feeding detection algorithm achieves good accuracy and matches the performance of state-of-the-art algorithms while requiring significantly less memory and computational resources. This work is an important first step towards long-term monitoring of high-level condition and health of marine animals in the wild.

Indoor localization using radio signals is challenging. A recently proposed algorithm based on WiFi channel state information is an effective solution. However, it relies on a computationally expensive grid search. We propose a new algorithm based on a modified matrix pencil method that reduces the computational complexity by two orders of magnitude without any loss of accuracy.

Carrier-frequency offset (CFO) estimation for uplink orthogonal frequency-division multiplexing access (OFDMA) systems is very challenging as it requires the estimation of multiple CFOs. In this paper, we propose a new framework referred to as sparse blind CFO estimation for interleaved uplink OFDMA. The proposed framework first discretizes the potential frequency offset ranges into discrete grid points, and formulates the original CFO estimation into a sparse signal recovery problem. Then, a novel two-stage matrix Bayesian compressive sensing-based CFO estimation method is proposed to solve the formulated problem. In the first stage, we employ a relatively large grid interval, and iteratively reconstruct a hyperparameter vector to generate a coarse estimation of multiple CFOs. The second stage reduces the grid interval, and the refined CFOs are estimated one by one through a novel low-complexity one-dimension searching algorithm. Numerical results show that the proposed method significantly outperforms conventional ones in terms of estimation accuracy, especially in the scenarios, such as low signal-to-noise ratios, large CFOs, and a large number of users.

Natural images tend to mostly consist of smooth regions with individual pixels having highly correlated spectra. This information can be exploited to recover hyperspectral images of natural scenes from their incomplete and noisy measurements. To perform the recovery while taking full advantage of the prior knowledge, we formulate a composite cost function containing a square-error data-fitting term and two distinct regularization terms pertaining to spatial and spectral domains. The regularization for the spatial domain is the sum of total variation of the image frames corresponding to all spectral bands. The regularization for the spectral domain is the ℓ1-norm of the coefficient matrix obtained by applying a suitable sparsifying transform to the spectra of the pixels. We use an accelerated proximal-subgradient method to minimize the formulated cost function. We analyze the performance of the proposed algorithm and prove its convergence. Numerical simulations using real hyperspectral images exhibit that the proposed algorithm offers an excellent recovery performance with a number of measurements that is only a small fraction of the hyperspectral image data size. Simulation results also show that the proposed algorithm significantly outperforms an accelerated proximal-gradient algorithm that solves the classical basis-pursuit denoising problem to recover the hyperspectral image.

Interference from unknown devices makes time-of-arrival (ToA) estimation using conventional signal processing methods unreliable. In this paper, we propose new ToA estimation techniques based on Bayesian compressive sensing (BCS) to improve the accuracy of the ToA estimation under the interference scenario. Our proposed BCS based ToA estimation schemes maximize the posterior probability of the channel impulse response (CIR) with given frequency domain received signals. Simulation results show that proposed BCS based ToA estimations exhibit significantly improved ToA detection accuracy and mean-squared error performance in interference scenarios. We also demonstrate a practical example of the ToA estimation using real measured indoor channels.