Knowledge distillation aims to train a compact student network using soft supervision from a larger teacher network and hard supervision from ground truths. However, determining an optimal knowledge fusion ratio that balances these supervisory signals remains challenging. Prior methods generally resort to a constant or heuristic-based fusion ratio, which often falls short of a proper balance. In this study, we introduce a novel adaptive method for learning a sample-wise knowledge fusion ratio, exploiting both the correctness of teacher and student, as well as how well the student mimics the teacher on each sample. Our method naturally leads to the \textit{intra-sample} trilateral geometric relations among the student prediction ($\mathcal{S}$), teacher prediction ($\mathcal{T}$), and ground truth ($\mathcal{G}$). To counterbalance the impact of outliers, we further extend to the \textit{inter-sample} relations, incorporating the teacher's global average prediction ($\mathcal{\bar{T}})$ for samples within the same class. A simple neural network then learns the implicit mapping from the intra- and inter-sample relations to an adaptive, sample-wise knowledge fusion ratio in a bilevel-optimization manner. Our approach provides a simple, practical, and adaptable solution for knowledge distillation that can be employed across various architectures and model sizes. Extensive experiments demonstrate consistent improvements over other loss re-weighting methods on image classification, attack detection, and click-through rate prediction.

To deal with notorious delays in communication systems, it is crucial to forecast key system characteristics, such as the communication load. Most existing studies aggregate data from multiple edge nodes for improving the forecasting accuracy. However, the bandwidth cost of such data aggregation could be unacceptably high from the perspective of system operators. To achieve both the high forecasting accuracy and bandwidth efficiency, this paper proposes an Adaptive Multi-Teacher Weighting in Teacher-Student Learning approach, namely AdaTeacher, for communication load forecasting of multiple edge nodes. Each edge node trains a local model on its own data. A target node collects multiple models from its neighbor nodes and treats these models as teachers. Then, the target node trains a student model from teachers via Teacher-Student (T-S) learning. Unlike most existing T-S learning approaches that treat teachers evenly, resulting in a limited performance, AdaTeacher introduces a bilevel optimization algorithm to dynamically learn an importance weight for each teacher toward a more effective and accurate T-S learning process. Compared to the state-of-the-art methods, Ada Teacher not only reduces the bandwidth cost by 53.85%, but also improves the load forecasting accuracy by 21.56% and 24.24% on two real-world datasets.

Advanced communication network functions, such as resource allocation and dynamic spectrum management, heavily rely on the accurate forecasting of traffic. Data-driven solutions, e.g., Neural Network (NN) based forecasting methods, have been proven to be effective only when sufficient data is available. However, Base Stations (BSs) have limited data in the real world, since big data for communication networks could be extremely expensive to collect, store, and migrate. Therefore, most existing traffic forecasting methods have limited accuracy in reality due to the lack of big data. To tackle this problem, our key observation is that, despite the data “amount” in a BS is limited, the data “source” is rich and diverse, i.e., in addition to Internet traffic logs, there are logs of Call and SMS. More importantly, our analysis shows a high correlation between different sources, which can be utilized to improve the forecasting accuracy. Motivated by this, we introduce AdaSource, a Multi-Source Adaptive Feature Boosting approach, which utilizes data source correlations for accurate traffic forecasting even on data-limited BSs. The core idea of AdaSource is a novel two-branch NN structure that adaptively trains multiple Encoder-Decoders for refining different data sources and multiple Encoder-Predictors for utilizing data source correlations to improve the accuracy. The experiments on a real-world dataset show that AdaSource improves the forecasting accuracy by up to 30.14%, compared to the state-of-the-art methods.

Although reinforcement learning (RL) shows advantages in cellular network load balancing, it suffers from a low generalization ability, preventing it from real-world applications. Specifically, if network traffic pattern changes, the learned RL policy cannot adapt accordingly, resulting in system performance degradation. To address this issue, we propose a Multi-teacher MOdel BAsed Reinforcement Learning algorithm (MOBA), which leverages multi-teacher knowledge distillation theory to learn a generalized load balancing policy for adapting the real-world traffic pattern changes. The key is that different teachers represent different traffic patterns, and can learn various system models. By distilling and transferring the teacher knowledge, the student network is able to learn a generalized system model that covers different traffic patterns and unseen situations. Moreover, to improve the robustness of multi-teacher knowledge transfer, we learn a set of student models and use an ensemble method to jointly predict system dynamics. Results show that, compared with state-of-the-art RL methods, MOBA improves the minimal throughput and total throughput of a cellular network by up to 28.6% and 23.2%. Results also show that MOBA improves the training efficiency by up to 64%.

Accurate traffic volume estimation and prediction are essential for advanced communication network functions, such as automatic operations and predictive resource allocation. Although machine learning (ML)-based approaches achieve great success in accomplishing this goal, existing approaches suffer from two drawbacks that limit their real-world applications. First, the ML-based prediction models developed in the past might be obsolete now, since the communication traffic patterns and volumes keep changing in the real world, leading to prediction errors. Second, most Base Stations (BSs) can only save a small amount of data due to the limited storage capacity and high storage costs, which prevents from training an accurate prediction model. In this paper, we propose a novel framework that adapts the prediction model to the constantly changing traffic with only a few current traffic data. Specifically, the framework first learns the knowledge of historical traffic data as much as possible by using a proposed two-branch neural network design, which includes a prediction and a reconstruction module. Then, the framework transfers the knowledge from an old (past) prediction model to a new (current) model for the model update by using a proposed continual knowledge distillation technique. Evaluations on a real-world dataset show that the proposed framework reduces the Mean Absolute Error (MAE) of traffic prediction by up to 9.62% compared to the state-of-the-art prediction methods.

Prediction of key system characteristics, such as the communication load, is required to overcome the delays in wireless communication systems. State-of-The-Art (SOTA) approaches mostly apply existing Neural Network (NN) structures, and extract latent features purely based on their sensitivity to the forecasting accuracy. This way of feature extraction may neglect some non-obvious yet informative dimensions in the model input, leading to inaccurate forecasting results. In this paper, we present an Adaptive Feature Boosting (AFB) approach, which integrates multiple AutoEncoders (AEs) to automatically extract robust and comprehensive latent features for communication load forecasting. The recurrent and residual connections among the AEs make sure that the extracted latent features are representative for all input dimensions. With more comprehensive information extracted from the history, the forecasting accuracy is thus improved. We evaluate AFB against existing approaches on a real-world dataset that contains Call Detail Records (CDRs) of the Milan city over a period of two months. The evaluation shows that our AFB-based approach achieves 35.2% more accurate load forecasting results than the SOTA deep approaches.