The growing resource demands of large-scale transformer models pose significant challenges for resource-constrained users, particularly in distributed environments. To address this issue, we propose a federated learning framework called Fed-Grow, which enables multiple participants to collaboratively learn a lightweight scaling operation that transfers knowledge from pretrained small models to a large transformer model. In Fed-Grow, we introduce the Dual-LiGO (Dual Linear Growth Operator) architecture, consisting of Local-LiGO and Global-LiGO components. Local-LiGO addresses model heterogeneity by adapting each participant's pre-trained model to a common intermediate form, while Global-LiGO facilitates knowledge sharing across participants without sharing local models or raw data, ensuring privacy preservation. This federated approach offers a scalable solution for growing large transformers in a distributed manner, where only the Global-LiGO is shared, significantly reducing communication overhead while maintaining comparable model performance under the same communication constraints. Experimental results demonstrate that Fed-Grow outperforms state-of-the-art methods in terms of accuracy and precision, while reducing the number of trainable parameters by 59.25% and communication costs by 73.01%. These improvements allow for higher efficiency in training large models in distributed environments, without sacrificing performance. To the best of our knowledge, Fed-Grow is the first method that enables cooperative transformer scaling in a distributed setting, making it a practical solution for resource-constrained users.

In this paper, we study a stable Age-of-Information (AoI) scheduling problem to handle the massive packet aggregation from arbitrary total ofnend devices to an edge device via a single hop wireless channel when information constantly arrives at the end side. Specifically, we consider the arrivals of the information with anonlineinjection mode, i.e., information is injected to end devices with an unknown injection rate in each time slot. After the information is injected, the AoI with respect to those end devices constantly increases until their fresh messages are received by the edge device, which characterizes the freshness of the information at destination. Based on the online injection mode, we propose the first distributed stable AoI scheduling algorithm combining NOMA (Non-Orthogonal Multiple-Access) technique in this paper, to minimize the expected average peak AoI (EAP-AoI) of our end-to-edge information system. Adopting NOMA technique enableskmessages decoded from a mixed signal by the edge device in our algorithm, with parameter$k\gt 1$. We prove that our algorithm is stable even under the asymptotically maximum injection rate of$O(k/n)$that any stable AoI scheduling algorithm may handle, and the EAP-AoI of our information system is bounded by$O(\sqrt [{3}]{nk})$time slots under the injection rate of$O(k/n)$. Comparing with two existing results, the EAP-AoI in our algorithm is$O\left ({{\frac {n^{2/3}}{k^{4/3}}}}\right)$and$O\left ({{\frac {n^{2/3}}{k^{1/3}}}}\right)$times smaller. Numerical results also verify the stability and efficiency of our algorithm.

Federated edge learning (FEEL) is a novel paradigm that enables privacy-preserving and distributed machine learning on end devices. However, FEEL faces challenges from data/system heterogeneity among the participating clients and resource constraints of edge networks, which affect the efficiency and accuracy of the learning process. In this paper, we propose a comprehensive framework for client selection in FEEL based on the concept of Value-of-Information (VoI), which measures how valuable a client is for the global model aggregation. Our framework consists of two independent components: a VoI estimator that uses reinforcement learning to learn the relationship between VoI and various heterogeneous factors of clients; and a greedy client selector that chooses the most valuable clients under network resource constraints. Compared with most of the previous works that use concrete criteria to evaluate and select heterogeneous clients, our VoI-based approach is more comprehensive. Extensive experiments on different datasets and learning tasks are conducted, which show that our framework outperforms several state-of-the-art methods in terms of accuracy.

As an amalgamation of digital twin and edge computing, the digital twin edge networks (DITENs) have drawn much attention from industry and academia to bridge the divide between physical edge networks and digital systems. Meanwhile, the physical hardware and open-access wireless communication environments in edge networks raise significant challenges in timely information aggregation and real-time status updates for digital twins, especially in the presence of inherent failure and hostile jamming. In this paper, we investigate the fault-tolerant Age-of-Information (AoI) optimization problem for digital twin edge networks against the severe fault phenomena in wireless communications. In contrast to previous works that focus on single fault or individual jamming behavior, we propose a comprehensive communication failure model over thefnon-overlapping wireless sub-channels, wheref1out of thefsub-channels are inherently failed andf2out of thefsub-channels are jammed by adversaries. Then, based on the multi-channel communication failure model, we present a distributed fault-tolerant communication algorithm to minimize the expected average peak AoI in DITENs. Using an adaptive transmission strategy, we prove that the AoI optimization issue fornend devices can be resolved within Θ(n) time steps whenf1+f2f. Both theoretical analyses and empirical simulations are conducted to verify the fault-tolerance and efficiency of our proposed algorithm despite the communication failures.

In this paper, we propose De-RPOTA, a novel algorithm designed for decentralized learning, equipped with mechanisms for resource adaptation and privacy protection through over-the-air computation. We theoretically analyze the combined effects of limited resources and lossy communication on decentralized learning, showing it converges towards a contraction region defined by a scaled errors version. Remarkably, De-RPOTA achieves a convergence rate of$\mathcal {O}\left ({{\frac {1}{\sqrt {nT}}}}\right)$in scenarios devoid of errors, matching the state-of-the-arts. Additionally, we tackle a power control challenge, breaking it down into transmitter and receiver sub-problems to hasten the De-RPOTA algorithm’s convergence. We also offer a quantifiable privacy assurance for our over-the-air computation methodology. Intriguingly, our findings suggest that network noise can actually strengthen the privacy of aggregated information, with over-the-air computation providing extra security for individual updates. Comprehensive experimental validation confirms De-RPOTA’s efficacy in communication resources limited environments. Specifically, the results on the CIFAR-10 dataset reveal nearly 30% reduction in communication costs compared to the state-of-the-arts, all while maintaining similar levels of learning accuracy, even under resource restrictions.

In this paper, we propose a novel decentralized learning algorithm, namely DLLR-OA, for resource-constrained over-the-air computation with formal privacy guarantee. Theoretically, we characterize how the limited resources induced model-components selection error and compound communication errors jointly impact decentralized learning, making the iterates of DLLR-OA converge to a contraction region centered around a scaled version of the errors. In particular, the convergence rate of the DLLR-OA algorithm in the error-free case [EQUATION] achieves the state-of-the-arts. Besides, we formulate a power control problem and decouple it into two sub-problems of transmitter and receiver to accelerate the convergence of the DLLR-OA algorithm. Furthermore, we provide quantitative privacy guarantee for the proposed over-the-air computation approach. Interestingly, we show that network noise can indeed enhance privacy of aggregated updates while over-the-air computation can further protect individual updates. Finally, the extensive experiments demonstrate that DLLR-OA performs well in the communication resources constrained setting. In particular, numerical results on CIFAR-10 dataset shows nearly 30% communication cost reduction over state-of-the-art baselines with comparable learning accuracy even in resource constrained settings.

In the field of remote sensing image processing, clouds heavily affect the quality of the remote sensing images and their application potential. Thus, in recent years, with the prevalence of deep learning techniques used in the field of image processing, many methods have been proposed for cloud removal using single remote sensing images. The existing single-image cloud removal methods suffer from poor generalization capabilities that prevent them from being applied to diverse remote sensing images. Thus, a novel method using a multimodal architecture is proposed which provides multiple most likely outputs for the image and selects the best one through perception-based image quality evaluator (PIQE). In addition, adversarial consistency loss is used to replace cycle consistency loss, which encourages the model to retain more texture information of the original image, and thus the quality of the generated image increases. Experiments demonstrate that the presented method can easily achieve a considerable increase in the peak signal-to-noise ratio and the structural similarity index compared with other methods.