Combining different data modalities enables deep neural networks to tackle complex tasks more effectively, making multimodal learning increasingly popular. To harness multimodal data closer to end users, it is essential to integrate multimodal learning with privacy-preserving approaches like federated learning (FL). However, compared to conventional unimodal learning, multimodal setting requires dedicated encoders for each modality, resulting in larger and more complex models. Training these models requires significant resources, presenting a substantial challenge for FL clients operating with limited computation and communication resources. To address these challenges, we introduce LW-FedMML, a layer-wise federated multimodal learning (FedMML) approach which decomposes the training process into multiple stages. Each stage focuses on training only a portion of the model, thereby significantly reducing the memory and computational requirements. Moreover, FL clients only need to exchange the trained model portion with the central server, lowering the resulting communication cost. We conduct extensive experiments across various FL and multimodal learning settings to validate the effectiveness of our proposed method. The results demonstrate that LW-FedMML can compete with conventional end-to-end FedMML while significantly reducing the resource burden on FL clients. Specifically, LW-FedMML reduces memory usage by up to$2.7\times $, computational operations (FLOPs) by$2.4\times $, and total communication cost by$2.3\times $. We also explore a progressive training approach called Prog-FedMML. While it offers lesser resource efficiency than LW-FedMML, Prog-FedMML has the potential to surpass the performance of end-to-end FedMML, making it a viable option for scenarios with fewer resource constraints.

Federated Learning is a new learning scheme for collaborative training a shared prediction model while keeping data locally on participating devices. In this paper, we study a new model of multiple federated learning services at the multi-access edge computing server. Accordingly, the sharing of CPU resources among learning services at each mobile device for the local training process and allocating communication resources among mobile devices for exchanging learning information must be considered. Furthermore, the convergence performance of different learning services depends on the hyper-learning rate parameter that needs to be precisely decided. Towards this end, we propose a joint resource optimization and hyper-learning rate control problem, namely${{\sf MS-FEDL}}$, regarding the energy consumption of mobile devices and overall learning time. We design a centralized algorithm based on the block coordinate descent method and a decentralized JP-miADMM algorithm for solving the${{\sf MS-FEDL}}$problem. Different from the centralized approach, the decentralized approach requires many iterations to obtain but it allows each learning service to independently manage the local resource and learning process without revealing the learning service information. Our simulation results demonstrate the convergence performance of our proposed algorithms and the superior performance of our proposed algorithms compared to the heuristic strategy.

Emerging cross-device artificial intelligence (AI) applications require a transition from conventional centralized learning systems toward large-scale distributed AI systems that can collaboratively perform complex learning tasks. In this regard, democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. The outlined principles are meant to study a generalization in distributed learning systems that go beyond existing mechanisms such as federated learning (FL). Moreover, such learning systems rely on hierarchical self-organization of well-connected distributed learning agents who have limited and highly personalized data and can evolve and regulate themselves based on the underlying duality of specialized and generalized processes. Inspired by Dem-AI philosophy, a novel distributed learning approach is proposed in this article. The approach consists of a self-organizing hierarchical structuring mechanism based on agglomerative clustering, hierarchical generalization, and corresponding learning mechanism. Subsequently, hierarchical generalized learning problems in recursive forms are formulated and shown to be approximately solved using the solutions of distributed personalized learning problems and hierarchical update mechanisms. To that end, a distributed learning algorithm, namely DemLearn, is proposed. Extensive experiments on benchmark MNIST, Fashion-MNIST, FE-MNIST, and CIFAR-10 datasets show that the proposed algorithm demonstrates better results in the generalization performance of learning models in agents compared to the conventional FL algorithms. The detailed analysis provides useful observations to further handle both the generalization and specialization performance of the learning models in Dem-AI systems.

A recent take toward federated analytics (FA), which allows analytical insights of distributed data sets, reuses the federated learning (FL) infrastructure to evaluate the summary of model performances across the training devices. However, the current realization of FL adopts single server-multiple client architecture with limited scope for FA, which often results in learning models with poor generalization, i.e., an ability to handle new/unseen data, for real-world applications. Moreover, a hierarchical FL structure with distributed computing platforms demonstrates incoherent model performances at different aggregation levels. Therefore, we need to design a robust learning mechanism than the FL that 1) unleashes a viable infrastructure for FA and 2) trains learning models with better generalization capability. In this work, we adopt the novel democratized learning (Dem-AI) principles and designs to meet these objectives. First, we show the hierarchical learning structure of the proposed edge-assisted Dem-AI mechanism, namelyEdge-DemLearn, as a practical framework to empower generalization capability in support of FA. Second, we validate Edge-DemLearn as a flexible model training mechanism to build a distributed control and aggregation methodology in regions by leveraging the distributed computing infrastructure. The distributed edge computing servers construct regional models, minimize the communication loads, and ensure distributed data analytic application’s scalability. To that end, we adhere to a near-optimal two-sided many-to-one matching approach to handle the combinatorial constraints in Edge-DemLearn and solve it for fast knowledge acquisition with optimization of resource allocation and associations between multiple servers and devices. Extensive simulation results on real data sets demonstrate the effectiveness of the proposed methods.

Nowadays, Federated Learning has emerged as the prominent collaborative learning approach among multiple machine learning techniques. This framework enables communication-efficient and privacy-preserving solution that a group of users interacts with a server to collaboratively train a powerful global model without exchanging users' raw data. However, federated learning might face the significant challenge with high communication cost when exchanging the huge model parameters. Moreover, training such a large model on devices is an obstacle under the battery limitation of mobile devices. To address this hindrance, we propose the federated learning with bi-level distillation, namely FedBD. The key idea of this proposal is to exchange the soft targets instead of transferring the model parameters between server and clients. The exchange knowledge was constructed based on the prediction outcomes for the shared reference dataset. By interchanging the knowledge of the learning models, our algorithm obtains the benefits of reducing both communication and computation costs. The proposed mechanism allows the different model architectures between server and learning agents. The experiments show that our proposed method can achieve comparable or even slightly higher accuracy than FedAvg algorithm on the image classification task while using fewer communication resources and power.

There is an increasing interest in a fast-growing machine learning technique called Federated Learning (FL), in which the model training is distributed over mobile user equipment (UEs), exploiting UEs' local computation and training data. Despite its advantages such as preserving data privacy, FL still has challenges of heterogeneity across UEs' data and physical resources. To address these challenges, we first propose FEDL, a FL algorithm which can handle heterogeneous UE data without further assumptions except strongly convex and smooth loss functions. We provide a convergence rate characterizing the trade-off between local computation rounds of each UE to update its local model and global communication rounds to update the FL global model. We then employ FEDL in wireless networks as a resource allocation optimization problem that captures the trade-off between FEDL convergence wall clock time and energy consumption of UEs with heterogeneous computing and power resources. Even though the wireless resource allocation problem of FEDL is non-convex, we exploit this problem's structure to decompose it into three sub-problems and analyze their closed-form solutions as well as insights into problem design. Finally, we empirically evaluate the convergence of FEDL with PyTorch experiments, and provide extensive numerical results for the wireless resource allocation sub-problems. Experimental results show that FEDL outperforms the vanilla FedAvg algorithm in terms of convergence rate and test accuracy in various settings.

Federated Learning (FL) is a distributed learning framework that can deal with the distributed issue in machine learning and still guarantee high learning performance. However, it is impractical that all users will sacrifice their resources to join the FL algorithm. This motivates us to study the incentive mechanism design for FL. In this paper, we consider a FL system that involves one base station (BS) and multiple mobile users. The mobile users use their own data to train the local machine learning model, and then send the trained models to the BS, which generates the initial model, collects local models and constructs the global model. Then, we formulate the incentive mechanism between the BS and mobile users as an auction game where the BS is an auctioneer and the mobile users are the sellers. In the proposed game, each mobile user submits its bids according to the minimal energy cost that the mobile users experiences in participating in FL. To decide winners in the auction and maximize social welfare, we propose the primal-dual greedy auction mechanism. The proposed mechanism can guarantee three economic properties, namely, truthfulness, individual rationality and efficiency. Finally, numerical results are shown to demonstrate the performance effectiveness of our proposed mechanism.

Multi-access Edge Computing (MEC) is recently acknowledged as one of the key pillars for the next revolution of mobile communications area to provide lower latency and more computation capability for cellular base stations (BSs). The trade-off of delay performance and energy cost which is fully controlled by the association decisions among systems of edge sites of mobile users and the service rate scaling of MEC servers to serve the user tasks is analyzed. In this paper, we formulate a joint user association and MEC server scaling optimization problem, namely MEC - MP. Accordingly, we first propose a centralized algorithm by iteratively solving the user association and MEC server scaling subproblems to obtain a suboptimal solution approach. We then propose a distributed algorithm based on a greedy user association strategy and decentralized solutions of the MEC server scaling problem.

There is an increasing interest in a new machine learning technique called Federated Learning, in which the model training is distributed over mobile user equipments (UEs), and each UE contributes to the learning model by independently computing the gradient based on its local training data. Federated Learning has several benefits of data privacy and potentially a large amount of UE participants with modern powerful processors and low-delay mobile-edge networks. While most of the existing work focused on designing learning algorithms with provable convergence time, other issues such as uncertainty of wireless channels and UEs with heterogeneous power constraints and local data size, are under-explored. These issues especially affect to various trade-offs: (i) between computation and communication latencies determined by learning accuracy level, and thus (ii) between the Federated Learning time and UE energy consumption. We fill this gap by formulating a Federated Learning over wireless network as an optimization problem FEDL that captures both trade-offs. Even though FEDL is non-convex, we exploit the problem structure to decompose and transform it to three convex sub-problems. We also obtain the globally optimal solution by charactering the closed-form solutions to all sub-problems, which give qualitative insights to problem design via the obtained optimal FEDL learning time, accuracy level, and UE energy cost. Our theoretical analysis is also illustrated by extensive numerical results.

Installation of backup power supply plays a vital role in maintaining communication services which can save billions of dollars as well as human lives during natural disasters. Due to the higher capital and operational expense compared to public power, pooling and sharing the backup power supplies can be an economical solution since the backup power capacity can be sized based on the aggregate demand of co-located operators. However, how to pool and share the backup power at multi-operator cellular sites in a fair manner should be considered due to the limited capacity and high user demands. In this paper, we adopt the Nash Bargaining Solution (NBS) of a bargaining problem which can guarantee the fairness of backup power sharing and design a decentralized algorithm approach with limited information exchange among the operators. Our simulation demonstrates that the sharing the backup power reduces the average delay and requires less BS power consumption than the non-sharing approach, especially for high traffic load scenarios. In addition, we also extend the formulation with respect to admission control for very high traffic demand cases.

Keeping wireless base stations operating continually and providing uninterrupted communications services can save billions of dollars as well as human lives during natural disasters and/or electricity outages. Toward this end, wireless operators need to install backup power supplies whose capacity is sufficient to support their peak power demand, thus incurring a significant capital expense. Hence, pooling together backup power supplies and sharing it among co-located wireless operators can effectively reduce the capital expense, as the backup power capacity can be sized based on the aggregate demand of co-located operators instead of individual demand. Turning this vision into reality, however, faces a new challenge: how to fairly share the backup power supply? In this paper, we propose fair sharing of backup power supply by multiple wireless operators based on the Nash bargaining solution (NBS). In addition, we integrate our analysis with multiple time slots for emergency cases in which the study the backup energy sharing based on model predictive control and NBS subject to an energy capacity constraint regarding future service availability. Our simulations demonstrate that sharing backup power/energy improves the communications service quality with lower cost and consumes less base station power than the non-sharing approach.

There have been many recent studies on the Demand Response (DR) of Datacenters (DCs). Nonetheless, (i) DR of Colocation Datacenters (CDCs), and (ii) the role of Demand Response Provider (CSP) have been largely overlooked. CDCs differ from big owner-operated DCs in that the operator has no control over their tenants, and thus, requiring a mechanism for the operator to give tenants incentives to reduce their electricity usage. CSP uses compensation price as a guidance for customers' response. To fill the gap, we propose an incentive mechanism for CDC DR that studies the interaction between the CSP, CDCs and tenants. Firstly, the strategic behaviors of these interactions are formulated as a three-stage Stackelberg game which contains a separate problem at each stage. In Stage I, the CSP solves an optimal compensation pricing problem. In Stage II, each CDC operator finds its own optimal procurement and reward strategy. In Stage III, the optimal tenants' energy reduction is calculated. Secondly, we examine both exact and approximate solution at Stage II, and propose an efficient algorithm to obtain the optimal CSP price in Stage I. Finally, the extensive numerical analysis (a) shows that the CLT-based approximation achieves similar solutions compared to the exact analysis, and (b) illustrates the comparisons between the optimal CSP individual cost and the social cost.

Recommender agents are widely used in online markets, social networks and search engines. The recent online news recommendation systems such as Google News and Yahoo! News produce real-time decisions for ranking and displaying highlighted stories from massive news and users access per day. The more relevant highlighted items are suggested to users, the more interesting and better feedback from users achieve. Therefore, the distributed online learning can be a promising approach that provides learning ability for recommender agents based on side information under dynamic environment in large scale scenarios. In this work, we propose a distributed algorithm that is integrated online K-Means user contexts clustering with online learning mechanisms for selecting a highlighted news. Our proposed algorithm for online clustering with lower bound confident clustering approximates closer to offline K-Means clusters than greedy clustering and gives better performance in learning process. The algorithm provides a scalability, cheap storage and computation cost approach for large scale news recommendation systems.

In this paper, the problem of resource allocation in cloud datacenters, that own highly complex and heterogeneous tasks and servers, is considered. To address this problem, a novel framework, dubbed joint operation cost and network traffic cost (JOT) framework, is proposed. This framework combines notions from Gibbs sampling and matching theory to find an efficient solution addressing the NP-hard problem JOT. The proposed model is shown to be capable of controlling the active server set, in a coordinated manner while allocating VMs in order to reduce both operation cost and network traffic cost of the cloud datacenter. We also conduct a case-study to validate our proposed algorithm and the results show that JOT can reduce the total incurred cost by up to 19% compared to the existing non-coordinated approach.