The applications of unmanned swarms have become increasingly widespread, gradually transforming production processes and daily life. Task allocation, the top-level design for unmanned swarm missions, is pivotal to maximizing the efficiency of the entire swarm. However, traditional optimization methods and intelligent algorithms, including Reinforcement Learning (RL), often struggle to adapt to the complex and unpredictable situations in these tasks. To address this challenge, we propose a novel Multi-Agent Proximal Policy Optimization (MAPPO) algorithm combined with the population-based learning and Gaussian Mixture Model (GMM)-based adjustment mechanisms (PG-MAPPO). In PG-MAPPO, the population-based learning mechanism is integrated to enable agents with diverse exploration preferences to uncover optimal collaboration patterns among Unmanned Aerial Vehicles (UAVs), thereby enhancing cooperative efficiency. The GMM-based adjustment mechanism dynamically adjusts UAV formations for each agent, significantly improving the swarm’s flexibility and adaptability in rapidly changing environments. To demonstrate the effectiveness of PG-MAPPO, a maritime rescue simulation containing multiple complex and dynamic scenarios is conducted. Experimental results show that our algorithm achieves higher rescue success rate with faster convergence and greater stability than state-of-the-art Multi-Agent Reinforcement Learning (MARL) methods in all scenarios. Notably, the PG-MAPPO algorithm improves the rescue success rate by 31.6% compared to the best-performing baseline under challenging conditions.

Satellite networks are becoming an integral part of future network infrastructure. However, premature depletion of energy in some satellites caused by the limited capacity of batteries and the uneven distribution of network traffic poses a significant challenge. Therefore, we have a keen interest in designing an available energy routing algorithm for LEO satellite networks to balance energy usage. Firstly, a satellite available energy model has been established, which considers the output power of satellite solar panels and the energy consumption of satellite under the influence of Earth’s obstruction. Then, our algorithm considered the different routing transmission requirements of network services and appropriately selected different decision factors to generate paths, which can reduce network routing energy consumption while meeting the delay requirements of services. Numerical result demonstrate that our algorithm can reduce energy consumption by about 12.53% and reduce packet loss rate by 23.72% compared with QoS-SR, which effectively improving the end-to-end delay of services and extending the life of satellite networks.

Piecewise linear neural network (PLNN) possesses universal approximation ability for continuous functions on the compact domain, and for a PLNN in which the hidden neuron (HN) is wireless powered through wireless power transfer (WPT) technology and activated only if its energy harvest arrives at the activation threshold, the more HNs are activated, the smaller approximation error may be enjoyed, while the more energy will be consumed. Therefore, how to control the energy supply for the PLNN to reach the minimum approximation error while keeping the acceptable level of energy harvest arises as an interesting issue. To address this issue, this paper first formulates the energy-harvesting of an individual HN as a Markov decision process (MDP) to further deduce the objective function of the PLNN approximating the given continuous function, then uncovers the optimal activation probability and expected energy harvest for each HN, and finally reveals the optimal energy supply for the HNs to satisfy the optimal energy harvest. A novel algorithm based on our discovered theoretical foundations to control the energy supply to minimize the approximation error within the acceptable level of energy harvest is proposed, and the simulations and field experiments are both given to test its performance. To our best knowledge, this is the initial work towards the joint optimization of energy efficiency and universal approximation performance of the wireless powered PLNN.Note to Practitioners—This paper addresses the interesting issue of how to guarantee wireless powered PLNN to approximate the continuous function with the minimum approximation error under the energy constraint. Through the insightful disclosure of the optimal activation probability, optimal expected energy harvest, and optimal control of energy supply for each HN, this paper facilitates the wireless powered PLNN to operate in an energy-efficiency and universal-approximation way. Since PLNN can model the nonlinear and complex system (NCS) with arbitrary accuracy on the compact domain, our proposed solution can be applied by the NCS modeling technology to enjoy high approximation accuracy while keeping the energy consumption at the acceptable level, which we believe could inspire the development of low-power neural networks (NNs).

For multiple smart sensors with limited energy supply, the relationship between the energy supply and sensor fault-free region is often unknown, and neither of the interior structure nor exterior circumstance modeling the smart sensors is easy to accurately achieve, so how to guarantee the fault-free smart sensors to harvest the most energy in the fault-free way is a very challenging topic. To address this issue, this paper first formulates the individual smart sensor as a single-input single-output (SISO) model-free system (MFS), with its energy supply and sensing error as the input and output, respectively, then makes use of Lyapunov function to deduce an upper bound of the fault-free region to realize the fault-detection of any smart sensor and disclose the relationship between the energy supply and fault-free region, and finally achieves the optimal energy supply guiding the overall energy harvest of all fault-free smart sensors to converge to the maximum with the convergence rate no larger than$\kappa$,$\kappa\in[0,1]$, while enjoying the real-time fault-detection. An algorithm based on the sound theoretical foundations is further proposed to implement the optimal energy supply. Theoretical analysis, simulations and field experiments jointly verify the performance of our method. To our best knowledge, it is the initial work towards this issue.Note to Practitioners—This paper addresses the interesting issue of how to guarantee multiple smart sensors to harvest the most energy in the free-fault way. Through the insightful disclosure of relationship between the energy supply and sensor fault-free region, and the optimal control of energy supply, this paper facilitates the overall energy harvest of all fault-free smart sensors that work under the environments, where the available energy is limited, to converge to the maximum while enjoying the real-time fault-detection, which we believe could push the development of Internet of Things (IoT) or Cyber-Physical System (CPS) that employs multiple smart sensors to sense the physical world. Simulations and field experimental investigations jointly show that the proposed solution outperforms the existing solutions.

This paper studies the edge computing system (ECS) in which caching the frequently reusable service (FRS) at the edge server (ES) is an effective way to reduce delays. The larger cache space available in the ES (we call it ES cache space) might buffer the larger scale of FRS, and subsequently decrease the delays while bringing higher cache storage costs. Meanwhile, the distribution of FRS is not always known in advance. Therefore, how much ES cache space should be supplied to make the optimal tradeoff between the delays and cache storage costs arises as an interesting issue in practice. To address this issue, this paper first formulates the problem of determining the amount of ES cache space supply as a constrained Markov decision process (CMDP), then adopts the Zipf’s distribution to estimate the probability distribution of FRS, and finally proposes an effective cache space control algorithm (CSCA) guiding the ES to determine the amount of ES cache space supply to minimize the cache storage costs while maintaining the delays at the acceptable level. Theoretical analysis, simulations and field experiments document and illustrate its performance. Note to Practitioners—This paper addresses the interesting trade-off between the delays and cache storage costs for the edge computing system that operates with limited cache storage budgets while must satisfy the required real-time performances. It helps to improve the operation efficiency of the systems with edge computing setting in the area of Internet of Things (IoT) or Cyber-Physical Systems (CPS) that employ the edge server to cache the frequently reusable services, arriving at the minimization of the accumulative cache storage costs while maintaining the accumulative delays at the acceptable level. Theoretical analysis, simulation and field experimental investigations jointly show that the solution proposed here outperforms existing solutions.

The widespread adoption of EVs brings practical interest to the operation optimization of the charging station. This paper considers the joint scheduling of pricing and charging control to enhance the operational capability of the station. The following contributions are made. First, a joint scheduling model of pricing and charging control is developed to maximize the expected social welfare of the charging station considering the quality of service and the price fluctuation sensitivity of EV drivers. It is formulated as a Markov decision process with a variance criterion to capture uncertainties during operation. Second, a two-phase on-line policy learning algorithm is proposed to solve this joint scheduling problem. In the first phase, it implements event-based policy iteration to find the optimal pricing scheme. In the second phase, it implements scenario-based model predictive control for smart charging under the updated pricing scheme. Third, by leveraging the performance difference theory, the optimality of the proposed algorithm is theoretically analyzed. The feasibility of the proposed method and the improved social welfare of the charging station are numerically demonstrated based on a typical charging station with distributed generation and storage.Note to Practitioners—The popularization of EVs requires the high-efficiency operation of the charging station. The joint scheduling of pricing and charging control can provide a promising way to achieve the balance between EV drivers’ satisfaction and the profit maximization of the charging station. However, it suffers from EV drivers’ uncertain responses to the pricing scheme and the coupled relationship between pricing and charging control. This multi-stage stochastic programming is non-trivial to solve. In this paper, we propose a two-phase on-line policy learning method for this joint scheduling problem. This algorithm can be implemented in the controller of the charging station. In the first phase, the event-based policy iteration can be implemented to iteratively improve the current best pricing scheme until convergence. In the second phase, the scenario-based model predictive control, which is reformulated as a mixed integer linear programming, can be quickly solved for smart charging. Case studies demonstrate the operation enhancement of the station.

The electric vehicle (EV) utilizing vehicle-to-grid (V2G) technology can serve as an important mobile storage in a microgrid of building. Since more and more buildings are equipped with charging piles and distributed renewable energy, it can potentially improve the operation efficiency of building microgrid and reduce the impact of EVs to the grid by V2G scheduling considering the uncertain supply and EV charging demand in the building. We consider this important problem in this paper and make the following contributions. First, we formulate this V2G scheduling problem as a constrained Markov decision process (CMDP). The objective function is to improve the overall building energy operation cost while constraining the expected cycling time to ensure the participation enthusiasm of the EV users. Second, a simulation-based primal-dual approach is developed to decompose the original problem into a continuous optimization subproblem on the supply side and a discrete optimization subproblem on the demand side. The demand side optimization can be further decoupled as a distributed single EV scheduling problem where simulation-based regularized rollout method can be applied to improve from existing base policies. Third, the structural property of the problem and the cost improvement of the proposed method are analyzed to speed up the optimization and ensure the solution quality. Numerical experiments based on real building load data and EV data are conducted to demonstrate the performance of this method.Note to Practitioners—With the rapid adoption of EVs in the microgrid of building, there goes the challenge of how to reduce their charging impact to the microgrid. The V2G scheduling between EVs and building microgrid can provide a promising way to reduce the impact and improve the building operation efficiency. However, it suffers from the uncertain EV charging demand and the conflict between synthesized coordination and ensuring the participation enthusiasm of drivers. In this paper, we propose a constrained V2G scheduling model to address this issue. After formulating the problem as a constrained Markov decision process, a simulation-based primal-dual approach is developed to decompose the problem where the derived control policy in the supply side can be deployed in the building energy controller and the derived control policy in the demand side can be deployed in the controller of each charging pile. Case studies show the performance of the proposed method and its win-win property for the building and EV users.

Due to the mobility and vulnerability of satellite nodes, the topology of satellite-terrestrial integrated networks (STINs) is highly dynamic. It requires a flexible reconstruction strategy to avoid severe performance loss. However, the current reconstruction strategies usually result in a heavy computational burden, especially in the case of a shortage of satellite resources. Hence, it is of great practical interest to design a flexible topology reconstruction strategy to balance the computational efficiency and network performance of STINs. We investigated this important problem in this paper and made the following contributions. First, a deep Q-learning model is proposed to dynamically optimize the node classification results and flexibly determine the range of network reconstruction, which could enhance the efficiency of the network recovery. Second, a modified SVM classification model is proposed to classify the node type with the hyperplane parameters dynamically adjusted, which can increase the accuracy of the classification result and improve the convergence efficiency of the reconstruction algorithm. Third, an artificial bee colony algorithm is proposed to flexible recover the performance of the damaged network, which considers the age of information and range of the topology cross iteration. Numerical results demonstrate that compared with existing algorithms, the average end-to-end delay decreased by about 13.68%, and other performance indicators of STINs where improved to a certain extent.

The matching between random wind supply and electric vehicle (EV) charging demand can reduce the requirement of traditional power sources and the emission of CO2. This problem is of great practical interest but involves system dynamics in multiple timescales. We consider this an important problem in this paper. In order to capture the randomness in the wind supply and EV charging demand, we formulate the problem as a bilevel Markov decision process. At the upper level, the charging demand of EVs in different locations is aggregated into multiple aggregators. The system operator dispatches power among the aggregators in a coarse timescale to maximize the wind power utilization. At the lower level, the aggregator schedules the charging process of individual EVs at a finer timescale to minimize the charging cost. In order to solve this large-scale problem, a bilevel simulation-based policy improvement (SBPI) method is developed. It is mathematically proved that the SBPI can improve from base policies in both levels. The performance of this multi-timescale and bilevel coordination approach is demonstrated through case studies in the city of Beijing.