With the explosive growth of e-commerce, efficient last-mile delivery has emerged as a critical challenge. Truck-drone collaborative delivery has garnered significant attention as a promising solution to this problem. In this work, we formulate the truck-drone collaborative delivery problem as a collaborative optimization problem, aiming to minimize the total completion time for delivering packages to customers by leveraging the complementary strengths of the drone’s speed and the truck’s endurance. We propose a two-stage framework to address this challenge. In the first stage, the Lin-Kernighan Helsgaun (LKH) algorithm is employed to generate a high-quality initial Traveling Salesman Problem (TSP) solution, serving as a robust starting point. In the second stage, a Sequence Allocate Policy (SAPPO), based on Proximal Policy Optimization, refines the TSP solution by optimizing the truck-drone collaborative path using a specially designed action space. Extensive experiments conducted on both random dataset and TSPLIB benchmarks demonstrate that our method significantly outperforms existing algorithms regarding delivery time, while exhibiting improved scalability and less training time.

Dynamic systems often encounter disturbances like sensor outliers, which violate the Gaussian noise assumption in traditional Kalman filters. While maximum correntropy Kalman filters (MCKF) address this issue by utilizing higher-order statistical information, their performance critically depends on the manual selection of a kernel scale parameter. Existing methods with fixed or empirically adjusted kernel scales struggle to handle disturbances of varying intensities, limiting practical applications. This paper presents a novel adaptive Maximum Correntropy Kalman Filter (MCKF) framework. The key contributions are as follows: (1) Adaptive Kernel Scale Optimization: The kernel scale is modeled as a probabilistic variable, and variational Bayesian inference is employed to jointly estimate the system state, enabling automatic kernel scale optimization during the recursive process. (2) Theoretical Analysis and Extension: The computational complexity of the proposed algorithm is analyzed in the context of linear systems, and its theoretical connection to traditional correlation entropy filters is established. Furthermore, the method is extended to nonlinear systems. (3) Performance Enhancement: Experimental evaluations on typical single-target tracking task and complex nonlinear scenario demonstrate that the proposed approach outperforms existing methods. Under various types of noise interference, the average position estimation error is reduced by 40–60%, highlighting its superior adaptability and robustness. In addition, the algorithm is successfully applied to real-world battery State of Charge (SOC) estimation in Internet of Things (IoT) scenarios, demonstrating its practical value in embedded energy management systems.

Auction mechanism, as a fair and efficient resource allocation method, has been widely used in varieties trading scenarios, such as advertising, crowdsensoring and spectrum. However, in addition to obtaining higher profits and satisfaction, the privacy concerns have attracted researchers’ attention. In this paper, we mainly study the privacy preserving issue in the double auction market against the indirect inference attack. Most of the existing works apply differential privacy theory to defend against the inference attack, but there exists two problems. First, ‘indistinguishability’ of differential privacy (DP) cannot prevent the disclosure of continuous valuations in the auction market. Second, the privacy-utility trade-off (PUT) in differential privacy deployment has not been resolved. To this end, we proposed an attack-defense game-based reinforcement learning privacy-preserving method to provide practically privacy protection in double auction. First, the auctioneer acts as defender, adds noise to the bidders’ valuations, and then acts as adversary to launch inference attack. After that the auctioneer uses the attack results and auction results as a reference to guide the next deployment. The above process can be regarded as a Markov Decision Process (MDP). The state is the valuations of each bidders under the current steps. The action is the noise added to each bidders. The reward is composed of privacy, utility and training speed, in which attack success rate and social welfare are taken as measures of privacy and utility, a delay penalty term is used to reduce the training time. Utilizing the deep deterministic policy gradient (DDPG) algorithm, we establish an actor-critic network to solve the problem of MDP. Finally, we conducted extensive evaluations to verify the performance of our proposed method. The results show that compared with other existing DP-based double auction privacy preserving mechanisms, our method can achieve better results in both privacy and utility. We can reduce the attack success rate from nearly 100% to less than 20%, and the utility deviation is less than 5%. Note to Practitioners—Privacy protection in trading markets, such as advertising, crowdsensing, and spectrum, is crucial. Traditional approaches like differential privacy have been unable to entirely guard sensitive data against inference attacks. To address this, we introduce a novel privacy-preserving mechanism for double auction markets. Our approach employs an attack-defense game model, where noise is added to bidders’ valuations and then used to launch an inference attack. This process allows for the evaluation of the noise’s effectiveness and iteratively refines the privacy protection method. Transformed into a reinforcement learning model and optimized through a DDPG network, our mechanism reduces computational complexity. It has been shown to significantly diminish the success rate of inference attacks, while maintaining a minimal utility deviation. Practitioners in auction-based markets can leverage our approach to enhance privacy protection without negatively impacting market performance. By integrating our mechanism into their operations, auctioneers can foster a safer and more efficient trading environment.

In recent years, the rapid adoption of electric vehicles (EVs) in the taxi industry has transformed traditional taxi-hailing systems into electric taxi (E-taxi) hailing systems. As a result, it is crucial to develop effective strategies for optimizing E-taxi management by considering both passenger-taxi matching and charging planning. In this paper, we first formalize the E-taxi management optimization problem as a Markov decision problem with dynamic state and heterogeneous action. We then propose a dual-stage heuristic coordinated reinforcement learning (RL) approach that incorporates advanced feature selection and heuristic allocation strategies. Our approach consists of two main stages. In the first stage, we introduce the feature-guided state dimensionality stabilization proximal policy optimization (PPO) method to address dynamic state dimensions by a feature selection method, and enabling E-taxis to decide whether to charge or pick up passengers. In the second stage, we propose a heuristic coordinated assignment method to further allocate charging stations and passengers for the E-taxis, and provide the RL network in the first stage with rewards based on the results. This approach effectively tackles the challenge of RL processing of heterogeneous action spaces (charge and pick up). We evaluate our proposed method in a real-world E-taxi environment and find that it significantly enhances the experience for both E-taxis and passengers. Specifically, due to our method’s rational planning for passenger pick-up and charging, E-taxis can increase their revenue by 20% compared to traditional RL methods or random scheduling approaches. As for passengers, since the taxis have more efficiently planned their charging behavior, the probability of their orders being answered increases by 15%, while their waiting time is reduced by 55%. These achievements contribute to the advancement of E-taxi management strategies and promote the widespread adoption of electric vehicles, ultimately supporting the transition to a more sustainable transportation system. Note to Practitioners—The increasing adoption of electric vehicles in the taxi industry has led to the need for effective E-taxi management strategies that consider both passenger-taxi matching and charging planning. In this study, we introduce a dual-stage heuristic coordinated reinforcement learning approach that addresses these challenges by integrating a feature-guided state dimensionality stabilization proximal policy optimization method and a heuristic coordinated assignment method. Our approach offers several practical benefits for E-taxi service providers, drivers, and passengers. For E-taxi service providers, the proposed method improves E-taxi dispatch efficiency, resulting in a more effective use of available resources and potentially increasing overall revenue. For E-taxi drivers, our approach leads to better planning of charging and passenger pick-up decisions, increasing their earnings by 20% compared to traditional methods, and reducing the average occurrence of low battery status from more than 4 times every 10 hours to less than 1 time. Passengers, on the other hand, experience improved service quality due to the more efficient E-taxi management. The probability of their orders being answered increases by 15%, and their waiting time is reduced by 100%. These improvements contribute to an enhanced user experience and may encourage further adoption of E-taxis as a sustainable transportation solution. The proposed method can be integrated into existing E-taxi hailing platforms, such as DiDi and Uber, to enhance their dispatch and charging management capabilities. As the global trend towards sustainable transportation continues to grow, our approach provides valuable insights and a practical solution for the efficient management of E-taxi fleets in modern urban environments.

The rise in urbanization and the demand for refined oil pose a significant challenge in improving transportation efficiency while ensuring the safety of gas station inventories. Traditional optimization methods, which rely on deterministic models or demand forecasting, enhance efficiency but struggle with uncertainties such as demand fluctuations. Multi-agent reinforcement learning (MARL) presents a promising approach for adaptive cooperative dispatching. This study models the refined oil dispatch problem as a partially observable constrained Markov game (CPOMG), in which agents make cooperative decisions under inventory constraints with limited observation, and proposes Hierarchical Action-Constrained MAPPO (HAC-MAPPO), a novel cooperation MARL algorithm to find the optimal cooperative dispatching policy of the game. Within the CPOMG framework, the HAC-MAPPO agents optimize order generation decisions based on the states of gas station inventories. The subsequent delivery routing for these orders is optimized using an integrated fuel routing tabu search algorithm, which is part of the environment’s state transition logic. HAC-MAPPO incorporates inventory safety thresholds to constrain order generation decisions across fuel types. It employs a dual-critic architecture and a shared hierarchical actor network for efficient decentralized policy learning. Crucially, the Lagrangian relaxation technique is applied to transform the constrained actor optimization objective (for order generation) into an unconstrained form. Evaluations carried out in real-world scenarios (based on actual geographic coordinates) and synthetic scenarios using multiple types of synthetic fuel consumption datasets demonstrate that the proposed model and algorithm significantly reduce inventory violations and improve dispatch efficiency and safety under varying demand patterns, outperforming baseline methods.

Dynamic systems in practice are often nonlinear and subject to complex disturbances such as sensor outliers, which invalidate the Gaussian noise assumptions underlying conventional Kalman filters. To address the limitations of conventional Kalman filters under such conditions, recent studies have explored nonlinear filters based on the maximum correntropy criterion (MCC), which exploit both second-order and higher-order moments of the innovation for enhanced robustness. However, their performance is highly sensitive to the choice of kernel scale, and existing strategies—whether fixed offline or empirically adjusted—fail to adapt to unknown disturbances of varying intensities. This paper proposes a novel unsupervised adaptive maximum correntropy unscented Kalman filter (AMCUKF), which introduces three major contributions: (1) Online Kernel Scale Adaptation—the kernel scale is modeled as a latent random variable governed by an inverse Gamma distribution, and variational Bayesian inference is applied to jointly estimate the system state and kernel scale in a recursive manner; (2) Theoretical Consistency and Closed-Form Solutions—the algorithm maintains consistency with traditional MCUKF but achieves dynamic adaptability through the conjugacy of Gaussian and inverse Gamma priors, enabling efficient closed-form updates; and (3) Validated Performance under Non-Gaussian Noise—comprehensive tests on three representative scenarios, including nonlinear numerical systems with stationary/non-stationary t-distributed and Cauchy measurement noise, a single-target tracking task, and real-world battery state-of-charge (SoC) estimation using the public LG 18650 HG2 dataset. Compared with UKF, EMCUKF and FMCUKF, the proposed AMCUKF achieves about 20–60% lower RMSE and 30–70% lower worst-case error across heavy-tailed noise simulations, while maintaining comparable computational cost and convergence within 1–3 iterations. In the real-world SoC estimation task, it further reduces the maximum estimation error from over 7% to below 1%, demonstrating strong practical robustness.

Nowadays location-based service (LBS) has become an important service in people’s daily life. Online taxi-hailing system (DiDi, Uber, etc.) is one of the most common LBS system. As the scale of online taxi services continues to expand, some problems have gradually emerged. Specifically, the uncertainty of taxis and passengers makes it difficult to match them effectively, and the passengers’ location privacy will be threatened from both internal and external of the system. In this article, we first proposed a Bayesian-based location privacy inference attack method from external adversary’s point of view. After that, an adaptive differential privacy-based dynamic incentive method was proposed. First, an adaptivity clocking area division method was proposed to resist the internal privacy threat. Second, a dynamic incentive bidding method was proposed to deal with the trading issue. Third, an exponential-based matching method was proposed to resist the external inference privacy threat. Further, theoretical proofs show that the proposed method satisfy the privacy properties of$k$-anonymity,$2~\mu _{1}\varepsilon $-differential privacy, and the economic properties of incentive compatibility, individual rationality. Finally, the experimental results show that the inference attack would achieve a maximum attack success rate of 95% while ensuring the accuracy within 150 m, and the proposed adaptive differential privacy-based dynamic incentive method can not only provide a good economic performance in terms of satisfaction ratio, social welfare, and travel distance, but also can resist internal privacy threat with less than 1% of privacy leakage probability and reduce the success rate of external privacy inference attacks to 25%.

Microgrids (MGs) are essential components of a smart grid, and they play an important role in assisting smart grid operation. Furthermore, multi-MG systems, which can further improve smart grid operation, raise significant challenges in terms of management and interoperation because of their increased complexity. To address this, we propose a multi-bid online incentive scheme for energy trading in multi-MG systems, in which MGs act as buyers and sellers to trade surplus energy. In the proposed system, buyer MGs can submit multiple bids to different sellers, and the distributed MG system operator matches buyer and seller MGs in a way that maximizes the utility of buyers. Theoretical analysis demonstrates that the proposed method meets the properties of incentive compatibility and individual rationality. Moreover, through detailed performance evaluation, the proposed scheme was capable of achieving better performance in comparison with existing schemes regarding utility, energy purchasing cost, and buyer MG’s satisfaction ratio. Our experimental results also demonstrated that the proposed scheme could shift the system peak-load, improving the robustness of the multi-MG system. Note to Practitioners—In this article, we addressed the energy trading issues in multi-microgrid (MG) system, and proposed a distributed incentive mechanism to achieve more flexible and efficient energy allocation. Most existing works on energy trading use centralized methods, which means all electrical users will submit only one bid to the platform, and finally the platform decides to match the trading parties. This will lead to users unable to actively choose trading partners, thus reducing the volume of the whole market. To this end, in this paper, we proposed a distributed incentive mechanism for the multiple microgrid system, in which the buyer microgrids are able to submit multiple bids to different sellers. Moreover, each seller microgrids act as a distributed energy market, and the distributed MG system operator matches the buyers and sellers which aims to maximize the utility of buyers. The proposed scheme is helpful in managing the energy trading among the multi-MG system and can be readily implemented in the real-world large-scale electricity trading market.

With the continuous expansion of the power grid nodes scale, traditional centralized state estimation method shows certain limitations in estimation efficiency and accuracy. Recently, some power grids adopt a distributed state estimation method, in which each partition independently estimates the partial state information by partitioning the entire power system. However, the deviation of the state estimation in certain partition will result in the deviation of the estimation results in the entire power grid system. In this paper, we propose the attack strategy against the distributed state estimation in smart grids from two perspectives, i.e. the attack against local physical measurement value of the power system partition and the attack against the measurement value of coordination center. Moreover, the theoretical analysis of the state estimation deviation caused by the proposed data integrity attack and the propagation processes of proposed attack vectors in measurement calculations are formalized. The effectiveness of the proposed attack strategy is verified in the IEEE-30 bus and IEEE-118 bus systems. Simulation results show that attacking against a certain partition of a distributed system can indirectly affect the state estimation results of other partitions and attacking against the measurement value of coordination center can directly threaten the state estimation results of the entire power grid. Note to Practitioners— This paper proposes two attack strategies against the distributed state estimation of power grid from two perspectives, i.e. the attack against local physical measurement value of the power system partition and the attack against the measurement value of coordination center. Most of the previous works fail to formalize the state estimation deviation of both partial and entire state estimation results of power grid after the attacker launches the attack against the distributed state estimation. We formalize the state estimation deviation caused by the proposed data integrity attack and the propagation processes of proposed attack vectors in measurement calculations. The effectiveness of the proposed attack strategy against the state estimation of power grid is verified in the IEEE-30 bus and IEEE-118 bus systems. Simulation results show that attacking against a certain partition of power grid can indirectly cause the deviation in the state estimation results of entire power grid and attacking against the measurement value of the coordination center can directly threaten the state estimation results of the entire power grid. In conclusion, the proposed attack strategies are helpful for the research community to design detection strategies in a targeted manner and can be conveniently applied to the real-world security management system of smart grid.

Smart meters, as a key component of Advanced Metering Infrastructure (AMI), collect fine-grained electricity consumption data for demand response in smart grids. While this data improves the grid’s accuracy, it also poses significant threats to users’ privacy. In this paper, we study the privacy issue in smart metering systems from both attacker and defender perspectives. First, we propose an improved Temporal Convolutional Network (TCN) based Non-Intrusive Load Monitoring (NILM) attack method, which infers electrical appliance usage from public load curves, addressing the gradient vanishing, gradient exploding, and other problems while improving attack accuracy. Second, we develop a rechargeable battery-assisted energy management system to hide load characteristics of electrical appliances by adding physical noise, thus resisting NILM attacks. To address the privacy-cost trade-off optimization problem, we propose a Practical Deep Reinforcement Learning-based Rechargeable Battery assist Privacy Preserving Method (PRoP) that learns optimal battery charging/discharging policies. We design a novel privacy measurement method and constraints to ensure the feasibility of system deployment and prove PRoP’s effectiveness in resisting NILM attacks. Comprehensive evaluations demonstrate that our improved TCN-based NILM method achieves an attack success rate of over 80% on various electrical appliances, improving attack performance (MAE, RMSE) by 20% compared to existing methods while reducing model training time. Moreover, our proposed PRoP achieves a better trade-off between privacy protection and electricity cost than existing battery-assisted methods, reducing costs by 5% and the attack success ratio to 36%, while increasing the MAE and RMAE obtained by NILM by 3 times.Note to Practitioners—Smart grid, which can support bidirectional information transmission, has a series of advantages, such as high efficiency and high stability. However, it also brings a significant threat to users’ electricity privacy. Although encryption-based privacy protection methods have been deployed on terminal devices of smart grids to prevent privacy leaks, this method can often only defend against intrusive attacks and has little effect on non-intrusive attacks. To this end, this paper studies the privacy issues caused by non-intrusive attacks. Specifically, to better study the protection method, we first investigate the attack mechanism and design an improved TCN-based Non-Intrusive Load Monitoring method. Then, we propose a Practical Reinforcement learning-based rechargeable battery-assisted Privacy preserving method (PRoP) to defend against this attack physically. The most practical contribution of this paper is that, compared with existing battery-assisted privacy protection methods, we do not blindly pursue algorithm performance but fully consider the practical factors of deployment, such as limiting battery capacity and constraining battery charging and discharging behavior. This can guide practitioners to better apply this technology in practice.

Due to high convenience and efficiency, electronic auction technology has been developed rapidly and has been applied to many online trading market applications. As more attention has been paid to information security, the privacy issues in the electronic auction have been widely studied. Differential privacy, as a lightweight privacy protection method, is an important direction in privacy preserving auction mechanism designing. However, most of the existing researches on differential privacy-based auction mechanism have not proposed a theoretical privacy inference attack method against the auction market. Therefore, the existence of privacy attacks is questionable, and the necessity and privacy protection performance of the existing differential privacy auction mechanism cannot be verified. To this end, in this paper we addressed the privacy attack issue and privacy protection issue in the auction market simultaneously. First, a Bayesian-based inference attack method against the double auction market was proposed from the perspective of the adversary. Theoretical analysis and evaluation results showed that the proposed inference attack method can effectively infer the bidding information of the target bidders, and attack success rate achieved approximately 95%. Second, an individual differential privacy-based auction mechanism was proposed from the perspective of the auction platform. Since not all the bidders will be attacked, we introduced the concept of individual differential privacy to provide targeted defense for specific bidders. Theoretical analysis demonstrated that the proposed auction mechanism satisfies$2\varepsilon $-individual differential privacy. And the extensive evaluation results showed that, compared with the existing differential privacy-based auction mechanism, our proposed mechanism provided the best privacy protection performance, that is, reduced the attack success rate to 20%, and ensured better auction performance, such as social welfare and satisfaction ratio, than the other mechanisms. Note to Practitioners—In this paper, we addressed the non-invasive privacy issues in the widely used electronic auction mechanism. Most of the previous works focused on designing differential privacy based auction mechanism against the non-invasive privacy attack, but neglecting the principle of non-invasive privacy attack methods. This makes it impossible to verify the privacy protection effectiveness of their proposed mechanisms. For this reason, a very large privacy budget may be selected to ensure the efficiency of privacy protection, which will lead to poor auction performance. To this end, we first proposed a Bayesian-based inference attack method against the double auction market. The proposed inference attack method allows the adversary infer the bidders’ private bidding information by the public auction results. Moreover, we then proposed an individual differential privacy auction mechanism, which aimed to achieve effective privacy protection while minimizing the added noise, thereby improving auction performance. The experiments demonstrate that the proposed Bayesian-based inference attack method achieves a good attack successful rate, and the proposed individual differential privacy auction mechanism will achieve the better efficiency of privacy protection as well as auction performance comparing with the exist differential privacy-based auction mechanism. In conclusion, this paper provides a verification method for the future research on the privacy protection of electronic auction mechanism. Meanwhile, this paper proposes an efficient privacy protection auction mechanism, which can be used in various trading scenarios.

An efficient energy scheduling strategy of a charging station is crucial for stabilizing the electricity market and accommodating the charging demand of electric vehicles (EVs). Most of the existing studies on energy scheduling strategies fail to coordinate the process of energy purchasing and distribution and, thus, cannot balance the energy supply and demand. Besides, the existence of multiple charging stations in a complex scenario makes it difficult to develop a unified schedule strategy for different charging stations. In order to solve these problems, we propose a multiagent reinforcement learning (MARL) method to learn the optimal energy purchasing strategy and an online heuristic dispatching scheme to develop a energy distribution strategy in this article. Unlike the traditional scheduling methods, the two proposed strategies are coordinated with each other in both temporal and spatial dimensions to develop the unified energy scheduling strategy for charging stations. Specifically, the proposed MARL method combines the multiagent deep deterministic policy gradient (MADDPG) principles for learning purchasing strategy and a long short-term memory (LSTM) neural network for predicting the charging demand of EVs. Moreover, a multistep reward function is developed to accelerate the learning process. The proposed method is verified by comprehensive simulation experiments based on real data of the electricity market in Chicago. The experiment results show that the proposed method can achieve better performance than other state-of-the-art energy scheduling methods in the charging market in terms of the economic profits and users’ satisfaction ratio.

Sharing private parking spaces during their idle time periods has shown great potential for addressing urban traffic congestion and illegitimate parking problems in smart cities. In this article, aiming to address the online parking-space sharing issue while ensuring the privacy of customer parking destination locations, we propose a novel destination privacy-preserving online parking sharing (DPOPS) incentive scheme. In particular, the online parking-space sharing problem is formalized as a social welfare maximization problem in a two-sided market, where parking-space providers (PSPs) and customers are regarded as sellers and buyers. Then, novel threshold value-based rules are designed to determine winners, payments, and reimbursement. Finally, winners are matched by solving a mixed-integer nonlinear programming problem, aiming to minimize the distance between customer’s destination and allocated parking space. In addition, the location privacy of the customers’ destinations is protected by the Laplace mechanism. We prove that DPOPS achieves several economically effective properties and approximate differential privacy. We analyze the upper bound of the efficiency loss of our scheme. Extensive evaluation results demonstrate that our scheme can not only achieve good performance regarding social welfare, PSP satisfaction ratio, privacy preservation, and computation overhead but also leads to shorter travel distances for customers comparing to the baseline scheme.Note to Practitioners—In this article, we address the online parking-space sharing issue with considering the parking-space providers (PSPs) and customers’ individual utility while preserving the location privacy of customers’ destinations. Most of the previous works focused on designing a centralized mechanism for allocating parking spaces without considering the protection of the customers’ location privacy. In particular, we propose an online parking-space sharing scheme called DPOPS, including a novel threshold value-based winner determination rule and a parking-space allocation rule. The proposed scheme DPOPS allows the PSPs and customers submit their bids and asks according to their own willingness and is able to improve the utilization of private parking spaces during their idle time periods. Moreover, the location privacy of customers’ destinations is protected by the Laplace mechanism. The experiments demonstrate that the proposed approach outperforms the exponential-based scheme in terms of PSP satisfaction ratio and the travel distance for parking-space customer. The proposed scheme is helpful in managing the vacant parking space in a competitive market and can be readily implemented in the real-world online parking-space sharing systems.

A smart grid integrates advanced sensors, efficient measurement methods, progressive control technologies, and other techniques and devices to achieve safe, efficient and economical operation of the grid system. However, the diversified and open environment of a smart grid makes energy and information of the smart grid vulnerable to malicious attacks. As a representative cyber-physical attack, the data integrity attack has an extremely severe impact on the grid operation for it can bypass the traditional detection mechanisms by adjusting the attack vector. In this paper, we first present the attack strategy against dynamic state estimation of power grid in the perspective of adversary and formulate the data integrity attack detection problem that has the characteristic of sequential decision making as a partially observable Markov decision process. Then, a deep reinforcement learning-based approach is proposed to detect against data integrity attacks, which utilizes the Long Short-Term Memory layer to extract the state features of previous time steps in determining whether the system is currently under attack. Moreover, the noisy networks are employed to ensure effective agent exploration, which prevents the agent from sticking to the non-optimal policy. The principle of a multi-step learning is adopted to increase the estimation accuracy of Q value. To address the sparse rewards problem, the prioritized experience replay is proposed to increase training efficiency. Simulation results demonstrated that the proposed detection approach surpasses the benchmarks in the comparison metrics: delay error rate and false rate.Note to Practitioners—In this paper, we present a deep reinforcement learning-based algorithm to defend against the data integrity attacks of smart grid. Most of the previous works discretized the system states and utilized the current state information to identify whether the system is under attack. For this reason, the detection policy may totally ignored the continuously changing characteristics of the grid states, which will lead to poor detection performance. Moreover, the attacked system states only accounts for a small part of the entire grid operation states, the probability of sampling the experience containing the attack state is extremely small, which limits the learning efficiency of previous RL-based detection approaches. In order to increase the accuracy of detection, we first present the attack strategy against power grid’s dynamic state estimation in the perspective of adversary and formulate the partially observable Markov decision process model of attack detection problem. Moreover, we propose a deep reinforcement learning-based detection approach combining the LSTM network to extract the system state features of the previous time steps to determine whether the system is currently being attacked. To address the sparse rewards problem, the prioritized experience replay is used to increase learning efficiency. The experiments demonstrate the effectiveness of proposed detection scheme compared with benchmarks in terms of detection delay as well as accuracy. In conclusion, the proposed detection scheme is helpful in defending against the data integrity attacks without obtaining the opponent’s strategy in advance and can be conveniently applied to the real-world security management system of smart grid.

The rapid and wide adoption of microgrids (MGs) and the increasing popularity of electric vehicles (EVs) have created a unique opportunity for the integration of these technologies. In this article, we address the issue of demand response of EVs during MG outages by leveraging Vehicle-to-Grid (V2G) technology. Particularly, we investigate an auction trading market that allows EVs with surplus energy to act as sellers, and EVs that want to be charged to act as buyers. A novel distributed double auction scheme is proposed to allow each buyer EV to submit multiple bids to seller EVs in different parking lots. Nonetheless, the locations of buyer EVs could be inferred by an adversary through analyzing the valuations, posing serious privacy and security risks. In this regard, a valuation-based attack scheme is investigated to validate the potential privacy risk. To defend against such an attack, we present a location privacy-preserving double auction scheme, in which the MicroGrid Central Controller (MGCC) acts as the auctioneer, solving the social welfare maximization problem of matching buyers to sellers, and the cloud is used to conduct calculations for the auctioneer, protecting the privacy of participants via homomorphic encryption. Theoretical analysis is conducted to validate our auction scheme in satisfying the designed economic and privacy properties (e.g., strategy-proofness and$k$-anonymity). The experimental results show that our auction scheme can not only mitigate the demand response problem in MGs, but also provides good performance with respect to social welfare, satisfaction ratio, computational and communication overhead, and privacy leakage.

The existing multi-agent reinforcement learning methods (MARL) for determining the coordination between agents focus on either global-level or neighborhood-level coordination between agents. However the problem of coordination between individual agents is remain to be solved. It is crucial for learning an optimal coordinated policy in unknown multi-agent environments to analyze the agent's roles and the correlation between individual agents. To this end, in this paper we propose an agent-level coordination based MARL method. Specifically, it includes two parts in our method. The first is correlation analysis between individual agents based on the Pearson, Spearman, and Kendall correlation coefficients; And the second is an agent-level coordinated training framework where the communication message between weakly correlated agents is dropped out, and a correlation based reward function is built. The proposed method is verified in four mixed cooperative-competitive environments. The experimental results show that the proposed method outperforms the state-of-the-art MARL methods and can measure the correlation between individual agents accurately.

Electric vehicle (EV) has become one of the most critical components in the smart grid with the applications of the Internet-of-Things (IoT) technologies. Real-time charging control is pivotal to ensure the efficient operation of EVs. However, the charging control performance is limited by the uncertainty of the environment. On the other hand, it is challenging to determine a charging control strategy that is able to optimize multiple objectives simultaneously. In this article, we formulate the EV charging control model as a Markov decision process (MDP) by constructing state, action, transition function, and reward. Then, we propose a deep-reinforcement-learning-based approach: charging control deep deterministic policy gradient (CDDPG) to learn the optimal charging control strategy for satisfying the user's requirement of battery energy while minimizing the user's charging expense. We utilize the long short-term memory (LSTM) network that extracts the information of previous energy price to determine the current charging control strategy. Moreover, Gaussian noise is added to the output of the actor network to prevent the agent from sticking into the nonoptimal strategy. In addition, we address the limitation of sparse rewards by using two replay buffers, of which one is used to store the rewards during the charging phase and another is used to store the rewards after charging is completed. The simulation results prove that the CDDPG-based approach outperforms the deep-$Q$ -learning-based approach (DQL) and the deep-deterministic-policy-gradient-based approach (DDPG) in satisfying the user's requirement for the battery energy and reducing the charging cost.

In this paper, to address the issue of demand response in the smart grid with island MicroGrids (MGs), we introduce an effective and secure auction market that allows electric vehicles (EVs) having surplus energy to act as sellers, and the EVs having insufficient energy in the island MGs to act as buyers. There are two primary challenges in designing an effective auction market in the smart grid. First, the auction market scheme shall be online, allowing buyers and sellers to enter the market at any time, and satisfy several critical economic properties (individual rationality, incentive compatibility, and so on.). Second, the sensitive information of participants shall be protected in the auction process. To address these challenges, we present a novel privacy-preserving online double auction scheme based on differential privacy. In our auction market, the MicroGrid Center Controller (MGCC) acts as the auctioneer, aiming at solving the social welfare maximization problem to match buyers and sellers. The principle of differential privacy is leveraged to protect the privacy of EVs' sensitive bidding information. Via theoretical analysis, we demonstrate that our designed auction scheme satisfies both economic and privacy-preserving properties, including individual rationality, incentive compatibility, weak budget balance, and ε-differential privacy. We conduct an extensive performance evaluation to measure the effectiveness of our proposed scheme. Our experimental results show that the proposed auction scheme can not only ensure the privacy of participants but also effectively facilitates demand response in the smart grid, with respect to social welfare, satisfaction ratio, social efficiency, and computational overhead.

In this paper, we address the issue of demand response (DR) in microgrids via vehicle-to-vehicle technology in the smart grid with consideration for location privacy protection supported by Internet of Vehicles. To enable effective DR, the online double auction is a viable approach to support energy trading between electric vehicles (EVs) that have surplus or insufficient energy, while the utility of each participant can be considered. Nonetheless, there are three primary challenges in designing such an online double auction approach. First, as EVs are allowed to enter the market at any time, the auctioneer should make the best decision without further information about bids and asks. Second, as EVs are allowed to enter the market in different places, the auctioneer should perform routing optimization for EV charging after determining the winner. Third, there is a risk of leakage in the location of EVs that needs to be protected. To tackle these issues, we present a new truthful online double auction scheme, which features multiunit energy trading among EVs, routing optimization for EV charging, and location privacy protection. We conduct a theoretical analysis and demonstrate that our online double auction scheme is capable of achieving several important economic properties as well as the privacy guarantee (i.e., k-anonymity). Our experimental results show that the proposed scheme can achieve good performance with respect to social welfare, satisfaction ratio, total profit of EV owners, peak load shifting, state of charge, driving distance satisfaction, and computing time, and can further ensure location privacy protection.

In this paper, we address the issue of the energy trading in the scenario of electric vehicles (EVs) charging in the smart grid. The EVs energy trading problems have attracted growing attention with the popularity of EVs. As the traditional first-reserve-first-serve scheme in the energy trading market impairs the benefits of both buyers and seller, we consider an auction scheme, called progressive second price (PSP), which has been proved to be an efficient way to conduct resource allocation in the trading market. Compared with other auction schemes, the PSP scheme can achieve both incentive compatibility and Nash equilibrium, which are important properties for the market. Nonetheless, the PSP auction scheme is not designed for online auction and it cannot guarantee that the seller can provide an enough number of charging piles to satisfy the demand of winners. To tackle these issues, in this paper we propose a novel online continuous PSP-based auction scheme, which is capable of not only achieving the property of online energy trading but also guaranteeing that the number of winners is limited to be no more than the number of charging piles. Further, we prove that our auction scheme achieves incentive compatibility and Nash equilibrium. The extensive experimental results demonstrate that our auction scheme achieves good performance with respect to social welfare, the seller satisfaction ratio, the buyer satisfaction ratio, as well as computation overhead.

With the development of online taxi-hailing systems (DiDi, Uber Lyft, etc.), how to effectively allocate taxis has attracted great attention in the recent past. Meanwhile, with the rapid increase of taxi-related crimes, the privacy of passengers' sensitive information such as location remains a critical concern. In this paper, we present a novel incentive-based scheme, which provides the differential privacy guarantee for passengers' locations in taxi-hailing systems. To be specific, to allocate limited taxis to passengers, we first present the Vickrey-Clarke-Groves (VCG)-based online auction mechanism for determining the winning passengers. Then, to match the taxis and winning passengers as well as to protect the location privacy of passengers, we present the new allocating rule based on the exponential differential privacy-based mechanism. Further, we prove that the proposed incentive-based scheme satisfies both economic properties and 2-ε differential privacy guarantee. Finally, we evaluate the performance of our proposed scheme. The experimental data confirms that our proposed scheme not only achieves better performance than the two baseline schemes with respect to social welfare and satisfaction ratio, but also is capable of protecting the location privacy of passengers with low privacy disclosure.

In this paper, we present theory and a design of the online double auction for the trading of energy within a smart grid with microgrids (MGs). The online double auction has the potential to enable the allocation of surplus electricity to the MGs that need electricity with the highest gain in the real-time market. Nonetheless, two critical issues remain challenging when designing an effective online double auction scheme in such a system. First, as the agents are allowed to arrive and depart at any time, the auctioneer needs to make decisions without the information of further bids and asks. Second, the economic properties of strategy-proof, individual rational, and (weak) budget balance should be satisfied. To address these issues and enable multiunit electricity trading among local MGs, in this paper, we propose a strategy-proof online double auction (SODA) scheme, in which the surplus and insufficient MGs in the system are treated as sellers and buyers, respectively, and the MG center controller is capable of maximizing the social welfare of MGs by appropriately matching buyers and sellers. Via theoretical analysis, we prove that SODA can achieve the properties of individual rationality, (weak) budget balance, strategy-proofness, and computational efficiency. Experiments also show that SODA is capable of reducing the energy purchasing cost of the MGs and shifting the peak-load, while achieving great performance with respect to social welfare, seller/buyer satisfaction ratio, social efficiency, and computation overhead.

In this paper, we investigate the data integrity attack against Optimal Power Flow (OPF) with the least effort from the adversary's perspective. The investigated attack can first select the minimum number of target nodes to compromise by analyzing the difference between the capacity of transmission line and the real transmission power, and then search for a critical attack vector (with a goal to minimize the amount of information to manipulate) as an optimal attack strategy. To defend against such an attack, we develop the defensive scheme by protecting the critical nodes. Based on various IEEE standard systems, we show the effectiveness of our investigated attack scheme and the corresponding defense schemes.

In this paper, we address the issue of data trading in big data markets. Data trading problems have attracted increased attention recently, as the economic benefits and potential of big data trading are substantial and varied. However, how to effectively trade data between the data owners (sellers) and data collectors/users (buyers) is far from settled, and requires careful design. Auction mechanisms have been applied across many fields, and have significant potential to facilitate data transactions in a fair, truthful, and secure way. Nonetheless, a truthful auction must ensure the property of incentive compatibility, meaning that the bidders can obtain highest utility if and only if they submit their bids and asks truthfully. Furthermore, a truthful and fair auction should also protect the optimal auction results from being manipulated by false-name bidding attacks, where users (participants) utilize multiple identities or accounts to influence the auction results. To tackle these issues, we propose a Multi-round False-name Proof Auction (MFPA) scheme, which enables data trading among data owners (sellers) and data collectors (buyers). We prove that our MFPA scheme achieves the properties of incentive compatibility, false-name bidding proofness, and computational efficiency. The experimental results demonstrate that MFPA achieves good performance in terms of social surplus, satisfaction ratio, and computation overhead.

In this paper, we address the problem of demand response of electrical vehicles (EVs) during microgrid outages in the smart grid through the application of Vehicle-to-Grid (V2G) technology. Particularly, we present a novel privacy-preserving double auction scheme. In our auction market, the MicroGrid Center Controller (MGCC) acts as the auctioneer, solving the social welfare maximization problem of matching buyers to sellers, and the cloud is used as a broker between bidders and the auctioneer, protecting privacy through homomorphic encryption. Theoretical analysis is conducted to validate our auction scheme in satisfying the intended economic and privacy properties (e.g., strategy-proofness and k-anonymity). We also evaluate the performance of the proposed scheme to confirm its practical effectiveness.

Microgrids (MGs) have attracted growing attention due to self-sufficiency and self-healing properties. Nonetheless, the intermittent nature and uncertainty of distributed energy resources and load demands remain challenging issues in balancing demands and managing energy resources in MGs. Existing research efforts mainly focus on developing techniques to enable interactions between local MGs and the utility grid, which leads to high line power losses and operation costs. In this paper, we present the Sto2Auc framework to address the issue of stochastic optimal bidding problem for a system with MGs. First, the optimal bidding problem is formulated as a two-stage stochastic programming process, which aims to minimize the system operation cost and obtain optimal energy capacity of MGs by the MG center controller (MGCC). Uncertainties arise from both energy supply and demand, which are considered in the stochastic model, and random parameters representing those uncertainties are captured by using the Monte Carlo method. Second, to enable optimal electricity trading between the insufficient and surplus MGs, we propose a distributed double auction (DDA)-based scheme, which is proven to converge to the optimal social welfare of the system with MGs, and achieves the economical properties of being strategy-proof, individually rational, and (weak) budget balanced. Extensive experiments on an MG system composed of IEEE-33 buses demonstrate the effectiveness of proposed scheme. The experimental results show that Sto2Auc framework is capable of reducing the operational cost of MG systems, while the implemented DDA scheme achieves good performance with respect to social welfare, demand insufficiency, and MGCC profit.

In this paper, we address the security issue of optimal power flow (OPF) (as a key component in the smart grid). To be specific, we investigate the data integrity attack against OPF with the least effort from the adversary's perspective, and propose effectively defense schemes to combat the data integrity attack, with respect to the number of nodes to compromise and the amount of information to manipulate. The investigated attack can first select the minimum number of target nodes to compromise by analyzing the difference between the capacity of transmission line and the real transmission power, and then search for a critical attack vector as an optimal attack strategy. To defend against such an attack, we develop the defensive schemes by not only protecting the critical nodes but also detecting the existence of attacks based on false measurement detection schemes. Based on various IEEE standard systems, we show the effectiveness of our investigated attack scheme and the corresponding defense schemes. The experimental results show that the discovered compromised nodes and critical attack vector could lead to the increase of the fuel cost from the power generation by compromising the least number of nodes and injecting the least amount of false information, in comparison with the random attack as the baseline attack strategy. In addition, our two developed defensive schemes are capable of making OPF resilient to the data integrity attack via protecting critical nodes and identifying the falsified measurements accurately in the system.

State estimation plays a critical role in self-detection and control of the smart grid. Data integrity attacks (also known as false data injection attacks) have shown significant potential in undermining the state estimation of power systems, and corresponding countermeasures have drawn increased scholarly interest. Nonetheless, leveraging optimal phasor measurement unit (PMU) placement to defend against these attacks, while simultaneously ensuring the system observability, has yet to be addressed without incurring significant overhead. In this paper, we enhance the least-effort attack model, which computes the minimum number of sensors that must be compromised to manipulate a given number of states, and develop an effective greedy algorithm for optimal PMU placement to defend against data integrity attacks. Regarding the least-effort attack model, we prove the existence of smallest set of sensors to compromise and propose a feasible reduced row echelon form (RRE)-based method to efficiently compute the optimal attack vector. Based on the IEEE standard systems, we validate the efficiency of the RRE algorithm, in terms of a low computation complexity. Regarding the defense strategy, we propose an effective PMU-based greedy algorithm, which cannot only defend against data integrity attacks, but also ensure the system observability with low overhead. The experimental results obtained based on various IEEE standard systems show the effectiveness of the proposed defense scheme against data integrity attacks.

Islanded Microgrids (IMG) have attracted much attention in the research and development of the smart grid, which is a large-scale distributed cyber-physical system. To overcome the limitations of the single IMG on energy efficiency and economic, In this paper, we first proposed a novel self-sufficient system, namely “Archipelago microgrid (MG)”, which is comprised of multi-microgrids while disconnected with the utility grid. We formalized the EV (electric vehicle) penetration problem as an optimization mixed integer nonlinear programming, which aims to minimize the emission and operation cost in the system. To enable a reasonable deployment of EV in each MGs, we developed two scheduling schemes, namely Unlimited Coordinated Scheme (UCS) and Limited Coordinated Scheme (LCS), respectively. A decentralized algorithm was also developed to solve the optimization model in LCS. A simulation study based on an modified IEEE-9 bus system with three MGs show that our proposed schemes can reduce both the environmental pollution created by CO2 emission and operation cost. Especially, with the consideration of peak load limits and resident preferences, the LCS scheme can obtain better results than the UCS scheme, leading to the reduction of the environmental pollution by 15.2% raised by CO2 emission, as well as the total cost by 10.8% in the system.

In this paper, we addressed the issue of a stochastic optimal bidding problem for a system with microgrids (MGs). The optimal bidding problem is formulated as a two-stage stochastic programming process, which aims to minimize the system operation cost and to expand energy interactions among local MGs that are geographically close. Uncertainties come from both energy supply and demand sides (e.g., wind, solar, and load demand) are considered in the stochastic model and random parameters to represent those uncertainties are captured by using the Monte Carlo method. To enable an optimal electricity trading between local MGs, we presented two bidding schemes: (i) Cournot equilibrium based Dynamic Backtrack Energy Trading (DBET), and (ii) double auction based Dual Decomposition Auction (DDA). Experimental results on an IEEE-33 bus based system with MGs were presented to show the effectiveness of our proposed schemes. Experimental results show that our proposed bidding schemes can reduce the operation cost of the system, while the DDA scheme achieves better performance in terms of system social welfare than the DBET scheme.

It is critical for a power system to estimate its operation state based on meter measurements in the field and the configuration of power grid networks. Recent studies show that the adversary can bypass the existing bad data detection schemes, posing dangerous threats to the operation of power grid systems. Nevertheless, two critical issues remain open: 1) how can an adversary choose the meters to compromise to cause the most significant deviation of the system state estimation, and 2) how can a system operator defend against such attacks? To address these issues, we first study the problem of finding the optimal attack strategy--i.e., a data-injection attacking strategy that selects a set of meters to manipulate so as to cause the maximum damage. We formalize the problem and develop efficient algorithms to identify the optimal meter set. We implement and test our attack strategy on various IEEE standard bus systems, and demonstrate its superiority over a baseline strategy of random selections. To defend against false data-injection attacks, we propose a protection-based defense and a detection-based defense, respectively. For the protection-based defense, we identify and protect critical sensors and make the system more resilient to attacks. For the detection-based defense, we develop the spatial-based and temporal-based detection schemes to accurately identify data-injection attacks.