We propose a deep stochastic actor-critic algorithm with an integrated network architecture and fewer parameters. We address stabilization of the learning procedure via an adaptive objective to the critic's loss and a smaller learning rate for the shared parameters between the actor and the critic. Moreover, we propose a mixed on-off policy exploration strategy to speed up learning. Experiments illustrate that our algorithm reduces the sample complexity by 50%-93% compared with the state-of-the-art deep reinforcement learning (RL) algorithms twin delayed deep deterministic policy gradient (TD3), soft actor-critic (SAC), proximal policy optimization (PPO), advantage actor-critic (A2C), and interpolated policy gradient (IPG) over continuous control tasks LunarLander, BipedalWalker, BipedalWalkerHardCore, Ant, and Minitaur in the OpenAI Gym.

Conventional MIMO precoding relying on a predefined codebook exhibits significantly poor performance in terms of bit error ratio (BER). This is because traditionally codebooks are designed relying on the simplifying assumption of having fixed channel probability distribution. However, owing to both hardware imperfections arising due to non-linear components and channel estimation imperfections, as well as channel aging, the fixed channel probability assumption is unrealizable in practical MIMO systems. Therefore, in this paper, we first propose a data-driven minimum BER (MBER) based precoder using deep reinforcement learning (DRL) without relying on channel model distribution. Then to facilitate the finite-rate MIMO feedback systems, we propose a data-driven user-specific codebook design for downlink users (UE), where UE populates MBER precoders and constructs the codebook by invoking an unsupervised clustering based quantizer. We demonstrate by simulations that our proposed precoder selection relying on data-driven codebook performs similarly to that of genie-aided channel state information (CSI), whilst our data-driven precoder design outperforms the conventional MIMO precoding by more than 5 dB for BER of 1$0^{-3}$ with CSI error variance of 0.02.

We study online privacy-preserving anomaly detection in a setting in which the data are distributed over a network and locally sensitive to each node, and a probabilistic data model is unknown. We design and analyze a data-driven solution scheme where each node observes a high-dimensional data stream for which it computes a local outlierness score. This score is then perturbed, encrypted, and sent to a network operator. The network operator then decrypts an aggregate statistic over the network and performs online network anomaly detection via the proposed generalized cumulative sum (CUSUM) algorithm. We derive an asymptotic lower bound and an asymptotic approximation for the average false alarm period of the proposed algorithm. Additionally, we derive an asymptotic upper bound and asymptotic approximation for the average detection delay of the proposed algorithm under a certain anomaly. We show the analytical tradeoff between the anomaly detection performance and the differential privacy level, controlled via the local perturbation noise. Experiments illustrate that the proposed algorithm offers a good tradeoff between privacy and quick anomaly detection against the UDP flooding and spam attacks in a real Internet of Things (IoT) network.

Conventionally, the user selection for non-orthogonal multiple access (NOMA) transmission is employed by techniques such as clustering, where disparities in channel and effective received power are exploited. However, these methods necessitate having channel state information (CSI) at the transmitter, which imposes significant overhead. Additionally, the successive interference cancellation (SIC) assisted detection at the receiver is sensitive to the CSI impairments which erodes the bit error ratio (BER) due to error propagation. In this paper, we propose a deep Q-network (DQN) based NOMA user selection by exploiting the users’ uplink received signal whilst also simultaneously circumventing the downlink CSI. Furthermore, in order to facilitate the NOMA detection of dynamic users in the cluster, we advocate a codebook design comprising of a plurality of deep neural network (DNN) weights for different numbers of users, where user performs the symbol detection without relying on the CSI. We demonstrate by simulations that our proposed DQN-assisted NOMA user selection despite its low search complexity on the order of $\mathcal{O}\left( 1 \right)$ performs similarly to that of the optimal user selection, where an exhaustive search of all possible user combinations with a complexity on the order of $\mathcal{O}\left( {{2^n}} \right)$ is performed. Additionally, we show by simulations that our DNN aided NOMA symbol detection performs close to that of the conventional SIC with perfect CSI and outperforms the conventional SIC with CSI impairments by more than 5 dB SNR at a BER of 10−4and Doppler of 1 kHz.

Timely detection of abrupt anomalies is crucial for real-time monitoring and security of modern systems producing high-dimensional data. With this goal, we propose effective and scalable algorithms. Proposed algorithms are nonparametric as both the nominal and anomalous multivariate data distributions are assumed unknown. We extract useful univariate summary statistics and perform anomaly detection in a single-dimensional space. We model anomalies as persistent outliers and propose to detect them via a cumulative sum-like algorithm. In case the observed data have a low intrinsic dimensionality, we find a submanifold in which the nominal data are embedded and evaluate whether the sequentially acquired data persistently deviate from the nominal submanifold. Further, in the general case, we determine an acceptance region for nominal data via Geometric Entropy Minimization and evaluate whether the sequentially observed data persistently fall outside the acceptance region. We provide an asymptotic lower bound and an asymptotic approximation for the average false alarm period of the proposed algorithm. Moreover, we provide a sufficient condition to asymptotically guarantee that the decision statistic of the proposed algorithm does not diverge in the absence of anomalies. Experiments illustrate the effectiveness of the proposed schemes in quick and accurate anomaly detection in high-dimensional settings.

Smart grid is a large complex network with a myriad of vulnerabilities, usually operated in adversarial settings and regulated based on estimated system states. In this study, we propose a novel highly secure distributed dynamic state estimation mechanism for wide-area (multi-area) smart grids, composed of geographically separated subregions, each supervised by a local control center. We first propose a distributed state estimator assuming regular system operation that achieves near-optimal performance based on the local Kalman filters and with the exchange of necessary information between local centers. To enhance the security, we further propose to 1) protect the network database and the network communication channels against attacks and data manipulations via a blockchain (BC)-based system design, where the BC operates on the peer-to-peer network of local centers, 2) locally detect the measurement anomalies in real-time to eliminate their effects on the state estimation process, and 3) detect misbehaving (hacked/faulty) local centers in real-time via a distributed trust management scheme over the network. We provide theoretical guarantees regarding the false alarm rates of the proposed detection schemes, where the false alarms can be easily controlled. Numerical studies illustrate that the proposed mechanism offers reliable state estimation under regular system operation, timely and accurate detection of anomalies, and good state recovery performance in case of anomalies.

For a safe and reliable operation of the smart grid, timely detection of cyber-attacks is of critical importance. Moreover, considering smarter and more capable attackers, robust detection mechanisms are needed against a diverse range of cyber-attacks. With these purposes, we propose a robust online detection algorithm for (possibly combined) false data injection and jamming attacks, that also provides online estimates of the unknown and time-varying attack parameters and recovered state estimates. Further, considering smarter attackers that are capable of designing stealthy attacks to prevent the detection or to increase the detection delay of the proposed algorithm, we propose additional countermeasures. Numerical studies illustrate the quick and reliable response of the proposed detection mechanisms against hybrid and stealthy cyber-attacks.

In this paper, online detection of false data injection attacks and denial of service attacks in the smart grid is studied. The system is modeled as a discrete-time linear dynamic system and state estimation is performed using the Kalman filter. The generalized cumulative sum algorithm is employed for quickest detection of the cyber-attacks. Detectors are proposed in both centralized and distributed settings. The proposed detectors are robust to time-varying states, attacks, and set of attacked meters. Online estimates of the unknown attack variables are provided, that can be crucial for a quick system recovery. In the distributed setting, due to bandwidth constraints, local centers can only transmit quantized messages to the global center, and a novel event-based sampling scheme called level-crossing sampling with hysteresis is proposed that is shown to exhibit significant advantages compared with the conventional uniform-in-time sampling scheme. Moreover, a distributed dynamic state estimator is proposed based on information filters. Numerical examples illustrate the fast and accurate response of the proposed detectors in detecting both structured and random attacks and their advantages over existing methods.

In this study, optimal channel switching (time sharing) strategies are investigated under average power and cost constraints in order to maximize the average number of correctly received symbols between a transmitter and a receiver that are connected via multiple additive Gaussian noise channels. The optimal strategy is shown to perform channel switching either among at most three channels with full channel utilization (i.e., no idle periods), or between at most two channels with partial channel utilization. In addition, it is stated that the optimal solution must operate at the maximum average power and the maximum average cost, which facilitates low-complexity approaches for calculating the optimal strategy. For two-channel strategies, an upper bound in terms of the noise standard deviations of the employed channels is provided for the ratio between the optimal power levels. Furthermore, a simple condition depending solely on the systems parameters is derived, under which partial channel utilization cannot be optimal. Numerical examples are presented to demonstrate the validity of the theoretical results.