Abnormal driving behavior is a leading cause of road accidents. Traditional detection methods, relying on classification or unsupervised learning, struggle with accuracy and generalization. Deep Reinforcement Learning (DRL) offers potential but faces challenges such as handling unlabeled and imbalanced data, designing effective reward functions, and ensuring efficient exploration. To address these, we propose an expert-guided DRL framework that integrates a CNN-BiLSTM-Self Attention (CBSA) model and an Isolation Forest (iForest) to guide Proximal Policy Optimization (PPO), enhancing detection accuracy and computational efficiency. Our framework consists of two stages. First, a deep learning model trained on labeled data provides expert guidance. Second, unlabeled data is processed through the pre-trained model and iForest, refining the DRL model via a sparse reward function to detect unknown anomalies while mitigating class imbalance and improving generalization. Experiments on an open-source dataset show our method out-performs baselines, achieving the highest recall (0.70) and Fl-score (0.63). Additionally, attention maps and anomaly heatmaps enhance interpretability, confirming its effectiveness for real-time abnormal driving behavior detection and improved driver safety.

Deep Reinforcement Learning (DRL) combines the perceptual capabilities of deep learning with the decision-making capabilities of Reinforcement Learning RL, which can achieve enhanced decision-making. However, the environmental state data contains the privacy of the users. There exists consequently a potential risk of environmental state information being leaked during RL training. Some data desensitization and anonymization technologies are currently being used to protect data privacy. There may still be a risk of privacy disclosure with these desensitization techniques. Meanwhile, policymakers need the environmental state to make decisions, which will cause the disclosure of raw environmental data. To address the privacy issues in DRL, we propose a differential privacy-based online DRL algorithm. The algorithm will add Gaussian noise to the gradients of the deep network according to the privacy budget. More important, we prove tighter bounds for the privacy budget. Furthermore, we train an autocoder to protect the raw environmental state data. In this work, we prove the privacy budget formulation for differential privacy-based online deep RL. Experiments show that the proposed algorithm can improve privacy protection while still having relatively excellent decisionmaking performance.

With the dramatic increase in malicious software, the sophistication and innovation of malware have increased over the years. In particular, the dynamic analysis based on the deep neural network has shown high accuracy in malware detection. However, most of the existing methods only employ the raw API sequence feature, which cannot accurately reflect the actual behavior of malicious programs in detail. The relationship between API calls is critical for detecting suspicious behavior. Therefore, this paper proposes a malware detection method based on the graph neural network. We first connect the API sequences executed by different processes to build a directed process graph. Then, we apply Bert to encode the API sequences of each process into node embedding, which facilitates the semantic execution information inside the processes. Finally, we employ GCN to mine the deep semantic information based on the directed process graph and node embedding. In addition to presenting the design, we have implemented and evaluated our method on 10,000 malware and 10,000 benign software datasets. The results show that the precision and recall of our detection model reach 97.84% and 97.83%, verifying the effectiveness of our proposed method.

Decoupling capacitor (decap) has been widely used to effectively reduce dynamic power supply noise. Traditional decap budgeting algorithms usually explore the sensitivity-based nonlinear optimizations or conjugate gradient (CG) methods, which can be prohibitively expensive for large-scale decap budgeting problems and cannot be easily parallelized. In this paper, we propose a hierarchical cross-entropy based optimization technique which is more efficient and parallel-friendly. Cross-entropy (CE) is an advanced optimization framework which explores the power of rare event probability theory and importance sampling. To achieve the high efficiency, a sensitivity-guided cross-entropy (SCE) algorithm is introduced which integrates CE with a partitioning-based sampling strategy to effectively reduce the solution space in solving the large-scale decap budgeting problems. Compared to improved CG method and conventional CE method, SCE with Latin hypercube sampling method (SCE-LHS) can provide 2× speedups, while achieving up to 25% improvement on power supply noise. To further improve decap optimization solution quality, SCE with sequential importance sampling (SCE-SIS) method is also studied and implemented. Compared to SCE-LHS, in similar runtime, SCE-SIS can lead to 16.8% further reduction on the total power supply noise.

Decoupling capacitor (decap) placement has been widely adopted as an effective way to suppress dynamic power supply noise. Traditional decap budgeting algorithms usually explore the sensitivity-based nonlinear optimizations or conjugate gradient methods, which can be prohibitively expensive for large-scale decap budgeting problems. We present a hierarchical cross entropy (CE) optimization technique for solving the decap budgeting problem. CE is an advanced optimization framework which explores the power of rare-event probability theory and importance sampling. To achieve high efficiency, a sensitivity-guided cross entropy (SCE) algorithm is proposed which integrates CE with a partitioning-based sampling strategy to effectively reduce the dimensionality in solving the large scale decap budgeting problems. Extensive experiments on industrial power grid benchmarks show that the proposed SCE method converges 2X faster than the prior methods and 10X faster than the standard CE method, while gaining up to 25% improvement on power grid supply noise. Importantly, the proposed SCE algorithm is parallel-friendly since the simulation samples of each SCE iteration can be independently obtained in parallel. We obtain up to 1.9X speedup when running the SCE decap budgeting algorithm on a dual-core-dual-GPU system.