In industrial wireless networks, millimeter wave (mmWave) communication is able to provide ultra-low latency and ultra-high bandwidth, emerging as a key technology to meet the real-time requirements of future industrial applications. In order to maintain the stability of the mmWave connection, the traditional approach is to perform beam alignment by frequent beam sweeping, but this results in additional beam overhead. Recently proposed vision-aided beam prediction models emerge as a solution to the above problem. However, the usage of models with complex structures, such as deep convolutional neural networks, requires substantial system storage and causes high computational costs. To this end, we propose a lightweight multimodal environment semantics aided beam prediction scheme. It extracts and combines the absolute and relative positions of the target from the locator and images as the environment semantics, then inputs them into an efficient and lightweight prediction model to output the optimal beam index. Experimental results demonstrate that, compared to the existing mainstream vision-aided or vision-position-aided beam prediction methods, the proposed model achieves a maximum top-1 beam prediction accuracy improvement of 7.6%, while saving the model size by about 83.4% and reducing the running time of the beam prediction stage by one order of magnitude.

Millimeter wave (mmWave) wireless communications are significant technologies that support Internet of Things (IoT) systems to achieve fast and stable data transmission, and the guarantee of its communication quality usually depends on accurate beam prediction. Due to the advantage of not relying on channel state information, beam prediction schemes using visual data and artificial intelligence become popular. Most of the current vision-aided beam prediction schemes directly predict the index of the optimal beam in the codebook. However, these methods are only applicable to certain codebooks and have limited generalization and scalability. To address the issues, we propose a vision-aided beam prediction scheme based on deep reinforcement learning (DRL). The scheme takes the original image as input and extracts the position and velocity of the user through the object detection algorithm. Subsequently, combined with the current state of the base station, it outputs a continuous angle value and finally matches it with the beam index in the codebook. Furthermore, we integrate the attention mechanism into the actor network of the deep deterministic policy gradient (DDPG) and propose a scheme of DDPG with attention mechanism (DDPG-A), which can perform differential processing on features, thereby enhancing algorithmic performance. The simulation test utilizing real datasets demonstrates that the proposed scheme has good generalization and scalability while considerably reducing the beam training overheads.

Age of Information (AoI) is a popular information freshness metric and is widely used to evaluate the real-time performance of industrial wireless networks (IWNs). In IWNs with heterogeneous traffic, there may be differentiated real-time requirements. In this article, the peak AoI (PAoI) is introduced to assess the information freshness of the traffic with the random updating pattern. Meanwhile, the deadline related to delay for the traffic with the period updating pattern is considered to ensure the timeliness of packet delivery. An optimization problem for minimizing the average PAoI violation rate under the delay constraints is investigated. To address the NP-hard optimization problem with constraints, the tool from Lyapunov optimization theory is utilized to transform the constraint satisfaction into a queue stability problem to obtain an unconstrained optimization problem. Then, by decomposing the decision of each slot, a two-step scheduling scheme is proposed, which consists of one main policy based on deep reinforcement learning and two subpolicies. Numerical results show that the proposed scheme can minimize the average PAoI violation rate while satisfying the delay constraints.

Vehicular edge computing (VEC), as an emerging computing paradigm, enables the timely processing of computing tasks at the network edge through on-vehicle servers, thereby meeting users’ demands for information freshness. In this paper, we introduce the Age of Information (AoI) to measure information freshness and investigate the scheduling problem minimizing the long-term average AoI in VEC-assisted Internet of Things systems. The main challenge lies in the strong coupling between link scheduling and server selection under the location constraints of VEC. To address this issue, we design a scheduling strategy based on deep reinforcement learning and improve the neural network structure using a branch network approach, reducing complexity by decreasing the number of actions represented in the network’s output layer. Moreover, we introduce an action masking scheme that accelerates the algorithm’s convergence in this system. Numerical results show that the proposed scheduling algorithm can achieve up to a 25.4% performance gain compared to existing advanced algorithms.

To measure the freshness of information more accurately and satisfy the timeliness demands for a wide range of time-sensitive applications, Age of Information (AoI) becomes a popular network performance metric. In this paper, we investigate the optimization of real-time performance in an Internet of Things (IoT) system with heterogeneous traffic, where periodic traffic for control and random traffic for update coexist. Generally, the periodically arriving data used for control is strictly deadline-sensitive, while the randomly arriving data used for updates has freshness requirements. Thus, we seek to develop a scheduling scheme that minimizes the long-term average AoI under the delay tolerance constraint, to meet the varying transmission timeliness requirements for different arrival models. The optimization problem is modeled as a constrained Markov decision process, and then it is decoupled into two classes of optimization sub-problems for single-node by relaxing the hard constraint of channel resources with the Lagrange multiplier. On this basis, a multi-node scheduling scheme consisting of a master policy and two sub-policies is designed. Specifically, two sub-policies are built for the two classes of optimization sub-problems, respectively, with Lyapunov and linear programming methods. For the hard constraint, a master policy is designed by introducing deep reinforcement learning for scheduling opportunities allocation, and a truncated sub-policy is proposed based on one of the sub-policies. Numerical results show that the proposed scheme can obtain a smaller long-term average AoI compared to other advanced algorithms while satisfying the delay constraint.

With the widespread application of 5G technology in the Industrial Internet of Things (IIoT), dividing nodes into different network slices according to delay tolerance requirements can facilitate reasonable resource allocation and guarantee quality of service (QoS). In this paper, we investigate the network slice resource allocation algorithm with delay tolerance based on traffic prediction. A traffic prediction algorithm is proposed combining convolutional neural network (CNN) with attention mechanism and bidirectional long-short term memory (Bi-LSTM) to obtain the spatiotemporal features. Based on the predicted traffic, the problem of minimizing the usage of physical resource blocks (PRBs) is studied, and a two-layer structure resource allocation algorithm based on deep reinforcement learning (DRL) is proposed. Specifically, Dueling Double DQN (D3QN) is used to allocate PRBs between slices, and a heuristic algorithm is used to allocate PRBs among nodes in the slice. Furthermore, we consider the joint optimization problem of PRBs and power. In light of the coupling between PRBs and power aggravates the high dimension of the action space, we propose a resource allocation algorithm which using the branch structure to decoupling the action space. Simulation results show that the proposed algorithms can satisfy the QoS and outperform the baseline algorithms.

In industrial wireless networks (IWNs), age of information (AoI) and delay are two significant metrics to measure data freshness and delivery timeliness. In this article, an IWN system with delay-sensitive and normal data is considered. To perform a joint optimization of the average AoI and the deadline-related overdue rate, we investigate the scheduling policy under time-varying channels. With the channel state knowledge available, we evaluate the expected gain obtained with the assumption that whether the data are scheduled or not for each sensor node, and develop a low-complexity scheduling policy. Under the hypothesis that channel state knowledge is not available, we utilize the model-free learning property of dueling double deep Q-network (D3QN) for the learning of scheduling policy and design a weighted expert knowledge-based exploration scheme that can achieve a higher convergence speed compared to the classical D3QN. Simulation results show the tradeoff between AoI and delay and demonstrate that the two proposed policies outperform existing state-of-the-art algorithms.

Real-time data delivery is significant for the Industrial Internet of Things (IIoT). Age of information (AoI), a popular real-time metric, is usually used to measure the data freshness of the IIoT systems. If the data most recently received by the destination at time $t$ was generated at time $t_{1}$ , then the AoI is $t-t_{1}$ . In this paper, we consider a multi-sensor multi-server IIoT system and develop scheduling algorithms to minimize the average AoI. The challenge lies in the strong coupling between link scheduling, server selection, and service preemption. To address this issue, we propose a guided exploration-based deep Q-Network (GE-DQN) algorithm utilizing a fixed advantage policy, which has a faster learning speed compared to classical deep Q-Network. Moreover, we use a shared decision module followed by several network branches to transform the structure of GE-DQN and propose a guided exploration-based Branching Dueling Q-Network (GE-BDQN) algorithm. Since the branch structure of GE-BDQN can decompose the high-dimensional action, GE-BDQN can reduce the approximate exponential growth of the number of output neurons with the increase of the number of sensors to linear growth compared to GE-DQN, ensuring the applicability of the algorithm under large-scale systems. From the simulation results, it can be found that the proposed two algorithms can achieve better average AoI compared to the advanced algorithms, and the GE-BDQN algorithm can achieve up to 36% performance gain.

Age of information (AoI) is a popular metric of data freshness, however, it neglects the usability of data. In light of this, we introduce a new metric,Age of Usage Information(AoUI), which can jointly capture the freshness and usability of correlated data in the Internet of Things in a fine-grained manner. Based on the proposed metric, we investigate the optimization problem of minimizing average AoUI, where the correlated nodes transmit data to the destination via noisy channels. To seek the optimal data scheduling policy, we first develop a virtual queue based (VQ) policy under the assumption that the priori knowledge of the channel state is known. Then, considering the case where the channel state is unknown, we utilize the model-free characteristic of double deep Q-network (DDQN) to design an improved exploration based DDQN (IE-DDQN) policy which does not require a priori knowledge of the channel state. Furthermore, we investigate the development of joint data scheduling and usage policy and introduce decoupled action branches to improve the structure of the neural network of DDQN proposing a decoupling action based DDQN (DA-DDQN) policy. Simulation results show that the proposed VQ, IE-DDQN, and DA-DDQN policies all exhibit superior performance compared to baseline algorithms such as the classical DDQN method and the greedy policy of scheduling the node with the largest product of AoUI and usable factor, which may be due to the consideration of the stability of the virtual queue, the improvement of the exploration process, and the reconstruction of the neural network structure, respectively.

Age of information (AoI) is a new metric that can measure the data freshness of the industrial Internet of things (IIoT) systems. Focusing on a hybrid scenario where periodic and random sampling devices exist simultaneously, we investigate AoI-aware scheduling schemes under deterministic delay constraint of devices with periodic sampling in noisy channels. We first consider that the probability of successful delivery of data obeys a known fixed probability, and develop a dynamic scheduling scheme utilizing the slot-based Lyapunov drift framework. Second, in the case where the prior knowledge of the probability of successful data delivery is unknown, we introduce deep reinforcement learning (DRL) to learn the model-free scheduling and propose a scheduling policy based on the dueling deep Q network (D3QN). Numerical results show that the proposed Lyapunov policy and D3QN policy can minimize the average AoI while subjecting to the delay constraint.

With its low delay and high reliability, 5G technology can meet the requirement of interconnection for Industrial Internet of things (IIoT). However, industrial heterogeneous networks have different quality of service (QoS) requirements, so 5G slicing technology is necessary to logically isolate them from each other. A main challenge lies in how to reasonably allocate network resources for each slice and the nodes in it to ensure the low delay and reliability requirements of IIoT. In this paper, we focus on the problem of minimizing the usage of physical resource blocks (PRBs) under delay constraints, and propose a dynamic allocation algorithm based on traffic prediction. In order to analyze the spatial and temporal features of network traffic, we combine convolutional neural network (CNN) with bidirectional long short-term memory (Bi-LSTM), and then add attention mechanism to form a traffic prediction model CNN-Bi-LSTM with attention mechanism (CBL-A). For dynamic resource allocation, according to the traffic prediction results, on the basis of Dueling Double DQN (D3QN), a heuristic PRBs scheduling policy (PSP) is embedded to obtain a D3QN model with PSP (D3QN-PSP), which reduces the action space and accelerates the convergence speed. The simulation results show that the proposed algorithm can minimize the consumption of PRBs while guaranteeing the delay and slice isolation constraints.

Age of Information (AoI) is a recently introduced metric to capture data freshness. In this article, we consider a multiuser single-destination Internet of Things (IoT) system with periodic state updating, and investigate the scheduling methods of minimizing the long-term average AoI with limited retransmission times. In view of the retransmission mode, the AoI optimization problems for retransmission with and without feedback are analyzed. For the retransmission without feedback, we formulate the expected decision loss (EDL) function of the AoI optimization problem with the finite retransmission times and propose a loss-greedy policy by minimizing the EDL at each step. For the retransmission with feedback, a potential optimal solution is to construct the AoI optimization problem as an infinite-horizon Markov decision process (MDP) and solve the corresponding Bellman optimal equations. However, this optimal solution is prone to suffer from the curse of dimensionality and is hard to implement. To address this issue, we decouple the infinite-horizon MDP into the finite-state MDP in each single frame, and then propose a low-complexity slot-based max-weight (SBMW) policy to minimize the long-term average AoI. Numerical results show that, compared with an ALOHA-like baseline policy, the proposed Loss-Greedy policy can achieve up to 48% reduction of the average AoI for the retransmission without feedback, while the SBMW policy can reduce the average AoI by 57% in the retransmission with feedback. The average performance gain of the proposed policies over the state-of-the-art policies is at least 10%.

Age of Information (AoI) is a newly rising metric for measuring the freshness of information. In this article, we consider a multidevice computing-enabled Internet of Things (IoT) system with a common destination, in which the status update sampled by the device can be offloaded directly to the destination for computing or computed by the device and then delivered to the destination, and jointly design offloading and scheduling policies to minimize the average weighted sum of AoI and energy consumption. The challenge lies in computing mode selection and its strong coupling with scheduling decisions. To address this issue, we formulate the optimization problem as a bilevel discrete-time Markov decision process (MDP) and approximate the optimal solution by relative value iteration. Furthermore, the threshold structure of the MDP policy is shown. However, with the expansion of the system scale, the MDP policy will suffer from the curse of dimensionality. In light of this, we develop a learning-based algorithm based on emerging deep reinforcement learning (DRL) to reduce the dimensionality of state space and utilize a late experience storage method to train two heterogeneous artificial neural networks (ANNs) synchronously during the training process. Simulation results show the structure of the MDP policy and verify the performance of the DRL policy is near-optimal.