In remote control systems, transmitting large data volumes (e.g., images, video frames) from wireless sensors to remote controllers is challenging when uplink capacity is limited (e.g., RedCap devices or massive wireless sensor networks). Furthermore, controllers often need only information-rich representations of the original data. To address this, we propose a semantic-driven predictive control combined with a channel-aware scheduling to enhance control performance for multiple devices under limited network capacity. At its core, the proposed framework, coined Time-Series Joint Embedding Predictive Architecture (TS-JEPA), encodes high-dimensional sensory data into low-dimensional semantic embeddings at the sensor, reducing communication overhead. Furthermore, TS-JEPA enables predictive inference by predicting future embeddings from current ones and predicted commands, which are directly used by a semantic actor model to compute control commands within the embedding space, eliminating the need to reconstruct raw data. To further enhance reliability and communication efficiency, a channel-aware scheduling is integrated to dynamically prioritize device transmissions based on channel conditions and age of information (AoI). Extensive simulations on inverted cart-pole systems demonstrate that the proposed framework achieves a 98.95% reduction in communication cost, a normalized prediction error of 0.004, and 74.48% control accuracy, outperforming conventional control baselines. Furthermore, the proposed framework maintains robust control performance on 15-step prediction horizons and supports up to 16× more devices than round-robin and opportunistic scheduling schemes with conventional control baselines under similar network conditions.

In this work, we study the joint scheduling and power allocation problem of vision-based remote control systems, where multiple devices upload their image states to a central controller and receive control actions. Due to the high dimensionality of the image states and to manage the lack of radio resources, we propose a novel self-supervised learning approach to predict the devices’ joint control and wireless dynamics in latent space, enabling wireless resource optimization without compromising the control objectives of the remote control system. Our method leverages two coupled joint-embedding predictive architectures (JEPAs): a control JEPA models the control transition dynamics and guides the predictions of a wireless JEPA, which captures the dynamics of the device’s channel state information (CSI) through cross-modal conditioning. We then train a deep reinforcement learning (RL) algorithm to derive a control policy from latent control dynamics and a power predictor to estimate scheduling slots with favorable channel conditions based on latent CSI representations. To enhance control reliability, we employ an efficient ensemble technique to estimate the uncertainty of JEPA predictions. The two JEPAs are used by the remote controller to forecast future latent trajectories of the devices’ control and wireless states, allowing the controller to proactively plan its scheduling policy using model predictive control (MPC). Simulation results, conducted in a customized image-based control environment with ray tracing, demonstrate that our proposed approach converges three times faster and reduces transmit power by over 50% while maintaining control performance comparable to baseline methods that do not account for wireless resource optimization.

In this work, we aim to optimize the radio resource management of a communication system between a remote controller and its device, whose state is represented through image frames, without compromising the performance of the control task. We propose a novel machine learning (ML) technique to jointly model and predict the dynamics of the control system as well as the wireless propagation environment in latent space. Our method leverages two coupled joint-embedding predictive architectures (JEPAs): a control JEPA models the control dynamics and guides the predictions of a wireless JEPA, which captures the dynamics of the device’s channel state information (CSI) through cross-modal conditioning. We then train a deep reinforcement learning (RL) algorithm to derive a control policy from latent control dynamics and a power predictor to estimate scheduling intervals with favorable channel conditions based on latent CSI representations. As such, the controller minimizes the usage of radio resources by utilizing the coupled JEPA networks to imagine the device’s trajectory in latent space. We present simulation results on synthetic multimodal data and show that our proposed approach reduces transmit power by over 50% while maintaining control performance comparable to baseline methods that do not account for wireless optimization.

In this study, we delve into the intricacies of semantic communication-control codesign (CoCoCo) for wireless mixed logical dynamical (MLD) systems operating under signal temporal logic (STL) specifications. Our novel contribution, the MLD-Koopman autoencoder (AE), emerges as a method to linearize the progression of system states within a feature space. This linearization effectively mitigates the communication and computation costs associated with MLD system control. To surmount the challenges posed by multiple correlated MLD systems that possess distinct logical control rules while sharing baseline dynamics, we present the compositional logical dynamical (CLD)-Koopman AE as a remedy to the scalability limitations of the MLD-Koopman AE. This innovative approach incorporates two pivotal models—the dynamics semantic Koopman (DSK) model, capturing semantic correlations among MLD systems, and the logical semantic Koopman (LSK) model, encoding logical control rules. These models portray the linear evolution of baseline dynamics and control rules within a feature space, facilitating predictions of future states for multiple MLD systems with constrained communication. Validation comes from simulations on large-scale inverted cart-pole systems, demonstrating the prowess of the CLD-Koopman AE in achieving an average state prediction performance 82.77% higher than other predictive benchmarks, particularly evident at a signal-to-noise ratio (SNR) of 10 dB.

Real-time remote control over wireless is an important yet challenging application in fifth-generation and beyond due to its mission-critical nature under limited communication resources. Current solutions hinge on not only utilizing ultrareliable and low-latency communication (URLLC) links but also predicting future states, which may consume enormous communication resources and struggle with a short prediction time horizon. To fill this void, in this article we propose a novel two-way Koopman autoencoder (AE) approach wherein: 1) a sensing Koopman AE learns to understand the temporal state dynamics and predicts missing packets from a sensor to its remote controller and 2) a controlling Koopman AE learns to understand the temporal action dynamics and predicts missing packets from the controller to an actuator co-located with the sensor. Specifically, each Koopman AE aims to learn the Koopman operator in the hidden layers while the encoder of the AE aims to project the nonlinear dynamics onto a lifted subspace, which is reverted into the original nonlinear dynamics by the decoder of the AE. The Koopman operator describes the linearized temporal dynamics, enabling long-term future prediction and coping with missing packets and closed-form optimal control in the lifted subspace. Simulation results corroborate that the proposed approach achieves a$38\times $lower mean squared control error at 0-dBm signal-to-noise ratio (SNR) than the nonpredictive baseline.

Remote state monitoring over wireless is envisaged to play a pivotal role in enabling beyond 5G applications ranging from remote drone control to remote surgery. One key challenge is to identify the system dynamics that is non-linear with a large dimensional state. To obviate this issue, in this article we propose to train an autoencoder whose encoder and decoder are split and stored at a state sensor and its remote observer, respectively. This autoencoder not only decreases the remote monitoring payload size by reducing the state representation dimension but also learns the system dynamics by lifting it via a Koopman operator, thereby allowing the observer to locally predict future states after training convergence. Numerical results under a non-linear cart-pole environment demonstrate that the proposed split learning of a Koopman autoencoder can locally predict future states, and the prediction accuracy increases with the representation dimension and transmission power.

This article studies the joint problem of uplink-downlink scheduling and power allocation for controlling a large number of control systems that upload their states to remote controllers and download control actions over wireless links. To overcome the lack of wireless resources, we propose a machine learning-based solution, where only one control system is controlled, while the rest of the control systems are actuated by locally predicting the missing state and/or action information using the previous uplink and/or downlink receptions via a Gaussian process regression (GPR). This GPR prediction credibility is determined using the age-of-information (AoI) of the latest reception. Moreover, the successful reception is affected by the transmission power, mandating a co-design of the communication and control operations. To this end, we formulate a network-wide minimization problem of the average AoI and transmission power under communication reliability and control stability constraints. To solve the problem, we propose a dynamic control algorithm using the Lyapunov drift-plus-penalty optimization framework. Numerical results corroborate that the proposed algorithm can stably control$2\times$more number of actuators compared to an event-triggered scheduling baseline with Kalman filtering and frequency division multiple access, which is$18\times$larger than a round-robin scheduling baseline.