Digital twins (DT) offer a low-overhead evaluation platform and the ability to generate rich datasets for training machine learning (ML) models before actual deployment. Specifically, for the scenario of ML-aided millimeter wave (mmWave) links between moving vehicles to roadside units, we show how DT can create an accurate replica of the real world for model training and testing. The contributions of this paper are twofold: First, we propose a framework to create a multimodal Digital Twin (DT), where synthetic images and LiDAR data for the deployment location are generated along with RF propagation measurements obtained via ray-tracing. Second, to ensure effective domain adaptation, we leveragemeta-learning, specificallyModel-Agnostic Meta-Learning(MAML), withtransfer learning(TL) serving as a baseline validation approach. The proposed framework is validated using a comprehensive dataset containing both real and synthetic LiDAR and image data for mmWave V2X beam selection. It also enables the investigation of how each sensor modality impacts domain adaptation, taking into account the unique requirements of mmWave beam selection. Experimental results show that models trained on synthetic data using transfer learning and meta-learning, followed by minimal fine-tuning with real-world data, achieve up to 4.09× and 14.04× improvements in accuracy, respectively. These findings highlight the potential of synthetic data and meta-learning to bridge the domain gap and adapt rapidly to real-world beamforming challenges.

Fast sector-steering in the mmWave band for vehicular mobility scenarios is a challenge because standard-defined exhaustive search over predefined antenna sectors cannot be assuredly completed within short contact times. This paper proposes machine learning to speed up sector selection using data from multiple non-RF sensors, such as LiDAR, GPS, and camera images in the mmWave radios with large codebooks. The contributions in this paper are threefold: First, we propose a multimodal deep learning architecture that fuses the inputs from these data sources and locally predicts the sectors for best alignment at a vehicle. Second, we propose FLASH-and-Prune, which combines the knowledge from multiple vehicles by aggregating the local model parameters and exploits model pruning to optimize the model parameter exchange overhead. Third, we present a pruning strategy that takes into account the distributed nature of federated learning to adaptively prune or retrieve model weights. We validate the proposed architecture on a real-world multimodal dataset collected from an autonomous car. We observe that FLASH-and-Prune incurs 29.25% and 35.89% less overhead in the uplink and downlink, respectively, compared to standard federated learning.

The use of alternate modalities, like images, for fast beamforming in the millimeter wave (mmWave)-band is being proposed to ensure high bandwidth connectivity in vehicular scenarios typically seen in the context of autonomous cars. Considering the dynamic deployment conditions, a car may encounter new environments which were not explicitly included in an apriori training dataset. In this paper, we propose to use the Model-Agnostic Meta-Learning (MAML) framework on the image data of the mmWave vehicle-to-infrastructure beam selection FLASH dataset, to overcome the generalization issues of a pre-trained model in unseen non-line-of-sight (NLOS) connectivity environments. MAML has additional advantages over traditional deep-learning techniques: (i) it uses a fraction of the data which, in turn, simplifies data collection and storage, and (ii) it results in equal or higher accuracy in optimal beam selection compared to the case when the new environment dataset is fully available during initial training. We show that our MAML implementation improves test accuracy of beam selection by up to 86% with fine-tuning when encountering an unseen NLOS environment compared to conventional supervised learning.

The use of non-RF data can potentially speed up millimeter wave-band sector-steering in vehicular mobility scenarios by gaining contextual knowledge of the environment. While several works have demonstrated the benefits of this approach, especially applying machine learning models on inputs from LiDAR and image sensors, adapting such models in ‘unseen’ environments remains an open problem. State-of-the-art techniques generally use a single, pretrained model for all different scenarios, which assumes that the network has ‘seen’ representative examples of all future scenarios. In this paper, we propose the TUNE framework, which solves this problem by: ($a$) transfer learning (TL) for better performance with similar convergence times in comparison to non-TL-generated model testing, (b) utilizing statistical properties to select the best-suited starting ‘seen’ scenario (and by extension the model trained for it), and (c) a refinement of the transfer learning framework by dynamically selecting the most pertinent layers for retaining, thus reducing the overhead compared to fully retraining a model. We validate TUNE on publicly available synthetic and real-world datasets for mmWave beam selection for V2X communication, revealing that TUNE generally outperforms non-TL methods in a variety of tasks where a different number of beams is available between the training and testing environments.

Fast sector-steering in the mmWave band for vehicular mobility scenarios remains an open challenge. This is because standard-defined exhaustive search over predefined antenna sectors cannot be assuredly completed within short contact times. This paper proposes machine learning to speed up sector selection using data from multiple non-RF sensors, such as LiDAR, GPS, and camera images. The contributions in this paper are threefold: First, a multimodal deep learning architecture is proposed that fuses the inputs from these data sources and locally predicts the sectors for best alignment at a vehicle. Second, it studies the impact of missing data (e.g., missing LiDAR/images) during inference, which is possible due to unreliable control channels or hardware malfunction. Third, it describes the first-of-its-kind multimodal federated learning framework that combines model weights from multiple vehicles and then disseminates the final fusion architecture back to them, thus incorporating private sharing of information and reducing their individual training times. We validate the proposed architectures on a live dataset collected from an autonomous car equipped with multiple sensors (GPS, LiDAR, and camera) and roof-mounted Talon AD7200 60GHz mmWave radios. We observe 52.75% decrease in sector selection time than 802.11ad standard while maintaining 89.32% throughput with the globally optimal solution.