Automated warehouses involve robots that move across the floor, avoiding obstacles while remaining connected via an access point (AP) to a central controller that instructs the robots. The complex propagation environment and presence of metallic surfaces results in spotty coverage, which changes over time as the location of stored products and machinery changes. Thus, maintaining an assured connectivity to APs while performing navigation is a challenge, although it is needed to relay local sensor data from the robots to the controller and receive directions from the latter.$\rm{DARWIN}$, involves creating a digital twin of the warehouse for training the robots by jointly optimizing the navigation and avoiding wireless dead-spots.$\rm{DARWIN}$has three key capabilities: First, it captures the features of both physical and RF environments in the digital world. Second, it allows real-time updating of the digital twin if significant disparity is detected compared to the physical environment. Finally, it includes a reinforcement learning algorithm that jointly optimizes navigation and network resource management, while accounting for handover and outage. We validate$\rm{DARWIN}$on an emulation environment consisting of Robot Operating System and Gazebo platforms along with real-world RF measurements. Results reveal that$\rm{DARWIN}$reduces the number of steps by 43% compared to choosing the closest AP, while detecting environmental changes with maximum 96% accuracy to maintain a high-fidelity digital twin.

Beamforming techniques use massive antenna arrays to formulate narrow Line-of-Sight signal sectors to address the increased signal attenuation in millimeter Wave (mmWave). However, traditional sector selection schemes involve extensive searches for the highest signal strength sector, introducing extra latency and communication overhead. This paper introduces a dynamic layer-wise and clustering-based federated learning (FL) algorithm for beam sector selection in autonomous vehicle networks called enhanced Dynamic Adaptive FL (eDAFL). The algorithm detects and selects the most important layers of a machine learning model for aggregation in FL process, significantly reducing network overhead and failure risks. eDAFL also consider an intra-cluster and inter-cluster approach to reduce overfitting and increase the abstraction level. We evaluate eDAFL on a real-world multi-modal dataset, demonstrating improved model accuracy by approximately 6.76% compared to existing methods, while reducing inference time by 84.04% and model size up to 52.20%.

This paper presents COPILOT, a ML-based approach that allows vehicles requiring ubiquitous high bandwidth connectivity to identify the most suitable road side units (RSUs) through proactive handoffs. By cooperatively exchanging the data obtained from local 3D Lidar point clouds within adjacent vehicles and with coarse knowledge of their relative positions, COPILOT identifies transient blockages to all candidate RSUs along the path under study. Such cooperative perception is critical for choosing RSUs with highly directional links required for mmWave bands, which majorly degrade in the absence of LOS. COPILOT proposes three modules that operate in an inter-connected manner: (i) As an alternative to sending raw Lidar point clouds, it extracts and transmits low-dimensional intermediate features to lower the overhead of inter-vehicle messaging; (ii) It utilizes an attention-mechanism to place greater emphasis on data collected from specific vehicles, as opposed to nearest neighbor and distance-based selection schemes, and (iii) it experimentally validates the outcomes using an outdoor testbed composed of an autonomous car and Talon AD7200 60GHz routers emulating the RSUs, accompanied by the public release of the datasets. Results reveal COPILOT yields upto 69.8% and 20.42% improvement in latency and throughput compared to traditional reactive handoffs for mmWave networks, respectively.

Fast sector-steering in the mmWave band for vehicular mobility scenarios is a challenge because standard-defined exhaustive search over predefined antenna sectors cannot be assuredly completed within short contact times. This paper proposes machine learning to speed up sector selection using data from multiple non-RF sensors, such as LiDAR, GPS, and camera images in the mmWave radios with large codebooks. The contributions in this paper are threefold: First, we propose a multimodal deep learning architecture that fuses the inputs from these data sources and locally predicts the sectors for best alignment at a vehicle. Second, we propose FLASH-and-Prune, which combines the knowledge from multiple vehicles by aggregating the local model parameters and exploits model pruning to optimize the model parameter exchange overhead. Third, we present a pruning strategy that takes into account the distributed nature of federated learning to adaptively prune or retrieve model weights. We validate the proposed architecture on a real-world multimodal dataset collected from an autonomous car. We observe that FLASH-and-Prune incurs 29.25% and 35.89% less overhead in the uplink and downlink, respectively, compared to standard federated learning.

Creating a digital world that closely mimics the real world with its many complex interactions and outcomes is possible today through advanced emulation software and ubiquitous computing power. Such a software-based emulation of an entity that exists in the real world is called a ‘digital twin’. In this paper, we consider a twin of a wireless millimeter-wave band radio that is mounted on a vehicle and show how it speeds up directional beam selection in mobile environments. To achieve this, we go beyond instantiating a single twin and propose the ‘$\MV$’ paradigm, with several possible digital twins attempting to capture the real world at different levels of fidelity. Towards this goal, this paper describes (i) a decision strategy at the vehicle that determines which twin must be used given the latency limitation, and (ii) a self-learning scheme that uses the$\MV$-guided beam outcomes to enhance DL-based decision-making in the real world over time. Our work is distinguished from prior works as follows: First, we use a publicly available RF dataset collected from an autonomous car for creating different twins. Second, we present a framework with continuous interaction between the real world and$\MV$of twins at the edge, as opposed to a one-time emulation that is completed prior to actual deployment. Results reveal that$\MV$offers up to$79.43\%$and$85.22\%$top-$10$beam selection accuracy for LOS and NLOS scenarios, respectively. Moreover, we observe$67.70-90.79\%$improvement in beam selection time compared to 802.11ad standard and 5G-NR standards.

The use of non-RF data can potentially speed up millimeter wave-band sector-steering in vehicular mobility scenarios by gaining contextual knowledge of the environment. While several works have demonstrated the benefits of this approach, especially applying machine learning models on inputs from LiDAR and image sensors, adapting such models in ‘unseen’ environments remains an open problem. State-of-the-art techniques generally use a single, pretrained model for all different scenarios, which assumes that the network has ‘seen’ representative examples of all future scenarios. In this paper, we propose the TUNE framework, which solves this problem by: ($a$) transfer learning (TL) for better performance with similar convergence times in comparison to non-TL-generated model testing, (b) utilizing statistical properties to select the best-suited starting ‘seen’ scenario (and by extension the model trained for it), and (c) a refinement of the transfer learning framework by dynamically selecting the most pertinent layers for retaining, thus reducing the overhead compared to fully retraining a model. We validate TUNE on publicly available synthetic and real-world datasets for mmWave beam selection for V2X communication, revealing that TUNE generally outperforms non-TL methods in a variety of tasks where a different number of beams is available between the training and testing environments.

Fast sector-steering in the mmWave band for vehicular mobility scenarios remains an open challenge. This is because standard-defined exhaustive search over predefined antenna sectors cannot be assuredly completed within short contact times. This paper proposes machine learning to speed up sector selection using data from multiple non-RF sensors, such as LiDAR, GPS, and camera images. The contributions in this paper are threefold: First, a multimodal deep learning architecture is proposed that fuses the inputs from these data sources and locally predicts the sectors for best alignment at a vehicle. Second, it studies the impact of missing data (e.g., missing LiDAR/images) during inference, which is possible due to unreliable control channels or hardware malfunction. Third, it describes the first-of-its-kind multimodal federated learning framework that combines model weights from multiple vehicles and then disseminates the final fusion architecture back to them, thus incorporating private sharing of information and reducing their individual training times. We validate the proposed architectures on a live dataset collected from an autonomous car equipped with multiple sensors (GPS, LiDAR, and camera) and roof-mounted Talon AD7200 60GHz mmWave radios. We observe 52.75% decrease in sector selection time than 802.11ad standard while maintaining 89.32% throughput with the globally optimal solution.

Accurate beam alignment in the millimeter-wave (mmWave) band introduces considerable overheads involving brute-force exploration of multiple beam-pair combinations and beam retraining due to mobility. This cost becomes often intractable under high mobility scenarios, where fast beamforming algorithms that can quickly adapt the beam configurations are still under development for 5G and beyond. Besides, blockage prediction is a key capability in order to establish mmWave reliable links. In this paper, we propose a data fusion approach that takes inputs from visual edge devices and localization sensors to (i) reduce the beam selection overhead by narrowing down the search to a small set containing the best possible beam-pairs and (ii) detect blockage conditions between transmitters and receivers. We evaluate our approach through joint simulation of multi-modal data from vision and localization sensors and RF data. Additionally, we show how deep learning based fusion of images and Global Positioning System (GPS) data can play a key role in configuring vehicle-to-infrastructure (V2I) mmWave links. We show a 90% top-10 beam selection accuracy and a 92.86% blockage prediction accuracy. Furthermore, the proposed approach achieves a 99.7% reduction on the beam selection time while keeping a 94.86% of the maximum achievable throughput.

We study an Open-World Class Discovery problem in which, given labeled training samples from old classes, we need to discover new classes from unlabeled test samples. There are two critical challenges to addressing this paradigm: (a) transferring knowledge from old to new classes, and (b) incorporating knowledge learned from new classes back to the original model. We propose Class Discovery Kernel Network with Expansion (CD-KNet-Exp), a deep learning framework, which utilizes the Hilbert Schmidt Independence Criterion to bridge supervised and unsupervised information together in a systematic way, such that the learned knowledge from old classes is distilled appropriately for discovering new classes. Compared to competing methods, CD-KNet-Exp shows superior performance on three publicly available benchmark datasets and a challenging real-world radio frequency fingerprinting dataset.

Perfect alignment in chosen beam sectors at both transmit- and receive-nodes is required for beamforming in mmWave bands. Current 802.11ad WiFi and emerging 5G cellular standards spend up to several milliseconds exploring different sector combinations to identify the beam pair with the highest SNR. In this paper, we propose a machine learning (ML) approach with two sequential convolutional neural networks (CNN) that uses out-of-band information, in the form of camera images, to (i) rapidly identify the locations of the transmitter and receiver nodes, and then (ii) return the optimal beam pair. We experimentally validate this intriguing concept for indoor settings using the NI 60GHz mmwave transceiver. Our results reveal that our ML approach reduces beamforming related exploration time by 93% under different ambient lighting conditions, with an error of less than 1% compared to the time-intensive deterministic method defined by the current standards.