Multimodal federated learning (MFL) aims to enrich model training in FL settings where clients are collecting measurements across multiple modalities. However, key challenges to MFL remain unaddressed, particularly in heterogeneous network settings where: (i) the set of modalities collected by each client is diverse, and (ii) communication limitations prevent clients from uploading all their locally trained modality encoders to the server. In this paper, we propose Multimodal Federated learning with joint Modality and Client selection (MFedMC), a communication-efficient MFL framework that tackles these challenges through a decoupled architecture and selective uploading. Unlike traditional holistic fusion approaches, MFedMC separates modality encoders and fusion modules: modality encoders are aggregated at the server for generalization across diverse client distributions, while fusion modules remain local to each client for personalized adaptation to individual modality configurations and data characteristics. Building on this decoupled design, our joint selection algorithm incorporates two main components: (a) A modality selection methodology for each client, which weighs (i) the impact of the modality, gauged by Shapley value analysis, (ii) the modality encoder size as a gauge of communication overhead, and (iii) the frequency of modality encoder updates, denoted recency, to enhance generalizability. (b) A client selection strategy for the server based on the local loss of modality encoders at each client. Experiments on five real-world datasets demonstrate that MFedMC achieves comparable accuracy to several baselines while reducing communication overhead by over 20×. A demo video and our code are available athttps://liangqiy.com/mfedmc/.

The detection of anomalous behavior of an engineered system or its components is an important task for enhancing reliability, safety, and efficiency across various engineering applications. However, designing an accurate anomaly detector can be very challenging in settings where anomalous labels are sparse or, in the worst case, missing in the training data. To mitigate this issue of the lack of anomalous labels in the domain of interest, existing approaches use transfer learning by leveraging information from anomalous samples in a closely related source domain. Although previous studies have shown good results from applying transfer learning, they do not specifically address the issue of high false-positive rates from such transfer. High false-positive rates can arise from misleading information present in uninformative/irrelevant features. Inspired by this observation, this paper focuses on identifying key input features, termed as such, due to their strong predictability in anomaly detection. A transfer learning approach is introduced that leverages the optimal$f_{\beta } $score for key feature estimation. This approach involves finding a weight vector to amplify key features and attenuate uninformative inputs during prediction. We demonstrate the capabilities of our proposed anomaly detection method as a quality check for newly manufactured automotive transmissions. Given the use of frequency domain order-based features in our use case, our proposed method is also easily extensible to the anomaly detection of other rotating machinery. Based on our findings we also find that our proposed anomaly detection algorithm, utilizing precise data-driven features, outperforms detectors based on experience/heuristics-based features currently used in automotive engineering applications. More importantly, our proposed framework can work with any downstream unsupervised anomaly detection algorithm, allowing us to freely choose the best algorithm for the anomaly detection task on hand. Note to Practitioners—Unlike legacy systems that have rich data sets for both healthy and anomalous behaviors over a broad spectrum of operating states, newly designed system or design variants of existing systems have little or no performance data. This makes the task of identifying indicators of anomalous behaviors very challenging for systems without legacy information. This work introduces a machine learning framework that uses label information from related, well-studied systems, to improve anomaly detection outcomes in a new or variant (but related) system. We demonstrate the effectiveness of our transfer approach through the End of Line (EoL) anomaly detection for quality checks on new design variants of an existing automotive transmission. Inspired by feature selection techniques, our proposed framework is easy to use and offers excellent interpretability, which is crucial for achieving a zero-failure system design.

Autoencoders have been extensively used in the development of recent anomaly detection techniques. The premise of their application is based on the notion that after training the autoencoder on normal training data, anomalous inputs will exhibit a significant reconstruction error. Consequently, this enables a clear differentiation between normal and anomalous samples. In practice, however, it is observed that autoencoders can generalize beyond the normal class and achieve a small reconstruction error on some of the anomalous samples. To improve the performance, various techniques propose additional components and more sophisticated training procedures. In this work, we propose a remarkably straightforward alternative: instead of adding neural network components, involved computations, and cumbersome training, we complement the reconstruction loss with a computationally light term that regulates the norm of representations in the latent space. The simplicity of our approach minimizes the requirement for hyperparameter tuning and customization for new applications which, paired with its permissive data modality constraint, enhances the potential for successful adoption across a broad range of applications. We test the method on various visual and tabular benchmarks and demonstrate that the technique matches and frequently outperforms more complex alternatives. We further demonstrate that implementing this idea in the context of state-of-the-art methods can further improve their performance. We also provide a theoretical analysis and numerical simulations that help demonstrate the underlying process that unfolds during training and how it helps with anomaly detection. This mitigates the black-box nature of autoencoder-based anomaly detection algorithms and offers an avenue for further investigation of advantages, fail cases, and potential new directions.

Sampling-based motion planning methods for manipulators in crowded environments often suffer from expensive collision checking and high sampling complexity, which make them difficult to use in real time. To address this issue, we propose a new generalizable control barrier function (CBF)based steering controller to reduce the number of samples needed in a sampling-based motion planner RRT. Our method combines the strength of CBF for real-time collision-avoidance control and RRT for long-horizon motion planning, by using CBF-induced neural controller (CBF-INC) to generate control signals that steer the system towards sampled configurations by RRT. CBF-INC is learned as Neural Networks and has two variants handling different inputs, respectively: state (signed distance) input and point-cloud input from LiDAR. In the latter case, we also study two different settings: fully and partially observed environmental information. Compared to manually crafted CBF which suffers from over-approximating robot geometry, CBF-INC can balance safety and goal-reaching better without being over-conservative. Given state-based input, our neural CBF-induced neural controller-enhanced RRT (CBFINC-RRT) can increase the success rate by 14% while reducing the number of nodes explored by 30%, compared with vanilla RRT on hard test cases. Given LiDAR input where vanilla RRT is not directly applicable, we demonstrate that our CBF-INCRRT can improve the success rate by 10%, compared with planning with other steering controllers. Our project page with supplementary material is at https://mit-realm.github.io/CBFINC-RRT-website/.

RGB-based pedestrian detection is a challenging task in low light and adverse weather conditions because image quality can degrade rather significantly. Including other modalities such as thermal and gated imaging sensors can, on the other hand, significantly improve the detection performance in these conditions. However, these sensors are expensive, and including them may cause design and manufacturing challenges. In this paper, we propose a new framework that utilizes Cross Modality Knowledge Distillation (CMKD) to improve the performance of RGB-only pedestrian detection in low light and adverse weather conditions. Specifically, we develop two CMKD methods that rely on feature-based knowledge distillation and adversarial training to transfer knowledge from a pedestrian detector (teacher) that is trained using multiple modalities to a single modality detector (student) that is trained using RGB images only. Experimental results using the "Seeing Through Fog" dataset show that both of our proposed methods outperform the baseline detector in terms of detection accuracy without increasing computational complexity during inference. In particular, the proposed methods reduce the performance gap between teacher and baseline models by up to 55%.

The performance of the current collision avoidance systems in Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS) can be drastically affected by low light and adverse weather conditions. Collisions with large animals such as deer in low light cause significant cost and damage every year. In this paper, we propose the first AI-based method for future trajectory prediction of large animals and mitigating the risk of collision with them in low light. In order to minimize false collision warnings, in our multi-step framework, first, the large animal is accurately detected and a preliminary risk level is predicted for it and low-risk animals are discarded. In the next stage a multi-stream CONV-LSTM-based encoder-decoder framework is designed to predict the future trajectory of the potentially high-risk animals. The proposed model uses camera motion prediction as well as the local and global context of the scene to generate accurate predictions. Furthermore, this paper introduces a new dataset of FIR videos for large animal detection and risk estimation in real nighttime driving scenarios. Our experiments show promising results of the proposed framework in adverse conditions. Our code is available online1.