Multimodal Federated learning (FL) is a collaborative and privacy preserving machine learning paradigm for multimodal data. With the impressive performance of large-scale pre-trained models, an increasing number of these models are being applied to FL. However, multimodal data in the real world is usually incomplete in modalities. Additionally, directly applying these large-scale pre-trained models in the federated learning framework will lead to the problem of high computational and communication costs. To address these problems, we propose a novel Personalized Multimodal Federated Learning method via Collaborative Prompting with Missing Modalities (MFLCP) . Specifically, we propose an efficient large-scale pre-trained personalized multimodal federated learning framework. To address the issue of incomplete modalities in multimodal data, we propose a modal projection-aware collaborative prompting strategy for incomplete multimodal federated learning. Different categories of prompts are designed for the missing categories, and the modality mapping part and leverage complementary semantic information from different modalities are designed to guide prompt learning, promoting better interaction between modalities. In addition, we propose a communication optimization method for efficient multimodal federated learning, which reduces the parameters of multimodal pre-trained models in the process of federated communication transmission, enhances the speed of local training, and significantly improves convergence speed by integrating large-scale pre-trained models in a lightweight manner. Meanwhile, we establish a personalized adaptive update mechanism for the federated local model, which can adaptively update the local model according to the characteristics of local data, effectively reduces the impact of data heterogeneity. Extensive experimental results on several benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art baselines.

Depth estimation is a crucial task in computer vision, applicable to various domains such as 3D reconstruction, robotics, and autonomous driving. In particular, thermal-based depth estimation has unique advantages, including night-time vision. However, the existing depth estimation method remains challenging in robust generalization due to limited data resources and spectral differences between thermal and RGB images. In this paper, we present a self-supervised approach to enhance thermal-based depth estimation by leveraging pre-trained visual models initially designed for RGB data. In detail, we design a novel two-stage training strategy, incorporating Low-rank Adapters and Convolutional Adapters, which not only significantly improves accuracy and robustness but also enables impressive zero-shot generalization capabilities. Our method outperforms existing thermal-based depth estimation models, opening new possibilities for cross-modal applications in computer vision and robotics research.

Neural implicit representations have revolutionized dense multi-view surface reconstruction, yet their performance significantly diminishes with sparse input views. A few pioneering works have sought to tackle this challenge by leveraging additional geometric priors or multi-scene generalizability. However, they are still hindered by the imperfect choice of input views, using images under empirically determined viewpoints. We propose PVP-Recon , a novel and effective sparse-view surface reconstruction method that progressively plans the next best views to form an optimal set of sparse viewpoints for image capturing. PVP-Recon starts initial surface reconstruction with as few as 3 views and progressively adds new views which are determined based on a novel warping score that reflects the information gain of each newly added view. This progressive view planning progress is interleaved with a neural SDF-based reconstruction module that utilizes multi-resolution hash features, enhanced by a progressive training scheme and a directional Hessian loss. Quantitative and qualitative experiments on three benchmark datasets show that our system achieves high-quality reconstruction with a constrained input budget and outperforms existing baselines.