Federated learning (FL) is well-suited for multimodal tasks due to its ability to protect privacy and support local training. However, the complexity of real-world sensor environments causes modality heterogeneity across clients. Some modalities may be missing altogether, making it difficult to construct a generalized global model. Existing multimodal federated learning methods often address modality-missing scenarios under simplified assumptions of modality heterogeneity, typically focusing on unimodal clients and modality-complete multimodal clients. Moreover, to mitigate performance degradation caused by missing modalities, some approaches assume the availability of auxiliary information at the server, which may be impractical in real-world scenarios. Therefore, we propose a novel heterogeneous multimodal Federated Learning with Mask-Restoration and Self-Guidance (FL-MRSG). The Mask-Restoration employs a masking strategy to simulate missing data during feature extraction, enabling the network to learn semantic features of missing modality. Furthermore, we introduce an innovative self-guidance mechanism that leverages the restored data as guidance information, enabling the network to distinguish between complete and missing data representations. In addition, we propose a personalized decoupled aggregation strategy to facilitate the collaborative training of a global model across heterogeneous modality clients. We extend the multimodal test set to arbitrary modality combinations to evaluate the robustness of the global model. Extensive experiments on MOSI and SIMS datasets demonstrate the effectiveness of the proposed FL-MRSG for arbitrary missing modalities.

The Dual Encoders (DE) framework maps image and text inputs into a coordinated representation space, and calculates their similarity directly. On the other hand, the Cross Attention (CA) framework performs modalities interactions after completing the feature embedding of images and text, and then outputs a similarity score. For scenarios with bulk query requests or large query sets, the latter is more accurate, but the former is faster. Therefore, this work finds a new way to improve the retrieval accuracy of the DE framework by borrowing the advantages of the CA framework. Drawing inspiration from image captioning, we introduce a text decoder in the model training stage to simulate the cross-modal interaction function, like the CA framework. The text decoder is eventually discarded, aligning our model with the DE framework. Finally, to ensure training stability and prevent overfitting, we modify the Self-Distillation from Last Mini-Batch and apply it to the retrieval areas. Extensive experiments conducted on the MSCOCO and Flickr30K datasets validate the effectiveness of our proposed methods. Notably, our model achieves competitive results compared to state-of-the-art approaches on the Flickr30K dataset.