As the demand for data privacy protection has grown, personalized federated learning has increasingly become a focal point of research. In various scenarios such as healthcare institutions and mobile devices, how to effectively address the heterogeneity of client data while protecting data privacy constitutes a significant challenge in this field. Current research on personalized federated learning primarily focuses on balancing global information and personalized information at the model parameter level. To address the challenge of balancing client data heterogeneity with personalized model training, this paper proposes a Collaborative Personalized Federated Learning algorithm, where Adaptive Feature Filtering (FedAFF) is employed to enhance model personalization while maintaining effective collaboration between clients. The core of our approach is an adaptive feature filtering framework based on an asymmetric dual-source design. This framework leverages a client-side gating mechanism, based on a one-dimensional convolutional neural network, to dynamically filter and fuse feature information derived from a frozen global model and a local personalized model according to the characteristics of each sample. Building on this, we further introduce an attention mechanism on the server-side for intelligent parameter aggregation. Specifically, our proposed adaptive feature filter is capable of generating a feature representation for each sample that balances global consistency and personalized attributes, while the Maximum Mean Discrepancy loss is employed to mitigate feature distribution shift. By optimizing the uploaded parameters through the attention mechanism, the model’s generalization capability is further enhanced. We conducted comprehensive comparative experiments across multiple datasets, comparing FedAFF with other advanced federated learning algorithms. The experimental results demonstrate its remarkable performance: the test accuracy of FedAFF increased by 34.54% compared to FedAvg, and averaged 6.42% higher precision than the state-of-the-art personalized federated learning methods.

While the field of multi-modal learning keeps growing fast, the deficiency of the standard joint training paradigm has become clear through recent studies. They attribute the sub-optimal performance of the jointly trained model to the modality competition phenomenon. Existing works attempt to improve the jointly trained model by modulating the training process. Despite their effectiveness, those methods can only apply to late fusion models. More importantly, the mechanism of the modality competition remains unexplored. In this paper, we first propose an adaptive gradient modulation method that can boost the performance of multi-modal models with various fusion strategies. Extensive experiments show that our method surpasses all existing modulation methods. Furthermore, to have a quantitative understanding of the modality competition and the mechanism behind the effectiveness of our modulation method, we introduce a novel metric to measure the competition strength. This metric is built on the mono-modal concept, a function that is designed to represent the competition-less state of a modality. Through systematic investigation, our results confirm the intuition that the modulation encourages the model to rely on the more informative modality. In addition, we find that the jointly trained model typically has a preferred modality on which the competition is weaker than other modalities. However, this preferred modality need not dominate others. Our code will be available at https://github.com/lihong2303/AGM_ICCV2023.

As deep learning advances, there is an ever-growing demand for models capable of synthesizing information from multi-modal resources to address the complex tasks raised from real-life applications. Recently, many large multi-modal datasets have been collected, on which researchers actively explore different methods of fusing multi-modal information. However, little attention has been paid to quantifying the contribution of different modalities within the proposed models. In this paper, we propose the SHapley vAlue-based PErceptual (SHAPE) scores that measure the marginal contribution of individual modalities and the degree of cooperation across modalities. Using these scores, we systematically evaluate different fusion methods on different multi-modal datasets for different tasks. Our experiments suggest that for some tasks where different modalities are complementary, the multi-modal models still tend to use the dominant modality alone and ignore the cooperation across modalities. On the other hand, models learn to exploit cross-modal cooperation when different modalities are indispensable for the task. In this case, the scores indicate it is better to fuse different modalities at relatively early stages. We hope our scores can help improve the understanding of how the present multi-modal models operate on different modalities and encourage more sophisticated methods of integrating multiple modalities.