The performance of multimodal models often deteriorates when modality absence occurs. The absence disrupts the learned inter-modal correlations, resulting in biased multimodal representations. This challenge is especially pronounced when the absence is pervasive, affecting both the training and inference phases. Recent studies have attempted to reconstruct the missing information; however, most of them require complete supervision, which is seldom available in scenarios of pervasive absence. The quality of reconstruction remains a critical issue. Alternatively, others aim to learn robust representations from the available modalities but the substantial variations and biases are not fully addressed. This paper introduces the Multimodal Generalization and Refinement (MGR) framework to mitigate the issue of pervasive modality absence. MGR begins by acquiring generalized multimodal representations and iteratively refines them to recognize and calibrate the biased representations. Initially, multimodal samples with absence are embedded through foundation models, and MGR integrates independent unimodal features to further enhance generalization. Additionally, a novel mixed-context prompt is adopted to identify biases in both features and correlations. A redistribution operation can then refine these biases through graph pooling, culminating in robust and calibrated multimodal representations, which are suitable for downstream tasks. Comprehensive experiments on four benchmark datasets demonstrate that the proposed MGR framework outperforms state-of-the-art methods, effectively mitigating the impact of pervasive modality absence.

Text-to-image (T2I) diffusion models have achieved remarkable progress in generating realistic images from textual descriptions. However, ensuring consistent high-quality image generation with complete backgrounds, object appearance, and optimal texture rendering remains challenging. This paper presents a novel fine-grained pixel-level image editing method based on pre-trained diffusion models. The proposed dual-branch architecture, consisting of Guidance and Generation branches, employs U-Net Denoisers and Self-Attention mechanisms. An improved DDIM-like inversion method obtains the latent representation, followed by multiple denoising steps. Cross-branch interactions, such as KV Replacement, Classifier Guidance, and Feature Correspondence, enable precise control while preserving image fidelity. The iterative refinement and reconstruction process facilitates finegrained editing control, supporting attribute modification, image outpainting, style transfer, and face synthesis with Clickand-Drag style editing using masks. Experimental results demonstrate the effectiveness of the proposed approach in enhancing the quality and controllability of T2I-generated images, surpassing existing methods while maintaining attractive computational complexity for practical real-world applications.

Federated Hetero-Task Learning (FHTL) enables the simultaneous learning of multiple heterogeneous tasks on federated learning clients, offering enhanced flexibility. However, the inconsistency between optimization objectives and evaluation metrics for these heterogeneous tasks poses challenges in achieving performance fairness among clients. This study proposes a fairness-aware FHTL method, FairFHTL. It employs adversarial multi-task representation learning at the client level to learn the task-independent shared model. Consequently, it solves optimization objectives inspired by fair resource allocation on the server side to determine the update direction of the global shared model, ultimately achieving task-independent fair performance balance. Extensive experiments on three multi-task datasets demonstrate that FairFHTL significantly enhances performance across the majority of tasks compared to conventional federated learning and FHTL methods. Moreover, compared with other fairness-aware federated learning approaches, FairFHTL maintains a more uniform performance distribution across all tasks.

Automatic generation of ophthalmic reports holds significant potential to lessen clinicians’ workload, enhance work efficiency, and alleviate the imbalance between clinicians and patients. Recent advancements in multimodal large language models, represented by GPT-4, have demonstrated remarkable performance in the general domain. However, training such models necessitates a substantial amount of paired image-text data, yet paired ophthalmic data is limited, and ophthalmic reports are laden with specialized terminologies, making it challenging to transfer the training paradigm to the ophthalmic domain. In this paper, we propose EyeGraphGPT, a knowledge graph enhanced multimodal large language model for ophthalmic report generation. Specifically, we construct a knowledge graph by leveraging the knowledge from a medical database and expertise from ophthalmic experts to model relationships among ophthalmic diseases, enhancing the model’s focus on key disease information. We then perform relation-aware modal alignment to incorporate knowledge graph features into visual features, and further enhance modality collaboration through visual instruction fine-tuning to adapt the model to the ophthalmic domain. Our experiments on a real-world dataset demonstrates that EyeGraphGPT outperforms previous state-of-the-art models, highlighting its superiority in scenarios with limited medical data and extensive specialized terminologies.

Federated learning (FL) enables collaborative learning across multiple biomedical data silos with multimodal foundation models while preserving privacy. Due to the heterogeneity in data processing and collection methodologies across diverse medical institutions and the varying medical inspections patients undergo, modal heterogeneity exists in practical scenarios, where severe modal heterogeneity may even prevent model training. With privacy considerations, data transfer cannot be permitted, restricting knowledge exchange among different clients. To trickle these issues, we propose a cross-modal prototype imputation method for visual-language understanding (Buffalo) with only a slight increase in communication cost, which can improve the performance of fine-tuning general foundation models for downstream biomedical tasks. We conducted extensive experiments on medical report generation and biomedical visual question-answering tasks. The results demonstrate that Buffalo can fully utilize data from all clients to improve model generalization compared to other modal imputation methods in three modal heterogeneity scenarios, approaching or even surpassing the performance in the ideal scenario without missing modality.

Federated Learning has emerged as a revolutionary innovation in the evolving landscape of global healthcare, fostering collaboration among institutions and facilitating collaborative data analysis. As practical applications continue to proliferate, numerous federations have formed in different regions. The optimization and sustainable development of federation-pretrained models have emerged as new challenges. These challenges primarily encompass privacy, population shift and data dependency, which may lead to severe consequences such as the leakage of sensitive information within models and training samples, unfair model performance and resource burdens. To tackle these issues, we propose FairFusion, a cross-federation model fusion approach that enhances privacy and fairness. FairFusion operates across federations within a Model Trip paradigm, integrating knowledge from diverse federations to continually enhance model performance. Through federated model fusion, multi-objective quantification and optimization, FairFusion obtains trustworthy solutions that excel in utility, privacy and fairness. We conduct comprehensive experiments on three public real-world healthcare datasets. The results demonstrate that FairFusion achieves outstanding model fusion performance in terms of utility and fairness across various model structures and subgroups with sensitive attributes while guaranteeing model privacy.

Multi-modality physiological signal-based emotion recognition has attracted increasing attention as its capacity to capture human affective states comprehensively. Due to multi-modality heterogeneity and cross-subject divergence, practical applications struggle with generalizing models across individuals. Effectively addressing both issues requires mitigating the gap between multimodal signals while acquiring generalizable representations across subjects. However, existing approaches often handle these dual challenges separately, resulting in suboptimal generalization. This study introduces a novel framework, termed Correlation-Driven Multi-Modality Graph Decomposition (CMMGD). The proposed CMMGD initially captures adaptive cross-modal correlations. It connects each unimodal graph to a multimodal mixed graph. To simultaneously address the dual challenges, it incorporates a correlation-driven graph decomposition module that decomposes the mixed graph into concordant and discrepant subgraphs based on the correlations. The decomposed concordant subgraph encompasses consistently activated features across modalities and subjects during emotion elicitation, unveiling a generalizable subspace. Additionally, we design a Multi-Modality Graph Regularized Transformer (MGRT) backbone specifically tailored for multimodal physiological signals. The MGRT can alleviate the over-smoothing issue and mitigate over-reliance on any single modality. Extensive experiments demonstrate that CMMGD outperforms the state-of-the-art methods by 1.79% and 2.65% on DEAP and MAHNOB-HCI datasets, respectively, under the leave-one-subject-out cross-validation strategy.

Model fusion is becoming a crucial component in the context of model-as-a-service scenarios, enabling the delivery of high-quality model services to local users. However, this approach introduces privacy risks and imposes certain limitations on its applications. Ensuring secure model exchange and knowledge fusion among users becomes a significant challenge in this setting. To tackle this issue, we propose PrivFusion, a novel architecture that preserves privacy while facilitating model fusion under the constraints of local differential privacy. PrivFusion leverages a graph-based structure, enabling the fusion of models from multiple parties without additional training. By employing randomized mechanisms, PrivFusion ensures privacy guarantees throughout the fusion process. To enhance model privacy, our approach incorporates a hybrid local differentially private mechanism and decentralized federated graph matching, effectively protecting both activation values and weights. Additionally, we introduce a perturbation filter adapter to alleviate the impact of randomized noise, thereby recovering the utility of the fused model. Through extensive experiments conducted on diverse image datasets and real-world healthcare applications, we provide empirical evidence showcasing the effectiveness of PrivFusion in maintaining model performance while preserving privacy. Our contributions offer valuable insights and practical solutions for secure and collaborative data analysis within the domain of privacy-preserving model fusion.

Physiological signal based ubiquitous computing has garnered significant attention. However, the heterogeneity among multimodal physiological signals poses a critical challenge to practical applications. To traverse this heterogeneity gap, recent studies have focused on establishing inter-modality correlations. Early works only consider coarse-level correlations between the embeddings of each modality. More recent graph-based approaches incorporate prior knowledge-based correlations, although they may not be entirely accurate. In this article, we propose the Graph Joint Fusion (GJFusion) network, which leverages channel-level inter-modality correlations based on a graph joint to mitigate the heterogeneous gap. Our proposed GJFusion first represents each modality as a graph, with each vertex corresponding to a signal channel, and the edges denoting their functional connectivity. We then join each modality by constructing inter-modality correlations for each salient channel using a sampling-based matching method. Discarded channels are transformed into a virtual vertex through a lightweight pooling operation. Subsequently, the fusion network integrates intra- and inter-modality features, enabling multimodal physiological signal fusion. To validate the effectiveness of our method, we select emotional state recognition as the downstream task and conduct comprehensive experiments on two benchmark datasets. The results demonstrate that our proposed GJFusion network surpasses the latest state-of-the-art methods, achieving relative accuracy improvements of 1.22% and 0.81% on the DEAP and MAHNOB-HCI datasets, respectively. Furthermore, visualization experiments of the salient brain regions reveal the presence of interpretable knowledge within the proposed GJFusion model.

Federated learning has emerged as a powerful technique for training robust models while preserving data privacy and security. However, real-world applications, especially in domains like healthcare, often face challenges due to non-independent and non-identically distributed (non-iid) data across different institutions. Additionally, the heterogeneity of data and the absence of a trusted central server further hinder collaborative efforts among medical institutions. Our paper introduces a novel federated learning approach called FedTAM, which incorporates cyclic model transfer and feature attention-based multi-teacher knowledge distillation. FedTAM is designed to tailor personalized models for individual clients within a decentralized federated learning setting, where data distribution is non-iid. Notably, this method enables student clients to selectively acquire the most pertinent and valuable knowledge from teacher clients through feature attention mechanism while filtering out irrelevant information. We conduct extensive experiments across five benchmark healthcare datasets and one public image classification dataset with feature shifts. Our results conclusively demonstrate that our method achieves remarkable accuracy improvements when compared to state-of-the-art approaches. This affirms the potential of FedTAM to significantly enhance federated learning performance, especially in challenging real-world contexts like healthcare.

Horizontal Federated Learning offers a means to develop machine learning models in the realm of medical application while preserving the confidentiality and security of patient data. However, due to the substantial heterogeneity of the devices in medical institution, traditional synchronous federated aggregation methods result in a noticeable decrease in training efficiency, thereby impacting the application and deployment of federated learning. Asynchronous Federated Learning (AFL) model aggregation methods can mitigate this problem but present new challenges in terms of convergence stability and speed. In this paper, we propose a cosine similarity-based layer-wise penalty term and asynchronous model aggregation method AFL-CS, which considers the global model convergence direction during local training. Compared with existing AFL aggregation methods, AFL-CS can achieve faster and more consistent convergence direction to superior performance especially in non-iid settings with high statistical heterogeneity, even reaching and exceeding synchronous FL.

Human activity recognition requires the efforts to build a generalizable model using the training datasets with the hope to achieve good performance in test datasets. However, in real applications, the training and testing datasets may have totally different distributions due to various reasons such as different body shapes, acting styles, and habits, damaging the model’s generalization performance. While such a distribution gap can be reduced by existing domain adaptation approaches, they typically assume that the test data can be accessed in the training stage, which is not realistic. In this article, we consider a more practical and challenging scenario: domain-generalized activity recognition (DGAR) where the test dataset cannot be accessed during training. To this end, we propose Adaptive Feature Fusion for Activity Recognition (AFFAR) , a domain generalization approach that learns to fuse the domain-invariant and domain-specific representations to improve the model’s generalization performance. AFFAR takes the best of both worlds where domain-invariant representations enhance the transferability across domains and domain-specific representations leverage the model discrimination power from each domain. Extensive experiments on three public HAR datasets show its effectiveness. Furthermore, we apply AFFAR to a real application, i.e., the diagnosis of Children’s Attention Deficit Hyperactivity Disorder (ADHD), which also demonstrates the superiority of our approach.

Abstract As an enabling technology for edge computing scenarios, indoor localization has a broad prospect in a variety of location-based applications, such as tracking, navigating, and monitoring in indoor environments. In order to improve the location accuracy, numerous machine learning (ML)-based indoor localization schemes with fingerprint fusion have been proposed recently, which take advantage of the fusion of signal gathered from multiple wireless technologies (e.g., WiFi and BLE) and require a site survey to construct the fingerprint database. However, most solutions are based on cloud framework and thus pose a serious privacy leakage because users’ sensitive information (e.g., locations) is computed from the fingerprint database by the untrusted localization service provider. Furthermore, the site survey is time-consuming and labor-intensive. In this paper, we propose a differentially private fingerprint fusion semi-supervised extreme learning machine for indoor localization in the edge computing, called Adp-FSELM. The Adp-FSELM firstly employs a multi-level edge network-based privacy-preserving system framework to meet the requirements of ML-based fingerprint indoor localization for lightweight, low latency, and real-time response. Then, the Adp-FSELM extends the $$\varepsilon$$ ε -differential privacy to the fingerprint fusion semi-supervised extreme learning machine for indoor localization in edge computing through a three-phase private process consisting of private labeled sample obfuscation, differentially private feature fusion, and differentially private model training. Theoretical and comprehensive experimental results in real indoor environments demonstrate that the Adp-FSELM provides a high $$\varepsilon$$ ε -differential privacy guarantee for users’ location privacy while reducing human calibration effort and effectively resists Bayesian inference attacks. Compared with the existing semi-supervised learning-based localization methods, the mean absolute error of location accuracy of the Adp-FSELM is restricted to 2.22% at most, and the additional time consumption can be almost ignored. Thus, our mechanism can balance the trade-off among location privacy, location accuracy, and time consumption.

Attention Deficit Hyperactivity Disorder (ADHD) is one of the most common mental disorders affecting children. Because the etiology of ADHD is complex and its symptoms are not specific, there is a lack of feasible quantitative diagnostic methods. Pursuing objective and non-invasive detection methods and standards is of great practical significance to prevent the development of the disease. In this study, we aim to address one specific concern about the objectivity and quantification of ADHD diagnosis. Over a year, we iteratively designed and tested WeDA, a scale-driven wearable diagnostic assessment system. This system contains an Android computer machine with a large touchscreen, a suite of 3D printed interactive devices, and six wearable motion sensors. We implement ten diagnostic tasks drawing on the symptoms of ADHD based on DSM-5. The experimental results of classifying children with ADHD and typically developing children and subjective evaluations from doctors, parents, and children validate the effectiveness and acceptability of WeDA.

People with autism are at risk for social isolation due to differences in their perception and engagement with the social world. In this work, we aim to address one specific concern related to socialization the understanding, awareness, and use of interpersonal space. Over the course of a year, we iteratively designed and tested a series of concepts for supporting children with autism in perceiving, understanding, and responding to physical proximity with other people. During this process, we developed ProCom, a prototype system for measuring proximity without requiring instrumentation of the environment or another person. We used a variety of low and high fidelity prototypes, culminating in ProCom, to assess the feasibility, utility, and challenges of this approach. The results of these iterative design engagements indicate that wearable assistive technologies can support people in developing awareness of physical proximity in social settings. However, challenges related to both personal and collective use remain

Sleep plays a vital role in good health and well-being throughout human life. A great deal of studies have been done to detect sleep stages. Most of the current sleep monitoring systems are invasive to users, e.g. requiring users to wear a device during sleep. In this paper, we use microphone to detect sleep stages including deep sleep, light sleep and rapid eye movement (REM), and propose a convolutional neural network using spectrogram as input. This paper's contribution mainly concentrates on the following two aspects: First, microphone is unobtrusive for sleep detection. Second, we build the mapping between acoustic signal and sleep stages with little manual intervention to extract features. Performance of the proposed method is validated on a realistic environmental dataset containing 52 nights of 5 participants. Experimental results show that the accuracy of sleep stages detection is superior to the representative off-the-shelf applications. Besides, we propose to utilize the attention maps to visualize acoustic data to better understand the relationship between sound and sleep stages. Experimental results show that the model has the ability to effectively reduce noises in classification by ignoring the high-frequency sounds and white noises.

Currently available traditional as well as videogame-based cognitive assessment techniques are inappropriate due to several reasons. This paper presents a novel psychosocial game suite, BrainStorm, for non-invasive cross-generational cognitive capabilities data collection, which additionally provides cross-generational social support. A motivation behind the development of presented game suite is to provide an entertaining and exciting platform for its target users in order to collect gameplay-based cognitive capabilities data in a non-invasive manner. An extensive evaluation of the presented game suite demonstrated high acceptability and attraction for its target users. Besides, the data collection process is successfully reported as transparent and non-invasive.

As an important technology in LBS (Location Based Services) field, Wi-Fi based indoor localization suffers signal fluctuation problem which prevents lifelong and high performance running. With the fluctuation of wireless signal over time, fingerprints collected at the same location become different; therefore existing model cannot fit the new collected data well, which decreases the localization accuracy. In this paper, a novel indoor localization method COSELM (Constraint Online Sequential Extreme Learning Machine) is proposed, utilizing incremental data to update the old model and overcome the fluctuation problem. The performance of COSELM is validated in real Wi-Fi indoor environment. Compared with OSELM, it can improve more than 5% localization accuracy on average; and in contrast to batch learning, COSELM can save more than 50% time consumption.