Multimodal data holds great potential in enhancing the accuracy of sentiment analysis. However, the multimodal sentiment analysis (MSA) task faces two practical challenges: i) multimodal heterogeneity significantly increases the difficulty of integrating multimodal information, resulting in insufficient multimodal learning; and ii) multimodal incompleteness easily triggers cross-modal inference biases, seriously hindering multimodal understanding. In this paper, we propose a novel multimodal framework namedcross-modal explicit invariancecoordinateddualconstraintreconstruction (Clive-Dcr) to mitigate MSA for heterogeneous incomplete multimodal data. Specifically, we present a language-oriented cross-modal invariance reasoning (LCIR) approach that can alleviate the inter-modal representation gaps under the supervision of fine-grained sentiment information. More importantly, LCIR can explicitly quantify inter-modal consistency and break the vagueness of heterogeneous modal information, thereby fully exploring the complementary advantages between invariant features and specific features. In the prediction head, we develop a multimodal purification learning mechanism to minimize the intervention of sentiment-irrelevant information on MSA. To enhance the robustness under incomplete modalities, we design a dual constraint reconstruction (DCR) strategy. During the reconstruction of missing features, DCR not only performs high-level feature distribution alignment but also extends the cross-modal invariance reasoning from the complete to the incomplete multimodal scenarios. Comprehensive experiments on three multimodal datasets verify that our Clive-Dcr achieves remarkable performance under complete and incomplete multimodal patterns.

With the continuous rise in the prevalence of skin diseases, accurate classification has become an urgent demand in clinical diagnosis and treatment. In recent years, deep learningbased methods have demonstrated excellent performance in skin lesion classification tasks. To further improve diagnostic accuracy, multimodal fusion approaches integrating dermoscopic images, clinical photographs, and metadata have been explored. However, many existing methods still fail to effectively capture the deep semantic associations among these modalities. This oversight leads to semantic fragmentation across modalities, loss of critical diagnostic cues, and ultimately hinders classification accuracy. To address this challenge, we propose a Metadata-Guided Multimodal Consistency Constraints (MG-MCC) framework, which leverages metadata as a semantic anchor and enforces collaborative constraints through three specially designed loss functions: Metadata-Aware Alignment (MA), Lesion Feature Purification (LFP), and Semantic Consistency Preservation (SCP). Specifically, the MA loss enforces precise cross-modal alignment between image features and metadata in a unified space; the LFP loss strengthens semantic associations between core lesion features and metadata to accurately separate background features; and the SCP loss avoids the loss of diagnostic ROI information caused by Instance Normalization. Extensive experiments conducted on two public skin lesion datasets demonstrate that the proposed MG-MCC framework significantly outperforms state-of-the-art methods in skin lesion classification tasks.

Integrating multimodal learning (ML) with polysomnography (PSG) has emerged as a research hotspot for reliable sleep staging. However, the complexity of these signals and the discomfort associated with wearing multi-lead devices somewhat limit the feasibility of daily and ubiquitous sleep monitoring. Unfortunately, most existing ML paradigms are constrained by consistent and fixed input patterns. When the number of modalities is less than required by the ML framework, it is easy to cause inference bias, resulting in a significant performance degradation. To this end, we propose an elastic multimodal sleep staging network (ElaSleepNet), consisting of multimodal information completion (MIC) and adaptive cross-modal (ACM) interaction. Specifically, MIC maximizes the consistency of multimodal signals on intra-epoch temporal level and inter-epoch contextual level, thereby enhancing the reasoning and completion abilities of the available modalities for the unavailable modalities. Moreover, we introduce learnable parameters and design the ACM attention mechanism, which allows handling multimodal information interaction while maintaining robustness in the absence of certain modalities. Our ElaSleepNet demonstrates its state-of-the-art on three multimodal sleep datasets. Compared with previous methods, ElaSleepNet can achieve better performance with fewer testing modalities, making it flexible for daily monitoring.

Multimodal physiological signals, such as EEG, EOG and EMG, provide rich and reliable physiological information for automated sleep staging (ASS). However, in the real world, the completeness of various modalities is difficult to guarantee, which seriously affects the performance of ASS based on multimodal learning. Furthermore, the exploration of temporal context information within PTSs is also a serious challenge. To this end, we propose a robust multimodal sleep staging framework named contrastive imagination modality sleep network (CIMSleepNet). Specifically, CIMSleepNet handles the issue of arbitrary modal missing through the combination of modal awareness imagination module (MAIM) and semantic & modal calibration contrastive learning (SMCCL). Among them, MAIM can capture the interaction among modalities by learning the shared representation distribution of all modalities. Meanwhile, SMCCL introduces prior information of semantics and modalities to check semantic consistency while maintaining the uniqueness of each modality. Utilizing the calibration of SMCCL, the data distribution recovered by MAIM is aligned with the real data distribution. We further design a multi-level cross-branch temporal attention mechanism, which can facilitate the mining of interactive temporal context representations at both the intra-epoch and inter-epoch levels. Extensive experiments on five multimodal sleep datasets demonstrate that CIMSleepNet remarkably outperforms other competitive methods under various missing modality patterns. The source code is available at: https://github.com/SQAIYY/CIMSleepNet.

Sleep apnea syndrome (SAS) is a common chronic respiratory disorder, which seriously harms human health. In order to realize the large-scale promotion of SAS detection, the SAS detection method based on wearable devices has attracted the attention of some researchers. However, in view of the poor signals collected by wearable devices and the less variation between signals of different classes (normal/SAS), the detection methods in past studies are often unsatisfactory. In this study, a wearable SAS detection method based on 1-D multitask multiattention residual shrinkage convolution neural network (1D-MMResSNet) model and cost-sensitive (CS) classifier was proposed. First, the photoplethysmography (PPG) sleep data of 92 subjects were collected by using wearable smart Bracelet devices. Second, a backbone network composed of the residual multiattention mechanism convolution block and residual shrinkage convolution block was proposed to realize feature selection and feature extraction for pulse rate variability signals. At the same time, the single supervised learning task of the network was improved, and a multitask learning method was proposed to enhance the ability of the network to mine subtle differences between data, thereby effectively learning the discriminative network features. Finally, the AdaCost cost-sensitive algorithm was introduced to alleviate the class imbalance problem in data samples. In segment detection, the accuracy, sensitivity, and specificity were 81.82%, 70.27%, and 85.81%, respectively; In individual detection, the accuracy, sensitivity, and specificity were 95.65%, 88.89%, and 97.30%, respectively. Experimental results show that the proposed model has excellent detection performance and is expected to be embedded in wearable devices.