This paper presents a novel multi-modal deep learning framework for industrial robot joint health assessment and prediction, leveraging non-invasive signal fusion and Bayesian optimization. The proposed method addresses the challenges of comprehensive joint state monitoring in complex industrial environments without disrupting normal operations. We integrate Hall-effect current sensors, external accelerometers, and joint encoders to collect multi-modal data, including motor currents, vibrations, and kinematic information. A novel Adaptive Multi-Receptive Field Attention Network (AMRFAN) is employed to extract features from each modality, while a synchrosqueezing transform (SST) is utilized to capture time-frequency characteristics. An attention mechanism dynamically adjusts the weights of different modalities, and a bidirectional long short-term memory (BiLSTM) network models the temporal dependencies in the fused features. To enhance model performance and generalization, we implement a Bayesian optimization framework for hyperparameter tuning. Furthermore, we incorporate a Bayesian neural network to quantify prediction uncertainties, providing reliability metrics for decision-making processes. Experimental results on a four-axis industrial robot demonstrate that our framework achieves a 98.3% accuracy in joint health state classification and a mean absolute error of 4.2 in remaining useful life prediction, outperforming state-of-the-art single-modality methods. The proposed approach offers a robust, adaptable solution for real-time health monitoring and predictive maintenance of industrial robot joints, potentially improving manufacturing efficiency and reliability.

Aspect-based multimodal sentiment analysis (ABMSA) is a recent and popular research area that uses multiple modalities like text and images to determine the sentiment orientation of opinion entities. The main challenge in multimodal sentiment analysis is dynamically modeling each modality and effectively fusing information across different modalities. Existing methods have not considered fine-grained texture features in images, and direct fusion introduces irrelevant and ineffective features unrelated to sentiment information. To address these limitations, we propose a new model for multimodal sentiment analysis, the multimodal prompt fusion interaction Transformer (MPFIT). We designed two key components: 1) the image assist module (IAM), which leverages the self-attention mechanism and statistical pooling to obtain weighted mean and standard deviation vectors, enabling the model to focus on image texture information and reduce image noise. 2) Multimodal prompt fusion (MPF) restricts multimodal fusion to interactions between small prompt tokens that capture vital information from different modalities, allowing the model to focus on features that are more relevant to sentiment information. Experimental results show that our model outperforms baseline models on two publicly available datasets, Twitter-2015 and Twitter-2017. We conducted ablation experiments to evaluate the impact of our key components.

Pre-trained vision-language (V-L) models exhibit significant generalization capabilities in detecting rumors. However, their reliance on single-modality prompts—either language or vision—limits their flexibility for dynamic adjustments in both representation spaces during rumor detection. To address these limitations, we propose a multimodal rumor detection framework that uses prompt learning in both the vision and language domains to align their representations better. Inspired by recent advances in efficiently tuning large language models, we introduce a set of trainable parameters in the input space, keeping the model backbone frozen. Additionally, we use distinct prompts at various early stages, which helps progressively model the relationships between features, enhancing comprehensive context learning. Extensive experiments with two real-world multimodal datasets demonstrate our framework’s superior ability to distinguish rumors from facts.

Camouflaged object detection (COD) aims to segment objects that closely resemble their surroundings. Accurately recognizing camouflaged objects in these complex environments is challenging due to factors such as low illumination, object occlusion, small size, and similar background. To this end, we propose a novel network for camouflaged object detection, the Multi-Level Feature Cross-Fusion Network (MFCF-Net). This framework aims to learn and utilize background features at different scales through cross-fusion, thereby improving detection accuracy. The core of our approach is to use a modified version of the Pyramid Vision Transformer (PVTv2) as a backbone network to effectively capture contextual information at different scales. Then, we design the Multi-scale Feature Enhancement (MFE) module to optimize features at each scale. In addition, to enhance the model’s ability to recognize camouflaged objects in complex contexts, we cross-fused these enhanced features. Finally, we designed the Balanced Multilevel Feature Cross-Fusion (BMFCF) module. This module improves the accuracy of camouflaged object detection by deeply learning and effectively utilizing contextual feature information and cross-fusing these multi-scale features. Extensive research results show that our MFCF-Net significantly outperforms 18 leading methods on four widely used standard datasets.