Kinesthetic and tactile information can represent the physical states of objects, encompassing roughness, stiffness, motion, force, and other attributes. The introduction of these can enhance imprecise recognition that relies solely on visual information in cases of light disturbance, occlusion and camouflage. Nevertheless, this task is still challenging due to the heterogeneity among visual, tactile and kinesthetic data. To address this issue, this paper delves into the alignment of heterogeneous data dimensions, the fusion of heterogeneous data features, and the optimization of learning rates for multi-source heterogeneous sensor learning models. Consequently, an effective Visual-Kinesthetic-Tactile Information Fusion (VikitaFusion) network is proposed, which comprises: 1) heterogeneous data extractors that align visual images with tactile and kinesthetic data through image-to-sequence projection; 2) a visual-kinesthetic-tactile Transformer-based domain fusion that mimics human multi-sensory fusion perception through a feature-level fusion block and dynamic fusion blocks; 3) a Periodic Triangulation Learning Rate (PTLR) method aimed at optimizing the learning rate for performance enhancement in multi-source heterogeneous sensor learning models. Extensive experiments demonstrate that VikitaFusion outperforms current state-of-the-art methods with higher recognition accuracy and a lower parameter size. Source code will be made publicly available after the peer review process.

Embodied AI calls for a reliable, cross-modal object recognition that deeply mines High-Quality (HQ) object appearance (i.e., visual information) and touch details (i.e., haptic information). While in real-world scenarios, cross-modal data is usually degraded due to data acquisition and delivery in complex environments. In this paper, we propose a Robust Visual-Haptic recognition (RobustVisH) model that identifies Low-Quality (LQ) visual-haptic data with transmission distortion for the first time. First, we introduce the WIreless Transmission Interference-based Multi-modal benchmark (WITIM) as a visual-haptic dataset under transmission interference. In particular, the dataset consists of WITIM/AU and WITIM/PHAC-2, in which the original signals are obtained from AU and PHAC-2, respectively. Second, we design a trainable weighted fusion and a Transformer encoder based on the bi-directional self-attention mechanism, enabling RobustVisH to form and learn fused visual-haptic features after modality-specific one-dimensional feature encoding. Third, we employ a covariate shift paradigm, transferring knowledge of RobustVisH from HQ data to LQ data, thereby increasing its robustness against transmission-interference inputs. Experimental results demonstrate that the proposed RobustVisH improves the accuracy of the state-of-the-art method by 2.06% and 9.28% on WITIM/AU and WITIM/PHAC-2, respectively. Source code is available at: https://github.com/lylibylily/RobustVisH.