Multi-modal knowledge graph reasoning (MKGR) seeks to conjecture plausible facts in MKGs by learning effective representations from various modalities (e.g., structure, text, and image). However, due to holistic redundancy (i.e., each modality carries task-irrelevant redundancy) and modality conflict (i.e., different modalities contain contradictory information), the reasoning performance of current methods is substantially impaired. In this paper, we propose a novel Consistency Discovery-guided Information Bottleneck (CDIB) framework to address the aforementioned challenges. Specifically, a modality compression module is first designed to learn modality-private entity representations of alleviating redundant information. Then, a consistency discovery module is developed to discover cross-modal consistency during multi-modal fusion to learn the comprehensive entity representations. To retain task-relevant information, an information preservation module is devised to further enrich the comprehensive entity representations to be predictive for MKGR. Extensive experiments indicate that CDIB achieves state-of-the-art reasoning ability on two benchmark datasets over current MKGR baselines, and also exhibits promising robustness against noise.

The development of industrial Internet of Things (IoT) generates massive unlabeled multimodal data. multimodal clustering (MMC) methods can unsupervisedly learn reasonable clustering structures by mining the complementary information in each modality, which has become one of the essential critical technologies in industrial IoT. However, current MMC methods ignore redundant information in the multimodal data. This indistinguishable redundant information hinders the mining of complementary information and thus reduces the performance of MMC. In this article, we propose a multimodal nonredundant clustering via sufficiency complementary mining that explores complementary information while maximizing the removal of redundant information. We first construct a multimodal autoencoding network to learn the single-modal and fusion representation of all modalities. Then, we design a sufficiency complementary mining (SCM) mechanism based on information theory to train this autoencoding network. SCM transforms the complementary information into the sufficiency of fusion representation for each modality while reconstructing the redundant information as a conditional mutual information between the fusion and each single-modal representation given original data. This ingenious design allows SCM to safely remove redundant information by minimizing this conditional mutual information under guaranteed sufficiency. Moreover, the clustering task and the multimodal autoencoding network are integrated into a unified framework to enable the learned complementary information to guide the clustering process. Extensive experimental results on eight large-scale real-world datasets show that our method outperforms the state-of-the-art MMC methods.

Cross-modal clustering (CMC) aims to enhance the clustering performance by exploring complementary information from multiple modalities. However, the performances of existing CMC algorithms are still unsatisfactory due to the conflict of heterogeneous modalities and the high-dimensional non-linear property of individual modality. In this paper, a novel deep mutual information maximin (DMIM) method for cross-modal clustering is proposed to maximally preserve the shared information of multiple modalities while eliminating the superfluous information of individual modalities in an end-to-end manner. Specifically, a multi-modal shared encoder is firstly built to align the latent feature distributions by sharing parameters across modalities. Then, DMIM formulates the complementarity of multi-modalities representations as an mutual information maximin objective function, in which the shared information of multiple modalities and the superfluous information of individual modalities are identified by mutual information maximization and minimization respectively. To solve the DMIM objective function, we propose a variational optimization method to ensure it converge to a local optimal solution. Moreover, an auxiliary overclustering mechanism is employed to optimize the clustering structure by introducing more detailed clustering classes. Extensive experimental results demonstrate the superiority of DMIM method over the state-of-the-art cross-modal clustering methods on IAPR-TC12, ESP-Game, MIRFlickr and NUS-Wide datasets.

Abstract Crowd counting is still a practical and challenging problem owing to scale variations and information loss. Most existing methods based on the straightforward fusion of different features from a deep neural network seem to eliminate this limitation. However, these features are difficult to be fused since they often differ significantly in modality and dimensionality. Unlike previous works, a multi‐level features extraction network with gating mechanism for crowd counting is proposed. Specifically, a multi‐channel gated unit to adaptively extract features in different levels of the network is proposed, which can avoid interference from confusing information. To fully aggregate features via multi‐level fusion, multi‐level features extraction scheme is presented. The multi‐level features extraction network learns to fuse features from multiple levels and reduce false predictions. Extensive experiments and evaluations clearly illustrate that the proposed approach achieves state‐of‐the‐art counting performance against other methods on four mainstream crowd counting benchmarks.