In many machine learning systems that jointly learn from multiple modalities, a core research question is to understand the nature of multimodal interactions: how modalities combine to provide new task-relevant information that was not present in either alone. We study this challenge of interaction quantification in a semi-supervised setting with only labeled unimodal data and naturally co-occurring multimodal data (e.g., unlabeled images and captions, video and corresponding audio) but when labeling them is time-consuming. Using a precise information-theoretic definition of interactions, our key contribution is the derivation of lower and upper bounds to quantify the amount of multimodal interactions in this semi-supervised setting. We propose two lower bounds: one based on the shared information between modalities and the other based on disagreement between separately trained unimodal classifiers, and derive an upper bound through connections to approximate algorithms for min-entropy couplings. We validate these estimated bounds and show how they accurately track true interactions. Finally, we show how these theoretical results can be used to estimate multimodal model performance, guide data collection, and select appropriate multimodal models for various tasks.

Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Past work probing compositionality in sentence embedding models faces issues determining the causal impact of implicit syntax representations.Given a sentence, we construct a neural module net based on its syntax parse and train it end-to-end to approximate the sentence's embedding generated by a transformer model.The distillability of a transformer to a Syntactic NeurAl Module Net (SynNaMoN) then captures whether syntax is a strong causal model of its compositional ability.Furthermore, we address questions about the geometry of semantic composition by specifying individual SynNa-MoN modules' internal architecture & linearity.We find differences in the distillability of various sentence embedding models that broadly correlate with their performance, but observe that distillability doesn't considerably vary by model size.We also present preliminary evidence that much syntax-guided composition in sentence embedding models is linear, and that non-linearities may serve primarily to handle non-compositional phrases.

Cervical cancer affects 570,000 women globally and is among the most common causes of cancer-related deaths. Cervical cancer is caused due to the Human Papilloma Virus (HPV) which leads to abnormal growth of cells in the cervix region. Regular testing for HPV in women has helped reduce the death rate in developed countries. However, developing nations are still struggling to provide low-cost solutions due to the lack of affordable medical facilities. The skewed ratio of the oncologists to patients has also aggravated the problem. Motivated by the Deep Learning solutions in Bio-medical imaging, we propose a novel CervixNet methodology which performs image enhancement on cervigrams followed by Segmenting the Region of Interest (RoI) and then classifying the RoI to determine the appropriate treatment. For the classification task, a novel Hierarchical Convolutional Mixture of Experts (HCME) algorithm is proposed. HCME is capable of tackling the problem of overfitting, given that small datasets are an inherent problem in the field of biomedical imaging. Our proposed methodology has outperformed all the existing methodologies on publicly available Intel and Mobile-ODT Kaggle dataset giving an Accuracy of 96.77% and kappa score of 0.951. Hence, the results obtained validate our approach to provide first level screening at a low cost.