Multimodal Emotion Recognition in Conversations (ERC) is a typical multimodal learning task in exploiting various data modalities concurrently. Prior studies on effective multimodal ERC encounter challenges in addressing modality imbalances and optimizing learning across modalities. Dealing with these problems, we present a novel framework named Ada2I, which consists of two inseparable modules namely Adaptive Feature Weighting (AFW) and Adaptive Modality Weighting (AMW) for feature-level and modality-level balancing respectively via leveraging both Inter- and Intra-modal interactions. Additionally, we introduce a refined disparity ratio as part of our training optimization strategy, a simple yet effective measure to assess the overall discrepancy of the model's learning process when handling multiple modalities simultaneously. Experimental results validate the effectiveness of Ada2I with state-of-the-art performance compared to baselines on three benchmark datasets, particularly in addressing modality imbalances.

Emotion recognition is a crucial task for human conversation understanding.It becomes more challenging with the notion of multimodal data, e.g., language, voice, and facial expressions.As a typical solution, the globaland the local context information are exploited to predict the emotional label for every single sentence, i.e., utterance, in the dialogue.Specifically, the global representation could be captured via modeling of cross-modal interactions at the conversation level.The local one is often inferred using the temporal information of speakers or emotional shifts, which neglects vital factors at the utterance level.Additionally, most existing approaches take fused features of multiple modalities in an unified input without leveraging modalityspecific representations.Motivating from these problems, we propose the Relational Temporal Graph Neural Network with Auxiliary Cross-Modality Interaction (CORECT), an novel neural network framework that effectively captures conversation-level cross-modality interactions and utterance-level temporal dependencies with the modality-specific manner for conversation understanding.Extensive experiments demonstrate the effectiveness of CORECT via its stateof-the-art results on the IEMOCAP and CMU-MOSEI datasets for the multimodal ERC task.