Conventional Multi-modal multi-label emotion recognition (MMER) assumes complete access to visual, textual, and acoustic modalities. However, real-world multi-party settings often violate this assumption, as non-speakers frequently lack acoustic and textual inputs, leading to a significant degradation in model performance. Existing approaches also tend to unify heterogeneous modalities into a single representation, overlooking each modality’s unique characteristics. To address these challenges, we propose RAMer (Reconstruction-based Adversarial Model for Emotion Recognition), which refines multi-modal representations by not only exploring modality commonality and specificity but crucially by leveraging reconstructed features, enhanced by contrastive learning, to overcome data incompleteness and enrich feature quality. RAMer also introduces a personality auxiliary task to complement missing modalities using modality-level attention, improving emotion reasoning. To further strengthen the model's ability to capture label and modality interdependency, we propose a stack shuffle strategy to enrich correlations between labels and modality-specific features. Experiments on three benchmarks, i.e., MEmoR, CMU-MOSEI, and M³ED, demonstrate that RAMer achieves state-of-the-art performance in dyadic and multi-party MMER scenarios.

Large Language Models (LLMs) have demonstrated impressive capabilities across a range of scientific tasks including mathematics, physics, and chemistry. Despite their successes, the effectiveness of LLMs in handling complex statistical tasks remains systematically under-explored. To bridge this gap, we introduce StatQA, a new benchmark designed for statistical analysis tasks. StatQA comprises 11,623 examples tailored to evaluate LLMs' proficiency in specialized statistical tasks and their applicability assessment capabilities, particularly for hypothesis testing methods. We systematically experiment with representative LLMs using various prompting strategies and show that even state-of-the-art models such as GPT-4o achieve a best performance of only 64.83%, indicating significant room for improvement. Notably, while open-source LLMs (e.g. LLaMA-3) show limited capability, those fine-tuned ones exhibit marked improvements, outperforming all in-context learning-based methods (e.g. GPT-4o). Moreover, our comparative human experiments highlight a striking contrast in error types between LLMs and humans: LLMs primarily make applicability errors, whereas humans mostly make statistical task confusion errors. This divergence highlights distinct areas of proficiency and deficiency, suggesting that combining LLM and human expertise could lead to complementary strengths, inviting further investigation into their collaborative potential. Our source code and data are available at https://statqa.github.io/.

Understanding human drivers’ intentions and predicting their future motions are significant to connected and autonomous vehicles and traffic safety and surveillance systems. Predicting multimodal vehicular trajectories at urban unsignalized intersections remains challenging due to dynamic traffic flow and uncertainty of human drivers’ maneuvers. In this paper, we propose a comprehensive trajectory prediction framework that combines a multimodal trajectory generation network with inverse reinforcement learning (IRL) and risk aversion (RA) modules. Specifically, we first construct a multimodal spatial-temporal Transformer network (mmSTTN) to generate multiple trajectory candidates, using trajectory coordinates as inputs. Accounting for spatio-temporal features, we formulate the IRL reward function for evaluating all candidate trajectories. The optimal trajectory is then selected based on the computed rewards, a process that mimics human drivers’ decision-making. We further develop the RA module based on the driving risk field for optimal risk-averse trajectory prediction. We conduct experiments and ablation studies using the inD dataset at an urban unsignalized intersection, demonstrating impressive human trajectory alignment, prediction accuracy, and the ability to generate risk-averse trajectories. Our proposed framework reduces prediction errors and driving risks by 25% and 30% compared to baseline methods. Results validate vehicles’ human-like risk-averse diverging-and-concentrating behavior as they traverse the intersection. The proposed framework presents a novel approach for forecasting multimodal vehicular trajectories by imitating human drivers and incorporating physics-based risk information derived from the driving field. This research offers a promising direction for enhancing the safety and efficiency of connected and autonomous vehicles navigating urban environments.